Offline Reinforcement Learning for Large Scale Language Action Spaces

1 INTRODUCTION . Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing ( NLP ) tasks , yet challenging due to large language action spaces and limited availability of human-annotated data . Recently , large-scale pre-trained language models ( LM ) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary ( Devlin et al. , 2019 ; Radford et al. , 2019 ; Brown et al. , 2020 ; Raffel et al. , 2019 ) . The current best performing end-to-end conversational agents for a task-oriented dialogue system utilize a pre-training on largescale corpus and fine-tuning on downstream tasks ( Ham et al. , 2020 ; Yang et al. , 2021 ; Lin et al. , 2020 ; Peng et al. , 2021 ) . This combination of pre-training and fine-tuning significantly improves overall performance in the task-oriented dialogues . However , supervised fine-tuning ( i.e . imitation learning of the dialogue corpus ) alone may not be sufficient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels . Thus , in order to optimize the task performance of the conversational agent , goaloriented training ( i.e . reinforcement learning ) is an essential and promising direction to pursue . Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as offline reinforcement learning ( RL ) problem ( Levine et al. , 2020 ; Fujimoto et al. , 2019 ; Jaques et al. , 2020 ) , which offers the prospect to optimize the policy solely from the fixed dataset without online environment interaction . Most of the existing offline RL methods are built on the off-policy ActorCritic framework , which performs iterative optimization of the policy ( i.e . actor ) and the actionvalue function ( i.e . critic ) ( Fujimoto et al. , 2019 ; Janner et al. , 2019 ; Kumar et al. , 2020 ) . Yet , a naive application of these offline RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language ( Lewis et al. , 2017 ; Zhao et al. , 2019 ; Jang et al. , 2020 ) . To mitigate the aforementioned problem , discrete latent representation models for language actions have been proposed to disentangle the semantics of the utterance and the natural language generation ( Zhao et al. , 2019 ; Yarats & Lewis , 2018 ) . In that framework , goal-based training was performed in the space of the discrete latent variables instead of directly optimizing utterances . While these approaches can prevent degenerate responses in principle , they can not be straightforwardly applied to widely used large-scale LMs since most of them are not trained for discrete latent variables . One could make modifications to the LM to work with discrete latent variables but it would need to be pre-trained from scratch with a tremendous amount of data and time , which is undesirable . In this paper , we present an offline RL algorithm for task-oriented dialogue , which can be adopted for any generative pre-trained language model . Our algorithm , GPT-Critic , does not rely on policy gradient unlike actor-critic methods and is essentially free from the issue of diverging from human language . It starts with fine-tuning the GPT-2 model and learning the action-value function ( critic ) using the dialogue corpus . Then , GPT-Critic generates a strategically promising action that is selected based on the value estimated by the critic . GPT-Critic updates the policy through behavior cloning of the critic-guided self-generated responses . This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus , where the action choice is restricted to the support in the dataset ( Wang et al. , 2020 ) . Since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2 , it thus inherits GPT-2 ’ s ability to generate human-like responses . In the experiments , we demonstrate that GPT-Critic outperforms the state-of-the-art end-to-end dialogue agent in the task-oriented dialogue benchmarks including MultiWOZ ( Budzianowski et al. , 2018 ) and ConvLab ( Zhu et al. , 2020 ) . 2 BACKGROUND . 2.1 OFFLINE REINFORCEMENT LEARNING FOR TASK-ORIENTED DIALOGUES . We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process ( POMDP ) defined by tuple 〈S , A , O , T , Z , R , γ〉 where S is the set of environment states s = 〈g , h〉 ( underlying state that consists of the user goal g and dialogue history h ) , A is the set of actions a ( a sequence of tokens which represents dialogue act and system response ) , O is the set of observations o ( user utterance ) , T ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) is the transition function , Z ( o|s′ , a ) = Pr ( ot+1 = o|st+1 = s′ , at = a ) is the observation probability , R ( g , h , a ) is the reward function indicating the utility of executing action a in history h and the user goal g , and γ ∈ ( 0 , 1 ) is a discount factor . The history at time step t , ht = { o0 , a0 , . . . ot−1 , at−1 , ot } , is a sequence of all previous observations and actions . Since the underlying state s ( e.g . user goal ) is not directly observable , the agent makes decisions based on the entire observation-action history . The policy π ( at|ht ) is mapping from history ht to a probability distribution over A . The goal is to find an optimal policy π∗ that maximizes the expected cumulative rewards , i.e . π∗ = arg maxπ Eπ [ ∑∞ t=0 γ tR ( g , ht , at ) ] . The action-value function of policy π is defined as Qπ ( h , a ) : = Eπ [ ∑∞ t=0 γ tR ( g , ht , at ) |h0 = h , a0 = a ] , where Qπ is a unique solution of the Bellman equation : Qπ ( h , a ) = Eg [ R ( g , h , a ) ] + γEπ [ Qπ ( h′ , a′ ) ] . Using offline RL for dialogue policy optimization , the agent optimizes the policy from the precollected dataset D = { { ( gj , hjt , a j t , r j t , h j t+1 ) T t=0 } Nj=1 } without online environment interaction during the intermediate stages of training . Prior offline RL algorithms ( Fujimoto et al. , 2019 ; Janner et al. , 2019 ; Kumar et al. , 2020 ) rely on off-policy actor-critic method , where the critic network is trained by minimizing the temporal differnce error with respect to the target policy π : arg min φ E ( ht , at , rt , ht+1 ) ∼D [ ( rt + γEat+1∼π ( ht+1 ) [ Qφ̄ ( ht+1 , at+1 ) ] −Qφ ( ht , at ) ) 2 ] ( 1 ) where φ̄ is the parameters of the target network . As discussed in the prior work ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) , optimizing this loss can be challenging in the offline RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution ( OOD ) actions to evaluate the value of the next state . 2.2 END-TO-END TASK-ORIENTED DIALOGUE SYSTEM . We focus on the MultiWOZ dataset ( Budzianowski et al. , 2018 ) , which is a representative benchmark for task-oriented dialogue . The MultiWOZ dataset is a fully-annotated corpus of human-human task-oriented conversations , which is collected via the Wizard-of-Oz setting ( Kelley , 1984 ) . The traditional approach to building a task-oriented dialogue system adopts a modular pipeline , which consists of the following four modules : 1 ) A natural language understanding ( NLU ) module ( Kim et al. , 2017 ; Zhu et al. , 2020 ) identifies the user ’ s intent and extracts the information of slots and their values , 2 ) A Dialogue state tracking ( DST ) module ( Williams et al. , 2013 ) infers the belief state , 3 ) A dialogue policy ( POL ) module decides the system action , 4 ) A natural language generation ( NLG ) module ( Wen et al. , 2015 ) generates the system response corresponding to the system action . Recently , end-to-end task-oriented dialogue methods leveraging the pre-trained language model have been proposed ( Yang et al. , 2021 ; Ham et al. , 2020 ; Lin et al. , 2020 ; Peng et al. , 2021 ; Hosseini-Asl et al. , 2020 ) , and significantly improves overall performance in the task-oriented dialogues . In this paper , our algorithm is built upon UBAR ( Yang et al. , 2021 ) , which is based on GPT-2 ( Radford et al. , 2019 ) and currently the state-of-the-art end-to-end dialogue agent for the MultiWOZ domain . 3 OFFLINE REINFORCEMENT LEARNING FOR END-TO-END TASK-ORIENTED DIALOGUE SYSTEMS . The corpus collected from human-human conversations inevitably contains unsuccessful dialogues in terms of task completion . For example , approximately 20 % dialogues of the MultiWOZ dataset fail to meet the user goal . Therefore , a naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot of unsuccessful dialogues : an agent that imitates failure would be inevitably suboptimal . Yet , dropping the unsuccessful dialogues from the corpus is also undesirable , since they may contain some task-specific information that is useful to properly respond to user requests . We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while improving the task performance . In this section , we present GPT-Critic , an offline RL algorithm for task-oriented dialogue . Our GPTCritic is analogous to Actor-Critic method : GPT ( Actor ) decides which action to take while the Critic informs how good the action was and provides a signal for policy improvement . Still , GPTCritic is distinct from the Actor-Critic methods in that it does not rely on the policy gradients , which are generally known to cause the issue of diverging from human language ( Lewis et al. , 2017 ; Zhao et al. , 2019 ) . Instead , we sample a set of action candidates using GPT-2 and pick the best one using the critic , which constitutes a revised dialogue corpus . Then , we perform supervised fine-tuning of the GPT-2 on the revised dialogue corpus . This learning procedure of our GPT-Critic does not hurt the agent ’ s capability to generate human-like sentences , given that the generated action candidates were all natural-looking sentences due to the power of large pre-trained LM . Our algorithm is built upon the GPT-2 but it can be adopted for any generative pre-trained language model . 3.1 POLICY EVALUATION . Our GPT-Critic starts by training the action-value function ( i.e . critic ) , which can evaluate the candidates for the response . The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value . The parameterization of the critic network Qφ is designed to share the parameters of the Transformer ( Vaswani et al. , 2017 ) layers of GPT-2 , where the parameters of the Transformer layers are only updated during the policy improvement step . The critic network of GPT-Critic is trained by minimizing the temporal difference error with respect to the dataset D : arg min φ E ( ht , at , rt , ht+1 , at+1 ) ∼D [ ( rt + γQφ̄ ( ht+1 , at+1 ) −Qφ ( ht , at ) ) 2 ] ( 2 ) where φ̄ is the parameters of the target network . Note that Eq . ( 2 ) is an on-policy evaluation on the dataset D , which can be optimized very stably since every at+1 is always an in-distribution sample of D. This is in contrast to Eq . ( 1 ) , which requires evaluation of out-of-distribution actions sampled from the target policy π . The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset . This kind of on-policy evaluation has been explored in the offline RL context for stable policy optimization ( Brandfonbrener et al. , 2021 ; Goo & Niekum , 2021 ) , but they are limited to only one-step policy improvement : once the policy π is improved by the initial on-policy Q-function ( i.e . π ( s ) = arg maxaQ ( s , a ) ) , the new policy deviates from the dataset policy , thus it requires off-policy evaluation for further policy iteration . In contrast , our GPT-Critic performs policy improvement by generating an improved dataset based on the learned critic , where we can perform on-policy evaluation on the new dataset again . As a consequence , GPT-Critic can enjoy the stable multi-step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset , which will be discussed in the following section .

Summary: This paper presents a reinforcement learning-based approach to building a task-oriented dialogue agent. Given a dialogue dataset annotated with rewards, the state-action value function is first trained by minimizing temporal differences. Then, a new training dataset is created by using the best actions selected among the candidates generated by the current policy (i.e., the language model). The policy is then updated by behavior cloning using the created dataset, and the whole process is repeated. The authors have conducted experiments using MultiWOZ and ConvLab and shown that this iterative process improves the performance of the agent.