Offline Reinforcement Learning for Large Scale Language Action Spaces

1 INTRODUCTION . Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing ( NLP ) tasks , yet challenging due to large language action spaces and limited availability of human-annotated data . Recently , large-scale pre-trained language models ( LM ) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary ( Devlin et al. , 2019 ; Radford et al. , 2019 ; Brown et al. , 2020 ; Raffel et al. , 2019 ) . The current best performing end-to-end conversational agents for a task-oriented dialogue system utilize a pre-training on largescale corpus and fine-tuning on downstream tasks ( Ham et al. , 2020 ; Yang et al. , 2021 ; Lin et al. , 2020 ; Peng et al. , 2021 ) . This combination of pre-training and fine-tuning significantly improves overall performance in the task-oriented dialogues . However , supervised fine-tuning ( i.e . imitation learning of the dialogue corpus ) alone may not be sufficient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels . Thus , in order to optimize the task performance of the conversational agent , goaloriented training ( i.e . reinforcement learning ) is an essential and promising direction to pursue . Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as offline reinforcement learning ( RL ) problem ( Levine et al. , 2020 ; Fujimoto et al. , 2019 ; Jaques et al. , 2020 ) , which offers the prospect to optimize the policy solely from the fixed dataset without online environment interaction . Most of the existing offline RL methods are built on the off-policy ActorCritic framework , which performs iterative optimization of the policy ( i.e . actor ) and the actionvalue function ( i.e . critic ) ( Fujimoto et al. , 2019 ; Janner et al. , 2019 ; Kumar et al. , 2020 ) . Yet , a naive application of these offline RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language ( Lewis et al. , 2017 ; Zhao et al. , 2019 ; Jang et al. , 2020 ) . To mitigate the aforementioned problem , discrete latent representation models for language actions have been proposed to disentangle the semantics of the utterance and the natural language generation ( Zhao et al. , 2019 ; Yarats & Lewis , 2018 ) . In that framework , goal-based training was performed in the space of the discrete latent variables instead of directly optimizing utterances . While these approaches can prevent degenerate responses in principle , they can not be straightforwardly applied to widely used large-scale LMs since most of them are not trained for discrete latent variables . One could make modifications to the LM to work with discrete latent variables but it would need to be pre-trained from scratch with a tremendous amount of data and time , which is undesirable . In this paper , we present an offline RL algorithm for task-oriented dialogue , which can be adopted for any generative pre-trained language model . Our algorithm , GPT-Critic , does not rely on policy gradient unlike actor-critic methods and is essentially free from the issue of diverging from human language . It starts with fine-tuning the GPT-2 model and learning the action-value function ( critic ) using the dialogue corpus . Then , GPT-Critic generates a strategically promising action that is selected based on the value estimated by the critic . GPT-Critic updates the policy through behavior cloning of the critic-guided self-generated responses . This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus , where the action choice is restricted to the support in the dataset ( Wang et al. , 2020 ) . Since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2 , it thus inherits GPT-2 ’ s ability to generate human-like responses . In the experiments , we demonstrate that GPT-Critic outperforms the state-of-the-art end-to-end dialogue agent in the task-oriented dialogue benchmarks including MultiWOZ ( Budzianowski et al. , 2018 ) and ConvLab ( Zhu et al. , 2020 ) . 2 BACKGROUND . 2.1 OFFLINE REINFORCEMENT LEARNING FOR TASK-ORIENTED DIALOGUES . We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process ( POMDP ) defined by tuple 〈S , A , O , T , Z , R , γ〉 where S is the set of environment states s = 〈g , h〉 ( underlying state that consists of the user goal g and dialogue history h ) , A is the set of actions a ( a sequence of tokens which represents dialogue act and system response ) , O is the set of observations o ( user utterance ) , T ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) is the transition function , Z ( o|s′ , a ) = Pr ( ot+1 = o|st+1 = s′ , at = a ) is the observation probability , R ( g , h , a ) is the reward function indicating the utility of executing action a in history h and the user goal g , and γ ∈ ( 0 , 1 ) is a discount factor . The history at time step t , ht = { o0 , a0 , . . . ot−1 , at−1 , ot } , is a sequence of all previous observations and actions . Since the underlying state s ( e.g . user goal ) is not directly observable , the agent makes decisions based on the entire observation-action history . The policy π ( at|ht ) is mapping from history ht to a probability distribution over A . The goal is to find an optimal policy π∗ that maximizes the expected cumulative rewards , i.e . π∗ = arg maxπ Eπ [ ∑∞ t=0 γ tR ( g , ht , at ) ] . The action-value function of policy π is defined as Qπ ( h , a ) : = Eπ [ ∑∞ t=0 γ tR ( g , ht , at ) |h0 = h , a0 = a ] , where Qπ is a unique solution of the Bellman equation : Qπ ( h , a ) = Eg [ R ( g , h , a ) ] + γEπ [ Qπ ( h′ , a′ ) ] . Using offline RL for dialogue policy optimization , the agent optimizes the policy from the precollected dataset D = { { ( gj , hjt , a j t , r j t , h j t+1 ) T t=0 } Nj=1 } without online environment interaction during the intermediate stages of training . Prior offline RL algorithms ( Fujimoto et al. , 2019 ; Janner et al. , 2019 ; Kumar et al. , 2020 ) rely on off-policy actor-critic method , where the critic network is trained by minimizing the temporal differnce error with respect to the target policy π : arg min φ E ( ht , at , rt , ht+1 ) ∼D [ ( rt + γEat+1∼π ( ht+1 ) [ Qφ̄ ( ht+1 , at+1 ) ] −Qφ ( ht , at ) ) 2 ] ( 1 ) where φ̄ is the parameters of the target network . As discussed in the prior work ( Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) , optimizing this loss can be challenging in the offline RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution ( OOD ) actions to evaluate the value of the next state . 2.2 END-TO-END TASK-ORIENTED DIALOGUE SYSTEM . We focus on the MultiWOZ dataset ( Budzianowski et al. , 2018 ) , which is a representative benchmark for task-oriented dialogue . The MultiWOZ dataset is a fully-annotated corpus of human-human task-oriented conversations , which is collected via the Wizard-of-Oz setting ( Kelley , 1984 ) . The traditional approach to building a task-oriented dialogue system adopts a modular pipeline , which consists of the following four modules : 1 ) A natural language understanding ( NLU ) module ( Kim et al. , 2017 ; Zhu et al. , 2020 ) identifies the user ’ s intent and extracts the information of slots and their values , 2 ) A Dialogue state tracking ( DST ) module ( Williams et al. , 2013 ) infers the belief state , 3 ) A dialogue policy ( POL ) module decides the system action , 4 ) A natural language generation ( NLG ) module ( Wen et al. , 2015 ) generates the system response corresponding to the system action . Recently , end-to-end task-oriented dialogue methods leveraging the pre-trained language model have been proposed ( Yang et al. , 2021 ; Ham et al. , 2020 ; Lin et al. , 2020 ; Peng et al. , 2021 ; Hosseini-Asl et al. , 2020 ) , and significantly improves overall performance in the task-oriented dialogues . In this paper , our algorithm is built upon UBAR ( Yang et al. , 2021 ) , which is based on GPT-2 ( Radford et al. , 2019 ) and currently the state-of-the-art end-to-end dialogue agent for the MultiWOZ domain . 3 OFFLINE REINFORCEMENT LEARNING FOR END-TO-END TASK-ORIENTED DIALOGUE SYSTEMS . The corpus collected from human-human conversations inevitably contains unsuccessful dialogues in terms of task completion . For example , approximately 20 % dialogues of the MultiWOZ dataset fail to meet the user goal . Therefore , a naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot of unsuccessful dialogues : an agent that imitates failure would be inevitably suboptimal . Yet , dropping the unsuccessful dialogues from the corpus is also undesirable , since they may contain some task-specific information that is useful to properly respond to user requests . We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while improving the task performance . In this section , we present GPT-Critic , an offline RL algorithm for task-oriented dialogue . Our GPTCritic is analogous to Actor-Critic method : GPT ( Actor ) decides which action to take while the Critic informs how good the action was and provides a signal for policy improvement . Still , GPTCritic is distinct from the Actor-Critic methods in that it does not rely on the policy gradients , which are generally known to cause the issue of diverging from human language ( Lewis et al. , 2017 ; Zhao et al. , 2019 ) . Instead , we sample a set of action candidates using GPT-2 and pick the best one using the critic , which constitutes a revised dialogue corpus . Then , we perform supervised fine-tuning of the GPT-2 on the revised dialogue corpus . This learning procedure of our GPT-Critic does not hurt the agent ’ s capability to generate human-like sentences , given that the generated action candidates were all natural-looking sentences due to the power of large pre-trained LM . Our algorithm is built upon the GPT-2 but it can be adopted for any generative pre-trained language model . 3.1 POLICY EVALUATION . Our GPT-Critic starts by training the action-value function ( i.e . critic ) , which can evaluate the candidates for the response . The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value . The parameterization of the critic network Qφ is designed to share the parameters of the Transformer ( Vaswani et al. , 2017 ) layers of GPT-2 , where the parameters of the Transformer layers are only updated during the policy improvement step . The critic network of GPT-Critic is trained by minimizing the temporal difference error with respect to the dataset D : arg min φ E ( ht , at , rt , ht+1 , at+1 ) ∼D [ ( rt + γQφ̄ ( ht+1 , at+1 ) −Qφ ( ht , at ) ) 2 ] ( 2 ) where φ̄ is the parameters of the target network . Note that Eq . ( 2 ) is an on-policy evaluation on the dataset D , which can be optimized very stably since every at+1 is always an in-distribution sample of D. This is in contrast to Eq . ( 1 ) , which requires evaluation of out-of-distribution actions sampled from the target policy π . The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset . This kind of on-policy evaluation has been explored in the offline RL context for stable policy optimization ( Brandfonbrener et al. , 2021 ; Goo & Niekum , 2021 ) , but they are limited to only one-step policy improvement : once the policy π is improved by the initial on-policy Q-function ( i.e . π ( s ) = arg maxaQ ( s , a ) ) , the new policy deviates from the dataset policy , thus it requires off-policy evaluation for further policy iteration . In contrast , our GPT-Critic performs policy improvement by generating an improved dataset based on the learned critic , where we can perform on-policy evaluation on the new dataset again . As a consequence , GPT-Critic can enjoy the stable multi-step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset , which will be discussed in the following section .

Summary: The paper works on offline reinforcement learning for natural language action space setting, particularly for task-oriented dialogue management. The paper nicely incorporate the policy network (to sample agent response) and the q network (to evaluate the agent response) into a single GPT-2 network and propose a policy interation algorithm to optimize both the q and policy network. During policy evaluation, the q network is updated with sampled system actions and responses. During policy improvement, the sampled system actions and responses with maximum q values are used as labels to update the policy network. The model achieves SoTA performance on MultiWOZ dataset.