Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History

1 INTRODUCTION . In collaborative dialog settings , agents work together and communicate to reach a common goal ( He et al. , 2017 ) , such as booking flight and making restaurant reservation . Historically , in collaborative setting , the dialog history and structure is modeled explicitly by tracking semantic content , for example , the set of used slot-value pairs ( Bowden et al. , 2017 ; Larionov et al. , 2018 ; Zeigler & Mazor , 1995a ; b ) . Prior work also models dialog history implicitly by using an encoder-decoder model ( Sordoni et al. , 2015 ; Shang et al. , 2015 ; Vinyals & Le , 2015 ; Li et al. , 2016 ; Wen et al. , 2015 ; Yao et al. , 2015 ) . Although these techniques show promising results in a collaborative setting , they have drawbacks when applied to non-collaborative settings , where agents have competing interests and goals but aim to reach an agreement , and they use various strategies and tactics to reach an agreement favorable to them . In non-collaborative dialog settings , leveraging effective sequences of tactics is as important as controlling for semantic content , and different tactic sequences lead to different outcomes ( Zhou et al. , 2019 ) . Learning latent dialog structure efficiently is challenging for dialog systems . Prior work mainly focused on applying hidden Markov models ( HMMs ) to capture contextual dependencies within dialogs ( Chotimongkol , 2008 ; Ritter et al. , 2010 ; Zhai & Williams , 2014 ) . Recently , Shi et al . ( 2019 ) proposed to use a discrete variational recurrent neural network ( D-VRNN ) for learning latent dialog structure because of its flexibility and nonlinear nature . In this paper , we take a different approach by using pre-trained FSTs to learn latent dialog structure . FSTs have been used in many traditional dialog systems and have proven to be effective across different domains ( Larionov et al. , 2018 ; Zeigler & Mazor , 1995a ; b , inter alia ) . We focus on modeling dialog in non-collaborative settings , and propose to explicitly leverage the dialog structure , including history of tactics and dialog acts , to improve dialog planning and generation . Specifically , we use weighted FSTs to learn dialog acts and tactics history and then integrate the learned FSTs to encoder-decoder pipeline to make the end-to-end system capture semantic and tactic history . FSTs have several advantages over a traditional recurrent neural network . First , an FST can explicitly track the entire path it traversed , which gives additional symbolic constraints and information about the dialog history . Due to the more informative history representation , an FST has a better prediction of the next step ( dialog act/strategy ) compared to an RNN , as we empirically confirm . A trained FST serves as a scaffold for dialog history tracking . Second , FST is more interpretable , as each state is explicitly represented by an action distribution . It is thus easier for humans to interpret model decision . To leverage pre-trained FSTs , we propose an FST-enhanced hierarchical encoder-decoder model ( FeHED ) . Our model , depicted in Figure 1 , consists of a natural language understanding ( NLU ) module , two pre-trained FSTs , and a natural language generation ( NLG ) module . The NLU module has a set of classifiers , where each one is responsible for detecting a dialog act or a negotiation/persuasion strategy/tactics from a given utterance . FSTs model a latent dialog structure , which can encode dialog history . One FST is trained on sequences of dialog acts ( FST-DA ) and the other FST is trained on sequences of strategies ( FST-S ) . The NLG module is a hierarchical encoderdecoder model ( HED ) , which conditions on the outputs from FST- { DA , S } and previous utterances to predict strategies and generate system response . We focus on ( 1 ) a bargaining scenario where a seller negotiates with a buyer over a given product through a chat interface online ( He et al. , 2018 ) , and ( 2 ) a persuasion dialog setting where a persuader , in an online chat , attempts to persuade the persuadee to donate their earnings to a charity ( Wang et al. , 2019 ) . We propose an automated agent that plays the role of the seller/persuader . Existing work only focuses on dialog acts , such as “ disagree ” , which capture shallow semantics . However , these dialog acts can not capture the pragmatics of the dialog acts . For example , whether the user disagrees politely or rudely impacts the dialog system ’ s behavior . To capture pragmatic content , we employ negotiation strategies and tactics introduced by Zhou et al . ( 2019 ) , motivated by negotiation literature . For persuasion dialog setting , we adopt a set of persuasion strategies from Wang et al . ( 2019 ) . Besides pragmatics , these strategies also capture domain-specific semantics that dialog acts do not cover . We evaluate our seller/persuader models using standard measures , BLEU ( Papineni et al. , 2002 ) and accuracy of predicting strategies . Additionally , we propose unigram and bigram accuracy of strategies to evaluate dialog models fairly . Experiment results show that FeHED significantly outperforms all the baseline models on four of the five metrics . Moreover , quantitative analysis shows that it is important to model both semantic and tactic history . Finally , qualitative analysis demonstrates that FSTs can track tactic history better than a RNN in non-collaborative settings . 2 MODEL . Figure 1 gives an overview of FeHED . There are four components in FeHED : a dialog act classifier , a strategy classifier , two finite state transducers ( FST-DA/S ) , and a hierarchical encoder-decoder model ( HED ) . The four components are connected . The output of the dialog act classifier and strategy classifier is the input of the FSTs . The FSTs ’ output is the input for HED along with utterance embedding . Finally , HED generates both the next strategy and the next utterance . 1 . Dialog Act Classifier converts utterances , denoted by u1 , u2 , .. , ut , into a sequence of dialog acts da1 , da2 ... , dat 2 . Strategy Classifier converts utterances into a sequence of strategies and tactics st1 , st2 , ... , stt used in each utterance . 3 . FST-DA/S takes a sequence of dialog acts da1 , da2 , ... , dat or strategies st1 , st2 , ... , stt ( green dotted lines in Figure 1 ) and returns a sequence of state embeddings sda/st1 , s da/st 2 , ... , s da/st t . 4 . HED conditions on sda/st1 , s da/st 2 , ... , s da/st t ( indicated by blue dotted lines ) and u1 , u2 , ... , ut to predict a set of possible strategies st′t+1 in the next utterance and generate the response ut+1 . We describe each component next . s da/st 1 , s da/st 2 , ... , s da/st t to the hierarchical encoder-decoder ( HED ; blue dotted lines ) . Lastly , HED generates the system response ut+1 and predicts the next strategies st′t+1 . The red dotted line indicates a posterior constraint . ⊕ represents concatenation . 2.1 DIALOG ACT CLASSIFIER . We follow the setting in He et al . ( 2018 ) and define a list of seven dialog acts , including introduction , initiate-price , insist on price , agree , disagree , inform and inquire . He et al . ( 2018 ) detect these dialog acts with a set of handcrafted regular expressions . Details about dialog acts are given in Appendix A.1 . 2.2 STRATEGY CLASSIFIERS . We use fifteen negotiation tactics , proposed by Zhou et al . ( 2019 ) , operationalizing strategies and tactics described in economics and behavioral science research on negotiation ( Pruitt , 1981 ; Bazerman et al. , 2000 ; Fisher & Ury , 1981 ; Lax & Sebenius , 2006 ; Thompson et al. , 2010 ) . These include rhetoric strategies , e.g. , talk informally , use certainty words , and behavioral strategies , e.g. , build rapport , address buyer ’ s concerns . Part of these strategies are implemented using regular expressions , for example , “ please ” is a salient word for communicate politely . Other strategies are implemented using linear classifiers , trained on negotiation dialogs . Details about negotiation strategies are given in Appendix A.2 . We adopt a set of persuasion strategies , developed by Wang et al . ( 2019 ) , based on the Elaboration Likelihood Model ( ELM ) theory from social psychology ( Petty & Cacioppo , 1986 ) . Persuasion strategies include persuasive appeal , e.g. , logical appeal , emotion appeal , and persuasive inquiry , e.g. , source-related inquiry , personal-related inquiry . These strategies are captured with hybrid recurrent convolutional neural network classifiers ( Wang et al. , 2019 ) . Details about persuasion strategies are given in Appendix A.3 . 2.3 FSTS FOR SEMANTIC AND STRATEGIC DIALOG HISTORY . Two FSTs were trained to learn latent dialog structure . One is trained with dialog act sequences , while the other one is trained with strategy sequences . Training data for FSTs is sequences of dialog acts or strategies , where each sequence is extracted from a dialog by using dialog act/strategy classifier . We initialize FST with a single ergodic node where all edges go back to itself . We then iterate over this initial FST and split the state ( s ) . We greedily select a state split , based on splitting on the incoming dialog act/strategy that minimizes the entropy of the two generated states . This follows the algorithm often used in node splitting in building decision trees ( Breiman et al. , 1984 ) . There is no well-defined stopping criteria , therefore , we need to choose the number of states as a hyperparameter . We show an example of trained FSTDA with 3 states in Appendix A.4 . By looking at incoming and outgoing edges of a node , we can understand which dialog state this node represents . For example , if the incoming edge of a state is “ seller answering a question ” and the outgoing edge is “ buyer inquiring ” , then in this state , the buyer is most likely to ask a question . The trained FST gives a probability mass function ( PMF ) for the likelihood of transfer from the current state to each of the possible dialog acts . We use this vector of probabilities as a state embedding ( sda/stt ) , which can be generated for each utterance and concatenated to other utterance representations . At test time , we use our dialog act or strategy classifiers to choose the predicted dialog act or strategy , and transition to the new state in the FST to get the PDF for the next set of dialog acts . The FST not only returns the current state embedding , but also all the embeddings of state it traversed since the start state . 2.4 HIERARCHICAL ENCODER-DECODER ( HED ) . Let ut = [ wt1 , ... , w t n ] , where w t i is the i-th word in current utterance ut . We use a standard GRU ( Cho et al. , 2014 ) to encode ut into a hidden state representation hut . hut = GRU u ( ut ) We concatenate the utterance embedding with the output of FST-DA and FST-S to enrich the utterance embedding to incorporate dialog history . h′ut = [ h u t ; s da t ; s st t ] . Finally , we use another GRU to combine all utterances till current time to encode the entire dialog into a hidden state hdt hdt = GRU d ( h′u1 , h ′u 2 , ... , h ′u t ) Then we predict the next utterance strategies stt+1 and finally , generate system response ut+1 using hdt . Strategy predictor Before generating system response , we add an intermediate step to predict the set of possible strategies stt+1 in it . The output stt+1 is a 15-dimensional binary-value vector , where each dimension represents whether a certain negotiation strategy occurred in ut+1 . We compute the probability of the j-th strategy occurring in ut+1 as : p ( stt+1 , j = 1|hdt ) = σ ( Wj [ hdt ] + bj ) where Wj and bj are learned parameters . We denote the negative log likelihood of strategies LST as the loss function of this task : LST = − ∑ { j|st′t+1 , j=1 } log ( stt+1 , j ) − ∑ { j|st′t+1 , j=0 } log ( 1− stt+1 , j ) , where st′t+1 is the ground truth strategies . Utterance decoder is a standard GRU decoder with attention ( Bahdanau et al. , 2015 ) . The input of this decoder is the dialog hidden state hdt concatenated with the predicted strategies stt+1 . It calculates a probability distribution pt+1j over vocabulary at time j conditioning on the previous word : pt+1j = softmax ( GRU de ( [ stt+1 ; h d t ] , w t+1 j−1 ) ) Cross entropy loss LNLG is used for this generation task : LNLG = − ∑ { wj∈u′t+1 } log ( pt+1j , wj ) , where u′t+1 is the target utterance . Finally , we combine the strategy prediction task and system utterance generation loss together . We also add a posterior constraint to enforce the generated utterances and the predicted negotiation strategies align with each other : Ljoint = LNLG + αLST + β ∑ j 1 ( stt+1 , j /∈ ut+1 ) where α , β are constants ( both were set to 1.0 in our experiments ) and the last term is a posterior constraint ( the red dotted line in Figure 1 ) that has a positive value if the strategies in the generated utterance ut+1 does not contain some strategies in the predicted strategies stt+1 . We jointly train strategy predictor and utterance decoder using Ljoint .

Summary: This paper proposed a framework for non-collaborative dialog systems. To track the history better, the authors propose to apply two pre-trained finite state transducers (FSTs) to take the sequence of dialog acts or strategies as inputs and output sequences of state embeddings for the hierarchical encoder-decoder (HED) part as inputs. The authors test their model on two tasks, a bargain task and a persuasion task.