Why Do Neural Response Generation Models Prefer Universal Replies?

1 INTRODUCTION . Past years have witnessed the dramatic progress on the application of generative sequential models ( also noted as seq2seq learning ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) ) on Neural Response Generation ( NRG ) fields ( Vinyals & Le , 2015 ; Serban et al. , 2017 ) . Seq2seq model has been proved to be capable of directly generating reply given an open domain query ( Li et al. , 2016c ; Xing et al. , 2017 ) . Both relevant words or phrases are automatically selected , and smoothness and fluency of responses are guaranteed through the end-to-end learning . Moreover , abundant impressive humanto-machine conversation cases have been presented in many previous studies ( Serban et al. , 2016 ; Shang et al. , 2015 ; Shao et al. , 2017 ) . Despite these promising results , current Sequence-to-Sequence ( Seq2Seq ) architectures for response generation are still far from steadily generating relevant and coherent replies . The essential issue identified by many studies is the Universal Replies : the model tends to generate short and general replies which contain limited information , such as “ That ’ s great ! ” , “ I don ’ t know ” , etc . ( Li et al. , 2016b ; d ; Mou et al. , 2016 ; Xing et al. , 2017 ) . Intuitively , this problem was attributed to the vast coverage of common replies in the training set and insufficient guiding knowledge in the models ’ response generation step ( Mou et al. , 2016 ; Shao et al. , 2017 ) . Hence , current efforts mainly focus on introducing external information to the model ( Mou et al. , 2016 ; Xing et al. , 2017 ) , and encouraging the model to generate diverse responses in searching space via variational beam search strategies during inference ( Shao et al. , 2017 ; Li et al. , 2016b ; d ) . Nevertheless , most previous analysis over the issue are empirical and lack of statistical evidence . Therefore , in this paper , we conduct an in-depth investigation on the performance of seq2seq models on the NRG task . In our inspections on the existing dialog corpora , it is shown that those repeatedly appeared replies have two essential traits : 1 ) Most of them are composed of highly frequent words ; 2 ) They cover a large portion of the dialog corpora that each universal reply stands for the response of various queries . Above characteristics of universal replies deviate the NRG from other successful applications of sea2seq model such as translation , and lead current generative NRG models to prefer common replies . To discuss the influences from the specific distributed corpus , we decompose the target sequence ’ s probability into two parts and analyze the probability respectively . To break down the mentioned characteristics of dialog corpora in the model training step , we propose a ranking-oriented regularization term to prune the scores of those irrelevant replies . Experimental results reveal that the model with such regularization can produce better results and avoid generating ambiguous responses . Also , case studies show that the issue of generic response is alleviated that these common responses are ranked relatively lower than more appropriate answers . The main contributions of this paper are concluded as follows : 1 ) We analyze the loss function of Seq2seq models on NRG task and conclude several critical reasons that the NRG models prefer universal replies ; 2 ) Based on the analysis , a max-marginal ranking regularization is presented to help the model converge to informative responses . 2 ANALYSIS OF SEQ2SEQ MODELS FOR NRG . Different from significant advances in machine translation ( Bahdanau et al. , 2015 ) and abstractive summarization ( Rush et al. , 2015 ; Nallapati et al. , 2016 ) , it remains challenging to apply Seq2Seq models in practical response generation . One widely accepted issue within current models is that Seq2Seq architectures are inclined to produce common and unrelated replies , even when the quality of training data is significantly improved and different Seq2Seq variants are proposed . The primary reason for this phenomenon lies in the fact that the semantic constraint from query to the possible responses is naturally weak , since the responses to a given query are not required to be semantically equivalent . In contrast , the references in machine translation or summarization are usually restricted to be equivalent to each other semantically or even lexically . Especially , for machine translation , words that appear in the target language should satisfy word level mapping from the source sentence , so the learned word alignment function could ensure the model to generate suitable translated words . Different from learning the semantic alignments between languages in NMT , in NRG the replies can be diversified as they only need to satisfy the causality with the given queries . Moreover , given a query , the sequential model is optimized to learn the shared information among all replies , thus the model is more likely to choose those high-frequent common replies , which is also mentioned in Ritter et al . ( 2011 ) . Taking the case in Table 1 for example , the topic of this query is about movie . It can be observed that the replies shown in the table are semantically diversified : the first two replies are related to the opinion of the respondent toward the movie , while the rest of the replies are about the director , content , and origin of the movie . By contrast , the two valid translations in French are very similar regarding their semantics , which can be attributed to the fixed word-level mapping between query and targets . 2.1 PROBLEM DECOMPOSITION . The sequence-mapping problem in NRG can be decomposed into two independent sub-learning problems : 1 ) Target word selection , in which a query is summarized and translated into the semantic space of responses , and then a set of target words is selected to represent the meaning ; 2 ) Word ordering , in which a grammatical coherent reply is generated based on the candidate word set ( Vinyals et al. , 2016 ) . The word selection and ordering of the target sequence are jointly learned which can also be reflected in the model ’ s loss function by two possible factored phases : − log p ( y|x ) = − log p ( S ( y ) |x ) − log p ( y|S ( y ) , x ) ( 1 ) where x stands for the given query and y is the corresponding response with n words . Besides , S ( y ) = { w1 , · · · , wn|wi ∈ y , i ∈ [ 1 , n ] } represents all predicted words without sequential order , so p ( S ( y ) |x ) is referred as the probability of the target word selection . Meanwhile , p ( y|S ( y ) , x ) indicates the probability of word ordering given this group of possible words . Thus , the objective can be redescribed from maximizing the probability of the ground truth response y under query x to maximizing these two joint probabilities simultaneously . After the above interpretation , we will further discuss the impact of the implicative constriction from two separated probabilities in Eq . 1 , which results in the potential failure of models in learning conversational patterns . 2.2 TARGET WORD SELECTION PROBABILITY . Assuming that we have a set of K ground-truth replies : { y1 , · · · , yK } to a given query x , the upper bound of the target word selection probability can be derived via Jensen ’ s Inequality ( Boyd & Vandenberghe , 2004 ) : K∑ k log p ( S ( yk ) |x ) = K∑ k log ∏ w∈S ( yk ) p ( w|x ) = ∑ w∈∪Kk S ( yk ) log p ( w|x ) ≤ LS log ∑ w∈∪Kk S ( yk ) p ( w|x ) LS ( 2 ) where ∪Kk S ( yk ) denotes all the words appearing in the entire response set , and LS = | ∪Kk S ( yk ) | . Thus , optimizing the first segment is proportional to maximizing the last conditional probabilities , and the optimal strategy is to assign probabilities according to the frequency of words in these K responses . Such strategy adopted by Seq2Seq can be verified by the long-tailed distribution of words in Fig . 1 , in which only few common words are assigned with preferred high probabilities . Given that , during the inference , the best strategy is to employ more frequently occurring words rather than rare ones such as “ background , ” “ art , ” and “ director ” in Table 1 . Furthermore , assuming that each response contains a fixed number of T words ( so that 1 ≤ LS ≤ K × T ) , we can find that the probability of each response for x is inversely proportional to K : LS log ∑ w∈∪Kk S ( yk ) p ( w|x ) LS = LS log E ( w|x ) × T K × T × LS ∝ log 1 ( K × LS ) LS ≤ log 1 K ( 3 ) where E ( w|x ) denotes the mean frequency of words appeared in these K replies , which is 1.32 for the cases in Table 1 . In general , the mean frequency is around 1 owing to the long-tailed Unigram distribution which satisfies Zipf ’ s law ( Zipf , 1935 ) . In other words , the target word selection probability is limited by K , so queries with more diverse answers are more challenging to learn . Meanwhile , it is difficult to obtain good predictions for lower-informational queries , as they contain more possible responses which are somewhat equivalent to a larger K ( Li et al. , 2016a ) . Nonetheless , the translation task requires word-level mappings as they are well-aligned in the semantic space , therefore source and target sentences are semantically equivalent . So that , translated candidates are confined toK ≈ 1 . Thus the upper bound can be approximated as the full probability . 2.3 WORD ORDERING PROBABILITY . 2.3.1 LEMMAS . Before discussing the word ordering probability , we present four lemmas and corresponding proofs . Moreover , all these lemmas are only available for the response generation task except Lemma 1 . According to the Zipf ’ s law ( Zipf , 1935 ) , the frequency of any word is inversely proportional to its rank in the frequency table , such that the probability p ( wi ) = Z/iα , where Z ≈ 0.1 , α ≈ 1 , and i is the frequency rank of the word wi . Then , denoting the vocabulary size as V and the total number of query-response pairs as N , we can formulate two characteristics of a universal reply y as follows : 1 ) A response is universal if it consists of only top-t ranked words . For any wordw in such response , p ( w ) ≥ 1/ ( 10t ) according to the Zipf ’ s law . 2 ) The amount of possible queriesM of y is directly proportional to the size of query-response pairs N , noted as 1 M ∝ N . To simplify , we suppose that t > 1000 to cover most universal replies , and the frequency of the response not belonging to the universal replies is a constant c ( 1 ≤ c M ) . Accordingly , we can derive the following lemmas . Lemma 1 p ( S ( y ) |y ) = 1 , p ( S ( y ) , y ) = p ( y ) , p ( x , y , S ( y ) ) = p ( x , y ) . Proof . Lemma 1 describes the obvious fact that the event “ the word set of the response equals to S ( y ) ” must happen when the event ” y stands for the response ” is established . Lemma 2 p ( x|yur ) = 1 , where 1 > 0 and is sufficiently small , and yur is a universal reply . Proof . Based on the second character of the universal reply and the fact thatN is a very large number for any large scaled datasets , Lemma 2 is established as : p ( x|yur ) = 1M ∝ 1 N = 1 Lemma 3 ∑ i p ( y ur i |S ( y ) ) → 1 , p ( yoj |S ( y ) ) = 2 , where 2 > 0 and is sufficiently small , yuri stands for the i-th universal reply and yoj is the j-th non-universal grammatical replies , meanwhile , S ( yuri ) ⊆ S ( y ) and S ( yoj ) ⊆ S ( y ) Proof . According to the following inequation ∑t i 1 i > ∫ t+1 1 1 xdx = ln ( t + 1 ) , we can get the conclusion that the probability of a chosen word belonging to the most frequent t words is large than 0.1 ∗ ln ( t + 1 ) > 0.69 . Since y contains T words , there is at least T ln ( t + 1 ) words belonging to the top-t ranked on average according to the binomial distribution . We suppose m responses are universal replies among the n possible responses when their words are constrained by S ( y ) . Besides , the proportion of m can be computed as : m n = Tln ( t+1 ) ∑ i=1 CiT∑T j=1 C j T ∗ 1 10 ln ( t+ 1 ) = 2T − ∑T i=Tln ( t+1 ) C i T 2T ∗ 1 10 ln ( t+ 1 ) > 1 20 ln ( t+ 1 ) > 0.34 ( 4 ) where C donates the combination . Since n/m is not a very large number , the total probability of these m replies can be deducted as : ∑ i p ( yuri |S ( y ) ) = ∑m i f ( y ur i ) ∑m i f ( y ur i ) + ∑n−m i f ( Y o i ) = M ∗m M ∗m+ c ∗ ( n−m ) = M M + n/m− c > M M + 3− c ( 5 ) where f ( y ) donates the frequency of a response y in the corpus . According to the Eq . 5 and the fact that M ∝ N is a very large number for any practical large-scale datasets , ∑ i p ( y ur i |S ( y ) ) → 1 can be established . Apparently , for any other candidate response yoj , its probability satisfies p ( yoj |S ( y ) ) < 1− ∑ i p ( y ur i |S ( y ) ) = 2 . Lemma 4 Assuming each informative query has K ground-truth replies and the query-response pairs are extracted from a multi-turn conversational corpus , a reply y not belonging to universal replies has K unique queries , noted as p ( x|y ) = 1K . Proof . Most query-response pairs are extracted from a practical large-scale multi-turn conversational corpus , so that any response always works as the post in another pair . That is , y also appearsK times as it also has K replies . Therefore , there also exist K unique posts for y .

Summary: The paper investigates the problem of universal replies plaguing the Seq2Seq neural generation models. The problem is indeed quite important because for problems with high entropy solutions the seq2seq models have been shown to struggle in past literature. While the authors do pick a good problem, that's where the quality of the paper ends for me. The paper goes on an endless meandering through a lot of meaningless probabilistic arguments.  First of all, factorizing a seq2seq model as done in equation 1 is plain wrong. The model doesn't operate by first selecting a set of words and then ordering them. On top of this wrong factorization, section 2.2 & 2.3 derives a bunch of meaningless lemmas with extremely crude assumptions. For example, for lemma 3, M is supposed to be some universal constant defined to be the frequency of universal replies while all other replies seem to have a frequency of 1. Somehow through this wrong factorization and some probabilistic jugglery, we arrive at section 3 where the takeaway from section 2 is the rather known one that the model promotes universal replies regardless of query.