Patent ID: 12260185

In the figures, elements having the same designations have the same or similar functions.

DETAILED DESCRIPTION

The embodiments are directed to a coarse-to-fine abstractive dialogue summarization neural network model or CorDial that is equipped with granular controllability. Initially, the CorDial model creates a summary draft that contains user intent information and important key phrases, if any, that may appear in the summary for each dialogue turn. This summary draft may be prefixed to the human-annotated summary while finetuning a summary generator. The summary draft provides some weak supervision because the final summary is conditioned on the generated summary draft.

The embodiments are also directed to a CorDial model that is trained to clip the dialogue text with special tokens. The CorDial model then matches each summary sentence to its corresponding clipped dialogue context in the dialogue text. In this way, the CorDial model generates a single sentence for each clipped dialogue context. Clipping dialogue text enables the CorDial model to generate a dialogue summary at different granularity by highlighting arbitrary numbers of text spans from a dialogue. This also makes the dialogue summary more interpretable.

In some embodiments, the CorDial model is built on top of another language model, such as a BART language model, that is pre-trained with unsupervised denoising objectives and fine-tuned on the News summarization corpus XSUM.

As used herein, the term “network” or “model” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system, and/or any training or learning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

FIG.1is a simplified diagram of a computing device100, according to some embodiments. As shown inFIG.1, computing device100includes a processor110coupled to memory120. Operation of computing device100is controlled by processor110. And although computing device100is shown with only one processor110, it is understood that processor110may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device100. Computing device100may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.

Memory120may be used to store software executed by computing device100and/or one or more data structures used during operation of computing device100. Memory120may include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

Processor110and/or memory120may be arranged in any suitable physical arrangement. In some embodiments, processor110and/or memory120may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor110and/or memory120may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor110and/or memory120may be located in one or more data centers and/or cloud computing facilities.

In some examples, memory120may include a non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory120includes instructions for a coarse-to-fine abstractive dialogue summarization model130or (CorDial model130). CorDial model130may be a neural network that includes one or more networks or modules and/or pre-trained language models that perform natural language processing tasks. CorDial model130may receive input, such as a dialogue conversational history140and generate output which may be a dialogue summary150of dialogue conversational history140. Dialogue conversational history140may include multiple dialogue turns that occurred in a dialogue between one or more speakers. Each dialogue turn corresponds to an utterance made by one speaker before an utterance is made by another speaker. In some embodiments, dialogue conversational history140may be defined as D={X1, X2, . . . , XN} where each Xiis a sequence of words in a dialogue turn and N is a total number of dialogue turns. In some instances, dialogue conversation history140may include more than two speakers, each speaker speaking during a corresponding dialogue turn. The dialogue summary150may be defined as an M-sentence dialogue summary Y={Y1, Y2, . . . , YM} that summarizes dialogue conversation history140, but that is typically more brief than the overall dialogue conversation history140.

FIGS.2A and2Bare block diagrams200A and200B of a CorDial model, according to some embodiments.FIG.2Aillustrates the CorDial model130in an inference stage where a trained CorDial model130receives dialogue conversational history140and generates a dialogue summary150which is a summary of the dialogue conversational history140. In some embodiments, CorDial model130may include a pre-trained generative language model205, such as BART-xsum, which may be pre-trained using unsupervised denoising objectives and further fine-tuned using News summarization corpus XSUM. The generative language model205structure may be based on a transformer architecture which includes an autoencoder that is divided into encoder210and decoder215. The encoder210may receive dialogue conversational history140and generate encodings. Decoder215may receive the encodings and generate dialogue summary150.

In some embodiments, generative language model205may receive dialogue conversation history140that is divided into dialogue segments202. Each dialogue segment202is a segment of dialogue conversation history140, and may include one or more dialogue turns. The number of dialogue segments224may correspond to the number of sentences that CorDial model130may generate for dialogue summary150. When generative language model205receives dialogue segment202, encoder210may generate segment encodings. Decoder215may receive and convert the segment encodings into a corresponding segment summary204. Segment summary204may include one sentence that summarizes dialogue segment202in some embodiments. Concatenation module220may receive segment summaries204that decoder215generates from multiple dialogue segments202and concatenates multiple segment summaries204into dialogue summary150. In some embodiments, concatenation module220may concatenate the segment summaries204associated with dialogue conversation history140linearly, that is in the order that generative language model205generates the segment summaries.

In some embodiments, encoder210of generative language model205may also generate summary draft206. The summary draft206may be used to train generative language model205as discussed inFIG.2B, but may be discarded during the inference stage discussed inFIG.2A.

In some embodiments, CorDial model130may include a dialogue-turn-level classifier225. Dialogue-turn-level classifier225may be trained to identify dialogue segments202in dialogue conversation history140by determinizing cutting points208between two dialogue turns in dialogue conversation history140. Each cutting point208separates two dialogue segments202in dialogue conversation history140.

Special highlighting tokens may be inserted into dialogue conversation history140at the identified cutting points208to indicate to generative language model205different dialogue segments202during the inference stage. Generative language model205may then generate segment summary204, for each dialogue segment202indicated by the special highlighting tokens. Concatenation module220may then concatenate the segment summaries204into dialogue summary150as discussed above.

In some embodiments, dialogue conversation history140may be manually divided into dialogue segments202. That is, CorDial model130may receive user input that divides dialogue conversation history140into dialogue segments202by inserting highlighting tokens into dialogue conversation history140.

To generate dialogue summary150, the CorDial model130may be trained. Unlike conventional dialogue summarization models, generative language model205of CorDial model130may be trained using a summary draft. Further dialogue turn level classifier225may be trained to identify cutting points208.FIG.2Bincludes a structure of CorDial model130that includes various training components, according to some embodiments. As illustrated inFIG.2B, CorDial model130may be trained using a similarity module235, parser240and label module245, each of which may be or include a neural network.

Similarity module235may receive dialogue conversation history140and training summary209. Training summary209may be a known summary for dialogue conversation history140that may be used to train CorDial model130. Training summary209and dialogue summary150that may be determined during the inference stage may or may not include the same text or be the same summaries.

Similarity module235may divide dialogue conversation history140into dialogue segments212and training summary209into segment summaries214. To divide dialogue conversation history140into dialogue segments212and training summary209into segment summaries214, similarity module235may include a similarity function, e.g. ROUGE-1 function. Similarity module235may divide dialogue conversation history140into M dialogue segments212, such that one dialogue segment212corresponds to one segment summary214. In an embodiment where M=1, the dialogue conversation history140may be dialogue segment212and segment summary214may be training summary209. In some embodiments, similarity function may match dialogue segment212with segment summary214by finding the dialogue segment that has the highest ROUGE score to one of the tested summary sentences in training summary209. The cutting point may be determined as follows:
tm=arg maxtSIM(Xcm:t, Ym)   Equation 1
where SIM may be a similarity function, e.g. ROUGE-1, cmmay be the accumulated turn index (c0=1 and cm=tm−1) that indicates a part of dialogue conversation history140that has been covered by a summary sentence, and tmis the cutting point in the dialogue conversation history140for the mthsummary sentence.

In some embodiments, parser240and label module245may receive dialogue segments212and/or segment summaries214generated from dialogue conversation history140and training summary209and create a summary draft250. Summary draft250may provide useful weak supervision that may be beneficial to the final summarization task that occurs in generative language model205. The summary draft250may include turn indexes that correspond to a dialogue turns in dialogue conversation history140, labels for action categories associated with the dialogue turns, and zero or more key phrase(s) associated with the dialogue turns.

In some embodiments, label module245may be a neural network. Label module245may assign labels using a Snorkle network. Specifically, label module245may receive dialogue segments212from dialogue conversation history140and assign a label for action category for each dialogue turn in dialogue conversation history140. Action categories may correspond to interrogative pronouns. In some embodiments, label module245may include a set of interrogative pronoun categories, and then assign an action label to each dialogue turn with its action category by a weakly-supervised labelling. The interrogative pronoun categories may be designed to identify functional units of all utterances, serving as the logic of the dialogue. Example action categories may be as follows:WHY: ask the reason of the state mentioned in the previous turn, e.g., “why” or “why not?”WHAT: request more details about the aforementioned object; the sentence usually starts with “what's” or “what about.”WHERE: ask the location of an appointment or event.WHEN: ask the time of an appointment or event, e.g. “when?” or “what time?”CONFIRM: ask the other speaker to establish the correctness of certain case; the sentence usually starts with patterns like “are you?”, “will you,” or “has he”?ABSTAIN: the utterance does not belong to any of the previous categories; this happens when speakers continue to state or comment without seeking for more information from the others.

Notably, training CorDial model130by assigning labels that are action categories is different from the conventional task-oriented dialogue systems which have clear and annotated intents (e.g., book flight and check account) and actions (e.g., inform and request).

In some embodiments, parser240may determine key phrases in dialogue conversation history140. Parser240may be a neural network and may be a constituency parser. Parser240may receive dialogue segment212from dialogue conversation history140and segment summaries214from training summary209. In some embodiments, parser240may parse each dialogue turn in dialogue segments212and each segment summary214in training summary209into one or more parsing trees. Parser240may then identify the longest common sub-sequence, if any, in the parsing trees between each dialogue turn in dialogue segments212and each segment summary in segment summaries214. If parser240identifies the longest common sub-sequence, the longest common sub-sequence becomes a key phrase or key phrase(s) for the dialogue turn. The key phrase(s) are included in summary draft250next to the label for action category for the corresponding dialogue turn. Notably, not every dialogue turn may contain key phrases, in which case the key phrase in summary draft250may be left empty or blank.

FIG.3is a diagram300of an example dialogue conversation history, summary draft, and dialogue summary, according to some embodiments. Specifically,FIG.3illustrates dialogue conversation history140that includes a dialogue between two participants: Morgan and Suzanne, over nine dialogue turns. The first dialogue turn begins with Morgan stating “Hey gorgeous, what's up?” In the seventh dialogue turn, participant Morgan asks “Do you feel like going to a concert next week? . . . ” In the eighth dialogue turn, participant Suzanne responds with “Really? OMG! That's wonderful! Thank you sweetheart!”

FIG.3also illustrates summary draft250. Summary draft250corresponds to dialogue conversation history140shown inFIG.3. Summary draft250includes dialogue turns indexed using turn index302. Because there are nine turns in dialogue conversation history140there are nine turn indexes302. As discussed above, summary draft250includes labels for action categories303and key phrase(s)305for each dialogue turn. The labels for action categories303shown inFIG.3include interrogative pronoun, such as WHY, WHAT, CONFIRM, and ABSTAIN. Further,FIG.3illustrates that dialogue turns corresponding to turn indexes 1, 5, 6, 8, and 9 may not have key phrase(s)305.

In some embodiments, CorDial model130may construct the summary draft250as a concatenated string that includes a sequence of turn indexes302, action categories303, and key phrase(s)305for each dialogue turn. The string may end with a special token “TLDR.” With reference toFIG.3, the summary draft250may be “1 what 2 abstain ‘s just one of . . . square garden’ 8 why 9 abstain TLDR”.

Going back toFIG.2B, summary draft250may be used to train CorDial model130. As discussed inFIG.2A, CorDial model130includes encoder210and decoder215. During training, encoder210may receive dialogue segments212and generates encodings from dialogue segments212. Decoder215may receive the encodings generated by encoder210, labels for action categories303and key phrase(s)305from summary draft250that correspond to turns in dialogue segments212. Using the encodings, labels for action categories303and key phrase(s)305, decoder215may generate segment summaries216. Typically, there may be one segment summary216for one dialogue segment212.

The training process may repeat for multiple iterations using different dialogue conversation histories140and training summaries209until generative language model205is trained. Once trained, generative language model205may generate dialogue summary150from dialogue conversation history140. An example dialogue summary150is shown inFIG.3.

In some embodiments, CorDial model130may be trained to control a number of sentences that may be included in dialogue summary150. In other words, during the inference stage discussed inFIG.2A, CorDial model130may generate a single sentence summary as dialogue summary150or divide dialogue conversation history140into multiple dialogue segments202and generate a segment summary204as an output for each dialogue segment202. If the dialogue conversation history140is divided into dialogue segments202, the number of output sentences in dialogue summary150may be the same as the number of dialogue segments202.

In some embodiments, during inference and training stages discussed inFIGS.2A and2B, dialogue segments202,212in the dialogue conversation history140may be identified by inserting special tokens <hl> and </hl> into dialogue conversation history140. The dialogue conversation history140example inFIG.3illustrates three dialogue segments. The first dialogue segment304is between special tokens310S and310E, the second dialogue segment306is between special tokens312S and312E, and the third dialogue segment308is between special tokens314S and314E. The number of segments may correspond to a number of sentences that may be included in dialogue summary150. For example, when dialogue conversation history140is divided into three dialogue segments, as shown inFIG.3, CorDial model130may generate a three-sentence summary, where the first segment summary316corresponds to the first dialogue segment304, the second segment summary318corresponds to the second dialogue segment306, and the third summary sentence320corresponds to the third dialogue segment308. In another example, an entire dialogue conversation history140may be a single segment that is augmented with a single pair of highlighting tokens. For example, the single segment may be between special tokens310S and314E. This results in a single segment summary (not shown) that is dialogue summary150.

In some embodiments, CorDial model130may be trained to control the number of dialogue segments202that may be generated from dialogue conversation history140. Because the number of dialogue segments202corresponds to the number of sentences in dialogue summary150, increasing the number of dialogue segments202increases the number of segment summaries, while decreasing the number of dialogue segments202decreases the number of segment summaries. In this way, CorDial model130may generate the dialogue summary150that is more interpretable.

As discussed above, CorDial model130may include dialogue turn level classifier225. Dialogue turn level classifier225may be trained to identify dialogue segments202in dialogue conversation history140during the inference state discussed inFIG.2A. During the training stage, dialogue turn level classifier225may be trained to determine where to place special tokens <hl> and </hl> in dialogue conversation history140. To determine where to place special tokens <hl> and </hl>, dialogue turn level classifier225may receive dialogue segments212and may predict whether each dialogue turn is a cutting point208. Cutting point208indicates where one dialogue segment ends and the next dialogue segment begins. With reference toFIG.3, a cutting point may be between dialogue turns4and5, where CorDial model130may insert special tokens310E and213S. Another cutting point may be between dialogue turns7and8where CorDial model130may insert special tokens312ES and3145.

In some instances, dialogue turn level classifier225may be a binary classifier. Specifically, dialogue-turn-level classifier225may be trained to receive dialogue segments212as input and predict whether each dialogue turn is a cutting point208. During training, each dialogue turn in dialogue segments212that make up dialogue conversation history140may be prefixed with a separation token (e.g., xsep=) and turned into a long sequence. Dialogue turn level classifier225may receive this long sequence and process the long sequence as follows:
H=C([xsep, X1, xsep, X2, . . . , xsep, XN])∈N×demb, {circumflex over (P)}=sigmoid(W1(H))∈NEquation 2
where C is dialogue level turn classifier225, H is the output of the dialogue level turn classifier225and may include representation of the separation tokens, and each of the separation tokens is a dembdimension vector, and W1∈demb×1is a trainable linear mapping. The P is a predicted segment probability that is trained with a binary cross-entropy loss. In some embodiments, a Bidirectional Encoder Representations from Transformers (BERT)-base model may be used as dialogue level turn classifier225. In some embodiments, the dialogue turn level classifier225may not need to be perfect because the labels may contain a certain noise. In some embodiments, that final ROUGE score produced by similarity module235may be similar for both “oracle” dialogue split and predicted dialogue split. The number of output summary sentences may be controlled by controlling the number of predicted dialogue splits. For example, if there is a three-sentence summarization150as shown inFIG.3, the dialogue may be split into three dialogue segments by selecting the top two highest segment probabilities in {circumflex over (P)}. Further, one segment summary may be generated by ignoring the segment classifier and clipping the whole dialogue with special tokens.

In some embodiments, CorDial model130may be trained using an “oracle” dialogue segmentation that adds highlighting tokens for each summary sentence, separately. For each summary sentence, CorDial model130may receive an entire dialogue conversation history140with a highlighted portion as input. From the dialogue conversation history140, CorDial model130may be trained to generate a corresponding summary draft250and segment summaries216, which may be segment summaries316,318, and320ofFIG.3. For example, for the first segment summary, such as segment summary316, CorDial model130may receive the whole dialogue conversation history140with the added highlighting tokens both in the beginning of turn one and in the end of the turn four. CorDial model130may then generate output that contains summary draft250from turn one to turn four and the first segment summary316that is “Suzanne is at work and is having a break now.” The CorDial model130is also trained on cross-entropy loss for the generated tokens.

FIG.4is a simplified diagram of a method400for training a CorDial model130to generate a dialogue summary from a dialogue, according to some embodiments. One or more of the processes402-408of method400may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes402-408.

At process402, dialogue conversation history is divided into dialogue segments. For example, similarity module235may divide dialogue conversation history140into dialogue segments212using on training summary209. Training summary209may also be divided into segment summaries214, such that one dialogue segment212corresponds to one segment summary214.

At process404, a summary draft is generated. For example, CorDial model130may generate a summary draft250from dialogue segments212in dialogue conversation history140. The summary draft250includes a turn index for each dialogue turn in dialogue conversation history140. For each dialogue turn, the summary draft250also includes a label for an action category and zero or more key phrase(s) that correspond to the dialogue turn. As discussed above, parser240may generate zero or more key phrase(s)255that are associated with the dialogue turn using dialogue segments212from dialogue conversation history140and segment summaries214from training summary209. As also discussed above, label module245may generate a label for action category that is associated with the dialogue turn.

At process406, segment summaries are generated. For example, generative language model205may receive dialogue segments212. For each dialogue segment in dialogue segments212, encoder210of generative language model205may generate encodings. The decoder215may receive encodings, labels for action categories and key phrase(s) for dialogue turns included in summary draft250and generate segment summary216for the dialogue segment212.

At process408, dialogue turn level classifier is trained to determined cutting points. For example, dialogue turn level classifier225is trained on dialogue segments212to determine cutting points208in dialogue conversation history140.

In some embodiments, method400may be repeated on multiple dialogue conversation histories140and the corresponding training summaries209, until CorDial model130may generate accurate dialogue summaries150. Once CorDial model130is trained, CorDial model130may be used in an inference stage to generate dialogue summary150from dialogue conversation history140.

FIG.5is a simplified diagram of a method500for training a CorDial model130to generate dialogue summary from a dialogue, according to some embodiments. One or more of the processes502-506of method500may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes402-408.

At process502, a dialogue conversation history is divided into multiple dialogue segments. For example, dialogue turn level classifier225may divide dialogue conversation history140into dialogue segments202by identifying cutting points230in between dialogue turns. The dialogue turns between the cutting points208are in the same dialogue segment202. In some embodiments, special highlighting tokens may be inserted into dialogue conversation history140at the cutting points208to identify dialogue segments202. In other embodiments, computing device100may receive input, such as highlighted text in dialogue conversation history140that identifies dialogue segments202in dialogue conversation history140. Based on the input, special highlighting tokens may be inserted into dialogue conversation history140.

At process504, segment summaries are generated. For example, generative language model205trained as discussed in method400may receive dialogue conversation history140with the highlighting tokens that identify dialogue segments202. For each dialogue segment202, that is the portion of dialogue conversation history140between the highlighting tokens, encoder210of generative language model205may generate encodings. The decoder215may receive260and generate segment summary204of the dialogue segment202.

At process506, the segment summaries are concatenated into a dialogue summary. For example, concatenation module220may combine the segment summaries204for the dialogue segments202into dialogue summary150. In some instances, concatenation module220may concatenate segment summaries204linearly into a dialogue summary150.

Some examples of computing devices, such as computing device100may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the processes of methods400and500. Some common forms of machine readable media that may include the processes of methods400and500are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the embodiments of this disclosure. Like numbers in two or more figures represent the same or similar elements.

In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.