Systems and methods for explainable and factual multi-document summarization

Embodiments described herein provide methods and systems for summarizing multiple documents. A system receives a plurality of documents and generates embeddings of the sentences from the plurality of documents. The embedded sentences are clustered in a representation space. Sentences from a reference summary are embedded and aligned with the closest cluster. Sentences from each cluster are summarized with the aligned reference sentences as a target. A loss is computed based on the summarized sentences and the aligned references, and the natural language processing model is updated based on the loss. Sentences may be masked from being used in the summarization by identifying sentences that are contradicted by other sentences within the plurality of documents.

TECHNICAL FIELD

The embodiments relate generally to machine learning systems and multi-document summarization.

BACKGROUND

Summarizing large amounts of information from many sources into concise snippets can be incredibly useful, but current neural summarization techniques still have many factual errors making these techniques hard to use in practice. Additionally, current techniques do not provide an explanation for a user to determine how the summary was developed.

DETAILED DESCRIPTION

As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

Embodiments described herein provide a multi-document summarization approach that is factual and explainable. Summarizing large amounts of information from many sources into concise snippets can be incredibly useful, but current neural summarization techniques still have many factual errors making these techniques hard to use in practice.

In multi-document summarization, a document can be fact checked by making use of the other documents in a document cluster. a clustering model is adopted to cluster sentences from various documents into a plurality of clusters, to use as an input for an abstractive model. A reference summary may then be aligned with the plurality of clusters by comparing each sentence from the reference summary with the clusters, e.g., using a pretrained model to generate cluster-wise reference summary for each cluster. Next, source sentences in each of the original documents are aligned with a closest reference sentence in the reference summary. Then the clustering among the source sentences can be transformed into a partition of the reference summary. The partition of the reference summary is then compared with the cluster-wise reference summary to finetune the clustering model.

Sentences which are identified as contradicted by another document may be masked so that they are not used as inputs to the summary. In order to improve explainability, masked out sentences may be identified to a user in an interface. The interface may also be used to visualize the clustering and summarization models, for example by displaying verified and contradicted sentences in the source documents.

In particular, masking out words or sentences from one article based on closely related sections of the other articles can lead to a summary that is more consistent with and faithful to all of the source documents and contains less hallucinations overall while still remaining fluent and similar in content to the reference summaries. In addition, this masking represents a valuable and interpretable explanation for the summary.

FIG.1is a simplified diagram of a computing device that implements the multi-document summarization, according to some embodiments described herein. As shown inFIG.1, computing device100includes a processor110coupled to memory120. Operation of computing device100is controlled by processor110. And although computing device100is shown with only one processor110, it is understood that processor110may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device100. Computing device100may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.

Processor110and/or memory120may be arranged in any suitable physical arrangement. In some embodiments, processor110and/or memory120may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor110and/or memory120may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor110and/or memory120may be located in one or more data centers and/or cloud computing facilities.

In some examples, memory120may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory120includes instructions for a Summarization module130that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. In some examples, the Summarization module130, may receive an input140, e.g., such as a collection of documents on a particular topic, via a data interface115. The data interface115may be any of a user interface that receives a question, or a communication interface that may receive or retrieve previously stored documents from the database. The Summarization module130may generate an output150, such as a summary of the input140.

In some embodiments, the Summarization module130may further includes the clustering module131, masking module132, and a generation module133. The clustering module331may be configured to use an encoder to generate an embedding of the sentences from the source documents, and then cluster those embedded sentences according to their relative distance in the representation space. A variety of clustering methods may be used such as K-means clustering or spectral clustering. K-means clustering partitions the sentences into K clusters in which each sentence belongs to the cluster with the nearest centroid. Spectral clustering clusters embedded sentences by identifying groups based on closeness between neighbors allowed for clusters that are aspherical. The masking module132is configured to mask sentences. Embedded sentences from each document may be compared to each other in order to find sentences which contradict each other. The masking module may be configured to mask those contradicted sentences that it identifies. The masking is done so that those contradicted sentences will not later be used in the generation of a summary, as they may contain information that is not factual. The generation module is configured to generate a summary of the documents based on the clustered and masked sentences. The generation model may be trained by first aligning sentences from a reference summary with the identified clusters. This may be done by embedding the sentences of the reference summary and finding the closest cluster in the representation space. The clustered sentences from the source documents may then be summarized with the target of the reference sentences aligned with each respective cluster. A loss may be computed based on the generated summary sentences and the reference sentences aligned with the cluster used to generate each respective summary sentence.

FIG.2is a simplified diagram showing an example method for summarizing multiple documents. Documents210,220, and230are multiple input documents, e.g., having text that covers similar material. Greater or fewer documents may be used in the methods described herein. Reference summary270is a summary of the documents, created either manually or otherwise. As is shown by the different hashings, portions of each document may describe the same or similar matter. For example, Document210may have a sentence that says, “The town parade will begin at 2:00,” and Document220may have a sentence that says, “The parade starts at 2:00.” Embeddings of the sentences from the source Documents210,220, and230may be generated, and then those embeddings may be clustered together based on their relative distance in the representation space, as opposed to grouped according to which document they are from. This is shown as Clusters240,250, and260which show that the similar sentences of each of the source documents210,220, and230are clustered together. As is illustrated, the sentences are divided into three clusters, however greater or fewer clusters may be used. A number of clustering methods may be used. For example, K-means clustering may be performed which partitions the sentences into K clusters in which each sentence belongs to the cluster with the nearest centroid. Another clustering method which may be used is Spectral clustering, which clusters embedded sentences by identifying groups based on closeness between neighbors allowed for clusters that are aspherical.

Sentences from the reference summary270may also be encoded into embeddings and then those embeddings are compared with embeddings of the clusters240250and260. In this way, sentences in the reference summary270may be respectively aligned with one of the identified clusters. As shown, multiple reference summary sentences may be aligned with a single cluster, such as in the illustrated example, sentences281,282, and283are aligned with cluster240, or in some cases a cluster may have no reference sentences aligned with it, such as cluster250in the illustrated example. Sentence284in the illustrated example is the only sentence from reference summary270which is aligned with cluster260. Aligning may be performed, for example, by choosing the closest cluster to each reference sentence, using Euclidean distance between the mean sentence embedding of the cluster and the sentence embedding of the reference sentence.

In one embodiment, a pretrained summarization model may be used to generate cluster-wise summaries from clusters240,250and260, respectively. The generated cluster-wise summaries may then be compared with the reference sentences281-284in the aligned summary280aligned with the cluster. As such, a loss function may be computed based on the generated cluster-wise summary and the aligned reference sentence(s)281-284. This loss function may be used to update parameters of the pretrained summarization model.

FIG.3provides an example logic flow diagram illustrating an example algorithm for training a multi-document summarization system. One or more of the processes described inFIG.3may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes305-340. In some embodiments, method300may correspond to the method used by the module130inFIG.1.

At step305, a plurality of documents and a reference summary associated with the plurality documents are received. Method300is described as receiving a plurality of documents with a single reference summary. In other aspects, multiple groups of documents may be received, each group having its own corresponding reference summary. Training of the model as described below may occur with documents either all together or in batches in order to train more efficiently.

At step310, embeddings of sentences from the plurality of documents are generated.

At step315, The sentences from the plurality of documents are clustered, based on the embeddings, into a plurality of clusters. A number of clustering methods may be used. For example, K-means clustering may be performed which partitions the sentences into K clusters in which each sentence belongs to the cluster with the nearest centroid. Another clustering method which may be used is Spectral clustering, which clusters embedded sentences by identifying groups based on closeness between neighbors allowed for clusters that are aspherical.

At step320, Sentences are masked which are contradicted by other sentences in the other documents so that they are not used in the generation of a summary. In some aspects, a sentence is only masked when contradicted by multiple other sentences and/or sentences from multiple documents. In addition to masking, the model may be configured to either prefer, or to only use sentences which are verified by other sentences in other input documents.

At step325, reference sentences from the reference summary are aligned with the plurality of clusters into a plurality of aligned reference sentence clusters, respectively. In order to align, sentences from the reference summary may also be embedded the same way the sentences from the source documents were embedded. Aligning may be performed, for example, by choosing the closest cluster to each reference sentence, using Euclidean distance between the mean sentence embedding of the cluster and the sentence embedding of the reference sentence.

At step330, a natural language processing model is used to generate a plurality of cluster-wise summaries corresponding to the plurality of clusters, respectively. For example, a pre-trained “PEGASUS” model may be used to generate the summaries. Clusters may be filtered out before the summaries are generated, meaning clusters with no aligned reference sentences associated with them are not used to generate cluster-wise summaries. When generating cluster-wise summaries, the model may only use a subset of the sentences in a cluster, for example the 10 sentences closest to the mean of the cluster. Other methods of selecting a subset of sentences from a cluster include “Oracle” ranking where instead of truncating to the sentences closest to the mean, you truncate to the sentences closest to the cluster's reference summary. When the method performs the masking step, sentences which are masked based on a contradiction are not used in generating a summary.

At step335, the plurality of cluster-wise summaries and the plurality of aligned reference sentence clusters are compared to compute a loss.

At step340, The natural language processing model is updated based on the computed loss.

FIG.4provides a simplified illustration of a user interface (UI) according to some embodiments. As illustrated, the UI may have some representation of input documents410,420, and430. Sentences which are contradicted (and therefore masked), may be indicated as such. Sentences which are verified by other documents may also be indicated. Some indication may also be given in the UI of which sentences are clustered together. The contradictions, verifications, and clusters, may also be shown to correspond with the final generated summary440. For example, the sentence which is at the centroid of a cluster may have a line drawn from that sentence in the display to the sentence(s) associated with that cluster in the summary440.

FIG.5provides an example table illustrating example performance of different clustering methods. The metric used is ROUGE, which is a set of metrics for evaluating summarization of texts based on a comparison to a reference summary, as described inLin, ROUGE: A Package for Automatic Evaluation of Summaries, in WAS, pages 1-8, 2004. The results compare a baseline concatenation model, K-means clustering, Spectral clustering, and different methods of filtering clusters and truncating sentences.

FIG.6provides an example table illustrating example performance of different multi-document summarization methods discussed herein. Results compare a baseline concatenation model, a clustering model, a clustering model that uses pre-trained entailment models to mask out source sentences, and an “oracle” model that uses a brute force method to obtain oracle masks. Note that article entailment is the percentage of articles that entail the summary as judged by a fine-tuned T5 model, and Cluster entailment is the percentage of instances for which all articles entail the summary as judged by the model. BERTScore measures the models based on their agreement with the reference summary, as described in Zhang et al., BERTScore: Evaluating Text Generation with BERT, pages 1-9, ICLR 2020. Reference summaries may not be entailed by each article individually so they may not have very good article entailment or faithfulness scores. Article FEQA is a metric used to measure faithfulness using question answering. Hallucination rate is the percentage of instances for which none of the articles entail the summary.

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the embodiments of this disclosure. Like numbers in two or more figures represent the same or similar elements.