Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistency
Year: 2023

Abstract:
Discourse-aware techniques, including entityaware approaches, play a crucial role in summarization. In this paper, we propose an entity-based SpanCopy mechanism to tackle the entity-level factual inconsistency problem in abstractive summarization, i.e. reducing the mismatched entities between the generated summaries and the source documents. Complemented by a Global Relevance component to identify summary-worthy entities, our approach demonstrates improved factual consistency while preserving saliency on four summarization datasets, contributing to the effective application of discourse-aware methods summarization tasks. 1

heading: 1 Introduction
text: Discourse-aware models play a crucial role in natural language processing applications, including machine translation (Guzmán et al., 2014) and text summarization (Xu et al., 2020). Among these applications, abstractive text summarization, the task of generating informative and fluent summaries of the given document(s), has attracted much attention in the NLP community. While early neural approaches focused more on designing customized architectures or training schema (Nallapati et al., 2016; Tan et al., 2017; Liu* et al., 2018), recent works have shown that both pre-trained generation models fine-tuned on in-domain datasets and zeroshot GPT-like decoder-only models generally have better performance (Lewis et al., 2020; Zhang et al., 2020; Raffel et al., 2020; Goyal et al., 2022).
However, even with state-of-the-art performance on standard automatic evaluation metrics such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020), the generated summaries still suffer from the problem of factual inconsistency, which
1The code is available at https://github.com/ Wendy-Xiao/Entity-based-SpanCopy
means the generated summaries may not be factually consistent with the content expressed in the source documents (Kryscinski et al., 2020; Bubeck et al., 2023). Inconsistencies may exist either at the entity, where summaries mention entities absent from source documents, or at the relation level, where summaries express relations between entities that differ from the source (Nan et al., 2021).
In this paper, we focus on the entity-level inconsistency problem, i.e. to make the model generate summaries with less entities which do not appear in the source document(s) i.e., ‘hallucinated’ entities. Note however, that hallucinated entities are not necessarily ‘unfaithful’ or ‘wrong’ (Cao et al., 2021), so the goal is to reduce them without excluding entities that do appear in the reference summary i.e., without penalizing saliency. Table 1 shows an
70
example of entity-level factual inconsistency from the XSum dataset. Although the content of the summary generated by the SOTA summarizer PEGASUS (Zhang et al., 2020) is roughly similar that of the ground-truth summary, it does not accurately summarize the original documents with the proper entities. Specifically, the entity ‘Hampshire’ is ‘hallucinated’, as it does not appear in the source document. Despite the fact that the city ‘Portsmouth’ is located in ‘Hampshire’ county, the entity itself is still an instance of factual inconsistency (i.e., an unnecessary generalization).
Prior work (Dong et al., 2020; King et al., 2022) mainly address the entity-level inconsistency problem in the post-processing stage. However, those methods either requires additional sophisticated models, e.g. Dong et al. (2020) uses a pre-trained QA model to ‘revise’ the generated summaries, or being built on arguably brittle heuristics (King et al., 2022). Recent work (Nan et al., 2021) proposes two ways to directly improve the end-to-end summarization model, either by training with an auxiliary task, which is to recognize the summaryworthy entities in the source document using the hidden states from the encoder, or jointly generating the entities and the summaries, i.e. generating a chain of entities in the summary followed by the summary. The latter one is in-line with recently proposed entity-aware guided summarization methods (He et al., 2020; Narayan et al., 2021). Yet, both methods do not explicitly encourage the model to generate the summaries with more valuable entities, as both of them aim to guide the model to detect the summary-worthy entities without any changes in the summary generation process. Instead, aiming for a lean and modular solution, we propose the discourse-aware SpanCopy Mechanism to explicitly copy the matched entities2 from the source documents when generating the summaries. One key advantage of our proposal is that it can be easily integrated into any pre-trained generative sequence-to-sequence model.
Since often only a few of the entities in the source documents can be included in the summary, which we call ‘summary-worthy entities’, we also explore an additional Global Relevance component to better recognize the summary-worthy entities by automatically generating a prior distribution over all the entities in the source documents.
2We particularly focus on the Named Entities in this paper, but our method can be easily applied to any kind of spans or entities.
We test our proposal on four summarization datasets in the news and scientific paper domain, comparing it with the established SOTA PEGASUS system (Zhang et al., 2020). In a first set of experiments, as a sanity check, we assess our models on arguably easier subsets of these datasets, where all the entities in the reference summaries belong to the source document. In these cases, SpanCopy should definitely dominate PEGASUS, which is confirmed by the results. In a second set of experiments, we fine-tune and test on the full datasets. On this realistic and more challenging task, we find that SpanCopy (without Global Relevance) can strongly improve the entity-level factual consistency (+2.28) on average across datasets, with essentially no change in saliency (−0.06).

heading: 2 Related Work

heading: 2.1 Abstractive Summarization
text: Early neural abstractive summarization models (Nallapati et al., 2016; Paulus et al., 2018; See et al., 2017) are mainly sequence-to-sequence models based on different variants of RNN, e.g. LSTM or GRU, with additional components targeting different properties of the summaries, like redundancy (Tan et al., 2017) and coverage (See et al., 2017). However, all the recurrent models suffer from serious weakness like long-term memory loss, and requiring excessive time to train.
To tackle these problems, researchers in the area of abstractive summarization started to use attention-based transformer models (Liu and Lapata, 2019a,b); recently reaching SOTA performance when pre-trained generative transformers are applied to the task, e.g. BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020) and PRIMERA (Xiao et al., 2022). The SpanCopy mechanism we propose in this paper can be advantageously injected into any pre-trained models.

heading: 2.2 Factual Consistency
text: Despite the large improvements with respect to automatic evaluation metrics, recent studies (Cao et al., 2018; Kryscinski et al., 2020) show that around 30% of the summaries generated by the SOTA summarization models contain factual inconsistencies. Ideally, the assessment of factual consistency should rely on human annotations (Maynez et al., 2020), but these are costly, time consuming and lack a unified standard. Thus promising automatic evaluation metrics for factual consisten-
cies of generated summaries have been explored in recent years. To assess relation-level factual consistency two kinds of metrics have been proposed: one based on classification (Kryscinski et al., 2020), and one based on Question-Answering (Maynez et al., 2020; Durmus et al., 2020). For entity-level factual consistency, the focus of this paper, Nan et al. (2021) propose a simple but effective evaluation metric, based on the matched named entities in both generated and ground-truth summaries. In our work, we use such metric to evaluate whether the generated summaries are consistent with both the source documents and the reference summaries at the entity-level.

heading: 2.3 Entity-aware Summarization
text: The use of entities as part of discourse-aware approaches has been shown to improve both saliency and factual consistency for the summarization task. Xiao et al. (2022) identify salient entities within document clusters and utilize them to select pseudo summaries during the pre-training phase, leading to superior performance on multiple datasets. Entities have also been employed in guided summarization, where researchers extract oracle entities from ground-truth summaries and use them to guide summary generation. For instance, Dou et al. (2021) introduce an additional encoder to encode guidance signals, sharing partial parameters with the original document encoder. In related research, He et al. (2020) propose a pre-training strategy that prepends source documents with oracle keywords
as prefixes, and Narayan et al. (2021) train models to first predict an entity chain before generating the final summary. Diverging from prefix-based strategies, our approach enables the model to learn explicit copying of entities to specific positions in the generated summary, further advancing discourseaware summarization techniques.

heading: 2.4 Copy Mechanism
text: See et al. (2017) first apply pointer-generator network in an abstractive summarization model, which facilitates copying words from the source documents by pointing, i.e., generating a distribution of probabilities to copy each word from the source. Following their work, Bi et al. (2020) propose PALM, in which the copy mechanism is applied on top of the transformer model, and with a novel pre-training schema, the model achieves SOTA on several generative tasks, such as abstractive summarization and generative QA. More recently, Li et al. (2021) further explores how to make use of the copy history to predict the copy distribution for the current step. However, all the aforementioned works focus on copying at the word level, which tends to be sparse and noisy. Instead, we aim to train the model to copy spans of text i.e., the named entities, in this paper.
Admittedly, some previous work has also investigated span-based copy mechanisms. Yet, those models either predict the start and end indices of a span (Zhou et al., 2018), or predict the BIO labels for each token (Liu et al., 2021). Even if such
strategies can copy any kinds of spans (clauses, n-grams, entities, phrases or longest common sequence) from the source document, they may introduce unnecessary noise and break the coherence of the generated text. In this work, we focus on copying the spans of the Named Entities, extracted by a high-quality NER tool, aiming to improve factual consistency of the generated summary without negatively affecting saliency.

heading: 3 Our SpanCopy Method

heading: 3.1 Transformer-based Summarizers
text: Typically, transformer-based summarization(Lewis et al., 2020; Zhang et al., 2020) consists of two steps (i) The Encoding Step (by the Encoder shown in yellow in Fig.1), which encodes the source input(s) into an hidden space; (ii) the Decoding Step, which computes a probability distributions on the output vocabulary to generate each token of the resulting summary. In this paper, to better describe our methods in the context of a generic summarization models, we split the Decoding process into two components, the Decoder itself (shown in green in Fig.1), which outputs the representations of predicted tokens, and the Generator (shown in purple in Fig.1), an MLP layer mapping the representations to the final probability distribution on the output vocabulary.
More formally, for a document with n tokens D = {td1, td2, ..., tdn}, and the corresponding summary with m tokens, S = {ts1, ts2, ..., tsm}, the output of the Encoder is a sequence of hidden states of all the tokens, i.e. {he1, he2, ..., hen}. And then the Decoder predicts a sequence of vector, {hd1, hd2, ..., hdm}, representing the tokens to be predicted. Finally, the Generator maps those vectors to the distributions over the vocabulary, i.e. {p1,p2, ...,pm}, where pi ∈ R|V |.
There has been recent research on models with a decoder-only structure for summarization (Goyal et al., 2022), where the decoder is responsible for both the encoding and decoding steps. In this approach, the tokens in the source documents are represented using the output of the decoder, rather than relying on the encoder. Our proposed method specifically applies to the decoder, making it compatible with both encoder-decoder models and decoder-only models. However, in this paper, we primarily focus on exploring the application of our method to encoder-decoder models, while leaving the investigation of decoder-only models
for future research.

heading: 3.2 SpanCopy Mechanism
text: A key problem with generic sequence-to-sequence transformer-based summarizers is that the decoding step is prone to generate factual inconsistencies, i.e. the model may make up entities or relations that are not entailed by the source documents. To address entity-level factual inconsistency, we introduce in the Decoding Step the SpanCopy mechanism, which can be conveniently plugged into any pre-trained models. Specifically, we first identify and match the entities in both source document and summary, and then instead of generating the entire summary word by word, we add an additional Span Copier to directly copy entities from the source document, with a Copy Gate predicting the likelihood of whether the model should generate the current token from the vocabulary or directly copy an entity from the source document.
Span Copier (shown in blue in Fig.1) is an attention module over all the entities in the input document. Suppose there are |E| entities in the input document, with each entity j being a span over tokens [djs , dje ], then the entities can be simply represented as ej = avg([hejs : h e je ]), where hei represents the output of the encoder for each token di. At each decoding step i, we compute the logit vector of copying each entity at the current step as:
oci = Q(h d i ) ·K(ej),oci ∈ R|E| (1)
indicating how likely it is to copy the entities from the source document at each step. Notice that to better balance the numeric difference caused by the size of selection space (|V | and |E|), we generate and combine the raw logit vectors3 from the Span Copier and Generator, and take softmax over the combined space to get the final probability.
Copy Gate (shown in red in Fig.1) is a classifier to map the hidden states to a singular value, i.e.
pcopyi = σ(MLP (h d i )), p copy i ∈ [0, 1] (2)
which indicates the probability of copying an entity at each step. On the contrary, 1− pcopyi represent the probability of generating a token from the vocabulary at step i.
3The vector of raw (non-normalized) predictions that the classification model generates
Then the final probability, combining both generation over the vocabulary and the copy mechanism over the entity space, is computed as
pfinali = softmax([(1− pcopyi ) · o g i , p copy i · oci ]) (3)
with pfinali ∈ R(|V |+|E|), where o g i ∈ R(|V |) is the logit vector of token generation and oci ∈ R(|E|) is the logit vector of entity copying. As a result, the first |V | dimensions of the final probability represent the probability of generating all the tokens from the vocabulary, while the following |E| dimensions contain the probabilities of copying the entities from the source document.
Note that the input of the original Decoder in the transformer model at each step is the embedding of the previous token (which is the ground-truth one during training, and the predicted one for inference), but a span of text longer than 1 does not naturally have an embedding to match. We simply use the average of the embedding of all the tokens in the entity, following previous work using average embedding to represent a span of text (Xiao and Carenini, 2019).

heading: 3.3 Loss
text: We use the standard loss for abstractive summarization, i.e. the cross entropy loss between the predicted probability and the ground truth labels,
L1 = ∑
i
Ls(p final i , ti) (4)
However, notice that, since the predicted probability distribution is over the combined space of vocabulary size and entity size (pfinali ∈ R|V |+|E|), the corresponding ground truth labels can be either indices of words to be generated from the vocabulary, or the indices of entities to be copied from the source document, i.e. ti ∈ [0, |V | + |E|]. Specifically, if ti < |V |, then the ti-th token should be generated, and if ti > |V |, the (ti − |V |)-th entity should be copied from the source document.

heading: 3.4 SpanCopy with Global Relevance
text: Among all the entities in the source documents, there are only a few summary-worthy entities that should be copied into the summary (e.g. around 10% in CNNDM and 1.5% in arXiv). To make the model better recognize such summary-worthy entities, we explore a Global Relevance (GR) component, which takes all the entities in the source document as inputs, and predicts how likely each
entity is to appear in the final summary. We use the generated ‘entity likelihood’ as a prior distribution for the Span Copier component, with GR also trained as an auxiliary task.
Global Relevance is a classifier mapping the hidden state of a source document entity into a value within [0, 1], indicating the probability that such entity should be included in the summary.
gr = σ(MLP (e)),gr ∈ R|E| (5)
Then pfinali in Eq.3 is updated with gr as
pfinali = softmax([(1− pcopyi ) · o g i
, pcopyi · oci · gr]) (6)
New Loss As an auxiliary task, we also train the model with the ground-truth GR labels to make it more accurate. Specifically, the label ygri = 1 if the i-th entity in the input document is included in the ground truth summary. Then we update the loss function with Lgr balanced by β:
L2 =(1− β) ∑
i
Ls(p final i , ti)
+ β ∑
j
Lgr(grj , y gr j )
(7)

heading: 4 Experiments and Analysis

heading: 4.1 Settings
text: SpanCopy can be plugged into any pre-trained generation model. In this paper, we use PEGASUS(Zhang et al., 2020) as our base model, since it has delivered top performance on multiple summarization datasets. We recognize named entities with an off-the-shelf NER tool4. The balance factor β of GR is set by grid search on small subsets of each dataset (2k for training and 200 for validation).

heading: 4.2 Evaluation Metrics
text: To evaluate the saliency and entity-level factual consistency of the generated summaries, we apply the following metrics:
Saliency metrics assess the similarity of the generated summary with the reference summary.
ROUGE scores (Lin, 2004) measure the n-gram overlaps between generated and ground truth summaries. We apply the metrics R-1, R-2 and R-L.
Summary-precision, -recall and -f1 (sump, sumr and sumf ) (Nan et al., 2021) measure the
4https://spacy.io/
precision/recall/f1 score of the matched entities in the generated summaries and the reference summaries. we use NE(Sref ) and NE(Sgen) to represent the named entities in the reference summaries and generated summaries, respectively.
sump = |NE(Sref ) ∩NE(Sgen)|/|NE(Sgen)| sumr = |NE(Sref ) ∩NE(Sgen)|/|NE(Sref )| sumf = 2 ∗ (sump + sumr)/sump ∗ sumr
These three metrics measure the entity-level saliency of the generated summaries, i.e. recognizing how many copied (and generated) entities are salient, and should be included in the summary.
Entity-level factual consistency metric: measures the named entity matching between the generated summaries and the source documents. (Nan et al., 2021) With NE(D) and NE(Sgen) representing the named entities in the source document and generated summaries, respectively, Source-precision(srcp) measures how many entities in the generated summaries are from the source documents, i.e. srcp = |NE(D) ∩ NE(Sgen)|/|NE(Sgen)|. It is an evaluation metric for entity-level factual consistency, as it directly measures how consistent the generated summaries are with the source.

heading: 4.3 Datasets
text: We test and compare our SpanCopy model with the original PEGASUS on four datasets, in the domains of news (CNNDM(Nallapati et al., 2016), XSum(Narayan et al., 2018)) and scientific papers (Pubmed and arXiv(Cohan et al., 2018)). As a sanity check, we initially assess our models on subsets of these datasets, where all the entities in the reference summaries belong to the source document (we call these filtered datasets). In these cases (srcp(gt) = 1), Span Copy and GR should dominate PEGASUS, because by design they tend to
generate entities from the source document. 5
The statistics of the filtered and original datasets, on the lengths and number of entities in the document and summaries, can be found in Table 2. srcp(gt) measures the entity-level factual consistency between the source document and the groundtruth summary, with lower value meaning that there are more novel entities in the ground-truth summaries. The table shows that the datasets in the news domain have higher density of the entities with respect to the lengths (number of words) of both documents and ground-truth summaries, i.e. Ndoc/Ldoc and Nsumm/Lsumm are larger for the news articles. a possible explanation is that news articles tend to describe an event or a story, which may contain more names of people, organizations, locations, etc., as well as dates. Interestingly, CNNDM and Pubmed contain less novel than the other two datasets (with higher srcp(gt)), something that the proposed SpanCopy mechanism may benefit from. Comparing the filtered datasets with the original ones, we can see that the number of entities in the summaries drops for all the datasets, especially for arXiv, as the more entities in the summary, the less likely they can be all matched to the source documents.

heading: 4.4 Results and Analysis
text: The results on the filtered and original datasets are shown in Table 3 and Table 4.
Filtered Datasets We first evaluate our models, with the backbone model, PEGASUS on the filtered datasets, which is an easier task, and the results can be found in Table 3. All the models are fine-tuned and tested on the filtered datasets. Since we only keep the examples with all the entities in the summaries being matched with the entities in the source documents, the theoretical ceiling of
5Statistics of the (filtered/original) datasets can be found in Appendix.B
srcp is 100. Comparing SpanCopy and PEGASUS, SpanCopy performs better than PEGASUS regarding both saliency and entity-level factual consistency. Plausibly, this is because all the entities in the ground-truth summary can be copied from the source document, in which case the SpanCopy mechanism can better learn to copy. The SpanCopy model with the GR component performs better re-
garding the entity-level saliency on three out of all the four datasets. On arXiv, the performance of SpanCopy with the GR component regarding both entity-level saliency and factual consistency is quite low. One likely reason might be that it is a rather difficult task to identify the salient entities in the arxiv dataset, as there is a large amount of entities in the source documents, but only very
few entities are summary-worthy (164.1 v.s. 2.3 as shown in Table 2), which might bring in excessive noise.
Original Datasets In a second set of experiments, we fine-tune and test on the full/original datasets. On this realistic and more challenging task results are encouraging. As shown in Table 4, when the SpanCopy model is compared to PEGASUS, it improves the factual consistency of generated summaries with the source documents (srcp) on all the datasets, maintaining a very similar performance on the saliency metrics, i.e. ROUGE and entity-level saliency. Comparing across the four datasets, SpanCopy outperforms PEGASUS on both the saliency and factual consistency metrics on the Pubmed dataset. For better comparison, we show the relative gains/loss regarding PEGASUS on all the datasets, as well as the overall average results in Table 5. It is clear that the SpanCopy model performs much better regarding entity-level factual consistency (+2.13) with essentially no change in saliency (−0.06 on average ROUGE and −0.04 on entity-level saliency). Admittedly, despite the success of the GR component on the filtered datasets on both word-level and entity-level saliency, it fails to deliver any gain on the original datasets. A plausible explanation is that GR makes the model focus excessively on the entities in the source document,
therefore penalizing generation of new, potentially summary-worthy, entities.
Comparing the entity-level factual consistency on the filtered datasets and the original datasets, the filtered datasets always have higher srcp than the original ones, and the gain is especially larger on the XSum and arXiv datasets, as both of them contain more entity-level hallucinations in the original datasets. Remarkably, the performance gain of the SpanCopy model over PEGASUS on the filtered XSum dataset is much larger on the original XSum datasets (7.98 v.s. 0.66) , which might be because original XSum is more abstractive, the entity-level guidance is especially helpful for the abstractive examples with consistent entities in the summary.

heading: 4.5 Qualitative Analysis
text: For illustration, we examine a real example from the CNNDM dataset in Table 6, which is a news article on the evacuation of Americans during the time of the crossfire of warring parties in Yemen. While all of the three system generated summaries are able to capture the main statement that ‘it’s too dangerous to evacuate the Americans’, the person ‘Ivan Watson’ mentioned by PEGASUS’s summary does not exist in the source document, i.e., it is an ‘hallucinated’ entity. Most likely, PEGASUS is generating such hallucination because ‘Ivan Watson’ is a senior CNN correspondent several time associated with Yemen in other news article in the training set, and the model automatically ‘picked the entity from the memory’ to generate the summary without tightly adhering to the given document. In contrast, both of our models do not contain entities that are not in the source document, as the SpanCopy mechanism tend to guide the model to use more the entities in the source document. In addition, with the GR component, although the generated summary contains more matched entities with the source document, it pushes the model too far towards copying entities which are not salient (e.g. The State Department).

heading: 5 Conclusion and Future Work
text: In this paper, we tackle the problem of entity-level factual consistency for abstractive summarization through a discourse-aware approach, by guiding the model to directly copy the summary-worthy entities from the source document, through the novel SpanCopy mechanism (with the optional GR component). This mechanism can be integrated into
any transformer-based generative frameworks, contributing to the advancement of discourse-aware neural summarization.
To validate the effectiveness of our approach, we conducted experiments on four diverse summarization datasets, including a sanity check on arguably easier subsets. The results confirmed that SpanCopy with GR performs better on both entity-level factual consistency and saliency. Notably, experiments on the original test sets demonstrated that the SpanCopy mechanism can effectively improve entity-level factual consistency while maintaining word-level and token-level saliency.
Despite the recent success of GPT-like decoderonly systems on the summarization task (Goyal et al., 2022), they still appear to suffer from hallucinations and inconsistencies in the generated text (Bubeck et al., 2023). As mentioned in Section 3, our method can be easily extended to the decoder-only models, we intend to investigate how the mechanism works with the models for addressing these limitations.
More long term, we plan to extend our discourseaware approach towards controllable generation with given entities. Specifically, instead of using the learned GR scores, the model could generate summaries with desired entities provided by human users.
Limitation
In our method, we employ an existing NER tool (Spacy) to label the entities in both the source documents and the summaries, and the performance of the NER tool may have an influence on the results of the model. Thus a good in-domain NER tool may be required when the work is extended to some specific domains, e.g. medical text.
In addition, we use PEGASUS(Zhang et al., 2020) as our base model in all the experiments on different datasets, as it has delivered top performance on multiple summarization datasets. We follow the original paper on the length limits of all the datasets, however, the length of the source documents in both scientific paper datasets are much longer than the length limit (3k/6k v.s. 1024), which leaves the room for further improvement with sparse attention techniques applied (Xiao et al., 2022; Guo et al., 2022).
Ethics Consideration
Although we tackle the problem of factual inconsistency for abstractive summarization, and improve the entity-level factual consistency of the generated summaries by applying the entity-level span copy mechanism, the generated summaries still contain unfactual information. Therefore, caution must be exercised when the model is deployed in practical settings.

heading: Acknowledgement
text: This research was supported by the Language & Speech Innovation Lab of Cloud BU, Huawei Technologies Co., Ltd.

heading: A Model and Training Details
text: We use PEGASUS as our backbone model, which contains 571M parameters, and the span copy mechanism has 2M additional parameters. We train the fine-tuned models from the huggingface model hub6 for 100k steps (16 data per step) with early stopping based on the ROUGE scores on the validation set, which takes around 24 hours with single V100 GPU.

heading: B Datasets
text: We compare the size of filtered and original datsets in Table 7.

heading: C Software and Licenses
text: Our code is licensed under Apache License 2.0. Our framework dependencies are:
• HuggingFace Datasets7, Apache 2.0
• NLTK 8, Apache 2.0
• Numpy9, BSD 3-Clause "New" or "Revised"
• Spacy10, MIT
• Transformers11, Apache 2.0
• Pytorch12, Misc
• Pytorch Lightning 13,Apache 2.0
• ROUGE 14, Apache 2.0 6https://huggingface.co/models 7https://github.com/huggingface/
datasets/blob/master/LICENSE 8https://github.com/nltk/nltk 9https://github.com/numpy/numpy/blob/ main/LICENSE.txt 10https://github.com/explosion/spaCy/ blob/master/LICENSE 11https://github.com/huggingface/ transformers/blob/master/LICENSE 12https://github.com/pytorch/pytorch/ blob/master/LICENSE 13https://github.com/PyTorchLightning/ pytorch-lightning/blob/master/LICENSE 14https://github.com/google-research/ google-research/tree/master/rouge

heading: Future Work
text: However, in this paper, we primarily focus on exploring the application of our method to encoder-decoder models, while leaving the investigation of decoder-only models
for future research