# Enhancing Abstractive Summarization with Extracted Knowledge Graphs and Multi-Source Transformers
## 2 Related Work  
### Abstractive Text Summarization  
Typically, a neural text summarization agent takes a source document \(X_{i}\) consisting of sentences represented by a list of word tokens, digests them through an encoder to generate a latent representation, and passes the representation to a decoder that outputs the predicted summary \(Y_{i}\). The goal of model training is to maximize the conditional likelihood of each article-summary pair \(<X_{i},Y_{i}>\) in the whole corpus \(<X,Y>\). There are several exemplary models on abstractive text summarization [1; 8; 9; 10], with BART [4], which is itself a revised version of the BERT transformer-basedlanguage model [3] with extra attention layers and revised training schemes, outperforming many previous models in the domain. BART treats abstractive summarization as a translation task, summarizing article text using the same token set of the input article, and has drastically different input and output sizes compared to real translation, where the token sets are different and the output sizes are similar to the input sizes.  
**Knowledge Graphs and Graph Attention Models.** Knowledge graphs, or knowledge bases, are directed multigraphs as a semantic network containing multiple types of entities and relations with the form  
\[<\mathit{Head}(v1),\mathit{Relation}(e1),\mathit{Tail}(v2)>,\]  
where Head and Tail are entities, usually proper nouns (nodes in graph), and Relations express connections between entities (edges in the graph). Given a graph G = (V, E), we would like to learn a representation of each vertex \(v_{i}\in V\), which is contextualized by attending over the other vertices to which \(v_{i}\) is connected in G.  
The graph attention model we use is based on a Graph Attention Network (GAT) [11], which aims to capture the global context of the graph in a more effective manner. Similar models that have been used to extract features from graph semantics serving the goal of text generation include the graph-to-sequence framework [12], Gated Graph Neural Networks (GGNNs) [13], and Graph Convolutional Networks (GCNs) [14]. The most recent attempt was presented in [15], in which graph networks were viewed as guidance signals that use a separate encoder, but with some shared parameters with the document encoder.  
**Incorporating Semantics in Representation Learning.** We have discovered that there is a particular type of representation learning and sequence generation, in which an internal semantics structure is extracted and incorporated from text based on a given input. Such typical graphical structures include knowledge graphs, sentence structures, and dependency parsing trees.  
In [16], their approach comprises a set of staged abstract processes. A Factual Statement Extractor extracts concepts from simplified sentences while maintaining grammatical information, then a linguistic analysis is conducted. Based on this, representations are built around verbs, taking subject and objects if they are present. In [17], concept sets are sampled from several large corpora of image/video captions and paired with human-written sentences as expected outputs, forming a textual description from the knowledge graph structure. The model in ref. [18] is based on a pretrained Transformer Language Model, GPT-2, where Subject-Verb-Object text is generated in templates as a new input and a new fake-true story loss is built to train the generator in parallel. Similar designs also appear in models for question answering [19], conversation generation [20], and text generation from structured text [21; 22].  
Knowledge graphs are a good source of latent semantics that can form a skeleton for text generation, stressing the text/token selection of abstractive summarization. This idea has been proven possible through papers such as [23; 24; 25; 26; 27], and will be continually built upon in the future.