KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding
Year: 2023

Abstract:
With the advent of pretrained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks. These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pretrained LMs. While existing approaches have leveraged external knowledge, it remains an open question how to jointly incorporate knowledge graphs representing varying contexts—from local (e.g., sentence), to document-level, to global knowledge—to enable knowledge-rich exchange across these contexts. Such rich contextualization can be especially beneficial for long document understanding tasks since standard pretrained LMs are typically bounded by the input sequence length. In light of these challenges, we propose KALM, a Knowledge-Aware Language Model that jointly leverages knowledge in local, document-level, and global contexts for long document understanding. KALM first encodes long documents and knowledge graphs into the three knowledge-aware context representations. It then processes each context with context-specific layers, followed by a “context fusion” layer that facilitates knowledge exchange to derive an overarching document representation. Extensive experiments demonstrate that KALM achieves state-of-the-art performance on six long document understanding tasks and datasets. Further analyses reveal that the three knowledge-aware contexts are complementary and they all contribute to model performance, while the importance and information exchange patterns of different contexts vary with respect to different tasks and datasets. 1 Code and data are publicly available at https://github. com/BunsenFeng/KALM.

heading: 1 Introduction
text: Large language models (LMs) have become the dominant paradigm in NLP research, while knowledge graphs (KGs) are the de facto standard of symbolic knowledge representation. Recent advances in knowledge-aware NLP focus on combining the two paradigms (Wang et al., 2021b; Zhang et al., 2021; He et al., 2021), infusing encyclopedic (Vrandečić and Krötzsch, 2014; Pellissier Tanon et al., 2020), commonsense (Speer et al., 2017), and domain-specific (Feng et al., 2021; Chang et al., 2020) knowledge with LMs. Knowledgegrounded models achieved state-of-the-art performance in tasks including question answering (Sun et al., 2022), commonsense reasoning (Kim et al., 2022; Liu et al., 2021), and social text analysis (Zhang et al., 2022; Hu et al., 2021).
Prior approaches to infusing LMs with knowledge typically focused on three hitherto orthogonal directions: incorporating knowledge related to local (e.g., sentence-level), document-level, or global context. Local context approaches argue that sentences mention entities, and the external knowledge of entities, such as textual descriptions (Balachandran et al., 2021; Wang et al., 2021b) and metadata (Ostapenko et al., 2022), help LMs realize they are more than tokens. Document-level approaches argue that core idea entities are repeatedly mentioned throughout the document, while related concepts might be discussed in different paragraphs. These methods attempt to leverage entities and knowledge across paragraphs with document graphs (Feng et al., 2021; Zhang et al., 2022; Hu et al., 2021). Global context approaches argue that unmentioned yet connecting entities help connect the dots for knowledge-based reasoning, thus knowledge graph subgraphs are encoded with graph neural networks alongside textual content (Zhang et al., 2021; Yasunaga et al., 2021). However, despite their individual pros and cons, how to integrate the three
2116
document contexts in a knowledge-aware way remains an open problem.
Controlling for varying scopes of knowledge and context representations could benefit numerous language understanding tasks, especially those centered around long documents. Bounded by the inherent limitation of input sequence length, existing knowledge-aware LMs are mostly designed to handle short texts (Wang et al., 2021b; Zhang et al., 2021). However, processing long documents containing thousands of tokens (Beltagy et al., 2021) requires attending to varying document contexts, disambiguating long-distance co-referring entities and events, and more.
In light of these challenges, we propose KALM, a Knowledge-Aware Language Model for long document understanding. Specifically, KALM first derives three context- and knowledge-aware representations from the long input document and an external knowledge graph: the local context represented as raw text, the document-level context represented as a document graph, and the global context represented as a knowledge graph subgraph. KALM layers then encode each context with context-specific layers, followed by our proposed novel ContextFusion layers to enable knowledge-rich information exchange across the three knowledge-aware contexts. A unified document representation is then derived from contextspecific representations that also interact with other contexts. An illustration of the proposed KALM is presented in Figure 1.
While KALM is a general method for long document understanding, we evaluate the model on six tasks and datasets that are particularly sensitive to broader contexts and external knowledge: political perspective detection, misinformation detection, and roll call vote prediction. Extensive experiments demonstrate that KALM outperforms pretrained LMs, task-agnostic knowledge-aware baselines, and strong task-specific baselines on all six datasets. In ablation experiments, we further establish KALM’s ability to enable information exchange, better handle long documents, and improve data efficiency. In addition, KALM and the proposed ContextFusion layers reveal and help interpret the roles and information exchange patterns of different contexts.

heading: 2 KALM Methodology

heading: 2.1 Problem Definition
text: Let d = {d1, . . . ,dn} denote a document with n paragraphs, where each paragraph contains a sequence of ni tokens di = {wi1, . . . , wini}. Knowledge-aware long document understanding assumes the access to an external knowledge graph KG = (E ,R,A, ϵ, φ), where E = {e1, . . . , eN } denotes the entity set, R = {r1, . . . , rM} denotes the relation set, A is the adjacency matrix where aij = k indicates (ei, rk, ej) ∈ KG, ϵ(·) : E → str and φ(·) : R → str map the entities and relations to their textual descriptions.
Given pre-defined document labels, knowledgeaware natural language understanding aims to learn document representations and classify d into its corresponding label with the help of KG.

heading: 2.2 Knowledge-Aware Contexts
text: We hypothesize that a holistic representation of long documents should incorporate contexts and relevant knowledge at three levels: the local context (e.g., a sentence with descriptions of mentioned entities), the broader document context (e.g., a long document with cross-paragraph entity reference structure), and the global/external context represented as external knowledge (e.g., relevant knowledge base subgraphs). Each of the three contexts uses different granularities of external knowledge, while existing works fall short of jointly integrating the three types of representations. To this end, KALM firstly employs different ways to introduce knowledge in different levels of contexts.
Local context. Represented as the raw text of sentences and paragraphs, the local context models the smallest unit in long document understanding. Prior works attempted to add sentence metadata (e.g., tense, sentiment, topic) (Zhang et al., 2022), adopt sentence-level pretraining tasks based on KG triples (Wang et al., 2021b), or leverage knowledge graph embeddings along with textual representations (Hu et al., 2021). While these methods were effective, in the face of LM-centered NLP research, they are ad-hoc add-ons and not fully compatible with existing pretrained LMs. As a result, KALM proposes to directly concatenate the textual descriptions of entities ϵ(ei) to the paragraph if ei is mentioned. In this way, the original text is directly augmented with the entity descriptions, informing the LM that entities such as "Kepler" are more than
mere tokens and help to combat the spurious correlations of pretrained LMs (McMilin). For each augmented paragraph d′i, we adopt LM(·) and mean pooling to extract a paragraph representation. We use pretrained BART encoder (Lewis et al., 2020) as LM(·) without further notice. We also add a fusion token at the beginning of the paragraph sequence for information exchange across contexts. After processing all n paragraphs, we obtain the local context representation T (0) as follows:
T (0) = {t(0)0 , . . . , t(0)n } = {θrand,LM(d′1), . . . ,LM(d′n)}
where θrand denotes a randomly initialized vector of the fusion token in the local context and the superscript (0) indicates the 0-th layer.
Document-level context. Represented as the structure of the full document, the documentlevel context is responsible for modeling crossparagraph entities and knowledge on a document level. While existing works attempted to incorporate external knowledge in documents via document graphs (Feng et al., 2021; Hu et al., 2021), they fall short of leveraging the overlapping entities and concepts between paragraphs that underpin the reasoning of long documents. To this end, we propose knowledge coreference, a simple and effective mechanism for modeling text-knowledge interaction on the document level. Specifically, a document graph with n+ 1 nodes is constructed, consisting of one fusion node and n paragraph nodes. If paragraph i and j both mention entity ek in the external KB, node i and j in the document graph are connected with relation type k. In addition, the
fusion node is connected to every paragraph node with a super-relation. As a result, we obtain the adjacency matrix of the document graph Ag. Paired with the knowledge-guided GNN to be introduced in Section 2.3, knowledge coreference enables the information flow across paragraphs guided by external knowledge. Node feature initialization of the document graph is as follows:
G(0) = {g(0)0 , . . . ,g(0)n } = {θrand,LM(d1), . . . ,LM(dn)}
Global context. Represented as external knowledge graphs, the global context is responsible for leveraging unseen entities and facilitating KGbased reasoning. Existing works mainly focused on extracting knowledge graph subgraphs (Yasunaga et al., 2021; Zhang et al., 2021) and encoding them alongside document content. Though many tricks are proposed to extract and prune KG subgraphs, in KALM, we employ a straightforward approach: for all mentioned entities in the long document, KALM merges their k-hop neighborhood to obtain a knowledge graph subgraph. We use k = 2 following previous works (Zhang et al., 2021; Vashishth et al., 2019), striking a balance between KB structure and computational efficiency while KALM could support any k settings. A fusion entity is then introduced and connected with every other entity, resulting in a connected graph. In this way, KALM cuts back on the preprocessing for modeling global knowledge and better preserve the information in the KG. Knowledge graph embedding methods (Bordes et al., 2013) are then adopted to initialize node features of the KG subgraph:
K(0) = {k(0)0 , . . . ,k (0) |ρ(d)|}
= {θrand,KGE(e1), . . . ,KGE(e|ρ(d)|)}
where KGE(·) denotes the knowledge graph embeddings trained on the original KG, |ρ(d)| indicates the number of mentioned entities identified in document d. We use TransE (Bordes et al., 2013) to learn KB embeddings and use them for KGE(·), while the knowledge base embeddings are kept frozen in the KALM training process.

heading: 2.3 KALM Layers
text: After obtaining the local, document-level, and global context representations of long documents, we employ KALM layers to learn document representations. Specifically, each KALM layer consists of three context-specific layers to process each context. A ContextFusion layer is then adopted to enable the knowledge-rich information exchange across the three contexts.

heading: 2.3.1 Context-Specific Layers
text: Local context layer. The local context is represented as a sequence of vectors extracted from the knowledge-enriched text with the help of pretrained LMs. We adopt transformer encoder layers (Vaswani et al., 2017) to encode the local context:
T̃ (ℓ) = {t̃(ℓ)0 , . . . , t̃(ℓ)n } = ϕ ( TrmEnc({t(ℓ)0 , . . . , t(ℓ)n }) )
where ϕ(·) denotes non-linearity, TrmEnc denotes the transformer encoder layer, and t̃(ℓ)0 denotes the transformed representation of the fusion token. We omit the layer subscript (ℓ) for brevity.
Document-level context layer. The documentlevel context is represented as a document graph based on knowledge coreference. To better exploit the entity-based relations in the document graph, we propose a knowledge-aware GNN architecture to enable knowledge-guided message passing on the document graph:
G̃ = {g̃0, . . . , g̃n = GNN ( {g0, . . . ,gn} )
where GNN(·) denotes the proposed knowledgeguided graph neural networks as follows:
g̃i = ϕ ( αi,iΘgi + ∑
j∈N (i) Θgj
)
where αi,j denotes the knowledge-guided attention weight and is defined as follows:
αi,j = exp
( ELU(aT [Θgi||Θgj ||Θf(KGE(agij))]) )
∑ k∈N (i) exp ( ELU(aT [Θgi||Θgk||Θf(KGE(agik))]) )
where g̃0 denotes the transformed representation of the fusion node, a and Θ are learnable parameters, agij is the i-th row and j-th column value of adjacency matrix Ag of the document graph, ELU denotes the exponential linear unit activation function (Clevert et al., 2015), and f(·) is a learnable linear layer. Θf(KGE(agij)) is responsible for enabling the knowledge-guided message passing on the document graph, enabling KALM to incorporate the entity and concept patterns in different paragraphs and their document-level interactions.
Global context layer. The global context is represented as a relevant knowledge graph subgraph. We follow previous works and adopt GATs (Veličković et al., 2018) to encode the global context:
K̃ = {k̃0, . . . , k̃|ρ(d)|} = GAT ( {k0, . . . ,k|ρ(d)|} )
where k̃0 denotes the transformed representation of the fusion entity.

heading: 2.3.2 ContextFusion Layer
text: The local, document, and global contexts model external knowledge within sentences, across the document, and beyond the document. These contexts are closely connected and a robust long document understanding method should reflect their interactions. Existing approaches mostly leverage only one or two of the contexts (Wang et al., 2021b; Feng et al., 2021; Zhang et al., 2022), falling short of jointly leveraging the three knowledge-aware contexts. In addition, they mostly adopted direct concatenation or MLP layers (Zhang et al., 2022, 2021; Hu et al., 2021), falling short of enabling context-specific information to flow across contexts in a knowledge-rich manner. As a result, we propose the ContextFusion layer to tackle these challenges. We firstly take a local perspective and
extract the representations of the fusion tokens, nodes, and entities in each context:
[ tL,gL,kL ] = [ t̃0, g̃0, k̃0 ]
We then take a global perspective and use the fusion token/node/entity as the query to conduct attentive pooling ap(·, ·) across all other tokens/nodes/entities in each context:
[ tG,gG,kG ] = [ ap ( t̃0, {t̃i}ni=1 ) ,
ap ( g̃0, {g̃i}ni=1 ) , ap ( k̃0, {k̃i}ni=1
)]
where attentive pooling ap(·, ·) is defined as:
ap ( q, {ki}ni=1 ) = n∑
i=1
exp ( q · ki ) ∑n
j=1 exp ( q · kj )ki
In this way, the fusion token/node/entity in each context serves as the information exchange portal. We then use a transformer encoder layer to enable information exchange across the contexts:
[ t̃L, g̃L, k̃L, t̃G, g̃G, k̃G ]
= ϕ ( TrmEnc ([ tL,gL,kL, tG,gG,kG ]))
As a result, t̃L, g̃L, and k̃L are the representations of the fusion token/node/entity that incorporates information from other contexts. We formulate the output of the l-th layer as follows:
T (ℓ+1) = {t̃(ℓ)L , t̃ (ℓ) 1 , . . . , t̃ (ℓ) n }, G(ℓ+1) = {g̃(ℓ)L , g̃ (ℓ) 1 , . . . , g̃ (ℓ) n }, K(ℓ+1) = {k̃(ℓ)L , k̃ (ℓ) 1 , . . . , k̃ (ℓ) n }
Our proposed ContextFusion layer is interactive since it enables the information to flow across different document contexts, instead of direct concatenation or hierarchical processing. The attention weights in TrmEnc(·) of the ContextFusion layer could also provide insights into the roles and importance of each document context, which will be further explored in Section 3.3. To the best of our knowledge, KALM is the first work to jointly consider the three levels of document context and enable information exchange across document contexts.

heading: 2.4 Learning and Inference
text: After a total of P KALM layers, we obtain the final document representation as [ t̃ (P) L , g̃ (P) L , k̃ (P) L ] . Given the document label a ∈ A, the label probability is formulated as p(a|d) ∝ exp ( MLPa([t̃ (P) L , g̃ (P) L , k̃ (P) L ]) ) . We then optimize KALM with the cross entropy loss function. At inference time, the predicted label is argmaxap(a|d).

heading: 3 Experiment

heading: 3.1 Experiment Settings
text: Tasks and Datasets. We propose KALM, a general method for knowledge-aware long document understanding. We evaluate KALM on three tasks that especially benefit from external knowledge and broader context: political perspective detection, misinformation detection, and roll call vote prediction. We follow previous works to adopt SemEval (Kiesel et al., 2019) and Allsides (Li and Goldwasser, 2019) for political perspective detection, LUN (Rashkin et al., 2017) and SLN (Rubin et al., 2016) for misinformation detection, and the 2 datasets proposed in Mou et al. (2021) for roll call vote prediction. For external KGs, we follow existing works to adopt the KGs in KGAP (Feng et al., 2021), CompareNet (Hu et al., 2021), and ConceptNet (Speer et al., 2017) for the three tasks.
Baseline methods. We compare KALM with three types of baseline methods for holistic evaluation: pretrained LMs, task-agnostic knowledgeaware methods, and task-specific models. For pretrained LMs, we evaluate RoBERTa (Liu et al., 2019b), Electra (Clark et al., 2019), DeBERTa (He et al., 2020), BART (Lewis et al., 2020), and LongFormer (Beltagy et al., 2020) on the three tasks. For task-agnostic baselines, we evaluate KGAP (Feng et al., 2021), GreaseLM (Zhang et al., 2021), and GreaseLM+ on the three tasks. Task-specific models are introduced in the following sections. For pretrained LMs, task-agnostic methods, and KALM, we run each method five times and report the average performance and standard deviation. For task-specific models, we compare with the results originally reported since we follow the exact same experiment settings and data splits.

heading: 3.2 Model Performance
text: We present the performance of task-specific methods, pretrained LMs, task-agnostic knowledge-
aware baselines, and KALM in Table 1. We select the best-performing task-specific baseline (Task SOTA) and pretrained language model (BestLM), while the full results are available in Tables 4, 5, and 6 in the appendix. Table 1 demonstrates that:
• KALM consistently outperforms all task-specific models, pretrained language models, and knowledge-aware methods on all three tasks and six datasets/settings. Statistical significance tests in Section A.4 further demonstrates KALM’s superiority over existing models.
• Knowledge-aware LMs generally outperform pretrained LMs, which did not incorporate external knowledge bases in the pretraining process. This suggests that incorporating external knowledge bases could enrich document representations and boost downstream task performance.
• GreaseLM+ outperforms GreaseLM by adding the global context, which suggests the importance of jointly leveraging the three document contexts. KALM further introduces information
exchange across contexts through the ContextFuion layer and achieves state-of-the-art performance. We further investigate the importance of three document contexts and the ContextFusion layer in Section 2.3.2.

heading: 3.3 Context Exchange Study
text: By jointly modeling three document contexts and employing the ContextFusion layer, KALM facilitates information exchange across the three document contexts. We conduct an ablation study to examine whether the contexts and the ContextFusion layer are essential in the KALM architecture. Specifically, we remove the three contexts one at a time and change the ContextFusion layer into MInt (Zhang et al., 2021), concatenation, and sum. Table 2 demonstrates that:
• All three levels of document contexts, local, document, and global, contribute to model performance. These results substantiate the necessity of jointly leveraging the three document contexts for long document understanding.
• When substituting our proposed ContextFusion
layers with three existing combination strategies, MInt (Zhang et al., 2021), direct concatenation, and summation, performance drops are observed across multiple datasets. This suggests that the proposed ContextFusionn layer successfully boost model performance by enabling information exchange across contexts.
In addition to boosting model performance, the ContextFusion layer probes how different contexts contribute to document understanding. We calculate the average of attention weights’ absolute values of the multi-head attention in the TrmEnc(·) layer of ContextFusion and illustrate in Figure 2, which shows that the three contexts’ contribution and information exchange patterns vary with respect to datasets and KALM layers. Specifically, local and global contexts are important for the LUN dataset, document and global contexts are important for the task of roll call vote prediction, and the SLN dataset equally leverages the three contexts. However, for the task of political perspective detection, the importance of the three aspects varies with the depth of KALM layers. This is especially salient on SemEval, where KALM firstly takes a view of the whole document, then draws from both local and document-level contexts, and closes by leveraging global knowledge to derive an overall document representation.
In summary, the ContextFusion layer in KALM successfully identifies the relative importance and information exchange patterns of the three contexts, providing insights into how KALM arrives at the conclusion and which context should be the focus
of future research. We further demonstrate that the role and importance of each context change as training progresses in Section A.1 in the appendix.

heading: 3.4 Long Document Study
text: KALM complements the scarce literature in knowledge-aware long document understanding. In addition to more input tokens, it often relies on more knowledge reference and knowledge reasoning. To examine whether KALM indeed improved in the face of longer documents and more external knowledge, we illustrate the performance of KALM and competitive baselines with respect to document length and knowledge intensity in Figure 3. Specifically, we use the number of mentioned entities to represent knowledge intensity and the number of sentences to represent document length, mapping each data point onto a two-dimensional space. It is illustrated that while baseline methods are prone to mistakes when the document is long and knowledge is rich, KALM alleviates this issue and performs better in the top-right corner. We further analyze KALM and more baseline methods’ performance on long documents with great knowledge intensity in Figure 6 in the appendix.

heading: 3.5 Data Efficiency Study
text: Existing works argue that introducing knowledge graphs to NLP tasks could improve data efficiency and help alleviate the need for extensive training data (Zhang et al., 2022). By introducing knowledge to all three document contexts and enabling knowledge-rich context information exchange, KALM might be in a better position to
tackle this issue. To examine whether KALM has indeed improved data efficiency, we compare the performance of KALM with competitive baselines when trained on partial training sets and illustrate the results in Figure 4. It is demonstrated that while performance did not change greatly with 30% to 100% training data, baseline methods witness significant performance drops when only 10% to 20% of data are available. In contrast, KALM maintains steady performance with as little as 10% of training data.

heading: 4 Related Work
text: Knowledge graphs are playing an increasingly important role in language models and NLP research. Commonsense (Speer et al., 2017; Ilievski et al., 2021; Bosselut et al., 2019; West et al., 2022; Li et al., 2022a) and domain-specific KGs (Feng et al., 2021; Li et al., 2022b; Gyori et al., 2017) serve as external knowledge to augment pretrained LMs, which achieves state-of-the-art performance on question answering (Zhang et al., 2021; Yasunaga et al., 2021; Mitra et al., 2022; Bosselut et al., 2021; Oguz et al., 2022; Feng et al., 2022b; Heo et al., 2022; Ma et al., 2022; Li and Moens, 2022; Zhou and Small, 2019), social text analysis (Hu et al., 2021; Zhang et al., 2022; Reddy et al., 2022), commonsense reasoning (Kim et al., 2022; Jung et al., 2022; Amayuelas et al., 2021; Liu et al., 2022), and text generation (Rony et al., 2022). These approaches (Lu et al., 2022; Zhang et al., 2019; Yu et al., 2022b; Sun et al., 2020; Yamada et al., 2020; Qiu et al., 2019a; Xie et al., 2022) could be mainly categorized by the three levels of the context where knowledge injection happens.
Local context approaches focus on entity men-
tions and external knowledge in individual sentences to enable fine-grained knowledge inclusion. A straightforward way is to encode KG entities with KG embeddings (Bordes et al., 2013; Lin et al., 2015; Cucala et al., 2021; Sun et al., 2018) and infuse the embeddings with language representations (Hu et al., 2021; Feng et al., 2021; Kang et al., 2022). Later approaches focus on augmenting pretrained LMs with KGs by introducing knowledgeaware training tasks and LM architectures (Wang et al., 2021b,a; Sridhar and Yang, 2022; Moiseev et al., 2022; Kaur et al., 2022; Hu et al., 2022; Arora et al., 2022; de Jong et al., 2021; Meng et al., 2021; He et al., 2021). Topic models were also introduced to enrich document representation learning (Gupta et al., 2018; Chaudhary et al., 2020; Wang et al., 2018). However, local context approaches fall short of leveraging inter-sentence and inter-entity knowledge, resulting in models that could not grasp the full picture of the text-knowledge interactions.
Document-level models analyze documents by jointly considering external knowledge across sentences and paragraphs. The predominant way of achieving document-level knowledge infusion is through "document graphs" (Zhang et al., 2022), where textual content, external knowledge bases, and other sources of information are encoded and represented as different components in graphs, often heterogeneous information networks (Hu et al., 2021; Feng et al., 2021; Zhang et al., 2022; Yu et al., 2022a). Graph neural networks are then employed to learn representations, which fuse both textual information and external KGs. However, document-level approaches fall short of preserving the original KG structure, resulting in models with reduced knowledge reasoning abilities.
0.95 SLN
LUN
0.95 Random
0.95 Time-Based
Global context approaches focus on the KG, extracting relevant KG subgraphs based on entity mentions. Pruned with certain mechanisms (Yasunaga et al., 2021) or not (Qiu et al., 2019b), these KG subgraphs are encoded with GNNs, and such representations are fused with LMs from simple concatenation (Hu et al., 2021) to deeper interactions (Zhang et al., 2021). However, global context approaches leverage external KGs in a stand-alone manner, falling short of enabling the dynamic integration of textual content and external KGs.
While existing approaches successfully introduced external KG to LMs, long document understanding poses new challenges to knowledgeaware NLP. Long documents possess greater knowledge intensity where more entities are mentioned, more relations are leveraged, and more reasoning is required to fully understand the nuances, while existing approaches are mostly designed for sparse knowledge scenarios. In addition, long documents also exhibit the phenomenon of knowledge co-reference, where central ideas and entities are reiterated throughout the document and co-exist in different levels of document contexts. In light of these challenges, we propose KALM to jointly leverage the local, document, and global contexts of long documents for knowledge incorporation.

heading: 5 Conclusion
text: In this paper, we propose KALM, a knowledgeaware long document understanding approach that introduces external knowledge to three levels of document contexts and enables interactive exchange across them. Extensive experiments demon-
strate that KALM achieves state-of-the-art performance on three tasks across six datasets. Our analysis shows that KALM provides insights into the roles and patterns of individual contexts, improves the handling of long documents with greater knowledge intensity, and has better data efficiency than existing works.

heading: Limitations
text: Our proposed KALM has two limitations:
• KALM relies on existing knowledge graphs to facilitate knowledge-aware long document understanding. While knowledge graphs are effective and prevalent tools for modeling real-world symbolic knowledge, they are often sparse and hardly exhaustive (Tan et al., 2022; Pujara et al., 2017). In addition, external knowledge is not only limited to knowledge graphs but also exists in textual, visual, and other symbolic forms. We leave it to future work on how to jointly leverage multiple forms and sources of external knowledge in document understanding.
• KALM leverages TagMe (Ferragina and Scaiella, 2011) to identify entity mentions and build the three knowledge-aware contexts. While TagMe and other entity identification tools are effective, they are not 100% correct, resulting in potentially omitted entities and external knowledge. In addition, running TagMe on hundreds of thousands of long documents is time-consuming and resource-consuming even if processed in parallel. We leave it to future work on how to leverage
knowledge graphs for long document understanding without using entity linking tools.

heading: Ethics Statement
text: KALM is a knowledge-aware long document understanding approach that jointly leverages pretrained LMs and knowledge graphs on three levels of contexts. Consequently, KALM might exhibit many of the biases of the adopted language models (Liang et al., 2021; Nadeem et al., 2021) and knowledge graphs (Fisher et al., 2020, 2019; Mehrabi et al., 2021; Du et al., 2022; Keidar et al., 2021). As a result, KALM might leverage the biased and unethical correlations in LMs and KGs to arrive at conclusions. We encourage KALM users to audit its output before using it beyond the standard benchmarks. We leave it to future work on how to leverage knowledge graphs in pretrained LMs with a focus on fairness and equity.

heading: Acknowledgements
text: We would like to thank the reviewers, the area chair, Vidhisha Balachandran, Melanie Sclar, and members of the Tsvetshop for their feedback. This material is funded by the DARPA Grant under Contract No. HR001120C0124. We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, and NSF grants No. IIS2125201 and IIS2203097. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily state or reflect those of the United States Government or any agency thereof.

heading: A Additional Experiments

heading: A.1 Context Exchange Study (cont.)
text: In Section 3.3, we conducted an ablation study of the three knowledge-aware contexts and explored how the ContextFusion layer enables the interpretation of context contribution and information exchange patterns. It is demonstrated that the three contexts play different roles with respect to datasets and KALM layers. In addition, we explore whether the role and information exchange patterns of contexts change when the training progresses. Figure 5 illustrates the results with respect to training epochs, which shows that the attention matrices started out dense and ended sparse, indicating that the role of different contexts is gradually developed through time.

heading: A.2 Long Document Study (cont.)
text: We present error analysis with respect to document length and knowledge intensity on more baseline methods, including language models (RoBERTa, BART, LongFormer), knowledgeaware LMs (KGAP, GreaseLM, GreaseLM+), and our proposed KALM in Figure 6. Our conclusion still holds true: KALM successfully improves performance on documents that are longer and contain more external knowledge, which are positioned in the top-right corner of the figure.

heading: A.3 Manual Error Analysis
text: We manually examined 20 news articles from the LUN misinformation detection dataset where KALM made a mistake. Several news articles focused on the same topic of marijuana legalization, and some others focused on international affairs such as the conflict in Iraq. These articles feature entities and knowledge that are much more recent
such as "pot-infused products" and "ISIS jihadists", which are emerging concepts and generally not covered by existing knowledge graphs. We present the relevant sentences in Table 3. This indicates the need for more comprehensive, up-to-date, and temporal knowledge graphs that grow with the world.

heading: A.4 Significance Testing
text: To examine whether KALM significantly outperforms baselines on the three tasks, we conduct oneway repeated measures ANOVA test for the results in Table 4, Table 5, and Table 6. It is demonstrated that the performance gain is significant on five of the six datasets, specifically SemEval (against the second-best KCD on Acc and MaF), SLN (against the second-best KGAP on MiF and MaRecall), LUN (against the second-best CompareNet on MiF, MaF and MaRecall), Random (against the second-best GreasesLM+ on BAcc and MaF), and Time-Based (against the second-best GreaseLM+ on BAcc and MaF).

heading: A.5 Task-Specific Model Performance
text: We present the full results for task-specific methods, pretrained language models, knowledge-aware task-agnostic models, and KALM on the three tasks and six datasets/settings in Tables 4, 5, and 6.
A.6 Is local context enough?
Though long document understanding requires attending to a long sequence of tokens, it is possible that sometimes only one or two sentences would give away the label of the document. We examine this by removing the document-level and global contexts in KALM, leaving only the local context to simulate this scenario. Comparing the localonly variant with the full KALM, there are 14.78%, 10.53%, 8.21%, 4.85%, 1.4%, and 3.18% performance drops across the six datasets in terms of macro-averaged F1-score. As a result, it is necessary to go beyond local context windows in long document understanding.

heading: B Experiment Details

heading: B.1 Dataset Details
text: We present important dataset details in Table 7. We follow the exact same dataset settings and splits in previous works (Zhang et al., 2022; Hu et al., 2021; Feng et al., 2022a) for fair comparison.

heading: B.2 Baseline Details
text: We compare KALM with pretrained language models, task-specific baselines, and task-agnostic knowledge-aware methods to ensure a holistic evaluation. In the following, we provide a brief description of each of the baseline methods. We also highlight whether one approach leverages knowledge graphs and the three document contexts in Table 9.
• HLSTM (Yang et al., 2016) is short for hierarchical long short-term memory networks. It was used in previous works (Li and Goldwasser, 2019, 2021) for political perspective detection.
• MAN (Li and Goldwasser, 2021) proposes to
• KCD (Zhang et al., 2022) proposes to leverage multi-hop knowledge reasoning with knowledge walks and textual cues with document graphs for political perspective detection.
• Rubin et al. (2016) proposes the SLN dataset and leverages satirical cues for misinformation detection.
• Rashkin et al. (2017) proposes the LUN dataset and argues that misinformation detection should have more fine-grained labels than true or false.
• GCN (Welling and Kipf, 2016) and GAT (Veličković et al., 2018) are leveraged along with the attention mechanism by Hu et al. (2021) for misinformation detection on graphs.
• CompareNet (Hu et al., 2021) proposes to leverage knowledge graphs and compare the textual
Baseline Random Time-Based
BAcc MaF BAcc MaF
task specific
ideal-point 86.46 80.02 / / ideal-vector 87.35 80.15 81.95 75.49 Vote 90.22 84.92 89.76 84.35 PAR 90.33 / 89.92 /
language model RoBERTa 89.94 (±0.2) 86.10 (±0.7) 90.40 (±0.8) 84.78 (±2.2) Electra 87.47 (±0.3) 80.23 (±0.7) 88.92 (±0.4) 82.50 (±1.7) DeBERTa 86.98 (±0.4) 80.07 (±1.2) 88.59 (±0.1) 81.38 (±1.0) BART 89.76 (±0.5) 85.52 (±0.6) 90.25 (±0.6) 85.21 (±2.1) LongFormer 88.69 (±0.4) 83.52 (±1.2) 89.32 (±1.4) 83.42 (±3.8)
task agnostic KELM 89.13 (±1.1) 84.76 (±2.0) 90.80 (±0.2) 86.62 (±0.4) KnowBERT-Wordnet 86.72 (±0.9) 79.33 (±2.4) 86.92 (±0.6) 78.90 (±1.9) KnowBERT-Wikidata 85.98 (±0.8) 78.48 (±1.0) 86.45 (±0.5) 78.21 (±0.7) KnowBERT-W+W 85.75 (±1.0) 78.70 (±2.4) 87.07 (±1.0) 78.42 (±2.2) Joshi et al. 91.43 (±0.5) 89.64 (±0.6) 92.63 (±1.6) 89.31 (±2.4) KGAP 77.98 (±0.5) 68.11 (±6.0) 77.90 (±0.6) 70.81 (±4.6) GreaseLM 89.99 (±1.5) 84.72 (±3.0) 88.21 (±2.7) 79.73 (±7.4) GreaseLM+ 91.01 (±0.2) 87.29 (±0.3) 91.69 (±0.1) 87.95 (±0.3) KALM (Ours) 92.36 (±0.3) 89.33 (±0.4) 94.46 (±0.4) 91.97 (±0.5)
content to external knowledge for misinformation detection.
• Ideal-point (Gerrish and Blei, 2011) and idealvector (Kraft et al., 2016) propose to use 1d and 2d representations of political actors for roll call vote prediction.
• Vote (Mou et al., 2021) proposes to jointly leverage legislation text and the social network information for roll call vote prediction.
• PAR (Feng et al., 2022a) proposes to learn legislator representations with social context and expert knowledge for roll call vote prediction.
• RoBERTa (Liu et al., 2019b), Electra (Clark
et al., 2019), DeBERTa (He et al., 2020), BART (Lewis et al., 2020), and LongFormer (Beltagy et al., 2020) are pretrained language models. We use the pretrained weights roberta-base, electra-small-discriminator, deberta-v3-base, bart-base, and longformer-base-4096 in Huggingface Transformers (Wolf et al., 2020) to extract sentence representations, average across the whole document, and classify with softmax layers.
• KELM (Agarwal et al., 2021) proposes to generate synthetic pretraining corpora based on structured knowledge bases. In this paper, we further pretrained the roberta-base checkpoint on the KELM synthetic corpus and report performance
on downstream tasks.
• KnowBERT (Peters et al., 2019) is one of the first works to leverage external knowledge bases to enrich language representations. We used the three pretrained models, KnowBERT-Wordnet, KnowBERT-Wikidata, and KnowBERT-W+W for document representation extraction and report performance on downstream tasks.
• Joshi et al. (2020) proposes to learn contextualized language representations by adding Wikipedia text to the input sequences and jointly learning text representations. This is similar to KALM’s setting with only the local context, where Wikipedia descriptions of entities are concatenated to input texts.
• KGAP (Feng et al., 2021) proposes to construct document graphs to jointly encode textual content and external knowledge. Gated relational graph convolutional networks are then adopted for document representation learning.
• GreaseLM (Zhang et al., 2021) proposes to encode textual content with language model layers,
encode knowledge graph subgraphs with graph neural networks and KG embeddings, and adopt MInt layers to fuse the two for question answering. In this paper, we implement GreaseLM by using MInt layers to fuse the local and global contexts.
• GreaseLM+ is our extended version of GreaseLM, which adds the document-level context while keeping the original MInt layer instead of our proposed ContextFusion layer.
• KALM is our proposed approach for knowledgeaware long document understanding. It jointly infuses the local, document-level, and global contexts with external knowledge graphs and adopts ContextFusion layers to derive an overarching document representation.

heading: B.3 Evaluation Metrics Details
text: We adopted these evaluation metrics throughout the paper: Acc (accuracy), MaF (macro-averaged F1-score), MiF (micro-averaged F1-score), MaPrecision (macro-averaged precision), MaRecall
(macro-averaged recall), and BAcc (balanced accuracy). These metrics are chosen based on which metrics are used in previous works regarding the three tasks.

heading: B.4 Hyperparameter Details
text: We present KALM’s hyperparameter settings in Table 8. We conduct hyperparameter searches for different datasets and report the best setups.
B.5 Where did the numbers come from? For task-specific baselines, we directly use the results reported in previous works (Zhang et al., 2022; Hu et al., 2021; Feng et al., 2022a) since we follow the same experiment settings and the comparison is thus fair. For pretrained LMs and task-agnostic baselines, we run each method five times with different random seeds and report the average performance as well as standard deviation. Figure 4 is an exception, where we only run each method one time due to computing constraints.

heading: B.6 More experiment details
text: We provide more details about the experiments that are worth further explaining.
• Table 6: We implement pretrained LMs and taskagnostic baselines for roll call vote prediction by using them to learn representations of legislation texts, concatenate them with the legislator representations learned with PAR (Feng et al., 2022a), and adopt softmax layers for classification.
• Table 2: We remove each context by only applying ContextFusion layers to the other two context representations. We follow the implementation of MInt described in Zhang et al. (2021). We implement concat and sum by using the concatenation and summation of the three context representations as the overall document representation.
• Figure 2: The multi-head attention in the ContextFusion layer provides a 6 × 6 attention weight matrix indicating how information flowed across different contexts. The six rows (columns) stand for the local view of the local context, the global view of the local context, the local view of the document-level context, the global view of the document-level context, the local view of the global context, and the global view of the global context, which are described in detail in Section
2.3.2. The values in each square are the average of the absolute values of the attention weights across all data samples in the validation set.

heading: B.7 Computational Resources Details
text: We used a GPU cluster with 16 NVIDIA A40 GPUs, 1,988G memory, and 104 CPU cores for the experiments. Running KALM with the best parameters takes approximately 1.5, 16, 3, 4, 1, and 1 hour(s) for the six datasets (SemEval, Allsides, SLN, LUN, random, time-based).

heading: B.8 Scientific Artifact Details
text: KALM is built with the help of many existing scientific artifacts, including TagMe (Ferragina and Scaiella, 2011), pytorch (Paszke et al., 2019), pytorch lightning (Falcon and The PyTorch Lightning team, 2019), transformers (Wolf et al., 2020), pytorch geometric (Fey and Lenssen, 2019), sklearn (Pedregosa et al., 2011), numpy (Harris et al., 2020), nltk (Bird et al., 2009), OpenKE (Han et al., 2018), and the three adopted knowledge graphs (Feng et al., 2021; Hu et al., 2021; Speer et al., 2017). We commit to make our code and data publicly available upon acceptance to facilitate reproduction and further research.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: right after the main paper on page 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: right after the main paper on page 9

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: introduction is in Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? throughout the paper

heading: 3 B1. Did you cite the creators of artifacts you used?
text: throughout the paper, wherever the adopted artifact is mentioned
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 7 in the appendix
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section B.7
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section B.4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 and Section A

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section B.8
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave it to future work on how to leverage knowledge graphs in pretrained LMs with a focus on fairness and equity