GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction (2024)

Abstract: Document-level relation extraction typically relies on text-based encoders and hand-coded pooling heuristics to aggregate information learned by the encoder. In this paper, we leverage the intrinsic graph processing capabilities of the Transformer model and propose replacing hand-coded pooling methods with new tokens in the input, which are designed to aggregate information via explicit graph relations in the computation of attention weights. We introduce a joint text-graph Transformer model and a graph-assisted declarative pooling (GADePo) specification of the input, which provides explicit and high-level instructions for information aggregation. GADePo allows the pooling process to be guided by domain-specific knowledge or desired outcomes but still learned by the Transformer, leading to more flexible and customisable pooling strategies. We evaluate our method across diverse datasets and models and show that our approach yields promising results that are consistently better than those achieved by the hand-coded pooling functions.

heading: 1 Introduction
text: Document-level relation extraction is an important task in natural language processing, which involves identifying and categorising meaningful relationships between entities within a document, as exemplified in Figure 1. This task is foundational to many applications, including knowledge base population and completion (Banko et al., 2007; Ji et al., 2020), information retrieval and extraction (Manning et al., 2008; Theodoropoulos et al., 2021), question answering (Chen et al., 2017; Feng et al., 2022) and sentiment analysis (Pang and Lee, 2008), to name a few.
Standard methods that approach this challenge generally employ pretrained text-based encoders (Devlin et al., 2019; Beltagy et al., 2019; Zhuang et al., 2021; Cui et al., 2021), which are responsible for capturing the nuances of information con-

heading: Subject: Breakout  Object: Atari
text: Subject entity Breakout (red) and object entity Atari (blue) express relations "developer" and "publisher". Other entities are indicated as Mention (white).
tained in the entity mentions and their contextual surroundings. Previous successful methods often then use hand-coded pooling heuristics to aggregate the information learned by the encoder, with some aimed at creating entity representations, while others directly exploiting the pattern of attention weights to capture context aware relations between entity mentions (Zhou et al., 2021; Xiao et al., 2022; Tan et al., 2022a; Ma et al., 2023). These pooling heuristics can be very effective at leveraging the information in a pretrained encoder. However, as shown in Conneau et al. (2017); Jia et al. (2019); Reimers and Gurevych (2019); Choi et al. (2021), the selection of an appropriate pooling function can be model-dependent, task-specific, resource-intensive and time-consuming to determine, thereby limiting flexibility.
In this paper, we address these issues with a new approach where we leverage the intrinsic graph processing capabilities of the Transformer model (Vaswani et al., 2017), leveraging insights from the work of Mohammadshahi and Henderson (2020); Henderson (2020); Mohammadshahi and Henderson (2021); Henderson et al. (2023). They argue that attention weights and graph relations are functionally equivalent and show how to incorporate
1
structural dependencies between input elements by simply adding relation features to the attention functions. Transformers easily learn to integrate these relation features into their pretrained attention functions, resulting in very successful graphconditioned models (Mohammadshahi and Henderson, 2021; Miculicich and Henderson, 2022; Mohammadshahi and Henderson, 2023). Given this effective method for integrating explicit graphs with pretrained attention functions, we propose to use the attention function itself for aggregation. We replace the rigid pooling methods with new tokens which act as aggregation nodes, plus explicit graph relations which steer the aggregation.
We introduce a joint text-graph Transformer model and a graph-assisted declarative pooling (GADePo) method1 that leverages these special tokens and graph relations, to provide an explicit high-level declarative specification for the information aggregation process. By integrating these graphs in the attention functions of a pretrained model, GADePo exploits the pretrained embeddings and attention patterns but still has the flexibility of being trained on data. This enables the pooling to be guided by domain-specific knowledge or desired outcomes but still learned by the Transformer, opening up a more customisable but still data-driven relation extraction process.
We evaluate our method across diverse datasets and models commonly employed in documentlevel relation extraction tasks, and show that our approach yields promising results that are consistently better than those achieved by the hand-coded pooling functions.
Contributions We propose a new method for exploiting pretrained Transformer models which replaces hand-coded aggregation functions with explicit graph relations and aggregation nodes. We introduce a novel form of joint text-graph Transformer model. We evaluate our approach across various datasets and models, showing that it yields promising results that are consistently better than those achieved by hand-coded pooling functions.

heading: 2 Related Work
text: In recent studies, the scope of relation extraction has been expanded to include not only individual sentences but entire documents. This extension, known as document-level relation extraction,
1https://github.com/idiap/gadepo
presents a more realistic and challenging scenario as it seeks to extract relations both within sentences and across multiple sentences (Yao et al., 2019). Transformer-based (Vaswani et al., 2017) models have shown great potential in addressing this task.
Wang et al. (2019) and Tang et al. (2020) show that the BiLSTM-based (Hochreiter and Schmidhuber, 1997) baselines lack the capacity to model complex interactions between multiple entities. They propose a more robust approach, which consists of using the pretrained BERT (Devlin et al., 2019) model and a two-step prediction process, i.e., first identifying if a link between two entities exists, followed by predicting the specific relation type.
GAIN (Zeng et al., 2020) leverages BERT as a text encoder and GCNs (Kipf and Welling, 2017) to process two types of graphs, one at mention level and another at entity level, showing notable performance in inter-sentence and inferential scenarios.
Mohammadshahi and Henderson (2020, 2021) propose the G2GT model and show how to leverage the intrinsic graph processing capabilities of the Transformer model by incorporating structural dependencies between input elements as features input to the self-attention weight computations.
SSAN (Xu et al., 2021) leverages this idea and considers the structure of entities. It employs a transformation module that creates attentive biases from this structure to regulate the attention flow during the encoding phase.
DocuNet (Zhang et al., 2021) reformulates the task as a semantic segmentation problem. It employs a U-shaped segmentation module and an encoder module to capture global interdependencies and contextual information of entities, respectively.
PL-Marker (Ye et al., 2022) introduces a method that takes into account the interplay between spans via a neighbourhood-oriented and subject-oriented packing approach, highlighting the importance of capturing the interrelation among span pairs in relation extraction tasks.
SAIS (Xiao et al., 2022) explicitly models key information sources such as relevant contexts and entity types. It improves extraction quality and interpretability, while also boosting performance through evidence-based data augmentation and ensemble inference.
KD-DocRE (Tan et al., 2022a) proposes a semisupervised framework with three key components. Firstly, an axial attention module enhances performance in handling two-hop relations by capturing the interdependence of entity pairs. Secondly,
an adaptive focal loss solution addresses the class imbalance issue. Lastly, the framework employs knowledge distillation to improve robustness and overall effectiveness by bridging the gap between human-annotated and distantly supervised data.
DREEAM (Ma et al., 2023) is a method designed to enhance document-level relation extraction by addressing memory efficiency and annotation limitations in evidence retrieval. It employs evidence as a supervisory signal to guide attention and introduces a self-training strategy to learn evidence retrieval without requiring evidence annotations.
SAIS (Xiao et al., 2022), KD-DocRE (Tan et al., 2022a), and DREEAM (Ma et al., 2023) have been built upon the foundations of ATLOP (Zhou et al., 2021). ATLOP introduces two innovative techniques, adaptive thresholding, and localised context pooling, to address challenges in multi-label and multi-entity problems. Adaptive thresholding employs a learnable entities-dependent threshold, replacing the global threshold used in previous approaches for multi-label classification (Peng et al., 2017; Christopoulou et al., 2019; Nan et al., 2020; Wang et al., 2020). Localised context pooling leverages the attention patterns of a pretrained language model to identify and extract relevant context crucial for determining the relation between entities, using specific hand-coded pooling functions.

heading: 3 Background
text: The foundational work of ATLOP (Zhou et al., 2021) has been the basis of many State-of-the-Art (SotA) models (Xiao et al., 2022; Tan et al., 2022a; Ma et al., 2023). Given the problems with handcoded pooling functions, discussed in Section 1, we aim to provide a new baseline that can serve as the foundation for future SotA models. For this reason, we evaluate our proposed models by comparing them to this established baseline. Our goal is to demonstrate that our method not only achieves results comparable to or better than ATLOP, but also offers a novel approach which addresses its limitations. To provide a better understanding of ATLOP and its components, we present a detailed breakdown in the left portion of Figure 2, which we elaborate on in this section.

heading: 3.1 Problem Formulation
text: The document-level relation extraction task involves analysing a document D that contains a set of entities ED={ei}|ED|i=1 . The main objective is to determine the presence or absence of various relation types between all entity pairs (es, eo)s,o∈ED,s ̸=o, where the subject and object entities are denoted as es and eo, respectively. A key aspect to consider is that an entity can appear multiple times in the document, resulting in a cluster of
multiple mentionsMe={mi}|Me|i=1 for each entity e. The set of relations is defined asR∪ ∅, where ∅ represents the absence of a relation, often referred to as "no-relation". Given the clusters of mentions Mes andMeo , the task consists of a multi-label classification problem where there can be multiple relations between entities es and eo.

heading: 3.2 Previous Method: ATLOP
text: Text Encoding A special token ∗ is added at the start and end of every mention. Tokens TD={ti}|TD|i=1 are encoded via a Pretrained Language Model (PLM) as follows:
H,A = PLM(TD), (1)
where H ∈ R|TD|×d and A ∈ R|TD|×|TD| represent the token embeddings and the average attention weights of all attention heads, respectively, extracted from the last layer of the PLM.
Entity Embedding (EE) For each individual entity e with mentionsMe={mi}|Me|i=1 , an entity embedding he ∈ Rd is computed as follows:
he = log
|Me|∑
i=1
exp(Hmi), (2)
where Hmi ∈ Rd is the embedding of the special token ∗ at the starting position of mention mi. The choice of the logsumexp pooling function is based on the research conducted by Jia et al. (2019). Their study offers empirical evidence that supports the use of this pooling function over others, as it facilitates accumulating weak signals from individual mentions, thanks to its smoother characteristics.
Localised Context Embedding (LCE) ATLOP introduces the concept of localised context embedding to accommodate the variations in relevant mentions and context for different entity pairs (es, eo). Since the attention mechanism in the PLM captures the importance of each token within the context, it can be used to determine the context relevant for both entities. The importance of each token can be computed from the cross-token dependencies matrix A obtained in Equation 1. When evaluating entity es, the importance of individual tokens is determined by examining the cross-token dependencies across all mentions associated with es, denoted asMes . Initially, ATLOP collects and averages the attention Ami ∈ R|TD| at the special token ∗ preceding each mention mi ∈Mes . This
process results in as ∈ R|TD|, which represents the importance of each token concerning entity es (and analogously ao for eo). Subsequently, the importance of each token for a given entity pair (es, eo), denoted as q(s,o) ∈ R|TD|, is computed using as and ao as follows:
q(s,o) = as ◦ ao a⊤s ao , (3)
where ◦ represents the Hadamard product. Consequently, q(s,o) represents a distribution that indicates the importance of each token for both tokens in (es, eo). Finally, the localised context embedding is computed as follows:
c(s,o) = H⊤q(s,o), (4)
So c(s,o) ∈ Rd corresponds to a weighted average over all token embeddings that are important for both es and eo.
Relation Classification and Loss Function The representations hes , heo and c
(s,o) are input to a relation classifier, and the full model is fine-tuned to predict the relation labels for (es, eo). The relation classifier and its loss function are detailed in Appendix Subsection A.1.

heading: 4 Proposed Method: GADePo
text: We propose to avoid the reliance on the EE (i.e., he) and LCE (i.e., c(s,o)) heuristic aggregation functions by leveraging Transformers’ attention functions to do aggregation. Given the observation of Henderson (2020); Mohammadshahi and Henderson (2020, 2021); Henderson et al. (2023) that attention weights and graph relations are functionally equivalent, we introduce the inductive biases of EE and LCE directly into the model’s input as graph relations.
Our proposed graph-assisted declarative pooling (GADePo) method replaces the hand-coded aggregation functions EE and LCE with a declarative graph specification. By using the intrinsic graph processing capabilities of the Transformer model, the specified graph serves as an explicit high-level directive for the information aggregation process of the Transformer. By inputting the graph relations to the Transformer’s self-attention layers, GADePo enables the aggregation to be steered by domainspecific knowledge or desired outcomes, while still allowing it to be learned by the Transformer, opening up the possibility for a more tailored and customised yet data-driven relation extraction.
Our GADePo model is illustrated in the right portion of Figure 2. We address both EE and LCE with the introduction of two special tokens, <ent> (i.e., entity) and <pent> (i.e., pair entity), and two explicit graph relations of types <ent>←→ ∗ and <pent> ←→ ∗ in both directions, where ∗ represents the special token at the starting position of a specific mention. The set of relations is specified as cij ∈ C which each identify the relation label from i to j. Each of these relation labels is associated with an embedding vector of dimension d, as are the special token inputs <ent> and <pent>. These two special tokens are added to the PLM’s vocabulary of input tokens, while relation label embeddings are input to the self-attention functions for every pair of related tokens. These new embeddings represent learnable parameters that are trained during the PLM fine-tuning on the downstream tasks. As reported in Appendix Subsection A.2, GADePo adds a negligible number of extra parameters, namely only the special token inputs and the graph directional relation inputs.
Special Token <ent> To tackle the EE pooling function, we add to the input tokens TD as many <ent> special tokens as entities in the document. This way each entity e has a corresponding entity token <ent> in the input. We connect each <ent> token with its corresponding cluster of mentions Me={mi}|Me|i=1 , and vice-versa. The two graph relations we use are thus <ent> −→ ∗ and ∗ −→ <ent>, where ∗ represents the special token at the starting position of mention mi. Each <ent> token receives the same <ent> embedding, with no positional encoding, since each one collectively represents a set of mentions from different positions in the input graph. These identical inputs are only disambiguated through the connections to and from mentions expressed as the <ent> −→ ∗ and ∗ −→ <ent> graph relations. These relations tell the self-attention mechanism to use the <ent> token to aggregate information from the associated mentions, and thus the <ent> tokens have a direct correspondence to the computed he in Equation 2.
Special Token <pent> ATLOP performs information filtering by calculating via Equation 4 a localised context embedding (LCE) c(s,o) that is dependent on the cross-token attention matrix A output by the PLM. The intuition behind it is that the dependencies between different tokens are encoded as attention weights. We propose a straight-
forward adjustment of the input graph used for the EE pooling to effectively model and capture these dependencies. To address the LCE pooling function, we add to the input tokens TD as many <pent> special tokens as the number of all possible pairs of entities. Each special token <pent> thus refers to a pair of entities (es, eo). We connect each <pent> token with each mention in the two clusters of mentionsMes={mi}|Mes |i=1 andMeo={mi} |Meo | i=1 and vice-versa. Since the attention weights used in LCE are computed from these mention embeddings, we expect that they are sufficient for the Transformer to learn to find the relevant contexts. The two graph relations we use are thus <pent> −→ ∗ and ∗ −→ <pent>. Analogously to the <ent> tokens, the <pent> tokens all receive the same <pent> embedding, with no positional embeddings, and thus are only disambiguated by their different <pent> −→ ∗ and ∗ −→ <pent> graph relations. These relations tell the <pent> token to pay attention to its associated mentions, which in turn allows it to find the relevant context shared by these mentions. Thus, each <pent> token can be seen as having a direct correspondence to the computed c(s,o) in Equation 4.
All equations relative to the relation classification and the corresponding loss function reported in Appendix Subsection A.1 remain valid as we merely substitute the hand-coded computations of he and c(s,o) with the embeddings of <ent> and <pent>, respectively.
Text-Graph Encoding We follow Mohammadshahi and Henderson (2020, 2021); Henderson et al. (2023) in leveraging the intrinsic graph processing capabilities of the Transformer model by incorporating graph relations as relation embeddings input to the self-attention function. For every pair of input tokens ij, the pre-softmax attention weight eij ∈ R is computed from both the respective token embeddings xi,xj ∈ Rd, and an embeddings of the graph relation cij between the i-th and j-th tokens. However, we change the attention weight computation to:
eij = xiWQ diag(LN(cijWC)) (xjWK) ⊤ √ d ,
(5) where WQ,WK ∈ Rd×d represent the query and key matrices, respectively. cij ∈ {0, 1}|C| represents a 0/1 encoded label of the graph relation between the i-th and j-th input elements, and WC ∈ R|C|×d represents the relations’ embedding
matrix, so cijWC is the embedding of the relation between i and j. Finally, LN stands for the LayerNorm operation and diag returns a diagonal matrix.
Compared to the standard attention function, where eij = xiWQ(xjWK)⊤/ √ d, the relation embedding determines a weighting of the different dimensions. This is a novel way to condition on the relation embedding compared to the original formulation, which only models query-relation interactions (Mohammadshahi and Henderson, 2020). This change is motivated by our task requiring a more flexible formulation which models queryrelation-key interactions via a multiplicative mechanism, without requiring a full d× d matrix of bilinear parameters. This way, a key will be relevant to a query only when both agree on the relation. In preliminary experiments, we explored various methods for biasing attention and found that the formulation presented in Equation 5 produced the best results.

heading: 5 Experiments

heading: 5.1 Datasets and Models
text: Re-DocRED (Tan et al., 2022b) is a revisited version of the DocRED (Yao et al., 2019) dataset. It is built from English Wikipedia and Wikidata and contains both distantly-supervised and humanannotated documents with named entities, coreference data, and intra- and inter-sentence relations, supported by evidence. It requires analysing multiple sentences to identify entities, establish their relationships, and integrate information from the entire document. We comply with the model used by the authors and employ the RoBERTaLARGE (Zhuang et al., 2021) model in our experiments.
HacRED (Cheng et al., 2021) is a large-scale, high-quality Chinese document-level relation extraction dataset, with a special focus on practical hard cases. As the authors did not provide specific information about the model used in their study, we conducted our experiments using the Chinese BERTBASE with whole word masking model (Cui et al., 2021).
Datasets statistics Re-DocRED and HacRED exhibit notable distinctions in their statistics, as summarised in Table 2. Re-DocRED comprises a larger number of facts, entities per document, and relations compared to HacRED. This indicates a potentially richer and more extensive dataset in terms of factual information and relationship types. However, HacRED contains more documents and may present a broader range of scenarios for relation extraction, including more challenging cases, as it has been specifically created with a focus on practical hard cases.

heading: Statistic Re-DocRED HacRED

heading: 5.2 Results and Discussion
text: We follow the standard practice from prior research and report the results of our experiments on the Re-DocRED and HacRED datasets in Table 1 and Figure 4. For all datasets and models, we provide our reimplementation of the ATLOP baseline (indicated as ATLOP⋆), which achieves or surpasses pre-
viously reported results for ATLOP, and compare the proposed GADePo model against this model. We evaluate all datasets using the F1 metric. For Re-DocRED, Ign F1 (or Ignored F1) is also reported, and refers to the F1 score that excludes relational facts that are shared between the training and development/test sets. This is done to avoid potential biases in the evaluation metrics due to overlap in content between the sets, which might not reflect the model’s ability to generalise to truly unseen data. For HacRED, we adhere to the format introduced by Cheng et al. (2021) and report also the Precision (P ) and Recall (R) metrics. We comply with previous research and report the test score achieved by the best checkpoint on the development set. In Appendix Subsection A.4, we additionally present the mean and standard deviation on the development set, calculated from five training runs with distinct random seeds. We also provide in Appendix Subsection A.4, the same set of experiments conducted on the original DocRED dataset. Training details and hyperparameters are outlined in Appendix Subsection A.3.
Re-DocRED Results We evaluate our proposed GADePo method against the previous ATLOP method in two stages, first comparing the use of <ent> tokens against the use of EE pooling (he), and then comparing our full model against the full ATLOP model, including <pent> tokens and LCE pooling (c(s,o)), respectively.
Table 1 highlights the effectiveness of our proposed method. When comparing he with <ent>, we observe a noticeable improvement in both Ign F1 and F1 scores, achieving 75.55% and 76.38% respectively, compared to 75.27% and 75.92% attained by ATLOP⋆. This demonstrates the practical utility of employing the special token <ent> for information aggregation. This is illustrated in the attention weights heatmap in Figure 3. Incorporating c(s,o) and <pent> into the comparison, GADePo maintains performance parity with the significantly
enhanced ATLOP⋆, which outperformed ATLOP• from Tan et al. (2022b). The latter improvement suggests that a more refined hyperparameter search can lead to performance gains, as evidenced by the increase in F1 score from 77.56% to 78.38%. GADePo achieves an F1 score of 78.40%, affirming its competitive edge and the effectiveness of employing <pent> for aggregation.
Table 3 illustrates the results obtained with prior finetuning on the distantly supervised dataset, which contains approximately 100K documents (Yao et al., 2019). Interestingly, distant supervision appears to have a slightly negative impact on the results of both methods when incorporating c(s,o) or <pent>. However, it proves to be highly beneficial when utilising solely he or <ent> for aggregation. This suggests that although distant supervision might introduce noise into the training process, it can also provide valuable information that improves model generalisation, particularly when leveraging simpler feature representations like he and <ent>, possibly due to their robustness in capturing essential information amidst noise.
HacRED Results We observe a similar pattern to Re-DocRED, with ATLOP⋆ displaying a slight performance advantage over ATLOP⋄ from Cheng et al. (2021) (Table 1). On this dataset, GADePo shows a significantly improved performance, primarily driven by a substantial increase in Recall (R), indicating that the GADePo model is more effective at identifying relevant instances. As already reported for the Re-DocRED dataset, the performance boost after the inclusion of c(s,o) and
<pent> into ATLOP⋆ and GADePo, respectively, highlight the significant contributions of these features. GADePo outperforms ATLOP⋆ with an F1 score of 78.65% compared to 77.59%. This larger improvement on HacRED suggests that GADePo is better at handling challenging cases, which is not surprising given its greater flexibility over the fixed pooling functions of ATLOP.
Data Ablation To evaluate the models’ sensitivity to dataset size, the performance evaluation depicted in Figure 4 compares ATLOP⋆ (he ; c(s,o)) and GADePo (<ent> ; <pent>) on the development set, considering different levels of training data availability on the Re-DocRED and HacRED datasets. Accuracies generally converge as the dataset sizes increase, but on the challenging cases of HacRED, GADePo maintains a substantial advantage across the full range. On Re-DocRED, GADePo catches up with and slightly outperforms ATLOP⋆ as data size increases. This lower performance on smaller datasets is presumably because GADePo must learn how to exploit the graph relations to the special tokens <ent> and <pent> and pool information through them, whereas for ATLOP this pooling is hand-coded. On the ReDocRED dataset, ATLOP⋆ appears to have relatively consistent variance, while GADePo exhibits higher variance in the smaller training sets, while on the HacRED dataset, GADePo is significantly more stable for smaller datasets.
The data ablation analysis shows that the performance of hand-coded pooling functions can be dataset-specific, which restricts their adaptability. In contrast, GADePo consistently outperforms its
hand-coded counterparts on larger datasets, and matches them on all but some smaller datasets, presumably due to its flexibility. This pattern suggests that GADePo has a greater potential for optimisation, particularly on larger datasets. This is supported by GADePo’s better performance on HacRED, which is both larger and designed to be more challenging than Re-DocRED.

heading: 6 Conclusion
text: In this paper we proposed a novel approach to document-level relation extraction, challenging the conventional reliance on hand-coded pooling functions for information aggregation. Our method leverages the power of Transformer models by incorporating explicit graph relations as instructions for information aggregation. By combining graph processing with text-based encoding, we introduced the graph-assisted declarative pooling (GADePo) specification, which allows for more flexible and customisable specification of pooling strategies which are still learned from data.
We conducted evaluations using diverse datasets and models commonly employed in documentlevel relation extraction tasks. The results of our experiments demonstrated that our approach achieves promising performance that is comparable to or better than that of hand-coded pooling functions. This suggests that our method can serve as a viable basis for other relation extraction methods, providing a more adaptable and tailored approach. In particular, recent methods have improved performance by exploiting information about evidence, which can naturally be incorporated in our graph-based approach.

heading: Limitations
text: While the proposed GADePo model offers a promising and innovative approach to relation extraction, there are issues which the current study does not address. According to the data in Appendix Table 2, the average number of entities per document across datasets is approximately 15. This means that, on average, there will be an additional 15 <ent> tokens and 105 <pent> tokens. Given that the maximum allowable input length for the models is 512 tokens, the inclusion of these extra tokens results in roughly a 3% and 20% increase in the overall input length for <ent> and <pent>, respectively. It’s evident that the majority of the increase in input length is due to the quadratic number of <pent> special tokens, but we believe that an appropriate pruning strategy could easily reduce this number to linear in the number of entities without degrading accuracy. One such pruning strategy could involve an <ent>-only model with a binary classifier which is trained to predict pairs of related entities. This model could then be used to prune the set of candidate entity pairs for the final relation classification, with <pent> tokens being instantiated only for these candidate pairs. We have chosen to leave this approach as a potential avenue for future work, opting instead to focus on demonstrating the promise of the current simpler formulation.

heading: Ethics Statement
text: We do not anticipate any ethical concerns related to our work, as it primarily presents an alternative approach to a previously proposed method. Our main contribution lies in introducing a novel methodology for relation extraction. In our experiments, we use the same datasets and pretrained models as previous research, all of which are publicly available. However, it is important to acknowledge that these datasets and models may still require further examination for potential fairness issues and the knowledge they encapsulate.

heading: Acknowledgements
text: We extend our special gratitude to the Swiss National Science Foundation (SNSF) and Research Foundation – Flanders (FWO) for funding this work under grants 200021E_189458 and G094020N.

heading: A Appendix

heading: A.1 ATLOP: Relation Classification and Loss Function
text: Relation Classification To predict the relation between the subject entity es and object entity eo,
ATLOP first generates context-aware subject and object representations as follows:
zs = tanh(Ws[hes ; c (s,o)] + bs) (6)
zo = tanh(Wo[heo ; c (s,o)] + bo), (7)
where zs, zo ∈ Rd, [·; ·] represents the concatenation of two vectors, and Ws,Wo ∈ Rd×2d together with bs, bo ∈ Rd are trainable parameters. Then, the entity pair representation is computed as:
x(s,o) = zs ⊗ zo, (8)
where x(s,o) ∈ Rd2 and⊗ stands for the vectorised Kronecker product. Finally, relation scores are computed as:
y(s,o) = Wrx (s,o) + br, (9)
where y(s,o) ∈ R|R|, with Wr ∈ R|R|×d2 and br ∈ R|R| representing learnable parameters. The probability of relation r ∈ R between the subject and object entities is computed as follows:
P (r|s, o) = σ(y(s,o)), (10)
where σ is the sigmoid function. To reduce the number of parameters in the classifier, a grouped function is used, which splits the embedding dimensions into k equal-sized groups and applies the function within the groups as follows:
zs = [z 1 s ; . . . ; z k s ] (11)
zo = [z 1 o ; . . . ; z k o ] (12)
x(s,o) = [x(s,o) 1 ; . . . ;x(s,o) k ] (13)
y(s,o) = k∑
i=1
W irx (s,o)i + br, (14)
where zis, z i o ∈ Rd/k, x(s,o) i ∈ Rd2/k, and W ir ∈ R|R|×d2/k. This way, the number of parameters can be reduced from d2 to d2/k.
Loss Function ATLOP introduces the adaptive thresholding loss concept. This approach involves training a model to learn a hypothetical threshold class TH , which dynamically adjusts for each relation class r ∈ R. During training, for each entity pair (es, eo), the loss enforces the model to generate scores above TH for positive relation classes
RP and scores below TH for negative relation classesRN . The loss is computed as follows:
L =− ∑
s ̸=o
∑
r∈RP
exp(y (s,o) r )∑
r ′∈RP∪{TH} exp(y (s,o) r′ )
− exp(y (s,o) TH )∑
r′∈RN∪{TH} exp(y (s,o) r′ )
(15)

heading: A.2 GADePo’s Extra Parameters
text: GADePo introduces few extra parameters to the PLM. The amount of parameters is reported in Table 4.
The introduction of these parameters results in only a minimal increase in the overall parameter count of the models. Specifically, GADePo’s augmentation amounts to a mere 0.036% increase over the BERTBASE model. In contrast, even a slight increase of just one unit in BERTBASE’s hidden dimensions would result in a 0.139% parameter increase, which is roughly four times greater than the augmentation introduced by GADePo. Given that such a small change is incompatible with other architectural constraints, such as the number of heads, it is implausible that this minimal augmentation would solely account for the observed performance gains.
This indicates that the performance improvements are largely due to the effective inductive bias introduced by GADePo, rather than the increase in parameter count. The same rationale applies to the results observed with RoBERTaLARGE.

heading: A.3 Training Details
text: We generally comply with the hyperparameters of ATLOP and set the output dimension in Equation 6 and Equation 7 to 768. We also set the block size in Equation 11 and Equation 12 to 64, i.e., k = 12.
In all our experiments we perform early stopping on the development set based on the Ign F1 + F1
score for DocRED and Re-DocRED, and F1 score for HacRED. The five different seeds we use are {73, 21, 37, 7, 3}.
We use RAdam (Liu et al., 2020) as our optimiser. On the RoBERTaLARGE based models we train for 8 epochs and set the learning rates to 3e−5 and 1e−4 for the PLM parameters and the new additional parameters, respectively. On the BERTBASE based models we train for 10 epochs and set the learning rates to 1e−5 and 1e−4 for the PLM parameters and the new additional parameters, respectively. We use a cosine learning rate decay throughout the training process.
In all our experiments the batch size is set to 4 for ATLOP and 2 for GADePo, with gradient accumulation set to 1 and 2, for ATLOP and GADePo, respectively. We clip the gradients to a max norm of 1.0. All models are trained with mixed precision.
We run our experiments on two types of GPUs, namely the NVIDIA V100 32GB for the RoBERTaLARGE based models and NVIDIA RTX 3090 24GB for the BERTBASE based models, respectively.
We use PyTorch (Paszke et al., 2019), Lightning (Falcon and The PyTorch Lightning team, 2019), and Hugging Face’s Transformers (Wolf et al., 2020) libraries to develop our models.

heading: A.4 Additional Results
text: Re-DocRED and HacRED Table 5 and Table 6 present additional results for Re-DocRED and HacRED, respectively. In addition to the results outlined in Section 5, these tables include the mean and standard deviation on the development set, calculated from five training runs with distinct random seeds, as reported in Appendix Subsection A.3.
DocRED results The DocRED (Yao et al., 2019) dataset consists of 56, 354 facts, 96 relations, 5, 053 documents, and 26.2 average number of entities per document. In line with the approach taken for Re-DocRED and HacRED, Table 7 and Figure 5 illustrate the results for DocRED.

heading: Future Work
text: We have chosen to leave this approach as a potential avenue for future work, opting instead to focus on demonstrating the promise of the current simpler formulation