Venue: NAACL 2025

# Label Drop for Multi-Aspect Relation Modeling in Universal Information Extraction

Lu Yang\({}^{1}\), Jiajia Li\({}^{2,3}\), En Ci\({}^{1}\), Lefei Zhang\({}^{1}\), Zuchao Li\({}^{1,*}\), Ping Wang\({}^{2,3}\)

\({}^{1}\)School of Computer Science, Wuhan University, Wuhan, China

\({}^{2}\)School of Information Management, Wuhan University, Wuhan, China

\({}^{3}\)Key Laboratory of Archival Intelligent Development and Service, NAAC

{yang_lu, cantata, cien.cs}@whu.edu.cn

{zhanglefei, zcli-charlie, wangping}@whu.edu.cn

###### Abstract

Universal Information Extraction (UIE) has garnered significant attention due to its ability to address model explosion problems effectively. Extractive UIE can achieve strong performance using a relatively small model, making it widely adopted. Extractive UIEs generally rely on task instructions for different tasks, including single-target instructions and multiple-target instructions. Single-target instruction UIE enables the extraction of only one type of relation at a time, limiting its ability to model correlations between relations and thus restricting its capability to extract complex relations. While multiple-target instruction UIE allows for the extraction of multiple relations simultaneously, the inclusion of irrelevant relations introduces decision complexity and impacts extraction accuracy. Therefore, for multi-relation extraction, we propose LDNet, which incorporates multi-aspect relation modeling and a label drop mechanism. By assigning different relations to different levels for understanding and decision-making, we reduce decision confusion. Additionally, the label drop mechanism effectively mitigates the impact of irrelevant relations. Experiments show that LDNet outperforms or achieves competitive performance with state-of-the-art systems on 9 tasks, 33 datasets, in both single-modal and multimodal, few-shot and zero-shot settings.1

Footnote 1: [https://github.com/Lu-Yang666/LDNet](https://github.com/Lu-Yang666/LDNet)

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote †: leftmargin=0pt]

+
Footnote†: footnotetext: [leftmargin=0pt]

+
Footnote †: leftmargin=0pt]


the extraction of multiple relations simultaneously. This approach aims to model correlations between relations and improve extraction efficiency. However, the incorporation of irrelevant relations introduces decision complexity and can have ramifications on extraction accuracy.

Hence, to address the challenges in multi-relation extraction, we propose LDNet, a novel approach that leverages multi-aspect relation modeling and a label drop mechanism. In LDNet, we assign different relations to different levels for understanding and decision-making. This approach allows the model to capture the unique characteristics and nuances of each relation separately. By organizing relations into distinct levels, LDNet reduces decision confusion, enabling more accurate and reliable extraction results. Additionally, LDNet incorporates a label drop mechanism to address the impact of irrelevant relations. During the extraction process, LDNet selectively drops irrelevant labels, focusing on the most relevant relations for extraction. This mechanism helps mitigate the interference caused by irrelevant relations, ensuring that the model can concentrate its attention and resources on extracting the necessary and meaningful information. By filtering out noise and irrelevant signals, LDNet enhances the overall extraction performance and reduces the potential for false positives.

To assess the effectiveness of LDNet, we conduct extensive experiments on a diverse range of IE tasks and benchmark datasets, both single-modal and multi-modal. The evaluation covers few-shot and zero-shot settings to examine the generalization capability of LDNet. The results demonstrate that LDNet outperforms or achieves competitive performance compared to previous state-of-the-art systems across 9 tasks and 33 datasets.

**Our Contribution** 1) We propose LDNet, a novel approach that leverages multi-aspect relation modeling and a label drop mechanism. 2) We employ model transfer learning, a valuable strategy for further enhancing model performance across various datasets. 3) We conduct experiments on 33 datasets across 9 tasks, in both single-modal and multi-modal, few-shot and zero-shot settings, and the results demonstrate the superiority of LDNet.

## 2 Related Work

Generative UIETANL Paolini et al. (2021) sees IE tasks as a sequence-to-sequence problem and utilizes T5 as the generative model. UIE Lu et al. (2022) also uses T5 as the backbone. In addition, UIE designs Structured Extraction Language (SEL) that can represent diversified IE tasks, thereby enabling it to perform on a wider range of IE tasks. InstructUIE Wang et al. (2023) further incorporates the idea of instruction-tuning and utilizes FlanT5-11B Chung et al. (2022) for IE tasks. DeepStruct Wang et al. (2022) and GenIE Josifoski et al. (2022) both formulate the generated sequence as subject-relation-object triplets, with DeepStruct having a larger model size (10B). LasUIE Fei et al. (2022) proposes a novel structure-aware generative language model to unleash the power of syntactic knowledge. FSUIE Peng et al. (2023) introduces fuzzy span loss and fuzzy span attention to reduce over-reliance on span boundaries. GOLLIe Sainz et al. (2024) improves zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. TMR Zheng et al. (2023) addresses text-image misalignment by introducing a back-translation method using diffusion-based generative models. KnowCoder Li et al. (2024) introduces a code-style schema representation method. While the above generative UIE approaches offer a powerful solution for diversified IE tasks, they do not possess any notable advantages when it comes to efficiency.

Extractive UIEDyGIE++ Wadden et al. (2019) utilizes a dynamic span graph to model long-range relations, and with graph propagation, the model can disambiguate challenging entity mentions. UniEX Ping et al. (2023) converts IE tasks into a token-pair problem, develops a traffic attention mechanism to integrate heterogeneous factors, and obtains the extraction target via a scoring matrix. These single-target extractive UIE approaches can achieve strong performance using a relatively small model; however, they lack the ability to model correlations between relations, thus limiting their capability to extract complex relations.

OneIE Lin et al. (2020) also uses a span graph, but unlike DyGIE++, it incorporates global features and adopts a CRF-based tagger to remove the constraint on the length of extracted mentions. UMGF Zhang et al. (2021) adopts a unified intra-modal and inter-modal graph fusion method to represent visual and textual features within the same embedding space. HVPNeT Chen et al. (2022) designs pyramidal features for images, employing 

visual representations as insertable visual prefixes to guide error-insensitive predictive decisions of textual representations. MetaRetriever (Yu et al., 2023) retrieves task-specific knowledge from pre-trained language models to enhance performance. Mirror Zhu et al. (2023) transforms multiple tasks into a multi-span cyclic graph and predicts relations by verifying whether a cycle exists between slots in a tuple. While these multiple-target extractive UIE approaches take interactions between relations into account, they also include irrelevant relations, which leads to decision complexity and inaccuracy.

It is worth noting that the existing UIE models have basically only conducted experiments on single-modal or multi-modal IE tasks, and have not handled single-modal and multi-modal IE tasks simultaneously like LDNet.

## 3 Methodology

LDNet's overall framework is built upon a pretrained language model and an image backbone, consisting of a multi-aspect relation modeling component and a label drop mechanism, as shown in Figure 1.

We formulate IE tasks as a multi-aspect span-based relation extraction problem. Specifically, we consider three kinds of relations among IE tasks: TA relation (trigger-to-argument relation), A2A relation (argument-to-argument relation), and AS relation (argument-span relation). The TA relation signifies the association of the trigger word with the identified span. The A2A relation describes the connection between two related spans, representing the semantic or contextual relation between the identified spans. The AS relation describes the connection within a span, enabling LDNet to analyze the internal structure and coherence within the span itself. As shown in Figure 1, the AS relation is formed between "the" and "cat", the trigger word "chased" connects to "the" in the span "the cat" and "mouse" in the span "the mouse" through the TA relation, and "cat" is linked to "the" in the span "the mouse" through the A2A relation.

### Multi-aspect Relation Modeling

LDNet regularizes text input format into three components: instruction, schema labels, and text. Given an input sequence \({\bf x}=[x_{1},x_{2},\ldots,x_{|x|}]\), LDNet computes the text representation \({\bf H}=[h_{1},h_{2},\ldots,h_{|x|}]\in\mathbb{R}^{|x|\times d_{n}}\) as follows:

\[{\bf H} =PLM([x_{1},x_{2},\ldots,x_{|x|}]) \tag{1}\] \[=[h_{1},h_{2},\ldots,h_{|x|}] \tag{2}\]

where \(PLM(\cdot)\) is a pretrained language model.

To inject image information, given an image \(I\), LDNet initially resizes it to \(224\times 224\), then divides it into \(n_{p}\) patches according to the patch size specified by the image backbone, and subsequently derives image feature representation

Figure 1: The overview framework of LDNet. LDNet constructs a unified input format, which combines instruction, schema labels, and text. The representation obtained from the PLM is fused with image representation obtained with the image backbone. The multi-modal representation is fed into the multi-aspect relation modeling component to produce probability matrices for TA, A2A, and AS relations, respectively. These matrices are then subjected to label drop to mask out non-existent relations. Finally, the probability matrices are fed into the decoding process to generate target structures.

 

using the image backbone:

\[{\cal V}=VisionTransformer(I) \tag{3}\]

LDNet then employs cross-attention, where the image feature representation functions as the query and the text representation serves as both the key and value. The output of the attention mechanism is then subjected to the hyperbolic tangent activation function, followed by a summation operation. Finally, LDNet changes the sequence length of the resulting image feature representation, yielding the final image representation \({\bf V}\in\mathbb{R}^{|x|\times d_{h}}\):

\[Q={\rm FFNN}^{\rm I}_{q}(MLP({\cal V})),~{}Q\in\mathbb{R}^{n_{p}\times d_{h}} \tag{4}\]

\[K={\rm FFNN}^{\rm I}_{k}({\bf H}),~{}V={\rm FFNN}^{\rm I}_{v}({\bf H}) \tag{5}\]

\[{\cal V}^{\prime}=\sum_{i=1}^{n_{p}}Tanh\left(Softmax\left(\frac{QK^{\rm T}}{ \sqrt{d_{h}}}\right)V\right) \tag{6}\]

\[{\bf V}=Tanh({\rm FFNN}_{\rm I}({\cal V}^{\prime})) \tag{7}\]

where \(MLP\left(\cdot\right)\) represents a three-layer multilayer perceptron, \({\rm FFNN}^{\rm I}_{0/k/v}\in\mathbb{R}^{d_{h}\times d_{h}}\) represents feed-forward network for generating query/key/value, and \({\rm FFNN}_{\rm I}\in\mathbb{R}^{1\times|x|}\) represents the feed-forward network changing the sequence length of the resulting image feature representation.

LDNet modulates the fusion of image representation and text representation via a hyperparameter \(\alpha\in[0,1]\), and the fused image-text representation \({\bf M}\) is expressed as:

\[{\bf M}={\bf H}+\alpha\cdot{\bf V} \tag{8}\]

\(\alpha\) is set to 0 when only doing single-modal IE tasks.

After obtaining the multi-modal representation \({\bf M}=[m_{1},m_{2},\ldots,m_{|x|}]\), LDNet utilizes Rotary Position Embedding (RoPE) Su et al. (2022) to achieve relative position encoding via combining the attention computation with absolute position encoding. The queries and keys for different relations are calculated as follows:

\[q^{r}_{i}={\rm FFNN}^{r}_{q}(m_{i}),~{}k^{r}_{j}={\rm FFNN}^{r}_{k}(m_{j}) \tag{9}\]

where \(r\in\{TA,A2A,AS\}\), \({\rm FFNN}^{r}_{q/k}\in\mathbb{R}^{d_{h}\times d_{i}}\) are feed-forward layers for different relations, and \(q^{r}_{i}\) and \(k^{r}_{j}\) are the \(i\)-\(th\) query and the \(j\)-\(th\) key for different relations.

Afterwards, \(q^{r}_{i}\) and \(k^{r}_{j}\) are each left-multiplied by the transformation matrices \(R_{i}\) and \(R_{j}\) used in RoPE respectively. The dot product of \(R_{i}\) and \(R_{j}\) satisfies \(R^{T}_{i}R_{j}=R_{j-i}\), thus incorporating relative position information. The probability \(s^{r}_{ij}\) of the relation \(r\) existing between the span from \(i\) to \(j\) is the scaled dot product of the transformed \(q^{r}_{i}\) and \(k^{r}_{j}\):

\[s^{r}_{ij}=\frac{(R_{i}q^{r}_{i})^{T}(R_{j}k^{r}_{j})}{\sqrt{d_{i}}}=\frac{{ q^{r}_{i}}^{T}R_{j-i}k^{r}_{j}}{\sqrt{d_{i}}} \tag{10}\]

By parallelly computing the scaled dot product over all token pairs of different relations separately, we can obtain three probability matrices:

\[S^{r}=\begin{bmatrix}s^{r}_{11}&s^{r}_{12}&\dots&s^{r}_{1|x|}\\ s^{r}_{21}&s^{r}_{22}&\dots&s^{r}_{2|x|}\\ \vdots&\vdots&\ddots&\vdots\\ s^{r}_{|x|1}&s^{r}_{|x|2}&\cdots&s^{r}_{|x||x|}\end{bmatrix} \tag{11}\]

where \(s^{r}_{ij}\) is the probability of the specific relation \(r\in\{TA,A2A,AS\}\) existing between the token pair \(\langle x_{i},x_{j}\rangle\).

During training, LDNet utilizes multi-label categorical cross-entropy loss as the loss function for multi-aspect relation modeling:

\[l^{r}_{MR,neg}=log\left(1+\sum_{\Omega_{neg}}e^{s^{r}_{ij}}\right) \tag{12}\]

\[l^{r}_{MR,pos}=log\left(1+\sum_{\Omega_{pos}}e^{-s^{r}_{ij}}\right) \tag{13}\]

\[L_{MR}=\sum_{r\in\{TA,A2A,AS\}}\left(l^{r}_{MR,neg}+l^{r}_{MR,pos}\right) \tag{14}\]

where \(\Omega_{neg}\) and \(\Omega_{pos}\) are the sets of negative and positive samples, respectively. Graph labels \(G^{r}\in\mathbb{R}^{|x|\times|x|}\) are used to distinguish between negative and positive samples. Negative samples consist of position pairs where \(G^{r}_{ij}=0\), while positive samples are pairs where \(G^{r}_{ij}=1\). \(G^{r}_{ij}\) represents the label of the token pair \(\langle x_{i},x_{j}\rangle\) for relation \(r\).

### Label Drop

To prioritize the relational token pairs, we employ label drop to filter out token pairs that are unlikely to have relations.

Specifically, we first design a label vector \({\bf l}^{r}=[l^{r}_{1},l^{r}_{2},\dots,l^{r}_{|x|}]\in\mathbb{R}^{1\times|x|}\) for each relation as the standard. We set the values of the elements in 

whose indices fall within the gold spans and their corresponding schema labels to 1, and the rest to 0.

LDNet transforms the representation \({\bf M}\) into the predicted matrix \(\hat{L}^{r}\in\mathbb{R}^{1\times|x|\times 1}\) via linear activation and Sigmoid function:

\[\hat{L}^{r}=Sigmoid\left({\rm FFNN}^{r}\left({\bf M}\right)\right) \tag{15}\]

where \({\rm FFNN}^{r}\in\mathbb{R}^{d_{h}\times 1}\) is the feed-forward network for different relations.

Later, LDNet squeezes the matrix \(\hat{L}^{r}\) into \(\hat{l}^{r}\in\mathbb{R}^{1\times|x|}\) and multiplies \(\hat{l}^{r}\) with every row vector in the corresponding probability matrix \(S^{r}\) computed in Section 3.1, respectively, to obtain the final probability matrix \(P^{r}\), which is later used for decoding:

\[p_{i\cdot}^{r}=\hat{l}^{r}\otimes s_{i\cdot}^{r} \tag{16}\]

where \(p_{i\cdot}^{r}\) represents the \(i\)-\(th\) row vector of \(P^{r}\), \(s_{i\cdot}^{r}\) represents the \(i\)-\(th\) row vector of \(S^{r}\), and \(\otimes\) represents dot product. The detailed logic of label drop mechanism can be seen in Appendix A.4.

LDNet calculates the binary cross-entropy loss between \(\hat{l}^{r}\) and I' to make the predicted vector \(\hat{l}^{r}\) approach the label vector I" of the same relation during training:

\[l_{LD}^{r}=-\frac{1}{n}\sum_{i}^{|x|}\left({\textbf{I}}_{i}^{r}\log(\hat{l}_{ i}^{r})+(1-{\textbf{I}}_{i}^{r})\log(1-\hat{l}_{i}^{r})\right) \tag{17}\]

\[L_{LD}=\sum_{r\in\{TA,A2A,AS\}}l_{LD}^{r} \tag{18}\]

After label drop, LDNet utilizes the three final probability matrices for relation decoding. During the process of relation decoding, LDNet extracts relation between token pair whose final probability is larger than the threshold of 0.5 and identifies potential relation structures. If LDNet detects a closed relation loop as shown in Figure 1, it adds the extracted span to the predicted answer.

### Model Transfer Learning

To further boost LDNet's performance, we propose a model transfer learning approach. We select the best-performance models fine-tuned on each dataset as the teacher models, and the generated probability distributions, namely \(P^{r}\) of all data entries, from these teacher models are used as soft labels when fine-tuning the corresponding pre-trained student models. Mean Squared Error (MSE) loss is employed to guide LDNet in reducing the discrepancy between the distributions of the student model and the teacher model:

\[L_{MT}=\sum_{r\in\{TA,A2A,AS\}}\frac{1}{|x|^{2}}\sum_{i=1}^{|x|}\sum_{j=1}^{| x|}\big{(}p_{ij}^{r}-\hat{p}_{ij}^{r}\big{)}^{2} \tag{19}\]

where \(p_{ij}^{r}\) represents the \(ij\)-\(th\) element of the \(P^{r}\) generated by the teacher model, and \(\hat{p}_{ij}^{r}\) represents the \(ij\)-\(th\) element of the \(\hat{P}^{r}\) generated by the student model. The detailed algorithm is in Algorithm 1. Thus, the complete objective for LDNet model training can be represented as follows:

\[L=L_{MR}+L_{LD}+L_{MT} \tag{20}\]

```
1:for\(i\) in range (0, epochs) do
2: (iterate over fine-tuning epochs)
3:for\(j\) in range (0, steps) do
4: Obtain student model distributions \(\mathcal{D_{S}}\).
5: Set loss \(L\gets L_{MR}+L_{LD}\).
6:for\(d_{s}\in\mathcal{D_{S}}\)do
7:if\(d_{s}\) finds the corresponding \(d_{t}\in\mathcal{D_{T}}\)then
8:\(L\gets L+MSE(d_{s},d_{t})\).
9:endif
10:endfor
11:\(\Theta=\theta-\gamma\frac{\hat{m}_{i\cdot}}{\sqrt{\hat{v}_{i}}+\epsilon}\) (use the AdamW optimizer to update parameters)
12:endfor
13:endfor
14:return\(\Theta\)
```

**Algorithm 1** Model Transfer Learning

## 4 Experiments

### Experiment Setup

We use DeBERTa-v3-large He et al. (2021) as the PLM, ViT Dosovitskiy et al. (2021) as the image backbone, and AdamW Loshchilov and Hutter (2019) as the optimizer. We conduct experiments on ACE04 Mitchell et al. (2005), ACE05 Walker et al. (2006), CoNLL03 Tjong Kim Sang and De Meulder (2003), CoNLL04 Roth and Yih (2004), NYT Riedel et al. (2010), SciERC Luan et al. (2018), CASIE Satyanich et al. (2020), 14-res and 14-lap Pontiki et al. (2014), 15-res Pontiki et al. (2015), 16-res Pontiki et al. (2016), Twitter2015 Liu et al. (2018), Twitter2017 Zhang et al. ( 

2018b), and MNRE (Zheng et al., 2021) datasets. We used the F1 score as the metric unless otherwise specified.

We compare LDNet with TANL (Paolini et al., 2021), DeepStruct (Wang et al., 2022a), UIE (Lu et al., 2022), InstructUIE (Wang et al., 2023), USM (Lou et al., 2023), Mirror (Zhu et al., 2023), FESUIE (Peng et al., 2023b), UniEX (Ping et al., 2023), MetaRetriever (Yu et al., 2023), and GoLLIE (Sainz et al., 2024) in single-modal IE tasks, and with Mirror in Multi-modal Information Extraction (MIE) tasks.

In different pre-training and fine-tuning strategies, we specifically compare LDNet with Mirror. The pre-training of LDNet is before fine-tuning on

\begin{table}
\begin{tabular}{c|r|r r r r r r r r r r} \hline Task & Datasets & PANL & UIE & DeepStrect & InstructUIE & USM & Mirror & FESUIE & UniEX & MetaRetriever & GoLILE & **LDNet** \\ \hline  & ACE04 & - & 86.89 & - & 87.62 & 87.66 & 86.16 & 87.12 & 86.10 & - & **88.69** \\ NER & ACE05 & 84.90 & 85.78 & 86.90 & 86.66 & 87.14 & 86.72 & 86.91 & 87.02 & 84.01 & **88.10** & 87.79 \\  & CoNLL03 & 91.70 & 92.99 & 93.00 & 92.94 & 93.16 & 92.97 & - & 92.65 & 92.38 & 92.80 & **93.44** \\ \hline  & ACE05 & 63.70 & 66.06 & 66.80 & - & 67.88 & 69.02 & **74.16** & 66.06 & 64.37 & 63.60 & 69.14 \\ RE & CoNLL04 & 71.40 & 75.00 & 78.30 & 78.48 & 78.84 & 75.22 & - & 73.40 & 73.66 & - & **80.79** \\  & NYT & - & 93.54 & 93.30 & 90.47 & 94.07 & 94.25 & - & - & - & - & **94.96** \\  & SciERC & - & 36.53 & - & 45.15 & 37.36 & 04.50 & - & 38.00 & 35.77 & - & **46.85** \\ \hline  & ACE05-Tgg & 68.40 & 73.36 & 69.80 & 77.13 & 72.41 & 74.44 & - & 74.08 & 72.38 & 72.20 & **83.56** \\ EE & ACE05-Arg & 47.60 & 54.79 & 56.20 & **72.94** & 55.83 & 57.87 & - & 53.92 & 52.62 & 66.00 & 61.14 \\  & CASINE-Reg & - & 69.33 & - & 67.80 & 71.73 & 73.09 & - & 71.46 & 69.76 & 59.30 & **73.36** \\  & CASINE-Arg & - & 61.30 & - & 63.53 & 63.26 & 61.27 & - & 62.91 & 60.37 & 50.00 & **63.88** \\ \hline  & 14-res & - & 74.52 & - & - & 77.26 & 76.05 & 74.17 & 74.77 & 73.41 & - & **79.17** \\ ABSA & 14-lap & - & 63.88 & - & - & 65.51 & 64.08 & 65.56 & 65.23 & 62.83 & - & **69.00** \\  & 15-res & - & 67.15 & - & - & 69.86 & 67.41 & 70.63 & 65.88 & 65.85 & - & **71.18** \\  & 16-res & - & 75.07 & - & - & 78.25 & 77.46 & 75.80 & 76.02 & 73.55 & - & **78.85** \\ \hline  & Twitter-2015 & - & - & - & - & - & 73.08* & - & - & - & - & **76.17** \\ MIE & Twitter-2017 & - & - & - & - & - & 83.91* & - & - & - & - & **87.57** \\  & MNRE & - & - & - & - & - & 70.58* & - & - & - & - & **75.79** \\ \hline \end{tabular}
\end{table}
Table 1: Results on 16 IE benchmarks. -Tgg and -Arg refer to trigger F1 score and argument F1 score, respectively. Mirror does not test on multi-modal IE datasets. The results marked with * are the performance we obtain using Mirror’s model and training checkpoint.

\begin{table}
\begin{tabular}{l|r r r r} \hline  & P & R & F1 \\ \hline _Discontinuous NER: CADEC_ & & & \\ BART-NER & 70.08 & 71.21 & 70.64 \\ W2NER & 74.09 & 72.35 & 73.21 \\ Mirror\({}_{w}\)/PT A host. & 74.83 & 65.45 & 69.83 \\ Mirror\({}_{w}\)/PT F flat. & 68.80 & 68.38 & 68.59 \\ LDPNe\({}_{w}\)/PT A host. & **54.82** & 71.16 & **77.39** \\ LDNets\({}_{w}\)/PT K host. & 71.89 & 68.18 & 69.98 \\ \hline _N-ary Tuples: HyperRED_ & & & \\ CubeRE & 66.39 & 67.12 & 66.75 \\ Mirror\({}_{w}\)/PT A host. & 71.29 & 62.46 & 66.58 \\ Mirror\({}_{w}\)/PT F flat. & 75.41 & 61.14 & 67.53 \\ LDPCNet\({}_{w}\)/PT F dark. & 69.39 & 66.56 & **67.95** \\ LDNet\({}_{w}\)/PT F dark. & 68.40 & 64.31 & 66.29 \\ \hline \end{tabular}
\end{table}
Table 2: Results of LDNet compared with Mirror. PT stands for pre-training, and Inst. represents the task instruction. \(\mathrm{LDNet}_{MT^{-}}\) denotes LDNet without model transfer learning.

\begin{table}
\begin{tabular}{c|c c c} \hline  & P & R & F1 \\ \hline _Discontinuous NER: CADEC_ & & & \\ BART-NER & 70.08 & 71.21 & 70.64 \\ W2NER & 74.09 & 72.35 & 73.21 \\ Mirror\({}_{w}\)/PT A host. & 74.83 & 65.45 & 69.83 \\ Mirror\({}_{w}\)/PT F flat. & 68.80 & 68.38 & 68.59 \\ LDNet\({}_{w}\)/PT A host. & **54.82** & 71.16 & **77.39** \\ LDNets\({}_{w}\)/PT A host. & 71.89 & 68.18 & 69.98 \\ \hline _N-ary Tuples: HyperRED_ & & & \\ CuoReB & 66.39 & 67.12 & 66.75 \\ Mirror\({}_{w}\)/PT F flat. & 71.29 & 62.46 & 66.58 \\ Mirror\({}_{w}\)/PT F flat. & 75.41 & 61.14 & 67.53 \\ LDNet\({}_{w}\)/PT F dark. & 69.39 & 66.56 & **67.95** \\ LDNet\({}_{w}\)/PT F dark. & 68.40 & 64.31 & 66.29 \\ \hline \end{tabular}
\end{table}
Table 3: Results on multi-span and n-ary information extraction tasks.

 

downstream datasets, the pre-training datasets and hyperparameters are in Appendix C.1. 'With and without pre-training (PT)' refers to whether LDNet underwent pre-training on a pre-training dataset before fine-tuning on the downstream dataset. 'With and without task instructions (Inst.)' indicates whether the instruction part of LDNet's input is an empty string. In the configuration 'w/ PT w/o Inst.', the instruction part is an empty string, while in 'w/ PT w/ Inst.', the instruction part is not empty and resembles a string like 'Please determine the two entities mentioned in the text and specify the nature of their relationship.' The best results we obtained in the main results of Table 1 are from the four configurations: 'w/ PT w/ Inst.', 'w/ PT w/o Inst.', 'w/o PT w/ Inst.', and 'w/o PT w/o Inst.'

BART-NER Lewis et al. (2020), W2NER Li et al. (2022), and Mirror are chosen as the baseline models for the multi-span discontinuous NER task. In the hyper RE task, we select CubeRE Chia et al. (2022) and Mirror as the baseline models. As for the MRC tasks, We compare LDNet to BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), DeBERTa-v3 He et al. (2021), and Mirror.

### Main Results

LDNet main results over 16 IE and MIE datasets are shown in Table 1 and Table 2. We can observe that:

1) By adopting multi-aspect relation modeling and applying label drop on separate probability matrix, LDNet offers an effective methodology for IE and MIE. LDNet achieves state-of-the-art performance across almost all datasets and tasks. Although LDNet slightly underperforms FSUIE on ACE05-RE and underperforms GoLLIE on ACE05-NER, it surpasses them in other tasks and covers a broader range of tasks, including MRC, classification, discontinuous NER, and hyper-RE. MRC and multi-modal results are in Appendix C.5.

2) Despite having a relatively small model scale, LDNet consistently delivers superior results across almost all IE tasks. LDNet outperforms DeepStruct (10B) in all tasks. The comparison of pre-trained language model parameter scales is included in Appendix C.4 and more comparisons with models of different scales can be found in the Appendix C.5. We also conduct ablation studies on LDNet, focusing on pre-training and fine-tuning strategies, as shown in Table 2. LDNet surpasses Mirror in almost all settings and across all datasets.

3) Model transfer learning provides a valuable strategy for enhancing performance across all datasets, enabling LDNet to leverage information sharing between teacher models and student models.

Besides the triplet-based IE and MIE tasks, LDNet also demonstrates its effectiveness in discontinuous NER and n-ary hyper RE tasks. We provide results with pre-training (w/ PT, w/ Inst) and without pre-training (w/o PT, w/ Inst). As presented in Table 3, LDNet achieves improvements over previous methods, increasing by 4.18% and 0.42% on the CADEC Karimi et al. (2015) and Hyper

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline \hline Model & Parameter Scale & Movie & Restaurant & AI & Literature & Music & Politics & Science & Avg. \\ \hline USM & 372M & 37.73 & 14.73 & 28.18 & **56.00** & 44.93 & 36.10 & 44.09 & 37.39 \\ InstrumentUIE & 11B & **63.00** & 20.99 & 49.00 & 47.21 & 53.61 & 48.15 & 49.30 & 47.32 \\ Mirror & 304M & 39.20 & 16.32 & 45.23 & 46.32 & 58.61 & 67.30 & 54.84 & 46.83 \\ LDNet & 304M & 41.92 & **22.91** & **49.02** & 55.11 & **61.10** & **69.03** & **59.83** & **51.27** \\ \hline Llama-3 & 8B & 7.48* & 6.15* & 7.40* & 5.81* & 3.41* & 8.55* & 4.43* & 6.18* \\ \hline \end{tabular}
\end{table}
Table 4: Zero-shot results on 7 NER datasets. The results for Llama-3 are obtained from our experiments and are for reference only.

\begin{table}
\begin{tabular}{c|c|c c c|c} \hline Task & Model & 1-shot & 5-shot & 10-shot & Avg. \\ \hline  & UHE & 57.53 & 75.32 & 79.12 & 70.66 \\ NER & USM & 71.11 & 83.25 & 84.85 & 79.65 \\ CoNLL03 & Mirror & 76.49 & 82.45 & 84.69 & 81.21 \\  & LDNet & **78.33** & **84.53** & **85.39** & **82.75** \\ \hline  & UHE & 34.88 & 51.64 & 58.98 & 48.50 \\ Re & USM & 36.17 & 53.20 & 60.99 & 50.12 \\ CoNLL04 & Mirror & 26.29 & 47.42 & 55.77 & 43.16 \\  & LDNet & **37.93** & **53.74** & **61.46** & **51.04** \\ \hline  & UHE & 42.37 & 53.07 & 54.35 & 49.93 \\ Event Trigger & USM & 40.86 & 55.61 & 58.79 & 51.75 \\ ACE05 & Mirror & 47.77 & 57.90 & 59.16 & 54.94 \\  & LDNet & **54.77** & **62.75** & **64.24** & **60.59** \\ \hline  & UHE & 14.56 & 31.20 & 35.19 & 26.98 \\ Event Arg & USM & 19.01 & 36.69 & 42.48 & 32.73 \\ ACE05 & Mirror & 23.18 & 37.74 & 39.20 & 33.38 \\  & LDNet & **25.42** & **39.12** & **43.04** & **35.86** \\ \hline  & UHE & 23.04 & 42.67 & 53.28 & 39.66 \\ ABSA & USM & 30.81 & 52.06 & 58.29 & 47.05 \\
16-res & Mirror & 36.21 & 51.65 & 58.59 & 48.82 \\  & LDNet & **40.43** & **55.29** & **60.20** & **51.97** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Few-shot results on 4 IE tasks. These datasets are not included in pre-training, and LDNet does not apply model transfer learning in this setting, thus avoiding the risk of information leakage.

 

RED (Chia et al., 2022) datasets, respectively.

### Few-shot Results & Zero-shot Results

Followed by Zhu et al. (2023), we analyze LDNet's quick adaptation ability on NER, RE, EE, and ABSA tasks under 1-shot, 5-shot, and 10-shot settings. We compared LDNet with strong baselines such as UIE, USM, and Mirror. Table 5 shows the superior performance of LDNet under low-resource settings. On average, LDNet improves results by 3.41%, 2.26%, and 1.68% under 1-shot, 5-shot, and 10-shot settings, respectively. Additionally, LDNet achieves average improvements of 1.27%, 0.92%, 3.52%, and 3.02% over the NER, RE, EE, and ABSA tasks, respectively.

We also examine LDNet's extendability in the zero-shot setting using NER datasets from 7 distinct fields (Liu et al., 2013, 2021). LDNet is compared with USM, InstructUIE, Mirror and Llama-3-8B (Dubey et al., 2024). The zero-shot results are presented in Table 4. LDNet consistently outperforms Mirror and Llama-3-8B across all evaluated datasets. Although LDNet scores 0.89 lower than USM on the Literature dataset and is slightly lower than InstructUIE on the Movie dataset, the parameter scale of LDNet's pre-trained language model is smaller than that of USM's, and much smaller than that of InstructUIE's. Additionally, LDNet still achieves the highest average performance overall.

### Analysis on Label Drop

To investigate the effectiveness of our label drop mechanism, we conduct ablation studies under two settings: with and without label drop. We fine-tune for 100 epochs on the IE datasets in both of the settings to fully exploit the potential of the label drop mechanism. And in order to better demonstrate the effectiveness of the label drop mechanism, we do not perform model transfer learning in the ablation study. The results are shown in Figure 2. Compared to the results of LDNet with only the multi-aspect relation modeling component, the model with the label drop mechanism shows improved performance on most datasets. It achieves an improvement of 1.93% on average for the ABSA task and a substantial increase of 11.88% on the ACE05 dataset for the EE task.

To further analyze the capability of the label drop mechanism, we conduct experiments on 4 text datasets involving different tasks. We randomly drop different portions of the probability matrix and test the performance. We test drop rates of 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 100%. We also fine-tune 100 epochs for the analysis. The results are shown in Figure 2. More detailed results can be found in the Appendix C.5.

We can see that LDNet exhibits stable and strong performance on the NER, RE, and ABSA tasks, under different drop rates, demonstrating the robustness of the label drop mechanism. For the EE task, the performance is more volatile, with the best performance occurring under the full drop setting, which is as expected. The fluctuations in performance on the ACE05 dataset in Figure 2 are due to the large variety of label texts in the schema labels. The label texts following [LM] include 33 types, such as 'transport', 'elect', 'start position', 'nominate', 'end position', 'attack', 'meet', 'marry',

Figure 2: Results of the ablation study on the label drop mechanism. Table on the left shows LDNet’s performance with and without the Label Drop mechanism, box plot in the middle and line chart on the right illustrate LDNet’s performance under different drop rates.

 

'phone write', and so on. The label texts following [LR] have even more varieties, totaling 104 types. When only part of the probability matrix is dropped, some irrelevant token pairs' TA, A2A, or AS relation probabilities may not be dropped and could still exceed the threshold. As a result, non-existent TA, A2A, and AS relations may be included in the decoding process, and if they form a cycle, non-existent relations are predicted. Additionally, since we are **randomly** dropping portions of the probability matrix, even with a low drop rate, there is still a chance to accurately filter out token pairs that are unlikely to have relations. The content of the schema labels can be found in Appendix A.1, and the specific process of relation cycling can be referenced in Appendix A.2 or Appendix A.3. The stable high performance of other datasets is attributed to the significantly fewer types of label texts; for example, the CoNLL03 dataset has only four types: 'miscellaneous', 'person', 'location', and 'organization'. Even with a very low drop rate like 10%, the difficulty of filtering out non-existent relations is much lower in such a small range.

We also evaluate the accuracy of the label drop mechanism separately. The label drop probabilities \({\hat{l}^{r}}\) can be used to assess the accuracy of the label drop model. The accuracy can be computed as follows: we repeat \({\hat{l}^{r}}\in\mathbb{R}^{1\times|x|}\) to match the shape of the label matrices to create probability matrices \(A^{r}\in\mathbb{R}^{|x|\times|x|}\), \(r\in\{TA,A2A,AS\}\). Values in \(A^{r}\) below 0.5 are considered 0, while values above 0.5 are considered 1. We then compare this to the label matrices, and the number of correct values divided by the total number of values gives the accuracy. We test the accuracy, and the results are shown in the Table 6. It can be seen that the label drop accuracy is generally high, with an average accuracy exceeding 90%.

### Ablation Results of Multi-aspect Relation Modeling

We further conduct experiments on the Multi-aspect Relation Modeling mechanism. In these experiments, we utilize only the Multi-aspect Relation Modeling mechanism of the LDNet model, without incorporating Label Drop or Model Transfer Learning. We report the performance of LDNet under the 'w/o PT w/ Inst.' setting, with the selected comparison baseline being the performance of Mirror under the same setting. The results are shown in the Table 7. As can be seen, \(\mathrm{LDNet}_{MR}\) still outperforms Mirror under the same setting, demonstrating the effectiveness of the Multi-aspect Relation Modeling mechanism. The presence of Multi-aspect Relation Modeling reduces decision confusion while also creating a more suitable environment for the label drop mechanism. So it can be shown in the table that \(\mathrm{LDNet}_{MT^{-}}\) achieves the best performance among the three: \(\mathrm{LDNet}_{MR}\), \(\mathrm{LDNet}_{LD}\), and \(\mathrm{LDNet}_{MT^{-}}\).

## 5 Conclusion and Discussion

In this paper, we propose LDNet, a novel network that combines multi-aspect relation modeling and a label drop mechanism. LDNet assigns different relations to different levels for understanding and decision-making, thereby reducing decision confusion. By introducing the label drop mechanism, LDNet alleviates the influence of irrevelant relations. Experimental results show that LDNet achieves highly competitive performance across 9 tasks, in both single-modal and multi-modal, few-shot and zero-shot settings, which verifies its effectiveness and universality.

\begin{table}
\begin{tabular}{c|c|c} \hline Task & Datasets & Label Drop Accuracy \\ \hline NER & CoNLL03 & 93.04 \\ \hline RE & ACE05 & 94.69 \\  & CoNLL04 & 87.98 \\ \hline  & 14-lap & 92.61 \\ ABSA & 15-res & 89.82 \\  & 16-res & 90.41 \\ \hline  & Avg. & 91.76 \\ \hline \end{tabular}
\end{table}
Table 6: Results of the accuracy of label drop mechanism.

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline \multirow{2}{*}{Task} & \multirow{2}{*}{Datasets} & Mirror & \(\mathrm{LDNet}_{MR}\) & \(\mathrm{LDNet}_{LD}\) & \(\mathrm{LDNet}_{LP}\) \\  & w/o PT & w/o PT & w/o PT & w/o PT \\  & w/Inst. & w/ Inst. & w/ Inst. & w/ Inst. \\ \hline NER & CoNLL03 & 92.11 & 92.84 & 93.24 & **93.30** \\ \hline RE & ACE05 & 64.88 & 65.76 & 68.09 & **69.10** \\  & CoNLL04 & 71.19 & 75.11 & 71.97 & **75.81** \\ \hline  & 14-lap & 59.56 & 62.94 & 60.75 & **65.32** \\ ABSA & 15-res & 60.26 & 61.25 & 60.69 & **66.30** \\  & 16-res & 73.13 & 76.81 & 75.76 & **77.97** \\ \hline  & Avg. & 70.19 & 72.12 & 71.75 & **74.30** \\ \hline \end{tabular}
\end{table}
Table 7: Results of the ablation study on the multi-aspect relation modeling mechanism. \(\mathrm{LDNet}_{MR}\) represents LDNet using only Multi-aspect Relation Modeling, \(\mathrm{LDNet}_{LD}\) represents LDNet using only Label Drop, and \(\mathrm{LDNet}_{MT^{-}}\) indicates LDNet using both Multi-aspect Relation Modeling and Label Drop.

 



### Limitations

1) The total quantity and variety of MIE datasets are not enough, so LDNet cannot be pre-trained on a relatively large-scale dataset for MIE as it can be for IE, and since LDNet is a universal IE and UIE solution, its performance on certain multi-modal datasets may not be as good as models specifically designed for multi-modal tasks. 2) Due to the maximum input length constraint, LDNet may experience a performance decline in document-level information extraction.

## Ethical Considerations

If the model is able to extract information of high quality, it may be able to extract personal privacy information such as names, addresses, and phone numbers from large text datasets. This information could potentially be used for illegal monitoring, harassment, and other malicious purposes. Establishing appropriate privacy protection mechanisms and usage restrictions can be applied to ensure that the extracted information is only used for legitimate purposes and not abused.

## References

* P. M. Andersen, P. J. Hayes, A. K. Huettner, L. M. Schmandt, I. B. Nirenburg, and S. P. Weinstein (1992)Automatic extraction of facts from press releases to generate news stories. In ANLC, pp. 170-177. Cited by: SS3.4.
* F. Chen and Y. Feng (2023)Chain-of-thought prompt distillation for multimodal named entity and multimodal relation extraction. ArXivabs/2306.14122. External Links: Link, 2306.14122 Cited by: SS5.
* X. Chen, N. Zhang, L. Li, Y. Yao, S. Deng, C. Tan, F. Huang, L. Si, and H. Chen (2022)Good visual guidance make a better extractor: hierarchical visual prefix for multimodal entity and relation extraction. In NAACL, pp. 1607-1618. Cited by: SS5.
* Y. Ken Chia, L. Bing, S. Mahani Aljunied, L. Si, and S. Poria (2022)A dataset for hyper-relational extraction and a cube-filling approach. In EMNLP, pp. 10114-10133. Cited by: SS5.
* H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. Shane Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Y. Zhao, Y. Huang, A. M. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei (2022)Scaling instruction-finetuned language models. CoRRabs/2210.11416. External Links: Link, 2101.11416 Cited by: SS5.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186. Cited by: SS5.
* W. B. Dolan and C. Brockett (2005)Automatically constructing a corpus of sentential paraphrases. In IWP@IJCNLP, Cited by: SS5.
* A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021)An image is worth 16x16 words: transformers for image recognition at scale. In ICLR, Cited by: SS5.
* A. Dubey, A. Jauhrj, A. Pandey, A. Kadian, A. Al-Dahale, A. Latman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS5.
* H. Fei, S. Wu, J. Li, B. Li, F. Li, L. Qin, M. Zhang, M. Zhang, and T. Chua (2022)Lasuie: unifying information extraction with latent adaptive structure-aware generative language model. In NeurIPS, Cited by: SS5.
* R. Grishman (2019)Twenty-five years of information extraction. Nat. Lang. Eng.25 (6), pp. 677-692. External Links: Document, ISSN 1573-021X Cited by: SS5.
* P. He, J. Gao, and W. Chen (2021)Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. CoRRabs/2111.09543. External Links: Link, 2111.09543 Cited by: SS5.
* G. E. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. CoRRabs/1503.02531. External Links: Link, 1503.02531 Cited by: SS5.
* X. Hu, Z. Guo, Z. Teng, I. King, and P. S. Yu (2023)Multimodal relation extraction with cross-modal retrieval and synthesis. In ACL, pp. 303-311. Cited by: SS5.
* M. Josifoski, N. De Cao, M. Peyrard, F. Petroni, and R. West (2022)GenIE: generative information extraction. In NAACL, pp. 4626-4643. Cited by: SS5.
* S. Karimi, A. Metke-Jimenez, M. Kemp, and C. Wang (2015)Cadec: a corpus of adverse drug event annotations. J. Biomed. Informatics55, pp. 73-81. Cited by: SS5.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, pp. 7871-7880

Jiaja Li, Lu Yang, Mingni Tang, Cong Chen, Zuchao Li, Ping Wang, and Hai Zhao. 2024a. The music masctro or the musically challenged, a massive music evaluation benchmark for large language models. _Preprint_, arXiv:2406.15885.
* Li et al. (2022) Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022. Unified named entity recognition as word-word relation classification. In _AAAI_, pages 10965-10973.
* Li et al. (2019) Liumian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. _CoRR_, abs/1908.03557.
* Li et al. (2024b) Zixuan Li, Yutao Zeng, Yuxin Zuo, Weicheng Ren, Wenxuan Liu, Miao Su, Yucan Guo, Yantao Liu, Xiang Li, Zhilei Hu, et al. 2024b. Knowcoder: Coding structured knowledge into llms for universal information extraction. _arXiv preprint arXiv:2403.07969_.
* Liang et al. (2021) Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tiem Yan Liu. 2021. R-drop: Regularized dropout for neural networks. In _NeurIPS_, pages 10890-10905.
* Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In _ACL_, pages 7999-8009.
* Liu et al. (2013) Jingjing Liu, Panupong Pasupat, Scott Cyphers, and James R. Glass. 2013. Asgard: A portable architecture for multilingual dialogue systems. In _ICASSP_, pages 8386-8390.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Liu et al. (2021) Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In _AAAI_, pages 13452-13460.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _ICLR_.
* Lou et al. (2023) Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xinapei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. _CoRR_, abs/2301.03282.
* Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In _ACL_, pages 1990-1999.
* Lu et al. (2022) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianguei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. In _ACL_, pages 5755-5772.
* Lu et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirizi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In _EMNLP_, pages 3219-3232.
* Mitchell et al. (2005) Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2005. Ace 2004 multilingual training corpus.
* Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In _ICLR_.
* Peng et al. (2023a) Tianshuuo Peng, Zuchao Li, Ping Wang, Lefei Zhang, and Hai Zhao. 2023a. A novel energy based model mechanism for multi-modal aspect-based sentiment analysis. _Preprint_, arXiv:2312.08084.
* Peng et al. (2023b) Tianshuo Peng, Zuchao Li, Lefei Zhang, Bo Du, and Hai Zhao. 2023b. FSUIE: A novel fuzzy span mechanism for universal information extraction. In _ACL_, pages 16318-16333.
* Ping et al. (2023) Yang Ping, JunYu Lu, Ruyi Gan, Junje Wang, Yuxiang Zhang, Pingjian Zhang, and Jiaxing Zhang. 2023. UniEX: An effective and efficient framework for unified information extraction via a span-extractive perspective. In _ACL_, pages 16424-16440.
* Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Aryyoub, Yanyan Zhao, Bing Qin, Orphe De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelinikov, Nuria Bel, Salud Maria Jimenez-Zafra, and Gulsen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In _SemEval-2016_, pages 19-30.
* Pontiki et al. (2015) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In _SemEval 2015_, pages 486-495.
* Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In _SemEval 2014_, pages 27-35.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _ACL_, pages 784-789.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _ECML-PKDD_, pages 148-163.
* Roth and Yih (2004) Dan Roth and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In _HLT-NAACL_, pages 1-8 

Oscar Sainz, Iker Garcia-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agrirre. 2024. Gollie: Annotation guidelines improve zero-shot information-extraction. _Preprint_, arXiv:2310.03668.
* Satyapanich et al. (2020) Taneeya Satyapanich, Francis Ferraro, and Tim Finin. 2020. Casie: Extracting cybersecurity event information from text. In _AAAI_, pages 8749-8757.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _EMNLP_, pages 1631-1642.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _JMLR_, pages 1929-1958.
* Su et al. (2022) Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yunfeng Liu. 2022. Global pointer: Novel efficient span-based approach for named entity recognition. _CoRR_, abs/2208.03054.
* Sun et al. (2024) Lin Sun, Kai Zhang, Qingyuan Li, and Renze Lou. 2024. Umie: Unified multimodal information extraction with instruction tuning. In _AAAI_, volume 38, pages 19062-19070.
* Tan et al. (2022) Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022. Document-level relation extraction with adaptive focal loss and knowledge distillation. In _ACL_, pages 1672-1681.
* Tian et al. (2023) Jinhao Tian, Zuchao Li, Jiajia Li, and Ping Wang. 2023. N-gram unsupervised compoundation and feature injection for better symbolic music understanding. _Preprint_, arXiv:2312.08931.
* Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the call-2003 shared task: Language-independent named entity recognition. In _CoNLL_, pages 142-147.
* Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In _EMNLP-IJCNLP_, pages 5784-5789.
* Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus.
* Wan et al. (2024) Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. 2024. Omniparser: A unified framework for text spotting key information extraction and table recognition. In _CVPR_, pages 15641-15653.
* Wang et al. (2019) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _ICLR_.
* Wang et al. (2022a) Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022a. DeepStruct: Pretraining of language models for structure prediction. In _ACL_, pages 803-823.
* Wang and Lu (2020) Jue Wang and Wei Lu. 2020. Two are better than one: Joint entity and relation extraction with label-sequence encoders. In _EMNLP_, pages 1706-1721.
* Wang et al. (2023) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunusai Du. 2023. Instructuie: Multi-task instruction tuning for unified information extraction. _CoRR_, abs/2304.08085.
* Wang et al. (2022b) Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie, Kewei Tu, and Wei Lu. 2022b. Named entity and relation extraction with multi-modal retrieval. In _EMNLP_, pages 5925-5936.
* Warstadt et al. (2019) Alex Warstadt, Amampreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. _Trans. Assoc. Comput. Linguistics_, 7:625-641.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _NAACL_, pages 1112-1122.
* Wu et al. (2020) Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In _MM_, pages 1038-1046.
* Yu et al. (2020) Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In _ACL_, pages 3342-3352.
* Cong et al. (2023) Xin Cong. Bowen Yu, Mengcheng Fang, Tingwen Liu, Haiyang Yu, Zhongkai Hu, Fei Huang, Yongbin Li, and Bin Wang. 2023. Universal information extraction with meta-pretrained self-retrieval. _Preprint_, arXiv:2306.10444.
* Zhang et al. (2021) Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In _AAAI_, pages 14347-14355.
* Zhang et al. (2018a) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xu Nanjing Huang. 2018a. Adaptive co-attention network for named entity recognition in tweets. In _AAAI_, pages 5674-5681.
* Zhang et al. (2022) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuuning Huang. 2018b. Adaptive co-attention network for named entity recognition in tweets. In _AAAI_, pages 5674-5681.
* Zhang et al. (2022) Zheng Zhang, Zili Zhou, and Yanna Wang. 2022. SSEGCN: Syntactic and semantic enhanced graph convolutional network for aspect-based sentiment analysis. In _NAACL_, pages 4916-4925 

Changmeng Zheng, Junhao Feng, Yi Cai, Xiaoyong Wei, and Qing Li. 2023. Rethinking multimodal entity and relation extraction from a translation point of view. In _ACL_, pages 6810-6824.
* Zheng et al. (2021) Changmeng Zheng, Zhiwei Wu, Junhao Feng, Ze Fu, and Yi Cai. 2021. Mnre: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts. In _ICME_, pages 1-6.
* Zhong and Chen (2021) Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In _NAACL_, pages 50-61.
* Zhu et al. (2023) Tong Zhu, Junfei Ren, Zijian Yu, Mengsong Wu, Guoliang Zhang, Xiaoye Qu, Wenliang Chen, Zhefeng Wang, Baoxing Huai, and Min Zhang. 2023. Mirror: A universal framework for various information extraction tasks. In _EMNLP_, pages 8861-8876.

## Appendix A Methodology

### Schema Labels

When it comes to the specific implementation, the schema labels are divided into two parts: a set of special tokens [LM], [LR], [LC], [TL], [TP], and [B] and their corresponding label text.

[LM], [LR], and [LC] are the special tokens that represent labels of entity mention, relation, and text classification respectively. Only one of the three special tokens [TL], [TP], and [B] will appear in a single data instance. [TP] is used for the MRC task, [B] is used for the classification task, and [TL] is used for all other tasks.

The special tokens [LM], [LR], [LC] will be followed by their corresponding label text, which are the tokenized strings for the entity mention, relation, and text classification labels. All the labels that appear in the dataset will be included in the schema labels, such as [LC], "_correct", [LC], "_wrong" for the classification task with "correct" and "wrong" as the two labels.

We use the special tokens in the schema labels as the trigger words and utilize them to guide the relation extraction process. A relation will only be extracted when the trigger word is activated. The label drop operation sets the elements in the label vector corresponding to the special tokens that in gold spans to 1, while the other schema label elements remain 0. For example, in the classification task, if a data instance is classified as "correct", the element corresponding to the [LC] token before "_correct" token will be set to 1, while the element corresponding to the "_correct" token will remain 0.

### Handling of Unknown Schemas

If the schema is unknown, LDNet removes the schema from the original NER input format, transforming it from the format of _instruction + schema + text_:

[I] Please identify possible entities from the given text and determine their types [LT] person [LM] location [LM] organization [TL] Jerry Smith is a friend of Tom to:

[I] Please identify possible entities from the given text and determine their types [TP] Jerry Smith is a friend of Tom

Here, [I] Please identify possible entities from the given text and determine their types is the instruction part, with [I] being a special token indicating the start of the instruction; [LM] person [LM] location [LM] organization represents the schema, with [LM] being a special token representing an entity type, such as [LM] person, indicating the person type; [TL] Jerry Smith is a friend of Tom is the text part, with [TL] being a special token indicating that the text following it requires not only span extraction but also the extraction of types within the schema. For instance, to extract the entity Jerry Smith, it's necessary to extract both its span and its corresponding type, **person**, which is associated with the span of the [LM] token in [LM] person in the schema. In the input without a schema, [TP] indicates that only entity spans need to be extracted from the text, without requiring the extraction of additional information such as types in schemas.

Thus, LDNet handles unknown schemas by processing the input as follows:

[I] Please identify possible entities from the given text and determine their types

[TP] Jerry Smith is a friend of Tom

From the text _"Jerry Smith is a friend of Tom"_, LDNet directly extracts the spans of entities such as Jerry Smith and Tom as the output.

### Handling of Discontinuous NER and Nested NER

Discontinuous NER LDNet handles discontinuous NER in a manner similar to how it handles regular NER. Suppose the input is: 

[I] Please identify possible entities from the given text and determine their types
* [LM] person [LM] title
* [LM] organization [TL] The CEO of Tesla , Elon Musk, made an announcement today LDNet will extract the following relations:

* TA Relation: Between [LM] before **person** and CEO of Tesla, and [LM] before **person** and Elon Musk.
* A2A Relation: Between CEO of Tesla and Elon Musk.
* AS Relation: Between CE0 of Tesla and CEO of Tesla, and Elon Musk and Elon Musk.

The closed loop formed by the TA relation between [LM] before **person** and CEO of Tesla, the AS relation between CEO of Tesla, the TESla, the A2A relation between CEO of Tesla and Elon Musk, the AS relation between Elon Musk and Elon Musk, and the TA relation between [LM] before **person** and Elon Musk allows LDNet to extract CEO of Tesla, Elon Musk as a discontinuous entity of the **person** type.

**Nested NER** The input format for nested NER is:

[I] Please identify possible entities from the given text and determine their types
* [LM] organization [LM] title
* [LM] person [LM] location
* [TL] Apple CEO Tim Cook gave a speech at Stanford University in California

\begin{table}
\begin{tabular}{c c c c} \hline \hline Parameter & Value & Parameter & Value \\ \hline warmup proportion & 0.1 & batch size & 4 \\ pre-training epochs & 3 & PLM learning rate & 2e-5 \\ fine-tuning epochs & 20 & PLM weight decay & 0.1 \\ fine-tuning epoch patience & 3 & others learning rate & 1e-4 \\ few-shot epochs & 200 & max gradient norm & 1.0 \\ few-shot epoch patience & 10 & \(d_{h}\) & 1024 \\ \(d_{v}^{~{}\diamond}\) & 1024 & batch size\({}^{\diamond}\) & 32 \\ PLM learning rate\({}^{\diamond}\) & 3e-5 & others learning rate\({}^{\diamond}\) & 1e-4 \\ fine-tuning epochs\({}^{\diamond}\) & 20 & PLM weight decay\({}^{\diamond}\) & 0.1 \\ warmup proportion\({}^{\diamond}\) & 0.01 & \(\alpha^{\Diamond}\) & 0.5 \\ \hline \hline \end{tabular}
\end{table}
Table 8: The parameters marked with \({}^{\diamond}\) are for the multi-modal experiments.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline NER Pre-training Dataset & Instruction & Instance & RE Pre-training Dataset & Instruction & Instance \\ \hline AnatEM & 42 & 5,861 & ADE\_corpus & 9 & 3,417 \\ bc2gm & 42 & 12,500 & FewRel & 9 & 20,000 \\ bc4chemd & 42 & 20,000 & GIDS & 9 & 8,526 \\ bc5cdr & 42 & 4,560 & kbp37 & 9 & 15,807 \\ Broad\_Tweet\_Corpus & 42 & 5,334 & New-York-Times-RE & 9 & 20,000 \\ FabNER & 42 & 9,435 & NYT11HRL & 9 & 20,000 \\ FindVehicle & 42 & 20,000 & semeval & 9 & 8,000 \\ GENIA & 42 & 15,023 & WebNLG & 9 & 5,019 \\ HarveyNER & 42 & 3,967 & Wiki-ZSL & 9 & 23,107 \\ MultiNERD & 42 & 20,000 & MNRE\({}^{\diamond}\) & 9 & 12,248 \\ \cline{2-5} NCBIiease & 42 & 5,432 & MRC Pre-training Dataset & Instruction & Instance \\ \cline{2-5} ontoNotes5 & 42 & 20,000 & BiPaR & 11,524 & 11,668 \\ TweetNER7 & 42 & 7,103 & ms\_marco\_v2.1 & 20,000 & 20,000 \\ WikiANN\_en & 42 & 20,000 & newsqa & 19,659 & 20,000 \\ WNUT-16 & 42 & 2,394 & squad\_v2 & 19,998 & 20,000 \\ Twitter2015\({}^{\diamond}\) & 42 & 4,000 & SubjQA & 4,060 & 13,990 \\ \cline{2-5} Twitter2017\({}^{\diamond}\) & 42 & 3,376 & EE Pre-training Dataset & Instruction & Instance \\ \cline{2-5}  & & & PHEE & 40 & 2,898 \\ \hline \hline \end{tabular}
\end{table}
Table 9: The datasets marked with \({}^{\diamond}\) are for the multi-modal experiments.

 

In this example, Apple is an organization entity representing the Apple company, and Apple CEO is a title entity representing the CEO position of Apple.

After obtaining the TA, A2A, and AS relation probability matrices, LDNet will extract the following relations:

* TA Relation: Between [LM] before **organization** and Apple, between [LM] before **title** and Apple, and [LM] before **title** and CEO.
* AS Relation: Between Apple and Apple; between Apple and CEO.

The closed loop formed by the TA relation between [LM] before **organization** and Apple, and the AS relation between Apple and Apple allows LDNet to extract Apple as an organization entity. Similarly, the TA relations between [LM] before **title** and Apple, the AS relation between Apple and CEO, and the TA relation between [LM] before **title** and CEO form a closed loop, allowing LDNet to extract Apple CEO as a title entity.

The A2A relation is not mandatory, as the information to be extracted may not involve two arguments, such as in NER tasks.

### Underlying Logic of Label Drop Mechanism

The underlying logic of label drop mechanism is that if the \(i\)-\(th\) token does not exist in the gold answer, after training, the value of \({\hat{l}^{r}_{i}}\) at position \(i\) in \({\hat{l}^{r}}\) will close to 0, it suppresses the value of \(s^{r}_{.i}\) in the \(i\)-th column of the \(S^{r}\) matrix. During subsequent decoding, if the value of \(s^{r}_{.i}\) is suppressed below a threshold, LDNet will not consider extracting the relation between position \(i\) and other positions, therefore filtering out token pairs that is impossible to have relations. As shown in Figure 1, for the convenience of observation, we only depict the prediction of the A2A relation. And to better illustrate the concept of label drop, we hypothesize an extreme scenario where \({\hat{l}^{r}}\) is the same as I'. For instance, the probability of a relation existing between "playwright" and "born" is set to 0.

### LDNet's Contributions Relative to Prior Multi-modal Approaches

Multi-modal representation is a conventional approach. LDNet utilizes multi-modal representation to enable its application to multi-modal data. LDNet is a universal information extraction method, its capability is general and not differentiated by whether the data is multi-modal or unimodal. Compared to prior multi-modal approaches, the contribution of LDNet lies not in combining multi-modal

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Classification Pre-training Dataset & Instruction & Instance & Classification Pre-training Dataset & Instruction & Instance \\ \hline ag\_news & 5 & 5,000 & ANLI & 29 & 15,000 \\ ARC & 3,361 & 3,370 & CoLA & 43 & 5,000 \\ CosmosQA & 4,483 & 5,000 & cos\_e & 5,000 & 5,000 \\ dbpedia & 6 & 5,000 & DREAM & 3,842 & 5,000 \\ hellaswag & 20 & 5,000 & IMDB & 26 & 5,000 \\ MedQA & 5,000 & 5,000 & MNLI & 29 & 5,000 \\ MRPC & 40 & 3,668 & MultiRC & 4,999 & 5,000 \\ OpenBookQA & 4,835 & 4,957 & QASC & 4,832 & 5,000 \\ DNLI & 31 & 5,000 & QQP & 40 & 5,000 \\ RACE & 4,482 & 5,000 & RACE-C & 4,782 & 5,000 \\ ReClor & 3,368 & 4,638 & RTE & 29 & 2,490 \\ SciQ & 4,989 & 5,000 & SNLI & 29 & 5,000 \\ SST-2 & 26 & 5,000 & Winogrande & 20 & 5,000 \\ WNLI & 31 & 635 & & & \\ \hline \hline \end{tabular}
\end{table}
Table 10: The Classification Pre-training Datasets.

\begin{table}
\begin{tabular}{c|c c c c c c c} \hline \hline Model & TANL & UIE & DeepStruct & InstructUIE & USM & Mirror & LDNet \\ \hline Computational Complexity & \(O(n^{3})\) & \(O(n^{2})\) & \(O(n^{2})\) & \(O(n^{2})\) & \(O(n^{2})\) & \(O(n^{2})\) & \(O(n^{2})\) \\ PLM & TS-base & TS-large & GLM & FlanT5 & RoBERTa-large & DeBERta+v3-large & DeBERta-v3-large \\ PLM Params & 220M & 770M & 10B & 11B & 372M & 304M & 304M \\ \hline \hline \end{tabular}
\end{table}
Table 11: Computational complexity and model parameters of various models.

 

information, but rather in its Label Drop mechanism, which effectively filters out irrelevant token pairs.

Individual Contributions of Image FeaturesSince images only provide clue information; the information extraction ultimately comes from the text. Therefore, it is not possible to explore the individual contributions of image features on the model's performance.

## Appendix B Related Work

Dropout StrategyDropout (Srivastava et al., 2014) is a powerful technique usually used to regularize the training of deep neural networks. R-Drop (Liang et al., 2021) forces the output distributions of different submodels, sampled by dropout, to be consistent with each other by minimizing their bidirectional KL-divergence. LDNet transfers the idea of dropout into IE tasks and applies label drop to remove unrelational token pairs, forcing the model to concentrate on relational ones.

Model Transfer LearningAveraging the predictions of all trained models is a simple yet effective way to enhance the performance of almost any machine learning algorithm. However, it can be cumbersome and computationally expensive. Therefore, (Hinton et al., 2015) proposed Knowledge Distillation (KD) to compress the knowledge of an ensemble into a single model. Tan et al. (2022) utilizes KD in document-level relation extraction. The system trains a teacher model on the distantly-supervised data and uses the distributions generated by the teacher model as soft labels to pre-train the student model. The authors found that distilling with the MSE loss performs better than using KL-divergence. LDNet incorporates this idea and follows the same setting, using MSE loss to minimize the difference between the distributions of the teacher model and the student model.

Multi-modal Information ExtractionUnlike traditional information extraction, which relies exclusively on single-modal data, Multi-modal Information Extraction (MIE) leverages auxiliary visual cues from images to supplement the missing context. MEGA Zheng et al. (2021) is the first to propose the MNRE dataset and introduces a dual image alignment method to capture aligned information between visual object and textual object. UMT Yu et al. (2020), which pioneers the use of Transformer in the MIE task, utilizes a multi-modal interaction module to integrate token representa

\begin{table}
\begin{tabular}{c|l|r r r r|r r r} \hline \hline Task & Datasets & GoLLIE Baseline & GoLLIE & GoLLIE-13B & GOLLIE-34B & LDNet Baseline & LDNe\(t_{doctru-v3-base}\) & LDNe\(t_{before=vs-large}\) \\ \hline \multirow{2}{*}{NER} & ACE05 & 89.10 & 88.10 & 89.40 & **89.60** & 85.70 & 88.18 & 87.79 \\  & CoNLLO3 & 92.90 & 92.80 & 93.00 & 93.10 & 92.73 & 93.43 & **93.44** \\ \hline RE & ACE05 & 63.80 & 63.60 & 67.50 & **70.10** & 67.86 & 69.33 & 69.14 \\ \hline \multirow{3}{*}{EE} & ACE05-Tgg & 71.7 & 72.2 & 70.9 & 71.9 & 73.05 & 73.42 & **83.56** \\  & ACECoSeg & 65.9 & 66.0 & 67.8 & **68.6** & 54.73 & 57.04 & 61.14 \\ \cline{1-1}  & CASIE-Tgg & 33.9 & 59.3 & 62.2 & 65.5 & 71.60 & 72.77 & **73.36** \\ \cline{1-1}  & CASIE-Arg & 47.9 & 50.0 & 52.6 & 55.2 & 61.04 & 63.35 & **63.88** \\ \hline \hline \end{tabular}
\end{table}
Table 13: Comparison to GoCLLIE models.

 

tions with visual representations. MoRe (Wang et al., 2022b) enhances textual information retrieval by leveraging images and titles from search engines, thereby improving the accuracy of multi-modal RE and NER tasks. Building on MoRe, MRE-RS (Hu et al., 2023) retrieves textual and visual evidence at different levels and further proposes a novel method to synthesize information for improved reasoning across the same and different modalities. CoTPD (Chen and Feng, 2023) demonstrates the elicitation of reasoning abilities from LLMs using CoT prompts across various dimensions and introduces a conditional prompt distillation method to transfer commonsense reasoning to a student model, enhancing its performance on text-only inputs. There are also some recently new methods, such as UMIE (Sun et al., 2024) and OmniParser (Wan et al., 2024).

## Appendix C Experiments details

### Hyperparameters and Pre-training Datasets

Specifically, we use vit-large-patch32-224-in21k as our image backbone. The experiments can be run using only 1 NVIDIA RTX 3090 with 24 GB memory. The hyperparameters and pre-training datasets are in Table 8, Table 9 and Table 10.

### Experimental Configurations

The LDNet configurations 'w/ PT w/o Inst.' and 'w/ PT w/ InfN.' are not considered variants of the model; the only difference lies in whether the instruction part of the model input is an empty string. In 'w/ PT w/o Inst.' configuration, the instruction part is an empty string, while in 'w/ PT w/ Inst.', the instruction part is not empty and is a string similar to 'Please determine the two entities mentioned in the text and specify the nature of their relationship.'

We here give reasons why performance is often better in 'without PT and Inst.' configurations. The situation that configurations without PT and Inst. perform better is related to the dataset and the backbone model, DeBERTa-v3-large. For certain datasets like CoNLL04, 14-lap, and 15-res, better performance is observed under the 'with PT' configuration. On the other hand, some datasets, such as ACE04 and ACE05, are inherently large, and fine-tuning on their training sets alone yields good results, and the absence of pre-training does not introduce interference from other data, leading to better performance without PT. As shown in Table 2, the performance difference between

\begin{table}
\begin{tabular}{l|c c c c c c c|c c c} \hline \multirow{2}{*}{**Modality**} & \multirow{2}{*}{**Methods**} & \multicolumn{6}{c|}{Twitter-2015} & \multicolumn{3}{c|}{Twitter-2017} & \multicolumn{3}{c}{MNRE} \\ \cline{3-11}  & & Prec. & Rec. & F1 & Prec. & Rec. & F1 & Prec. & Rec. & F1 \\ \hline \multirow{8}{*}{Test+Image} & AdapCoAtt (Zhang et al., 2018a) & 69.87 & 74.59 & 72.15 & 85.13 & 83.20 & 84.10 & - & - & - \\  & VisualBERT (Li et al., 2019) & 68.84 & 71.39 & 70.09 & 84.06 & 85.39 & 84.72 & 57.15 & 59.48 & 58.30 \\  & OCSGA (Wu et al., 2020) & 74.71 & 71.21 & 72.92 & - & - & - & - & - & - \\  & UMT (Yu et al., 2020) & 71.67 & 75.23 & 73.41 & 85.28 & 85.34 & 85.31 & 62.93 & 63.88 & 63.46 \\  & UMGF (Zhang et al., 2021) & 74.49 & 75.21 & 74.85 & 86.54 & 84.50 & 85.51 & 64.38 & 66.23 & 65.29 \\  & MEGA (Zheng et al., 2021) & 70.35 & 74.58 & 72.35 & 84.03 & 84.75 & 84.39 & 64.51 & 68.44 & 66.41 \\  & HYPNeT (Chen et al., 2022) & 73.87 & 76.82 & 75.32 & 85.84 & 87.93 & 86.87 & 83.64 & 80.78 & 81.85 \\  & MoRe (Wang et al., 2022b) & - & - & - & - & - & 65.25 & 67.32 & 66.27 \\  & TMR (Zheng et al., 2023) & 75.26 & 76.49 & 75.87 & 88.12 & 88.38 & 88.25 & 90.48 & 87.66 & 89.05 \\ \cline{2-11}  & LDNet & 76.79 & 75.56 & 76.17 & 87.51 & 87.64 & 87.57 & 75.45 & 76.15 & 75.79 \\ \hline \end{tabular}
\end{table}
Table 15: Comparison of LDNet’s performance on MIE tasks with some MIE baselines.

 

with and without Inst.' is generally small. The distinction between these two configurations lies in whether the instruction part of LDNet's input is an empty string, and any difference should be related to DeBERTa-v3-large's instruction-following capability.

### Method for Few-shot Experiments

Regarding the few-shot experiments, we employed fine-tuning. Following the settings of Mirror to ensure the fairness of the comparison, we fine-tuned for several epochs on the training set of the few-shot dataset before assessing few-shot capabilities.

### Model Parameters and Computational Complexity

All models' computational complexities and the scales of the pre-trained language models used are listed in the Table 11.

Here, we define the following notations to analyze the computational complexity of the models:

* Expected answer length: \(k\)
* Text length: \(n\)
* Instruction length: \(l\)
* Schema length: \(s\)
* Hidden dimension: \(d\)
* Biaffine dimension: \(b\)
* Number of transformer layers: \(t\)

First, the computation for the hidden states of a transformer architecture model (excluding the computation for predicting token logits) is \((24nd^{2}+4dn^{2})\cdot t\). The computation for predicting token logits is \(2dvn\).

TANL, Deepstruct, UIE, and InstructUIE, which are generative unified information extraction methods, mainly focus on the computation involved in token prediction.

* **TANL** outputs a sequence that includes the answer to be extracted in the original input sequence, so its computation for sequence generation is \([(24nd^{2}+4dn^{2})\cdot t+2dvn]\cdot(n+k)\). Since the decoding algorithm used by TANL has a complexity of \(O(n^{2})\), its overall computational complexity is \(O(n^{3})\).
* **Deepstruct** outputs a sequence that is the answer sequence itself, so its computation is \([(24nd^{2}+4dn^{2})\cdot t+2dvn]\cdot k\), with a computational complexity of approximately \(O(n^{2})\).

\begin{table}
\begin{tabular}{c|c c c c c c c c c c c} \hline Dataset & Strategy & 10\% & 20\% & 30\% & 40\% & 50\% & 60\% & 70\% & 80\% & 90\% & 100\% \\ \hline CoNLL03 & w / PT \& Ins. & 94.74 & 94.35 & 94.29 & 94.49 & 95.04 & 94.68 & 94.08 & 94.56 & 94.29 & 94.60 \\  & w / PT \& w/o Inst. & 94.69 & 94.11 & 95.06 & 93.49 & 94.57 & 94.62 & 94.92 & 94.65 & 94.88 & 95.11 \\ \hline NYT & w / PT \& Inst. & 93.72 & 93.51 & 93.36 & 93.16 & 93.59 & 93.07 & 93.38 & 93.48 & 92.86 & 94.43 \\  & w / PT \& w/o Inst. & 93.35 & 93.86 & 93.17 & 93.39 & 93.96 & 93.70 & 93.54 & 93.25 & 93.54 & 94.41 \\ \hline ACE05-Tgg & w / PT \& Inst. & 71.79 & 72.59 & 68.33 & 74.45 & 67.23 & 70.07 & 69.14 & 75.58 & 68.66 & 78.74 \\  & w / PT \& w/o Inst. & 67.10 & 66.67 & 69.57 & 68.61 & 76.43 & 79.55 & 70.92 & 71.25 & 78.61 & 80.23 \\ \hline ACE05-Arg & w / PT \& Inst. & 49.80 & 48.73 & 60.42 & 50.24 & 58.95 & 47.06 & 49.64 & 51.22 & 42.49 & 55.78 \\  & w / PT \& w/o Inst. & 50.95 & 46.15 & 42.93 & 52.22 & 50.00 & 60.47 & 48.70 & 54.94 & 55.77 & 62.76 \\ \hline
14-res & w / PT \& Inst. & 84.62 & 83.86 & 84.29 & 85.58 & 84.82 & 83.93 & 85.37 & 84.75 & 85.10 & 83.58 \\  & w / PT \& w/o Inst. & 84.54 & 82.97 & 84.21 & 84.88 & 84.60 & 84.82 & 84.49 & 84.13 & 82.66 & 84.79 \\ \hline \end{tabular}
\end{table}
Table 16: Results of the label drop mechanism with different drop rates.

\begin{table}
\begin{tabular}{c|c c c c c c c c c c} \hline Dataset & Metric & 10\% & 20\% & 30\% & 40\% & 50\% & 60\% & 70\% & 80\% & 90\% & 100\% \\ \hline \multirow{3}{*}{Twitter-2015} & P & 76.22 & 76.52 & 78.22 & 77.61 & 76.36 & 75.93 & 75.56 & 79.45 & 76.74 & 76.79 \\  & R & 76.32 & 75.60 & 74.36 & 73.61 & 76.19 & 75.06 & 75.86 & 74.03 & 76.21 & 75.56 \\  & F1 & 76.27 & 76.05 & 76.24 & 75.56 & 76.28 & 75.49 & 75.71 & 76.64 & 76.47 & 76.17 \\ \hline \multirow{3}{*}{Twitter-2017} & P & 87.80 & 87.33 & 89.10 & 88.25 & 88.7 & 87.69 & 86.39 & 87.33 & 86.49 & 87.51 \\  & R & 87.34 & 86.75 & 86.68 & 86.75 & 87.27 & 87.49 & 87.86 & 87.79 & 87.19 & 87.64 \\  & F1 & 87.57 & 87.04 & 87.88 & 87.50 & 87.99 & 87.59 & 87.12 & 87.56 & 86.84 & 87.57 \\ \hline \multirow{3}{*}{MNRE} & P & 74.21 & 75.16 & 76.05 & 75.29 & 75.80 & 74.74 & 74.73 & 74.54 & 73.57 & 75.45 \\  & R & 73.79 & 74.97 & 72.99 & 72.49 & 74.72 & 74.41 & 74.04 & 72.37 & 72.61 & 76.15 \\ \cline{1-1}  & F1 & 74.00 & 75.06 & 74.49 & 73.86 & 75.26 & 74.57 & 74.39 & 73.44 & 73.09 & 75.79 \\ \hline \end{tabular}
\end{table}
Table 17: Results of the label drop mechanism with different drop rates on MIE datasets.

 

* **UIE** takes an input sequence that includes the schema and directly generates a Structured Extraction Language (SEL) as output. Its computation is \([(24(n+s)d^{2}+4d(n+s)^{2})\cdot t+2dv(n+s)]\cdot k\), with a computational complexity of approximately \(O(n^{2})\).
* **InstructUE** includes the instruction, schema, and text in the input sequence. Its computation is \([(24(n+s+l)d^{2}+4d(n+s+l)^{2})\cdot t+2dv(n+s+l)]\cdot k\), with a computational complexity of approximately \(O(n^{2})\).

USM, Mirror, and LDNet belong to extractive unified information extraction methods, hence they do not include the computation for predicting token logits.

* **USM** includes schema and text in the input. In addition to encoder representations, it computes token-token linking scores, label-token linking scores, and token-label linking scores. Its computation is \((24(n+s)d^{2}+4d(n+s)^{2})\cdot t+8(n+s)d^{2}+2d(n+s)^{2}\), with a computational complexity of \(O(n^{2})\).
* **Mirror** includes instruction, schema, and text in the input. After computing the representations with the pre-trained model, it uses a biaffine transformation to generate score matrices. Its computation is \((24(n+s+l)d^{2}+4d(n+s+l)^{2})\cdot t+4ndb+4nd+6nb^{2}+6bn^{2}\), with a computational complexity of \(O(n^{2})\).
* **LDNet** includes instruction, schema, and text in the input. The Multi-aspect Relation Modeling module's computation is \((8nd^{2}+2dn^{2})\cdot 3\). The label drop mechanism's computation is \(3n^{2}\). Therefore, the total computation is \((24(n+s+l)d^{2}+4d(n+s+l)^{2})\cdot t+(8nd^{2}+2dn^{2})\cdot 3+3n^{2}\), with a computational complexity of \(O(n^{2})\).

### Additional Experiments

Comparison to Models with Smaller PLMsWe separately list the pretrained language model parameters smaller than the DeBERTa-v3-Large (304M) used by LDNet and compare them with LDNet using DeBERTa-v3-Base (86M). The results are shown in the Table 12.

Comparison to GoLLIE ModelsIn the Table 13, we compare the results of GoLLIE and LDNet. For ease of comparison, we have included only the results where both models are evaluated.

MRC and Classification ResultsTo demonstrate the compatibility of LDNet, we conducted experiments on SQuAD v2 Rajpurkar et al. (2018) and the 7 GLUE datasets Warstadt et al. (2019); Wang et al. (2019); Williams et al. (2018); Socher et al. (2013); Dolan and Brockett (2005). As shown in Table 14, LDNet outperforms Mirror across all datasets. Additionally, LDNet surpasses BERTlarge on SST-2 and QNLI, outperforms RoBERTa-large on CoLA, RTE, and MRPC, and achieves competitive results with DeBERTa v3-large on CoLA and MRPC. It is important to note that LDNet is a universal solution for IE and does not undergo full fine-tuning like Language Model Models (LLMs). Therefore, LDNet has limitations when it comes to tasks such as MRC and classification.

Figure 3: Results of the ablation study on the label drop mechanism on MIE tasks.

 

Consequently, it is reasonable to observe a slight performance gap in these tasks. However, it is worth noting that LDNet exhibits a smaller performance gap compared to Mirror.

Multi-modal ResultsWe put detailed results of multi-modal experiments in Table 15. It can be seen that LDNet performs better than most baselines, and is slightly lower in some metrics on certain specific baselines. But these methods are all focused only on MIE, unlike LDNet which is a universal solution for both IE and MIE. And some methods like HVPNeT utilizes additional visual prefixes and uses a specially-designed pyramid structure, and TMR have introduced more datasets for training, so there may be some performance gap. Although LDNet performs slightly lower than TMR on some datasets, it surpasses TMR on the Twitter-2015 dataset. Moreover, LDNet is a universal information extraction solution that covers a broader range of tasks, such as Discontinuous NER and Hyper RE.

Specific Results of Label Drop MechanismWe put the specific results of the label drop mechanism with different drop rates in Table 16. It can be seen that after 100 rounds of fine-tuning, the performance of the two strategies on the CoNLL03, NYT, and 14-res datasets is not much different for LDNet, and in some drop rate cases, LDNet can perform better without the instruction, demonstrating the effectiveness of the label drop mechanism in the absence of specified instructions, which can be extended to other datasets without annotated instructions.

We also conduct ablation experiments on multimodal datasets, and the experiments on multimodal datasets are trained for 20 rounds, just like the main experiments. The results we release are the w/ PT & Inst. results. From Table 17 and Figure 3, we can see that the performance with the label drop mechanism is better than without it, which demonstrates the effectiveness of the label drop mechanism in MIE tasks. Under different drop rates, the F1 scores on the MIE datasets do not fluctuate greatly, indicating that the label drop mechanism still has robustness in MIE