Towards Robust Universal Information Extraction: Dataset, Evaluation, and Solution
Authors: Jizhao Zhu, Akang Shi, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
Accepted at: acl25

Towards Robust Universal Information Extraction: Dataset, Evaluation, and Solution

Jizhao Zhu1,2, Akang Shi1,2, Zixuan Li1,3*, Long Bai1,3, Xiaolong Jin1,3,4*, Jiafeng Guo1,3,4, Xueqi Cheng1,3,4
1Key Laboratory of Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences
2School of Computer Science, Shenyang Aerospace University, Shenyang, China
3State Key Laboratory of AI Safety
4School of Computer Science, University of Chinese Academy of Sciences
{zhujz@sau.edu.cn, shiakang@stu.sau.edu.cn,lizixuan@ict.ac.cn}

Abstract

In this paper, we aim to enhance the robustness of Universal Information Extraction (UIE) by introducing a new benchmark dataset, a comprehensive evaluation, and a feasible solution. Existing robust benchmark datasets have two key limitations: 1) They generate only a limited range of perturbations for a single Information Extraction (IE) task, which fails to evaluate the robustness of UIE models effectively; 2) They rely on small models or handcrafted rules to generate perturbations, often resulting in unnatural adversarial examples. Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench, which utilizes LLMs to generate more diverse and realistic perturbations across different IE tasks. Based on this dataset, we comprehensively evaluate existing UIE models and reveal that both LLM-based models and other models suffer from significant performance drops. To improve robustness and reduce training costs, we propose a data-augmentation solution that dynamically selects hard samples for iterative training based on the model’s inference loss. Experimental results show that training with only \(15\%\) of the data leads to an average \(8.1\%\text{ relative}\) performance improvement across three IE tasks. Our code and dataset are available at: https://github.com/ICT-GoKnow/RobustUIE.

1 Introduction

Information Extraction (IE) aims to extract structured knowledge from unstructured text based on predefined types of entities, relations, and events. It plays a fundamental role in downstream applications such as knowledge graph construction (Ji et al., 2021), information retrieval (Zhu et al., 2023), and reasoning (Guan et al., 2020). Universal Information Extraction (UIE), which seeks to unify the extraction of various knowledge types through a single model, has achieved significant progress in recent years. Most existing studies have primarily focused on enhancing the overall performance of UIE models, typically evaluated on fixed test sets. However, they often overlook the robustness (and generalization ability) of UIE models, which are crucial when handling real-world text.

To measure the robustness of IE models, some studies focus on constructing benchmark datasets by generating adversarial examples with small perturbations. For example, RockNER (Lin et al., 2021) employs a rule-based approach and BERT (Devlin et al., 2019) to generate two kinds of perturbations for Named Entity Recognition (NER); Li et al. (2021) generate adversarial examples for Relation Extraction (RE) by random replacing words with synonyms or similar words generated by some Natural Language Processing (NLP) tools. Liu et al. (2020) replace verbs and context using similar words generated by GloVe (Pennington et al., 2014) for Event Detection (ED). Overall, existing benchmark datasets typically have two limitations: 1) They generate limited kinds of perturbations for individual IE tasks, making it difficult to comprehensively evaluate the robustness on UIE models across various IE tasks; 2) They generate adversarial examples typically using small models or handcrafted rules, often resulting in unnatural samples.

Considering the powerful NLP capabilities of Large Language Models (LLMs) (OpenAI., 2023; Qin et al., 2024; Li et al., 2024a), we leverage them in this paper to generate more diverse and realistic perturbations. After human verification of the annotation accuracy of LLMs, we obtain a new benchmark dataset for Robust UIE, called RUIE-Bench. RUIE-Bench contains \(\mathbf{12,700}\) samples and includes \(\mathbf{14}\) distinct kinds of perturbation across three mainstream IE tasks, i.e., NER, RE, and ED. Based on RUIE-Bench, we conduct a comprehen-
sive evaluation of existing UIE models, including 8 open-source LLMs, 6 closed-source LLMs, 4 traditional IE models, and 4 fine-tuned UIE models. We obtain some intriguing observations from the experimental results, such as open-source LLMs have a significant performance gap compared with closed-source LLMs in both original tests and perturbation tests. LLM-based UIE models demonstrate better robustness than traditional IE models under certain perturbations. However, both types of models suffer from significant performance drops.

To improve the robustness of UIE models, a common solution is data augmentation. Since the training cost is also a key factor, especially for LLM-based UIE models, we further propose a Loss-guided Data Augmentation (LDA) solution to enhance the robustness of models using a limited number of samples. Specifically, we first generate additional adversarial examples for training. Then, the inference loss on these samples is leveraged to dynamically select the most challenging ones to fine-tune the UIE model. Using the fine-tuned model, we iteratively calculate the inference loss and select hard samples for the next round of training. The experimental results demonstrate that training KnowCoder (Li et al., 2024b) with just 15% of the augmented data using LDA yields a **8.1%** relative improvement in average performance on RUIE-Bench, compared with the state-of-the-art models. This performance is comparable to the fully trained model using the entire augmented dataset. Additionally, when evaluated on the unseen dataset, KnowCoder with LDA outperforms the fully trained model by an average of **8.9%** across three IE tasks.

In summary, our contributions are as follows:

• We construct RUIE-bench, which contains **12,700** samples with **14** distinct perturbations generated by LLMs across various IE tasks, which is the most comprehensive benchmark dataset with the most diverse perturbations for robust UIE.

• Based on the RUIE bench, we comprehensively evaluate existing IE models. The evaluation results highlight that current IE models exhibit robustness issues against perturbations.

• To improve the robustness with limited samples, we further propose a loss-guided data augmentation solution, which achieves performance comparable to training with the full dataset by using only 15% of the data. Moreover, when evaluated on unseen datasets, LDA outperforms the fully trained model with **8.9%** F1 on average across three IE tasks.

2 Related Work

Universal Information Extraction can be classifier into two kinds of methods, classification-based (Lin et al., 2020a; Nguyen et al., 2022) and generation-based UIE methods (Lu et al., 2022; Wang et al., 2023; Li et al., 2024b). The former mainly adopts the end-to-end joint extraction mode, enhancing cross-task interactions with global dependency modeling for unified extraction. The latter aims to generate structural information rather than extracting structural information from plain text. Recently, some UIE methods employ various types of prompts to enable LLMs to understand schemas and extract the corresponding knowledge. For example, InstructUIE (Wang et al., 2023) applies a text-style prompt for schema understanding. In contrast, KnowCoder (Li et al., 2024b) uses a code-style prompt to transform UIE into code generation, achieving state-of-the-art performance.

Robustness Research for IE primarily focuses on constructing benchmark datasets for individual IE tasks. For NER, RockNER (Lin et al., 2021) replaces the original entities with entities of relevant types from Wikidata\footnote{https://www.wikidata.org}, and employs BERT (Devlin et al., 2019) to substitute context words; Jin et al. (2023) adopt disentanglement and word attribution methods to identify keywords and injects character-level perturbations for these words; Srinivasan and Vajjala (2023) apply multiple perturbations, including random entity replacement, Bert-based contextual modifications, and paraphrase generation. For RE, Li et al. (2021) generate adversarial examples by randomizing word substitutions using RoBERTa (Liu, 2019) or using synonyms; Nolano et al. (2024) generate adversarial examples by replacing entities in triples using various strategies, including same-type and different-type substitutions. For ED, Liu et al. (2020) use GloVe (Pennington et al., 2014) to replace similar words to generate adversarial examples.

To enhance the robustness of IE, the existing method (Lin et al., 2021) expands the original training data by generating additional examples.
Furthermore, Li et al. (2021); Jin et al. (2023); Srinivasan and Vajjala (2023) employ perturbation techniques to generate adversarial examples for training. While these methods typically require a large number of adversarial examples, none focus on improving model robustness using only a small number of augmented samples.

3 The Construction of RUIE-Bench

In this section, we first introduce the methods of utilizing LLMs to generate adversarial examples. Subsequently, we provide the concrete construction process of the RUIE-Bench dataset.

3.1 LLM-based Adversarial Example Generation

In UIE, adversarial examples refer to samples intentionally perturbed to mislead UIE models while maintaining their original semantics or appearance. Mathematically, given an input sample \( s \) with its corresponding label \( y \), an adversarial example \( s' \) is generated by applying a small perturbation, which is constrained to ensure that it remains close to the input space. The goal of adversarial examples is to cause a model to predict an incorrect label.

To generate more diverse and realistic adversarial examples, we employed LLMs to simulate different kinds of perturbations for NER, RE, and ED tasks by designing various prompts for different IE tasks. Additionally, we utilized two general rule-based perturbations, which were applied across all IE tasks. A comprehensive demonstration of these adversarial examples is presented in Figure 1. In what follows, we will introduce these different kinds of perturbations for IE tasks in detail.

Replace Entity, Triple, and Trigger. A robust UIE model should be able to identify entities, relations, and events based on the context of the sentence corresponding to the task rather than memorizing each entity, relational triple, or event trigger and their corresponding types. To prevent the model from memorizing specific patterns instead of reasoning based on context, we introduce perturbations to existing sentences.

Such perturbations need to ensure the consistency of the original type during replacement. Although previous studies (Lin et al., 2021; Nolano et al., 2024) have introduced similar perturbations in NER and RE samples, such as replacing entities based on rules, these methods may lead to incorrect replacement. Moreover, for ED samples, replacing triggers while ensuring the type remains unchanged is an extremely challenging task. Given this, we instruct GPT-4 (OpenAI., 2023) to replace entities, relational triples, or event triggers while preserving their types and ensuring that other content remains unchanged. The corresponding prompts are provided in Appendix A.

Change Context. A robust UIE model should keep its performance even when the contextual content of the sample changes due to various perturbations. To evaluate the robustness of the UIE model against contextual variations, we introduce perturbations to context words. It should be noted that this method is only used for samples of NER and ED tasks since altering context words in RE may disrupt the semantic relations between entity pairs.

In the previous methods, the mask language model BERT (Devlin et al., 2019) is used for changing context, but these methods often generate semantically inaccurate or syntactically incorrect words, and only one word can be generated for a single mask position. Therefore, we use LLMs to change the context in sentences. Specifically, we first get rid of punctuation, entities, event triggers, and stop words in sentences, leaving only meaningful context words. And randomly choose up to four words to replace with the [MASK] token. Then, GPT-4 (OpenAI., 2023) is instructed to generate three predictions for each [MASK] token and randomly select one for replacement. The prompts can be found in Appendix A.

Extend Sentence. Generally, a robust UIE model is capable of accurately extracting the required information, even in the face of complex sentence structures or long text situations. In order to better evaluate the robustness of the UIE model in handling complex sentences or long text situations, we enhance the semantic depth of the sentences by adding semantically relevant content (such as contextual details, historical facts, or explanatory clauses), thereby increasing the complexity of the sentences.

Prior research has not explored the robustness of IE models under similar perturbations. Considering that added content must maintain semantic coherence and meaningfulness, we employ LLMs to implement this perturbation. For specific tasks, we instruct GPT-4 (OpenAI., 2023) to adhere to corresponding constraints. For NER, new sentences must preserve the original entity boundaries
<table>
  <tr>
    <th>Task</th>
    <th>Perturbation Type</th>
    <th>Original Example</th>
    <th>Adversarial Example</th>
  </tr>
  <tr>
    <td rowspan="4">NER</td>
    <td>Replace Entity</td>
    <td>The University of Ferrara [ORG] dates back to 1391.</td>
    <td>The University of Salamanca [ORG] dates back to 1391.</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>Before 2008, their creation was not permitted within a London borough [LOC].</td>
    <td>Before conducting business, their activities was not permitted within a London borough [LOC].</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>The University of Ferrara [ORG] dates back to 1391.</td>
    <td>The University of Ferrara [ORG] dates back to 1391, making it one of the oldest universities in the world.</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>Before 2008, their creation was not permitted within a London borough [LOC].</td>
    <td>Before 2008, their creation was not permitted within a London borough [LOC].</td>
  </tr>
  <tr>
    <td rowspan="6">Lowercase Conversion</td>
    <td>Replace Triple</td>
    <td>Residence</td>
    <td>Residence</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>Vladimir Lebedev of Russia won the bronze medal.</td>
    <td>Vladimir Lebedev of Russia won the bronze medal, showcasing his exceptional skills in the competition.</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>Edward James, who grew up in East Los Angeles, has become a policeman.</td>
    <td>Edward James, who grew up in East Los Angeles, has become a policeman.</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>In recent months, Clairett had been in touch with Ohio State Coach Jim Tressel.</td>
    <td>In recent months, clairett had been in touch with ohio state coach jim tressel.</td>
  </tr>
  <tr>
    <td>Replace Trigger</td>
    <td>Putin last visited [Conflict] Bush at his Texas ranch in November 2001.</td>
    <td>Putin last encountered [Conflict] Bush at his Texas ranch in November 2001.</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>Another appeal [Justice] is now pending in the Federal Court.</td>
    <td>Supreme Court's appeal [Justice] is now pending in the Federal judicial system.</td>
  </tr>
  <tr>
    <td rowspan="3">ED</td>
    <td>Extend Sentence</td>
    <td>"War [Conflict] is not justified," Fischer told reporters.</td>
    <td>"War [Conflict] is not justified," Fischer told reporters. A controversial statement with widespread attention.</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>Putin last visited [Conflict] Bush at his Texas ranch in November 2001.</td>
    <td>Putin last visited [Conflict] Bush at his Texas ranch in November 2001.</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>Another appeal [Justice] is now pending in the Federal Court.</td>
    <td>Another appeal [Justice] is now pending in the Federal court.</td>
  </tr>
</table>

Figure 1: Illustration of generated adversarial examples with different kinds of perturbations.

and types without introducing new entities. For RE, the original relational triples must remain unchanged, with no additional relational information introduced. For ED, the original event triggers in sentences must be retained without incorporating new event information. The prompts are provided in Appendix A.

Typo Injection. In reality, typos are common. However, a robust UIE model should continue to make accurate predictions when dealing with these typos. To simulate these common text spelling mistakes, we introduce typo injection. Namely, spelling mistakes are added to the words prone to errors. Initially, we tried using LLMs to inject, but it often introduced unrealistic errors. Therefore, we switched to a rule-based approach to achieve this.

We focus on sentences with more than eight words and select longer words (over six characters), as longer sentences and words are more prone to spelling errors. Additionally, we filter out stop words and high-frequency vocabulary to avoid selecting common words. We randomly choose 1-3 words from the remaining words for typo injection and avoid changing the first character for each selected word and introduce errors by randomly replacing characters, deleting characters, inserting characters, or swapping adjacent characters.

Lowercase Conversion. Consider that in practice, users overwhelmingly spell in lowercase, especially in informal environments such as social media, email, or search queries. Therefore, lowercase conversion is used to simulate non-standard input, which helps evaluate the robustness of the UIE model in response to changes in text format.

In this method, all characters of each word are converted to lowercase, except for the first letter of the first word. This tests whether the model can still accurately extract information under non-standard input conditions, forcing it to rely on semantic understanding rather than surface features. By doing so, it not only assesses the model’s robustness but also highlights how upper and lowercase expressions affect task performance.

3.2 Dataset Construction

To construct the RUIE-Bench dataset, we select seven datasets across the three subtasks of UIE. For NER, we use the ACE05-Ent (Walker and Consortium, 2005), CoNLL03 (Sang and Meulder, 2003), and WikiANN (Pan et al., 2017) datasets; for RE, we select the ACE05-Rel (Walker and Consortium, 2005) and NYT (Riedel et al., 2010) datasets; for ED, we use the ACE05-Evt (Walker and Consortium, 2005) and CASIE (Satyapanich et al., 2020) datasets. To balance sample sizes across these subtasks, we perform stratified sampling from the test sets of respective datasets, adhering to the principle of maintaining distributional consistency. Specifically, we conduct stratified sampling on all label types (including NULL-type) within each test set, obtaining 1,000 NER samples, 800 RE samples,
<table>
  <tr>
    <th>Benchmark Dataset</th>
    <th>RUIE-Bench (Ours)</th>
    <th>RockNER (Lin et al., 2021)</th>
    <th>DWR (Jin et al., 2023)</th>
    <th>Adv.re (Nolano et al., 2024)</th>
    <th>CSMG (Liu et al., 2020)</th>
  </tr>
  <tr>
    <td>Supported tasks</td>
    <td>NER & RE & ED</td>
    <td>NER</td>
    <td>NER</td>
    <td>RE</td>
    <td>ED</td>
  </tr>
  <tr>
    <td>Covered datasets</td>
    <td>7</td>
    <td>1</td>
    <td>3</td>
    <td>1</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Number of perturbation types</td>
    <td>14</td>
    <td>3</td>
    <td>1</td>
    <td>4</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Number of adversarial examples</td>
    <td>12,700</td>
    <td>13,169</td>
    <td>-</td>
    <td>6,277</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Methods for generating perturbations</td>
    <td>LLM & Rule</td>
    <td>Small model & Rule</td>
    <td>Small model</td>
    <td>Small model & Rule</td>
    <td>Small model</td>
  </tr>
</table>

Table 1: Comparison between RUIE-Bench and existing IE robust evaluation benchmark datasets. The - indicates that the detailed statistics of the datasets are not reported in their papers.

and 900 ED samples. For adversarial example generation, we apply the perturbation methods detailed in Section 3.1. For each original sample, corresponding adversarial samples are generated for every perturbation type, while retaining those samples that cannot be perturbed. During generation, we implement strict quality control through manual verification: any sample containing errors is immediately discarded and regenerated until accurate samples are obtained. Through this rigorous process, we successfully construct the RUIE-Bench dataset. We present a comprehensive comparison between RUIE-Bench and existing IE robustness evaluation benchmark datasets in Table 1. For detailed statistics and further information regarding RUIE-Bench, please refer to Appendix B.

4 Loss-guided Data Augmentation

Data augmentation is a common strategy to improve model robustness (Rebuffi et al., 2021; Wang et al., 2022; Li and Spratling, 2023). Existing methods primarily focus on synthesizing in-distribution adversarial samples for training. However, none of them focus on improving efficiency by selecting a minimal number of training samples to enhance robustness. This is especially critical for LLMs, where training costs are significantly higher due to their scale and complexity.

Inspired by Song et al. (2023); Buchnik and Cohen (2020); Werner (2023), we propose a loss-guided data augmentation solution for robust UIE. The core idea is to focus on samples where the model’s performance is currently suboptimal, as indicated by higher loss values, thereby potentially accelerating convergence and improving overall model performance. First, we train the initial model on the original training set and use it to compute the inference loss of the augmented samples. For each IE task, samples with high inference loss were selected to fine-tune the model. Next, based on the obtained fine-tuned model, the inference loss of the augmented samples is recalculated, followed by another round of sample selection and fine-tuning. This iterative process was repeated t times until a robust model was obtained.

Specifically, we first fine-tune the initial model \( M \) using the original training data to obtain \( M_0 \), and employ LLMs to generate augmented data \( D_{aug} \) for the original training set. During each iteration, we use \( M_{t-1} \) to compute inference loss \( L_i \) for each augmented sample. Then, based on selection ratio \( \beta \), the samples with higher loss are selected to form a new training dataset \( D^{(t)}_{retrain} \), which is subsequently used to fine-tune \( M_{t-1} \) to obtain model \( M_t \). After each iteration, the model is evaluated on the validation set. The algorithm terminates and returns the final model \( M_t \) when the improvement on the validation set falls below the convergence threshold \( \delta \). Algorithm 1 presents the training process of the proposed strategy. The details of augmented data generation are presented in Appendix C.

Algorithm 1 Loss-guided Data Augmentation

Input: Training data \( D \), Initial model \( M \), Selection ratio \( \beta \), Convergence threshold \( \delta \)

Output: Fine-tuned model \( M_t \)

1: Use LLMs to generate augmented data \( D_{aug} \) based on \( D \);
2: Fine-tune \( M \) on data \( D \) to obtain model \( M_0 \);
3: \( t \leftarrow 1 \);
4: do
5:   for each sample \( (x_i, y_i) \in D_{aug} \) do
6:     Compute loss: \( L_i = L(\theta_{t-1}; x_i, y_i) \)
7:   end for
8:   Sort samples in \( D_{aug} \) by loss \( L_i \) in descending order;
9:   Select top \( \beta \) samples to form \( D^{(t)}_{retrain} \);
10:  Fine-tune model \( M_{t-1} \) on \( D^{(t)}_{retrain} \) to obtain new model \( M_t \);
11: \( \beta \leftarrow \beta/2;\quad t \leftarrow t + 1 \);
12: while performance improvement of \( M_t \) on the validation set falls below threshold \( \delta \);
13: return \( M_t \)
5 Experiment Setup

We use RUIE-Bench to evaluate the robustness of the current models, including UIE ones, traditional IE ones, and LLMs. Meanwhile, we construct an unseen dataset to measure their generalization ability.

5.1 Metrics

We report the span-based offset Micro F1, following previous methods (Lu et al., 2022; Lin et al., 2020b). For NER, an entity is considered correct if both its boundaries and type are accurately predicted. For RE, a relation is deemed correct if the triple, including the relation type, head entity, and tail entity, matches the gold annotations. For ED, an event trigger is considered correct if both the event type and trigger are aligned with the gold annotations.

5.2 Training Details

To explore effective data augmentation methods, the KnowCoder-7B-base2 is selected as the initial model \( M \). Then, we fine-tune it on the seven datasets constructed for RUIE-Bench, obtaining the model \( M_0 \), which is denoted as KnowCoder-7B_partial. Additionally, the perturbation methods proposed in Section 3.1 are employed to generate augmented data \( D_{aug} \) based on the original training set. Next, we fine-tune the initial model using both original training data and all augmented data, obtaining the model KnowCoder-7B-Robust. Finally, we fine-tune the initial model using high-loss augmented samples selected according to the LDA strategy, with an initial selection ratio of 10%. After two iterations, we successfully construct the model KnowCoder-7B-Robust_LDA, utilizing a total of 15% augmented samples.

During the fine-tuning phase, we employ LoRA (Hu et al., 2021) for efficient parameter tuning. The LoRA rank and LoRA alpha parameters are set to 32 and 64, respectively. The learning rate is set to \( 3 \times 10^{-4} \), with a warm-up rate of 0.1 and a dropout rate of 0.1. The sequence length is limited to 2048 and the batch size is set to 256. Additionally, for LDA training, the selection ratio \( \beta \) is set to 10%, and the convergence threshold \( \delta \) is set to 0.3, meaning that iteration stops when the Micro F1 score improvement of the new model is less than 0.3. During validation phase, we use greedy search with a temperature of 0 and set the maximum output length to 640. All experiments are conducted on 8 × NVIDIA-A100 80G.

5.3 Baselines

We adopt the state-of-the-art UIE models to validate the robustness, including UIE (Lu et al., 2022), IstructUIE (Wang et al., 2023), YAYI-UIE (Xiao et al., 2024), and KnowCoder (Li et al., 2024b). We also employ traditional IE models for robustness evaluation across the NER, RE, and ED tasks. For NER, we choose Stanza (Qi et al., 2020) and TNER (Ushio and Camacho-Collados, 2021); for RE, we select PFN (Yan et al., 2021); and for ED, we choose EEQA (Du and Cardie, 2020). Additionally, we evaluate the robustness of two categories of LLMs: open-source models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct(Yang et al., 2024), Llama3-8B-Instruct (Dubey et al., 2024), Glm-4-9B-Chat (GLM et al., 2024), CodeLlama-7B-Instruct (Rozière et al., 2024), Internlm2.5-7B-Chat (Team, 2023), and Vicuna-7B-v1.5 (Zheng et al., 2023); Commercial model API services, including GPT-3.5-turbo, GPT-4-turbo (OpenAI., 2023), DeepSeek-V3 (Liu et al., 2024), DeepSeek-R1 (Guo et al., 2025), GLM4-Plus (GLM et al., 2024) and Qwen2.5-Max (Yang et al., 2024). For the evaluation of LLMs, we employ the 10-shot approach to instruct LLMs to conduct IE tasks with the specific prompts provided in Appendix D.

6 Results and Analyses

6.1 Results on the RUIE-Bench dataset

We report the Micro F1 scores of all the models across the three IE tasks on RUIE-Bench in Table 2. These results cover both the original test set and various perturbation settings. For the sake of space limitation in the table, we use abbreviations for these perturbations. For NER, we use P1-P5 to represent the perturbations of Replace Entity, Change Context, Extend Sentence, Typo Injection, and Lowercase Conversion, respectively. For RE, the perturbations of Replace Triple, Extend Sentence, Typo Injection, and Lowercase Conversion are denoted as P6-P9, respectively. For ED, we employ P10-P14 to represent Replace Trigger, Change Context, Extend Sentence, Typo Injection, and Lowercase Conversion, respectively. We also report the overall performance drop of all the models under the three tasks for all perturbations in
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">NER</th>
    <th colspan="5">RE</th>
    <th colspan="5">ED</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>None</th><th>P1</th><th>P2</th><th>P3</th><th>Drop_avg</th>
    <th>None</th><th>P6</th><th>P7</th><th>P8</th><th>Drop_avg</th>
    <th>None</th><th>P10</th><th>P11</th><th>P12</th><th>P14</th><th>Drop_avg</th>
    <th>Avg</th>
    <th>Rank</th>
  </tr>
  <tr>
    <td colspan="19">Opennnse LLMs</td>
  </tr>
  <tr>
    <td>Qwen-3.5-16-Instruction</td><td>58.6</td><td>51.6</td><td>37.7</td><td>55.0</td><td>46.6</td><td>7.9%</td><td>22.6</td><td>19.2</td><td>21.3</td><td>17.1</td><td>8.8</td><td>26.5%</td><td>29.8</td><td>27.5</td><td>28.3</td><td>26.5</td><td>28.5</td><td>4.9%</td><td>34.6</td><td>15</td>
  </tr>
  <tr>
    <td>Qwen-2.7B-Instruction</td><td>53.3</td><td>49.4</td><td>51.2</td><td>50.5</td><td>51.2</td><td>4.1%</td><td>15.6</td><td>13.4</td><td>14.0</td><td>13.8</td><td>3.8</td><td>27.9%</td><td>18.5</td><td>18.0</td><td>17.4</td><td>17.4</td><td>17.6</td><td>1.8%</td><td>27.3</td><td>15</td>
  </tr>
  <tr>
    <td>Qwen-2.5B-Instruction</td><td>49.5</td><td>47.5</td><td>46.7</td><td>45.3</td><td>45.5</td><td>40.2</td><td>9.0%</td><td>8.9</td><td>7.6</td><td>8.6</td><td>7.4</td><td>2.0</td><td>28.1%</td><td>11.9</td><td>11.4</td><td>10.7</td><td>11.2</td><td>11.0</td><td>1.6%</td><td>22.2</td><td>18</td>
  </tr>
  <tr>
    <td>Llama-2-7B-Instruct</td><td>55.4</td><td>52.6</td><td>52.9</td><td>51.1</td><td>53.5</td><td>25.7</td><td>14.9%</td><td>17.3</td><td>15.0</td><td>15.7</td><td>13.6</td><td>2.5</td><td>32.4%</td><td>11.9</td><td>11.8</td><td>11.9</td><td>10.5</td><td>11.3</td><td>1.0%</td><td>24.9</td><td>16</td>
  </tr>
  <tr>
    <td>Glm-4B-Chat</td><td>57.2</td><td>54.0</td><td>55.8</td><td>51.8</td><td>56.6</td><td>43.2</td><td>9.0%</td><td>8.8</td><td>7.4</td><td>7.8</td><td>1.8</td><td>31.2%</td><td>16.6</td><td>8.0</td><td>6.8</td><td>5.5</td><td>6.1</td><td>10.5%</td><td>21.1</td><td>17</td>
  </tr>
  <tr>
    <td>Qwen-2.5B-Chat</td><td>54.6</td><td>51.6</td><td>52.6</td><td>51.3</td><td>52.6</td><td>40.1</td><td>7.8%</td><td>13.3</td><td>11.3</td><td>10.1</td><td>9.8</td><td>1.3</td><td>28.0%</td><td>10.4</td><td>9.4</td><td>9.0</td><td>8.0</td><td>8.4</td><td>10.4%</td><td>20.0</td><td>19</td>
  </tr>
  <tr>
    <td>CodeLlama-7B-Instruct</td><td>46.3</td><td>45.0</td><td>45.0</td><td>38.9</td><td>42.4</td><td>14.5</td><td>19.7%</td><td>13.7</td><td>11.6</td><td>12.2</td><td>11.3</td><td>2.8</td><td>30.8%</td><td>9.5</td><td>9.6</td><td>9.4</td><td>7.1</td><td>8.6</td><td>9.8</td><td>6.3%</td><td>19.8</td><td>20</td>
  </tr>
  <tr>
    <td>Vicuna-7B-v1.5</td><td>39.0</td><td>38.2</td><td>37.4</td><td>35.0</td><td>38.0</td><td>16.7</td><td>15.2%</td><td>11.2</td><td>11.0</td><td>10.1</td><td>7.6</td><td>0.8</td><td>34.1%</td><td>6.9</td><td>7.0</td><td>6.8</td><td>5.4</td><td>5.8</td><td>6.2</td><td>9.5%</td><td>16.7</td><td>21</td>
  </tr>
  <tr>
    <td colspan="19">Commercial LLMs API Services</td>
  </tr>
  <tr>
    <td>DeepSeekR1</td><td>61.1</td><td>63.2</td><td>66.8</td><td>65.0</td><td>65.0</td><td>57.2</td><td>5.2%</td><td>37.4</td><td>33.1</td><td>35.4</td><td>32.2</td><td>20.6</td><td>18.9%</td><td>45.8</td><td>41.8</td><td>45.5</td><td>44.1</td><td>45.7</td><td>3.4%</td><td>47.7</td><td>8</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td><td>62.3</td><td>59.8</td><td>61.5</td><td>61.3</td><td>58.7</td><td>55.0</td><td>4.9%</td><td>31.3</td><td>29.0</td><td>29.6</td><td>26.2</td><td>10.0</td><td>24.3%</td><td>35.3</td><td>34.6</td><td>34.5</td><td>31.4</td><td>32.4</td><td>3.5%</td><td>40.5</td><td>11</td>
  </tr>
  <tr>
    <td>Qwen-2.5-Max</td><td>64.0</td><td>56.6</td><td>63.8</td><td>62.4</td><td>61.0</td><td>58.1</td><td>5.6%</td><td>34.9</td><td>30.9</td><td>33.9</td><td>28.3</td><td>13.2</td><td>23.8%</td><td>38.8</td><td>36.3</td><td>38.0</td><td>38.1</td><td>37.1</td><td>3.8%</td><td>43.2</td><td>9</td>
  </tr>
  <tr>
    <td>GLM4-Plus</td><td>63.2</td><td>59.8</td><td>63.0</td><td>61.0</td><td>60.9</td><td>49.7</td><td>6.6%</td><td>32.2</td><td>29.2</td><td>31.3</td><td>26.1</td><td>5.3</td><td>28.6%</td><td>37.5</td><td>34.9</td><td>37.4</td><td>30.3</td><td>35.1</td><td>3.7%</td><td>40.9</td><td>10</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>57.8</td><td>57.5</td><td>59.8</td><td>58.2</td><td>58.2</td><td>33.4</td><td>13.6%</td><td>13.3</td><td>10.0</td><td>11.6</td><td>8.4</td><td>4.5</td><td>29.6%</td><td>35.7</td><td>33.9</td><td>33.5</td><td>31.5</td><td>33.3</td><td>3.5%</td><td>38.0</td><td>12</td>
  </tr>
  <tr>
    <td>Qwen-2.5-turbo</td><td>57.8</td><td>57.5</td><td>59.8</td><td>59.5</td><td>59.9</td><td>33.1</td><td>15.3%</td><td>13.8</td><td>10.6</td><td>11.7</td><td>8.5</td><td>5.4</td><td>35.9%</td><td>31.4</td><td>28.6</td><td>29.8</td><td>25.6</td><td>27.4</td><td>3.0%</td><td>35.1</td><td>14</td>
  </tr>
  <tr>
    <td colspan="19">Traditional IE Models</td>
  </tr>
  <tr>
    <td>Stanza</td><td>80.7</td><td>70.1</td><td>77.1</td><td>71.5</td><td>78.1</td><td>51.1</td><td>13.8%</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TNER</td><td>83.0</td><td>73.3</td><td>78.0</td><td>73.9</td><td>81.0</td><td>73.2</td><td>8.6%</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>PPN</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>76.3</td><td>58.6</td><td>73.8</td><td>68.4</td><td>20.4</td><td>27.5%</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DEQA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>67.5</td><td>52.7</td><td>63.7</td><td>63.2</td><td>58.1</td><td>10.9%</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td colspan="19">UIE Models</td>
  </tr>
  <tr>
    <td>UIE</td><td>83.9</td><td>74.3</td><td>81.1</td><td>78.6</td><td>81.7</td><td>70.3</td><td>8.7%</td><td>84.1</td><td>63.1</td><td>81.1</td><td>77.0</td><td>75.9</td><td>23.5%</td><td>70.5</td><td>52.5</td><td>65.0</td><td>65.7</td><td>65.2</td><td>6.4%</td><td>89.8</td><td>5</td>
  </tr>
  <tr>
    <td>OntoIE-11B</td><td>73.9</td><td>65.7</td><td>69.3</td><td>68.2</td><td>70.3</td><td>75.2</td><td>7.5%</td><td>68.4</td><td>48.3</td><td>66.3</td><td>56.1</td><td>56.1</td><td>51.5%</td><td>59.3</td><td>49.2</td><td>55.9</td><td>58.2</td><td>57.5</td><td>5.8%</td><td>70.4</td><td>6</td>
  </tr>
  <tr>
    <td>AYAI-UIE-13B</td><td>80.7</td><td>69.7</td><td>75.3</td><td>72.6</td><td>79.2</td><td>75.6</td><td>7.8%</td><td>66.4</td><td>47.3</td><td>65.0</td><td>59.8</td><td>38.4</td><td>20.7%</td><td>43.8</td><td>36.8</td><td>41.9</td><td>41.3</td><td>43.2</td><td>6.8%</td><td>67.5</td><td>7</td>
  </tr>
  <tr>
    <td>KnowCoder-7B</td><td><b>87.4</b></td><td>76.4</td><td>81.3</td><td>79.6</td><td>84.7</td><td>81.5</td><td>7.7%</td><td>84.0</td><td>57.3</td><td>80.5</td><td>76.4</td><td>73.3</td><td>14.4%</td><td><b>72.9</b></td><td>53.5</td><td><b>68.5</b></td><td>69.2</td><td><b>68.3</b></td><td>7.0%</td><td>74.4</td><td>3</td>
  </tr>
  <tr>
    <td>KnowCoder-7B_{small}</td><td>84.4</td><td>73.8</td><td>80.1</td><td>81.1</td><td>81.2</td><td>79.1</td><td>6.1%</td><td>81.4</td><td>60.6</td><td>79.1</td><td>74.5</td><td>52.8</td><td>18.0%</td><td>68.2</td><td>54.6</td><td>65.1</td><td>64.5</td><td>64.7</td><td>6.1%</td><td>71.3</td><td>4</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-Robust</td><td>85.9</td><td>81.3</td><td>83.5</td><td>86.4</td><td>86.1</td><td>84.7</td><td>1.6%</td><td>83.1</td><td>66.0</td><td><b>82.9</b></td><td>81.1</td><td>79.8</td><td>6.8%</td><td>69.8</td><td>65.5</td><td>67.2</td><td><b>69.9</b></td><td>68.0</td><td>6.9%</td><td><b>77.1</b></td><td>1</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-Robust_{small}</td><td>86.1</td><td>81.2</td><td>84.9</td><td>86.5</td><td>85.6</td><td>83.8</td><td>1.9%</td><td>82.2</td><td>66.5</td><td>82.5</td><td>81.3</td><td>81.3</td><td>5.2%</td><td>69.1</td><td>65.7</td><td>67.9</td><td>68.5</td><td>68.8</td><td>1.5%</td><td>77.1</td><td>1</td>
  </tr>
</table>

Table 2: The performance of all baselines and our models on RUIE-Bench.

the “Drop_avg” column. The robustness evaluation results of all the models are ranked, and the final ranking is shown in the “Rank” column. Although there are differences in the evaluation methods employed by different categories of models, we can still draw some interesting observations from the results:

(1) The models with data augmentation training show the best performance. The model KnowCoder-7B-Robust_{LDA} trained with only 15% of the augmented data using LDA achieves results comparable with KnowCoder-7B-Robust. It convincingly verifies the effectiveness of the proposed LDA training strategy. Furthermore, a comprehensive comparison between these two models is in Appendix E.

(2) LLM-based models experience relatively smaller performance drops than other models, suggesting that LLMs have stronger generalization ability. This indicates that using LLMs to improve the robustness of UIE models is a promising approach for future work.

(3) All the LLMs exhibit a significant performance drop under various perturbations, especially in the NER and RE tasks. This indicates that LLMs face serious robustness issues when dealing with UIE tasks in few-shot prompting scenarios. However, LLM-based reasoning models such as DeepSeekR1 and Qwen2.5-Max demonstrate relatively better robustness compared to other LLMs, suggesting that incorporating stronger reasoning capabilities may enhance the stability of few-shot UIE performance under perturbations.

(4) From the results of the Qwen models with different parameter scales, it is evident that there is a significant positive correlation between the model’s parameter scale and its robustness in NER and RE tasks. In other words, an increase in the number of model parameters often accompanies an improvement in robustness. For ED, although there is no clear positive correlation between parameter scale and robustness, the smallest model still demonstrates the weakest robustness.

6.2 Results on the Unseen Dataset

To verify whether the model trained on data with different perturbations can generalize to unseen datasets, we create an unseen dataset that does not include any samples from RUIE-Bench. Furthermore, we ensure that the types in this dataset are a subset of those in RUIE-Bench. For NER, we select OntoNotes 5.0 (Hovy et al., 2006) and random sample some instances as unseen data. Similarly, we obtain the unseen data for RE from CoNLL04 (Roth and tau Yih, 2004) and GIDS (Jat et al., 2018). For the ED task, since no datasets with the same event types exist, we use GPT-4 (OpenAI., 2023) to generate 100 unseen samples, which are then manually verified for correctness.

Table 3 shows the results of different models on the unseen dataset. From the table, we can find that: 1) The two LLM-based reasoning models, DeepSeekR1 and Qwen2.5-Max, show rela-
Figure 2: Performance comparison of different models under various perturbations on NER datasets. Red and blue indicate performance drop and improvement, respectively. KC is short for the KnowCoder model.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Unseen Dataset</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>NER</th>
    <th>RE</th>
    <th>ED</th>
  </tr>
  <tr>
    <td>GLM4-Plus</td>
    <td>56.6</td>
    <td>41.4</td>
    <td>56.1</td>
    <td>51.4</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>59.2</td>
    <td>39.8</td>
    <td>55.5</td>
    <td>51.5</td>
  </tr>
  <tr>
    <td>GPT-4-turbo</td>
    <td>58.9</td>
    <td>38.8</td>
    <td>59.7</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-Max</td>
    <td>62.1</td>
    <td>43.4</td>
    <td>59.8</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>63.0</td>
    <td>43.6</td>
    <td>60.0</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>KnowCoder-7B_partial</td>
    <td>64.9</td>
    <td>40.4</td>
    <td>52.2</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-Robust</td>
    <td>62.8</td>
    <td>47.4</td>
    <td>56.6</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-RobustLDA</td>
    <td><b>67.2</b></td>
    <td><b>54.6</b></td>
    <td><b>60.1</b></td>
    <td><b>60.6</b></td>
  </tr>
</table>

Table 3: Results on the unseen dataset.

tively better generalization ability compared to the other LLMs. 2) Compared with the models without data augmentation training, the average performance of the models of KnowCoder-7B-Robust and KnowCoder-7B-RobustLDA in the UIE tasks is significantly improved. It is justified that training with the adversarial examples can enhance the model’s generalization ability. 3) It is worth noting that the KnowCoder-7B-RobustLDA trained with only 15% of the augmented data using the LDA strategy achieves an average 8.9% performance improvement compared with the KnowCoder-7B-Robust using the complete set of augmented data fine-tuning. We guess that full training will lead to overfitting and thus show poor prediction ability on the unseen dataset.

6.3 Detailed Analysis

To further verify the effect of different perturbations on the traditional models and UIE models, we report the performance drop and improvement under various perturbations on the seven datasets in RUIE-Bench. For the NER, RE, and ED datasets, the detailed results are shown in Figure 2, Figure 3, and Figure 4, respectively. Through analysis, we can summarize the following observations.

(1) Under the three different perturbations of

Figure 3: Performance comparison of different models under various perturbations on RE datasets.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">ACE05-RE</th>
    <th colspan="3">NYT</th>
  </tr>
  <tr>
    <th>NER</th>
    <th>RE</th>
    <th>ED</th>
    <th>NER</th>
    <th>RE</th>
    <th>ED</th>
  </tr>
  <tr>
    <td>GLM4-Plus</td>
    <td>64.2</td>
    <td>56.6</td>
    <td>-</td>
    <td>54.8</td>
    <td>59.3</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>53.4</td>
    <td>57.7</td>
    <td>-</td>
    <td>59.9</td>
    <td>1.6</td>
    <td>PA</td>
    <td>2.34</td>
    <td>2.68</td>
    <td>2.69</td>
    <td>2.55</td>
    <td>3.35</td>
    <td>2.15</td>
  </tr>
  <tr>
    <td>GPT-4-turbo</td>
    <td>57.2</td>
    <td>2.9</td>
    <td>-</td>
    <td>51.2</td>
    <td>2.8</td>
    <td>PA</td>
    <td>2.5</td>
    <td>2.0</td>
    <td>1.9</td>
    <td>2.9</td>
    <td>2.0</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-Max</td>
    <td>62.0</td>
    <td>8.0</td>
    <td>-</td>
    <td>63.2</td>
    <td>2.0</td>
    <td>PA</td>
    <td>8.5</td>
    <td>6.7</td>
    <td>9.1</td>
    <td>8.6</td>
    <td>8.2</td>
    <td>2.1</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>59.1</td>
    <td>1.6</td>
    <td>-</td>
    <td>57.7</td>
    <td>2.3</td>
    <td>PA</td>
    <td>8.0</td>
    <td>6.9</td>
    <td>16.4</td>
    <td>3.7</td>
    <td>14.3</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>KnowCoder-7B_partial</td>
    <td>64.9</td>
    <td>40.4</td>
    <td>52.2</td>
    <td>52.5</td>
    <td>52.5</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-Robust</td>
    <td>62.8</td>
    <td>47.4</td>
    <td>56.6</td>
    <td>55.6</td>
    <td>55.6</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td>KnowCoder-7B-RobustLDA</td>
    <td><b>67.2</b></td>
    <td><b>54.6</b></td>
    <td><b>60.1</b></td>
    <td><b>60.6</b></td>
    <td><b>60.6</b></td>
    <td><b>60.6</b></td>
  </tr>
</table>

Figure 4: Performance comparison of different models under various perturbations on ED datasets.

Replace Entity (P1), Replace Triple (P6), and Replace Trigger (P10) to the original test set results in a significant drop in model performance. This suggests that during training, the models may have memorized specific patterns of entities, relations, or events, rather than learning to reason based on contextual information. We provide representative case studies for all perturbations in Appendix F, including Replace Entity, Triple, and Trigger.

(2) Under certain perturbation settings, such as Lowercase Conversion, we observe that the model’s performance drops significantly on some datasets while other datasets remain unaffected. This is because the annotations in the affected datasets contain a relatively high number of uppercase characters. Additionally, we find that the models using LLMs show better performance on these datasets. This suggests that LLMs already have strong generalization abilities to handle such
simple noise.

(3) KnowCoder-7B-Robust_{LDA} shows a remarkable improvement across most of the datasets under nearly all perturbations. This observation strongly supports the effectiveness and feasibility of the LDA strategy. Furthermore, an interesting finding is observed in ACE05-Ent, WikiANN, and ACE05-Evt datasets: some models without data augmentation training (such as InstructUIE and YAYI-UIE) show similar robustness to KnowCoder-7B-Robust_{LDA}. This is because the performance of these models on the original test set is relatively poor, and thus, the perturbations have little effect on their performance.

7 Conclusion

In this paper, we introduced RUIE-Bench, a benchmark dataset designed to evaluate the robustness of UIE models. The dataset includes 14 adversarial perturbations for three core IE tasks, i.e., NER, RE, and ED. Through comprehensively benchmarking of existing models, the results reveal that these models struggle to maintain robustness when faced with these adversarial perturbations, highlighting the urgent need for robustness improvement for UIE. Motivated by this, we proposed a Loss-guided Data Augmentation (LDA) method that iteratively selects challenging samples for training. The results demonstrate that LDA achieves performance comparable to fully trained models on RUIE-Bench and even exhibits superior generalization capabilities on unseen datasets. This work aims to provide a valuable benchmark for evaluating robustness in UIE tasks and offer a practical methodology for enhancing model robustness.

Limitations

Generating more realistic perturbations remains an exploratory direction for future work. Although we propose various perturbation generation methods for UIE, they still fail to cover the diverse noise present in real-world scenarios. Meanwhile, due to cost and resource constraints, we have not conducted robustness evaluations on more LLMs. Moreover, the performance improvement achieved by the loss-guided data augmentation method may be constrained by the quality of the augmented data. In addition to these technical limitations, more robust UIE systems may also introduce societal risks, such as misuse for misinformation, surveillance, or amplification of bias. Addressing both the technical and ethical challenges will be a priority for future work, including more realistic perturbation design, broader model evaluation, and responsible deployment practices.

Acknowledgements

This work is funded by the National Natural Science Foundation of China under grants 62306299, 62441229 and 62406308, the CAS Special Research Assistant Fund, Lenovo-CAS Joint Lab Youth Scientist Project, the project under Grants No. JCKY2022130C039, and the Strategic Priority Research Program of the CAS under Grants No. XDB0680102. We thank anonymous reviewers for their insightful comments and suggestions.

References

Eliav Buchnik and Edith Cohen. 2020. Graph learning with loss-guided training. In Proceedings of the 3rd Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA), pages 1–13.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671–683, Online. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jaurhi, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.

Saiping Guan, Xiaolong Jin, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng. 2020. NeuInfer: Knowledge inference on N-ary facts. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6141–6151, Online. Association for Computational Linguistics.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma,
Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Eduard H. Hovy, Mitchell P. Marcus, Martha Palmer, Lance A. Ramshaw, and Ralph M. Weischedel. 2006. Ontonotes: The 90% solution. In North American Chapter of the Association for Computational Linguistics.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar. 2018. Improving distantly supervised relation extraction using word and entity based attention. Preprint, arXiv:1804.06987.

Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S Yu. 2021. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514.

Xiaomeng Jin, Bhanukiran Vinzamuri, Sriram Venkatapathy, Heng Ji, and Pradeep Natarajan. 2023. Adversarial robustness for large language NER models using disentanglement and word attributions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12437–12450, Singapore. Association for Computational Linguistics.

Huaxia Li, Haoyun Gao, Chengzhang Wu, and Miklos A Vassarhelyi. 2024a. Extracting financial data from unstructured sources: Leveraging large language models. Journal of Information Systems, pages 1–22.

Lin Li and Michael Spratling. 2023. Data augmentation alone can improve adversarial training. arXiv preprint arXiv:2301.09879.

Luqiu Li, Xiang Chen, Hongbin Ye, Zhen Bi, Shumin Deng, Ningyu Zhang, and Huajun Chen. 2021. On robustness and bias analysis of bert-based relation extraction. In Knowledge Graph and Semantic Computing: Knowledge Graph Empowers New Infrastructure Construction: 6th China Conference, CCKS 2021, Guangzhou, China, November 4-7, 2021, Proceedings 6, pages 43–59. Springer.

Zixuan Li, Yutao Zeng, Yuxin Zuo, Weicheng Ren, Wenxuan Liu, Miao Su, Yucan Guo, Yantao Liu, Lixiang Liang, Zhilei Hu, Long Bai, Wei Li, Yidan Liu, Pan Yang, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2024b. KnowCoder: Coding structured knowledge into LLMs for universal information extraction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8758–8779, Bangkok, Thailand. Association for Computational Linguistics.

Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. RockNER: A simple method to create adversarial examples for evaluating the robustness of named entity recognition models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3728–3737, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020a. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999–8009, Online. Association for Computational Linguistics.

Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020b. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999–8009, Online. Association for Computational Linguistics.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengji Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Jian Liu, Yubo Chen, Kang Liu, Yantao Jia, and Zhicheng Sheng. 2020. How does context matter? on the robustness of event detection with context-selective mask generalization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2523–2532, Online. Association for Computational Linguistics.

Yinhan Liu. 2019. Roberta: A robustly optimized bert prettraining approach. arXiv preprint arXiv:1907.11692, 364.

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755–5772, Dublin, Ireland. Association for Computational Linguistics.

Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022. Learning cross-task dependencies for joint extraction of entities, events, event arguments, and relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9349–9360, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Gennaro Nolano, Moritz Blum, Basil Ell, and Philipp Cimiano. 2024. Pointing out the shortcomings of relation extraction models with semantically motivated adversaries. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12809–12820, Torino, Italia. ELRA and ICCL.
OpenAI. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics.

Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819.

Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. 2021. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935–29948.

Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In ECML/PKDD.

Dan Roth and Wen tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Conference on Computational Natural Language Learning.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artiom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossé, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code llama: Open foundation models for code. Preprint, arXiv:2308.12950.

Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. Preprint, arXiv:cs/0306050.

Taneya Satyapanich, Francis Ferraro, and Tim Finin. 2020. Casie: Extracting cybersecurity event information from text. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8749–8757.

Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. 2023. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, pages 32483–32498. PMLR.

Akshay Srinivasan and Sowmya Vajjala. 2023. A multilingual evaluation of NER robustness to adversarial inputs. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 40–53, Toronto, Canada. Association for Computational Linguistics.

InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities.

Asahi Ushio and Jose Camacho-Collados. 2021. T-NER: An all-round python library for transformer-based named entity recognition. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 53–62, Online. Association for Computational Linguistics.

C. Walker and Linguistic Data Consortium. 2005. ACE 2005 Multilingual Training Corpus. LDC corpora. Linguistic Data Consortium.

Haohan Wang, Zeyi Huang, Xindi Wu, and Eric Xing. 2022. Toward learning robust and invariant representations with alignment regularization and data augmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1846–1856.

Xiao Wang, Wei Zhou, Can Zu, Han Xia, Tianze Chen, Yuan Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jiuhua Kang, J. Yang, Siyuan Li, and Chun-sai Du. 2023. Instructue: Multi-task instruction tuning for unified information extraction. Preprint, arXiv:2304.08085.

Tino Werner. 2023. Loss-guided stability selection. Advances in Data Analysis and Classification, pages 1–26.

Xinglin Xiao, Yijie Wang, Nan Xu, Yuqi Wang, Hanxuan Yang, Minzheng Wang, Yin Luo, Lei Wang, Wenji Mao, and Daniel Zeng. 2024. Yayi-uie: A chat-enhanced instruction tuning framework for universal information extraction. Preprint, arXiv:2312.15548.

Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. 2021. A partition filter network for joint entity and relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 185–197, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.

A Prompts for Adversarial Example Generation

Replace Entity, Triple, and Trigger To generate adversarial examples, we replace the entities, relation triples, and event triggers in the samples following specific rules designed for LLMs. The prompts used are shown in Figure 5.

Change Context We change the context of NER and ED samples based on the rules mentioned in Section 3.1 to generate high-quality adversarial examples. The prompt used for this context modification is presented in Figure 6.

Extend Sentence Using the prompt in Figure 7, we generate extended versions of NER, RE, and ED samples.

B RUIE-Bench Details

Sampling Details Considering that generating adversarial examples for all test sets of NER, RE and ED tasks would result in a large number of samples and costs, as well as significant evaluation expenses, we conduct sampling on the test sets of the selected datasets to balance evaluation costs and accuracy. The main principle we follow is to ensure that the sampled subsets maintain the same distribution. Specifically, for NER, we select 1,000 samples, including 134 from ACE05-Ent, 294 from CoNLL03, and 572 from WikiANN. For RE, we choose 800 samples, with 230 from ACE05-Rel and 570 from NYT. For ED, we choose 900 samples, with 676 from ACE05-Evt and 224 from CASIE.

Statistics Based on the sampled data, we utilize the perturbation methods described in Section 3.1 to generate adversarial examples, which are then used to construct the RUIE-Bench dataset. The detailed statistics of the resulting dataset are presented in Table 4.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Original Data</th>
    <th colspan="2">RUIE-Bench Data</th>
  </tr>
  <tr>
    <th>Type</th>
    <th>Test size</th>
    <th>Type</th>
    <th>Sampling size</th>
    <th>Data size</th>
  </tr>
  <tr>
    <td rowspan="3">NER</td>
    <td>ACE05</td>
    <td>7</td>
    <td>1,060</td>
    <td>7</td>
    <td>134</td>
    <td>670</td>
  </tr>
  <tr>
    <td>CoNLL03</td>
    <td>4</td>
    <td>3,453</td>
    <td>4</td>
    <td>294</td>
    <td>1,470</td>
  </tr>
  <tr>
    <td>WikiANN</td>
    <td>3</td>
    <td>10,000</td>
    <td>3</td>
    <td>572</td>
    <td>2,860</td>
  </tr>
  <tr>
    <td rowspan="2">RE</td>
    <td>ACE05</td>
    <td>6</td>
    <td>2,050</td>
    <td>6</td>
    <td>230</td>
    <td>920</td>
  </tr>
  <tr>
    <td>NYT</td>
    <td>24</td>
    <td>5,000</td>
    <td>24</td>
    <td>570</td>
    <td>2,280</td>
  </tr>
  <tr>
    <td rowspan="2">ED</td>
    <td>ACE05</td>
    <td>33</td>
    <td>676</td>
    <td>33</td>
    <td>676</td>
    <td>3,380</td>
  </tr>
  <tr>
    <td>CASIE</td>
    <td>5</td>
    <td>3,208</td>
    <td>5</td>
    <td>224</td>
    <td>1,120</td>
  </tr>
</table>

Table 4: Statistics of the RUIE-Bench dataset.

Computational complexity We use GPT-4 to generate adversarial examples. The primary computational cost during generation is concentrated on the self-attention mechanism, whose computational complexity is typically \( O(l^2) \) (where \( l \) denotes the length of the generated sequence). However, due to caching optimizations and other strategies employed during generation, the time required to generate a single token can be approximated as constant.

To facilitate quantification of the generation time, we conduct the following analysis: Assume the dataset contains a total of \( n \) examples. During adversarial example generation, an average of \( l \) tokens are generated per example. LLM takes a constant time \( k \) to generate each token. Then, the total time \( T \) can be expressed as: \( T = n \cdot l \cdot k \). During the data generation phase, we did not use any GPU-based computational resources. Instead, we relied on web-based API services to perform the generation. To construct RUIE-Bench, the LLM generated a total of 7,300 adversarial examples. Excluding time consumed due to response failures and other factors, the total generation time was approximately 341 minutes. On average, generating a single example took approximately 2.8 seconds, with each example containing an average of 38 generated tokens.

C Augment Data Generation

We utilize the original training sets from all datasets included in RUIE-Bench as the foundation for constructing augmented data. Considering the large scale of these training sets and the diverse types of perturbations involved, we perform random sampling to select 30% of the training data from each dataset. On this sampled subset, we apply all perturbation injection methods defined in RUIE-Bench to generate a variety of augmented samples. To efficiently generate a substantial volume of augmented data while controlling computational costs,
we opt to use Deepseck-V3 (Liu et al., 2024) as our data generation model instead of the more resource-intensive GPT-4 (OpenAI., 2023). The detailed statistics for all datasets before and after augmentation are summarized in Table 5.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Dataset</th>
    <th colspan="2">Data size</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>Augment</th>
  </tr>
  <tr>
    <td rowspan="3">NER</td>
    <td>ACE05</td>
    <td>7,299</td>
    <td>10,948</td>
  </tr>
  <tr>
    <td>CoNLL03</td>
    <td>14,041</td>
    <td>21,061</td>
  </tr>
  <tr>
    <td>WikiANN</td>
    <td>20,000</td>
    <td>30,000</td>
  </tr>
  <tr>
    <td rowspan="2">RE</td>
    <td>ACE05</td>
    <td>10,051</td>
    <td>12,061</td>
  </tr>
  <tr>
    <td>NYT</td>
    <td>56,196</td>
    <td>67,435</td>
  </tr>
  <tr>
    <td rowspan="2">ED</td>
    <td>ACE05</td>
    <td>19,216</td>
    <td>28,824</td>
  </tr>
  <tr>
    <td>CASIE</td>
    <td>11,189</td>
    <td>16,784</td>
  </tr>
</table>

Table 5: Statistics of Augmented Data.

D Few-shot Prompts for UIE

Taking the NER task as an example, we illustrate the prompt design used for extraction, as shown in Table 7. The prompt is composed of five main components: (1) Task Objective, which states the goal of the task; (2) Entity Types, which define the categories of entities to be extracted along with their descriptions; (3) Output Formatting, which specifies the expected structure of the output; (4) Examples, which present a few demonstration instances randomly sampled from the training set to guide the model; and (5) Current Task, which includes the input sentence to be processed. The RE and ED tasks follow a similar prompt structure, differing only in the defined types and output formats, as shown in Table 8 and Table 9, respectively.

E Comparison of Data Augmentation Training Models

To evaluate the effectiveness of the model trained with the LDA strategy, we conduct a comprehensive comparison against two baselines: the non-augmented training model, denoted as KnowCoder_partial, and the model trained with the full set of augmented data, referred to as KnowCoder-Robust. The results are summarized in Table 6, where the best performance under each specific perturbation for every dataset is highlighted in bold. As shown in the table, models trained with any form of data augmentation consistently outperform the non-augmented baseline across all perturbation types and datasets. Notably, KnowCoder-Robust_LDA achieves performance on par with KnowCoder-Robust, indicating that the LDA strategy can effectively support data augmentation training with significantly reduced augmentation cost.

F Case Study

As shown in Figures 8 and 9, we present extraction cases covering all types of perturbations to analyze model behavior under various perturbation settings. Specifically, for the Mask Context, Extend Sentence, Typo Injection, and Lowercase Conversation perturbations, we illustrate examples under the NER task. Correctly entities, relation triples, and event triggers are highlighted in red.

In the cases of Replace Entity, Triple, and Trigger perturbations, it is relatively easy to identify the correct entities, relations, or events based on contextual information, as these cases contain clear contextual clues that point to the correct interpretation. However, in practice, most models fail to make correct predictions on these adversarial examples. Only KnowCoder-Robust_LDA trained with data augmentation is able to generate accurate predictions. This suggests that these models tend to rely on memorization rather than using context for inference.

In the Change Context and Extend Sentence perturbation cases, the adversarial examples involve only simple modifications, such as replacing a single word or slightly extending the sentence. Nevertheless, models without data augmentation training produce incorrect or redundant predictions. Similarly, in the Typo Injection and Lowercase Conversion cases, the adversarial examples introduce only minor typographical errors or convert parts of the text to lowercase, yet models without augmentation still yield incorrect or missing predictions. These observations highlight the models’ sensitivity to input variations.

These cases collectively demonstrate that current LLMs and information extraction models suffer from insufficient robustness.
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Perturbation Type</th>
    <th colspan="3">Model</th>
  </tr>
  <tr>
    <th>KnowCoder<sub>partial</sub></th>
    <th>KnowCoder-Robust</th>
    <th>KnowCoder-Robust<sub>fix</sub></th>
  </tr>
  <tr>
    <td rowspan="6">NER</td>
    <td rowspan="6">ACE05</td>
    <td>None</td>
    <td>79.0</td>
    <td>82.5</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>Replace Entity</td>
    <td>72.3</td>
    <td>78.9</td>
    <td>76.3</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>74.2</td>
    <td>80.6</td>
    <td>78.9</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>72.1</td>
    <td>82.7</td>
    <td>81.4</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>76.8</td>
    <td>82.7</td>
    <td>80.1</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>77.0</td>
    <td>82.0</td>
    <td>82.4</td>
  </tr>
  <tr>
    <td rowspan="6">Conll03</td>
    <td rowspan="6">None</td>
    <td>91.7</td>
    <td>93.0</td>
    <td>93.2</td>
  </tr>
  <tr>
    <td>Replace Entity</td>
    <td>80.5</td>
    <td>90.9</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>87.2</td>
    <td>90.2</td>
    <td>90.6</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>91.0</td>
    <td>93.0</td>
    <td>94.0</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>90.0</td>
    <td>92.5</td>
    <td>91.4</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>85.1</td>
    <td>89.7</td>
    <td>87.5</td>
  </tr>
  <tr>
    <td rowspan="6">WikiANN</td>
    <td rowspan="6">None</td>
    <td>81.9</td>
    <td>83.3</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>Replace Entity</td>
    <td>70.8</td>
    <td>76.9</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>77.8</td>
    <td>80.8</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>78.2</td>
    <td>83.9</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>79.4</td>
    <td>83.6</td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>76.4</td>
    <td>82.6</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td rowspan="6">RE</td>
    <td rowspan="6">ACE05</td>
    <td>None</td>
    <td>58.6</td>
    <td>63.5</td>
    <td>59.3</td>
  </tr>
  <tr>
    <td>Replace Triple</td>
    <td>54.1</td>
    <td>61.6</td>
    <td>57.7</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>54.1</td>
    <td>62.3</td>
    <td>62.1</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>53.9</td>
    <td>59.7</td>
    <td>61.3</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>56.5</td>
    <td>56.9</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td rowspan="6">NYT</td>
    <td rowspan="6">None</td>
    <td>90.6</td>
    <td>91.1</td>
    <td>91.5</td>
  </tr>
  <tr>
    <td>Replace Triple</td>
    <td>62.9</td>
    <td>67.9</td>
    <td>70.0</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>89.2</td>
    <td>91.2</td>
    <td>90.8</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>52.8</td>
    <td>89.8</td>
    <td>89.4</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>51.3</td>
    <td>89.0</td>
    <td>89.3</td>
  </tr>
  <tr>
    <td rowspan="6">ED</td>
    <td rowspan="6">ACE05</td>
    <td>None</td>
    <td>69.1</td>
    <td>70.2</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>Replace Trigger</td>
    <td>55.5</td>
    <td>65.7</td>
    <td>66.0</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>66.1</td>
    <td>67.1</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>65.8</td>
    <td>70.5</td>
    <td>70.0</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>66.8</td>
    <td>68.5</td>
    <td>68.8</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>67.8</td>
    <td>69.6</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td rowspan="6">CASIE</td>
    <td rowspan="6">None</td>
    <td>67.0</td>
    <td>68.7</td>
    <td>68.2</td>
  </tr>
  <tr>
    <td>Replace Trigger</td>
    <td>53.5</td>
    <td>64.9</td>
    <td>65.5</td>
  </tr>
  <tr>
    <td>Change Context</td>
    <td>63.7</td>
    <td>67.5</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td>Extend Sentence</td>
    <td>62.9</td>
    <td>68.2</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td>Typo Injection</td>
    <td>62.0</td>
    <td>67.2</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td>Lowercase Conversion</td>
    <td>64.0</td>
    <td>69.1</td>
    <td>68.6</td>
  </tr>
  <tr>
    <td>All</td>
    <td>All</td>
    <td>Average</td>
    <td>71.3</td>
    <td>77.1</td>
    <td>77.1</td>
  </tr>
</table>

Table 6: Comparison of KnowCoder<sub>partial</sub> and different data augmentation training models on RUIE-Bench.

NER Task

[example data]
The above is a Named Entity Recognition data entry, where "sentence" contains the sentence information, and "entities" contains the entity label information. Now, based on the following rules, the entities in the sentence need to be changed:
1. Change the entity while keeping the original entity type.
2. The changed entities should be difficult and uncommon, and the number of entity words can vary.
3. Only change the entity content, do not change other content.
4. Changed entities should be updated in both "sentence" and "entities". Please output in the following format and do not output any extra content. {"sentence": "", "entities": []};

RE Task

[example data]
The above is a Relation Extraction data entry, where "sentence" contains the sentence information, and "relations" contains the relational triple information. Now, based on the following rules, the head and tail entities in the relational triples need to be changed:
1. Change the head and tail entities while keeping the original entity type.
2. The changed entity should be significantly different from the original entity and can vary in length.
3. Only change the head and tail entities, do not change other content.
4. Changed content should be updated in both "sentence" and "relations". Please output in the following format and do not output any extra content. {"sentence": "", "relations": []};

ED Task

[example data]
The above is a Event Detection data entry, where "sentence" contains the sentence information, and "events" contains the event trigger information. Now, based on the following rules, the event triggers in the sentence need to be changed:
1. Change the trigger while keeping the original event type.
2. The changed trigger should be significantly different from the original trigger and can vary in length.
3. Only change the trigger content, do not change other content.
4. Changed triggers should be updated in both "sentence" and "events". Please output in the following format and do not output any extra content. {"sentence": "", "events": []};

Figure 5: Prompts for Replace Entity, Triple, and Trigger.
NER Task
{example data}
The above is a Named Entity Recognition data entry, where the "sentence" contains the sentence information, and the "entities" contains the entity label information. The sentence containing 1-4 [MASK] tokens. These [MASK] tokens mask certain words, with each [MASK] token potentially masking one or more words. You need to generate some challenging words to replace these [MASK] tokens to create a difficult sentence. The generated words can not contain any entity information.
You need to provide 3 predictions for each [MASK] token. Please output in the following format, without any additional content:
[MASK]1: 1: "", 2: "", 3: ""
......
[MASK]n: 1: "", 2: "", 3: ""

ED Task
{example data}
The above is an Event Detection data entry, where the "sentence" contains the sentence information, and the "events" contains the event trigger information. The sentence containing 1-4 [MASK] tokens. These [MASK] tokens mask certain words, with each [MASK] token potentially masking one or more words. You need to generate some challenging words to replace these [MASK] tokens to create a difficult sentence. The generated words can not contain any event trigger information.
You need to provide 3 predictions for each [MASK] token. Please output in the following format, without any additional content:
[MASK]1: 1: "", 2: "", 3: ""
......
[MASK]n: 1: "", 2: "", 3: ""

Figure 6: Prompts for Change Context.

NER Task
{example data}
The above is a Named Entity Recognition data entry, where the "sentence" contains the sentence information, and the "entities" contains the entity label information.
Now, you need to ensure that the original sentence remains unchanged while adding semantically related content at the beginning or end of the sentence, with a preference for the end. The additional content should not be too simplistic and not not introduce new entity information.
Please output in the following format, without any additional content: {"sentence": "", "entities": []}, where "sentence" should be the expanded sentence, and "entities" should contain the original entity label information.

RE Task
{example data}
The above is a Relation Extraction data entry, where the "sentence" contains the sentence information, and the "relations" contains the relational triple information.
Now, you need to ensure that the original sentence remains unchanged while adding semantically related content at the end or beginning of the sentence, with a preference for the end. The additional content should not be too simplistic, and should not introduce new relational triple information.
Please output in the following format, without any additional content: {"sentence": "", "relations": []}, where "sentence" should be the expanded sentence, and "relations" should contain the original relational triple information.

ED Task
{example data}
The above is an Event Detection data entry, where the "sentence" contains the sentence information, and the "events" contains the event trigger information.
Now, you need to ensure that the original sentence remains unchanged while adding semantically related content at the end or beginning of the sentence, with a preference for the end. The additional content should not be too simplistic and not contain new event information.
Please output in the following format, without any additional content: {"sentence": "", "events": []}, where "sentence" should be the expanded sentence, and "events" should contain the original event trigger information.

Figure 7: Prompts for Extend Sentence.
<table>
  <tr>
    <th>Original NER example</th>
    <th>Replace Entity</th>
    <th>Adversarial example</th>
    <th>Original ED example</th>
    <th>Replace Trigger</th>
    <th>Adversarial example</th>
  </tr>
  <tr>
    <td>Sentence: France managed third place with 0.68 percent in the 10-nation world government bond index.<br>Entities: ["name": "France", "type": "LOC"]</td>
    <td></td>
    <td>Sentence: Atlanta managed third place with 0.68 percent in the 10-nation world government bond index.<br>Entities: ["name": "Atlanta", "type": "LOC"]</td>
    <td>Sentence: Over a hour of discussion, we asserted the will of both parties (Israel and the Arab world) to do everything to return to the negotiating table.<br>Events: ["trigger": "talk", "type": "Contact"]</td>
    <td></td>
    <td>Sentence: Over a hour of discussion, we asserted the will of both parties (Israel and the Arab world) to do everything to return to the negotiating table.<br>Events: ["trigger": "discussion", "type": "Contact"]</td>
  </tr>
  <tr>
    <td>Model</td>
    <td>Original example prediction</td>
    <td>Adversarial example prediction</td>
    <td>Model</td>
    <td>Original example prediction</td>
    <td>Adversarial example prediction</td>
  </tr>
  <tr>
    <td>LLama-3-BB-Instruct</td>
    <td>["name": "France", "type": "LOC"]</td>
    <td>["name": "Atlanta", "type": "LOC"]</td>
    <td>LLama-3-BB-Instruct</td>
    <td>["trigger": "talk", "type": "Contact"]</td>
    <td>NULL</td>
  </tr>
  <tr>
    <td>GLM4-Pi4</td>
    <td>["name": "France", "type": "LOC"]</td>
    <td>["name": "Atlanta", "type": "ORG"]</td>
    <td>GLM4-Pi4</td>
    <td>["trigger": "talk", "type": "Contact"]</td>
    <td>["trigger": "negotiating", "type": "Contact"]</td>
  </tr>
  <tr>
    <td>Stanza</td>
    <td>["name": "France", "type": "LOC"]</td>
    <td>["name": "Atlanta", "type": "ORG"]</td>
    <td>EEQA</td>
    <td>NULL</td>
    <td>NULL</td>
  </tr>
  <tr>
    <td>KnowCoder-7B</td>
    <td>["name": "France", "type": "LOC"]</td>
    <td>["name": "Atlanta", "type": "ORG"]</td>
    <td>KnowCoder-7B</td>
    <td>["trigger": "talk", "type": "Contact"]</td>
    <td>["trigger": "discussion", "type": "Contact"]</td>
  </tr>
  <tr>
    <td>KnowCoder-TB-RobustLM</td>
    <td>["name": "France", "type": "LOC"]</td>
    <td>["name": "Atlanta", "type": "LOC"]</td>
    <td>KnowCoder-TB-RobustLM</td>
    <td>["trigger": "talk", "type": "Contact"]</td>
    <td>["trigger": "discussion", "type": "Contact"]</td>
  </tr>
</table>

Figure 8: Example cases for the Replace Entity, Triple and Trigger.

<table>
  <tr>
    <th>Original NER example</th>
    <th>Change Context</th>
    <th>Adversarial example</th>
    <th>Original NER example</th>
    <th>Typo Injection</th>
    <th>Adversarial example</th>
  </tr>
  <tr>
    <td>Sentence: It was the second costly blunder by Hyderabad in four hours.<br>Entities: ["name": "Hyderabad", "type": "LOC"]</td>
    <td></td>
    <td>Sentence: It was the second costly blunder by Hyderabad in four minutes.<br>Entities: ["name": "Hyderabad", "type": "LOC"]</td>
    <td>Sentence: Slaughter steers and heifers not tested, compared with Thursday’s close, USDA said.<br>Entities: ["name": "USDA", "type": "ORG"]</td>
    <td></td>
    <td>Sentence: Slaughter steers and heifers not tested, compared with Thursday’s close, USDA said.<br>Entities: ["name": "USDA", "type": "ORG"]</td>
  </tr>
  <tr>
    <td>Model</td>
    <td>Original example prediction</td>
    <td>Adversarial example prediction</td>
    <td>Model</td>
    <td>Original example prediction</td>
    <td>Adversarial example prediction</td>
  </tr>
  <tr>
    <td>LLama-3-BB-Instruct</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>["name": "Hyderabad", "type": "ORG"]</td>
    <td>LLama-3-BB-Instruct</td>
    <td>["name": "USDA", "type": "ORG"]</td>
    <td>NULL</td>
  </tr>
  <tr>
    <td>GLM4-Pi4</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>["name": "Hyderabad", "type": "ORG"]</td>
    <td>GLM4-Pi4</td>
    <td>["name": "USDA", "type": "ORG"]</td>
    <td>NULL</td>
  </tr>
  <tr>
    <td>Stanza</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>["name": "Hyderabad", "type": "ORG"]</td>
    <td>Stanza</td>
    <td>["name": "USDA", "type": "ORG"]</td>
    <td>["name": "USDA", "type": "ORG"]</td>
  </tr>
  <tr>
    <td>KnowCoder-7B</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>["name": "Hyderabad", "type": "ORG"]</td>
    <td>KnowCoder-7B</td>
    <td>["name": "USDA", "type": "ORG"]</td>
    <td>["name": "Slaughter", "type": "PER"]</td>
  </tr>
  <tr>
    <td>KnowCoder-TB-RobustLM</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>["name": "Hyderabad", "type": "LOC"]</td>
    <td>KnowCoder-TB-RobustLM</td>
    <td>["name": "USDA", "type": "ORG"]</td>
    <td>["name": "USDA", "type": "ORG"]</td>
  </tr>
</table>

Figure 9: Example cases for the Mask Context, Extend Sentence, Typo Injection and Lowercase Conversation.
PROMPT FOR FEW-SHOT NER.

## Task Objective
Perform Named Entity Recognition (NER) on input sentences to extract entities of these types:

##Entity Types:
{entity_type 1}: {description 1}
{entity_type 2}: {description 2}
......
{entity_type n}: {description n}
The entity type here refer to the entity types specific to a given dataset, where the description represents the entity type information. When used, it should be replaced with the actual entity types and corresponding descriptions based on the specific dataset.

## Output Formatting
1. Return a JSON list of entities
2. Each entity must include:
   - **Type**: Entity category (exact uppercase labels)
   - **Name**: Original text span
3. Return empty list if no entities found

## Examples (10-shot)
1. Input: {example sentence 1}
   Output: {recognition result 1}
2. Input: {example sentence 2}
   Output: {recognition result 2}
......
10. Input: {example sentence 10}
   Output: {recognition result 10}
The example sentences are selected from the training set, and the recognition results should fully comply with the defined Output Formatting.

## Current Task
Input: {test sentence}
Output:
The input here should be the sentences to be tested, and the output should be the model’s recognition results.

Table 7: Prompt for Few-Shot NER.
PROMPT FOR FEW-SHOT RE.

## Task Objective
Perform Relation Extraction (RE) on input sentences to extract relational triples of these types:

## Relation Types:
{relation_type 1}: {description 1}
{relation_type 2}: {description 2}
......
{relation_type n}: {description n}
The relation type here refer to the relation types specific to a given dataset, where the description represents the relation type information. When used, it should be replaced with the actual relation types and corresponding descriptions based on the specific dataset.

## Output Formatting
1. Return a JSON list of relational triples
2. Each relational triple must include:
   - **Head**: Original head entity span
   - **Type**: Relation category (exact uppercase labels)
   - **Tail**: Original tail entity span
3. Return empty list if no relational triples found

## Examples (10-shot)
1. Input: {example sentence 1}
   Output: {recognition result 1}
2. Input: {example sentence 2}
   Output: {recognition result 2}
......
10. Input: {example sentence 10}
    Output: {recognition result 10}
The example sentences are selected from the training set, and the recognition results should fully comply with the defined Output Formatting.

## Current Task
Input: {test sentence}
Output:
The input here should be the sentences to be tested, and the output should be the model’s recognition results.

Table 8: Prompt for Few-Shot RE.
PROMPT FOR FEW-SHOT ED.

## Task Objective
Perform Event Detection (ED) on input sentences to extract events of these types:

## Event Types:
{event_type 1}: {description 1}
{event_type 2}: {description 2}
......
{event_type n}: {description n}
The event type here refer to the event types specific to a given dataset, where the description represents the event type information. When used, it should be replaced with the actual event types and corresponding descriptions based on the specific dataset.

## Output Formatting
1. Return a JSON list of events
2. Each event must include:
   - **Type**: Event category (exact uppercase labels)
   - **Trigger**: Event trigger span
3. Return empty list if no event found

## Examples (10-shot)
1. Input: {example sentence 1}
   Output: {recognition result 1}
2. Input: {example sentence 2}
   Output: {recognition result 2}
......
10. Input: {example sentence 10}
   Output: {recognition result 10}
The example sentences are selected from the training set, and the recognition results should fully comply with the defined Output Formatting.

## Current Task
Input: {test sentence}
Output:
The input here should be the sentences to be tested, and the output should be the model’s recognition results.

Table 9: Prompt for Few-Shot ED.

Reviewer paper summaries: In the paper, a new benchmark for Robust Universal Information Extraction (RUIE-Bench) that covers three IE tasks (Named Entity Recognition, NER; Relation Extraction, RE; Event Detection, ED) with 14 types of LLM-generated perturbations is introduced to evaluate and enhance the robustness of UIE models. RUIE-Bench consists of 11,580 samples that include LLM-generated perturbations. These perturbations are categorized into five types: replacement, context changing, sentence extending, typo injection, and lowercase conversion. Significant performance drops are observed in existing UIE baselines. To improve both robustness and data efficiency, the paper introduces a novel Loss-guided Data Augmentation (LDA) approach. This method leverages inference loss to identify and select the perturbed augmentation samples on which the model performs worst. LDA leads to an average improvement of 7.5% with 15% of the data.
Summary of strengths: 1. The paper introduces a novel benchmark RUIE-Bench to evaluate the robustness of existing UIE baselines. RUIE-Bench covers three major IE tasks (NER, RE, and ED) with 14 distinct perturbation types.
2. Significant drops in performance are observed in the RUIE-Bench, indicating the weak robustness of UIE baselines, such as memorizing specific patterns instead of reasoning based on context.
3. The paper proposes a Loss-guided Data Augmentation (LDA) approach that leads to an average improvement of 7.5% on perturbed data with 15% of the training samples, achieving the enhancement of both robustness and data efficiency.
Summary of weaknesses: 1. In Table 2, the performance of fine-tuned KnowCoder-7B models (partial, Robust, and $Robust_{LDA}$) declines in the Non-perturbation splits (~3% on average), suggesting that enhancing robustness may compromise accuracy.
2. In Table 2, the performance drops significantly in P6 Replace Triple for RE (27% for KnowCoder-7B) and P10 Replace Trigger for ED (20% for KnowCoder-7B). The gap remains after robustness fine-tuning. Why do UIE models fall short of these perturbations? Do human experts examine and analyze the LLM-generated perturbations?
3. The LLM-based perturbations can be seen as a mapping from the original sentence to the perturbed data. Through fine-tuning with perturbed data samples, UIE models can learn such mapping instead of improving general robustness. Based on this concern, an ablation study is necessary to evaluate the robustness of the enhanced model on unseen perturbations.
4. Why is the performance of KnowCoder-7B not reported in Table 3?
Comments, suggestions and typos: 1. How many rounds does LDA need to converge? How is the number 15% of the data computed? What is the overall acceleration ratio compared to training on whole data?
2. Line 55: replaces → replace. Line 168: uses → use.
Overall assessment scores: 3.5
Metareview score: 4
Metareview: The paper addresses the lack of robust evaluation benchmarks that can work for multiple Information Extraction (IE) tasks. The tasks considered in the paper are NER, Relation Extraction, and Event Detection. Authors propose using Large Language Models (LLMs), specifically GPT-4, to generate more diverse and realistic perturbations. The paper introduces a new benchmark dataset that spans six existing datasets and applying 14 types of perturbations. The paper also proposes a data augmentation solution called Loss-guided Data Augmentation (LDA) to efficiently improve model robustness, particularly for computationally expensive LLMs.

The experimental setup involves assessing the robustness of models against the generated perturbations. The models evaluated include open-source LLMs, closed-source LLMs, traditional IE models, and fine-tuned UIE models. The evaluation on RUIE-Bench revealed significant performance drops for all tested models when faced with perturbations. LDA trained models exhibit better robustness and on unseen dataset, the performance exceeds that of training model on all perturbed samples.

Reviewers expressed concerns regarding computational complexity, case studies related to robustness, societal impact, accuracy tradeoff and scope of datasets. The authors have meaningfully addressed all these aspects in the rebuttal or have committed to do so in the final version. 

Overall, the paper does a comprehensive job of presenting a new dataset, evaluating existing models, and proposing a new strategy for improving the model.
Summary of reasons to publish: 1. The paper introduces RUIE-Bench, a significantly more comprehensive benchmark for Robust UIE than previously available, covering multiple tasks (NER, RE, ED) and utilizing LLMs for generating diverse, realistic perturbations. This will serve as a very useful benchmark for the community.
2. Comprehensive evaluation of various existing models (LLMs, traditional IE, UIE) on this new benchmark, revealing crucial insights into their robustness limitations. 
3. The proposed Loss-guided Data Augmentation (LDA) method offers an effective and computationally efficient way to enhance UIE model robustness. The method seems to be quite general and could possibly benefit robustness in other domains as well.