Easy-to-Hard Learning for Information Extraction∗
Year: 2023

Abstract:
Information extraction (IE) systems aim to automatically extract structured information, such as named entities, relations between entities, and events, from unstructured texts. While most existing work addresses a particular IE task, universally modeling various IE tasks with one model has achieved great success recently. Despite their success, they employ a one-stage learning strategy, i.e., directly learning to extract the target structure given the input text, which contradicts the human learning process. In this paper, we propose a unified easy-tohard learning framework consisting of three stages, i.e., the easy stage, the hard stage, and the main stage, for IE by mimicking the human learning process. By breaking down the learning process into multiple stages, our framework facilitates the model to acquire general IE task knowledge and improve its generalization ability. Extensive experiments across four IE tasks demonstrate the effectiveness of our framework. We achieve new state-of-the-art results on 13 out of 17 datasets. Our code is available at https://github.com/DAMO-NLP-SG/ IE-E2H.

heading: 1 Introduction
text: Information extraction (IE) is a crucial task in natural language processing (NLP) that involves extracting structured knowledge from unstructured text data (Bing et al., 2013, 2015), enabling various applications such as information retrieval (Ruambo and Nicholaus, 2019), knowledge graph construction (Oramas et al., 2016; Wang et al., 2019), and question answering (Khot et al., 2017). Depending on what kind of information is to be extracted, IE
∗ This work was supported by Alibaba Group through Alibaba Research Intern Program. It was also partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719). This work was done when Chang Gao was an intern at Alibaba DAMO Academy.
†Wenxuan Zhang is the corresponding author.
consists of a wide range of tasks, including named entity recognition (NER) (Li et al., 2022a), joint entity and relation extraction (RE) (Taillé et al., 2020; Chia et al., 2022), event extraction (EE) (Li et al., 2022b), and aspect-based sentiment analysis (ABSA) (Zhang et al., 2022b).
Traditionally, IE has been approached with specialized models that are designed to handle specific IE tasks. For example, NER is often formulated as a sequence labeling (Ma and Hovy, 2016; Xu et al., 2021b) or span-based classification (Wang et al., 2020) problem. The more complex RE or EE task is usually solved with pipeline approaches that split the original task into several sequential subtasks and design specific models for each subtask (Subburathinam et al., 2019; Yang et al., 2019; Peng et al., 2020). These models often require extensive task-specific knowledge to design dedicated model architectures and thus suffer from poor generalization. Recently, motivated by pre-trained generative models such as T5 (Raffel et al., 2020) that handle multiple tasks with the unified text-to-text format, there has been a shift towards the use of unified models for IE as well, which can tackle all IE tasks with a single model structure. For example, TANL (Paolini et al., 2021) tackles various IE tasks with a text-to-text generative model by framing them as translation between augmented natural languages. UIE (Lu et al., 2022) models heterogeneous IE structures into a uniform representation via a structural extraction language.
Despite the success of existing unified models on various IE tasks, they typically adopt a one-stage learning paradigm, i.e., directly learning to predict the target structure given the input text. In contrast, humans often learn to tackle a task in an easy-tohard manner. They learn basic concepts or skills before solving more complex problems and often tackle harder examples to gain a better understanding of the problem. Taking the RE task as an example, it aims to extract relational triplets, where each
11913
triplet consists of a head entity, a relation, and a tail entity. To tackle it, humans first learn some basic skills, such as identifying entities, recognizing relations, and associating entities and relations, before extracting complex relational triplets. This process facilitates humans to learn meaningful substructures and the dependencies among them. Moreover, in practical scenarios, humans usually encounter harder cases, i.e., long input context of multiple sentences containing more entities and relations. By solving hard cases, humans improve their understanding of the task and problem-solving skills. By comparison, models are only trained with the provided training data. The gap between the model and human learning strategies hinders IE models from further development.
To bridge the gap, we propose an easy-to-hard (E2H) learning framework for IE tasks in this paper. E2H mimics the human learning procedure to learn each IE task in stages, i.e., the easy stage, the hard stage, and the main stage. The easy stage aims to help the model acquire basic skills of the task, and the hard stage aims to assist the model in handling broad-range variations of the task via training the model with diverse and harder data. Finally, the main stage focuses on the main task at hand for training. Thus an immediate question is how to prepare the data with different levels of difficulty for the easy and hard stages. It is labor-intensive and challenging to construct such data manually. In this work, we attempt only to leverage the existing data of the main task for constructing the data.
Specifically, for the easy stage, we observe that the target IE structure often has meaningful substructures. Therefore, we identify several basic skills for each task according to the substructures of its target structure. Returning to the RE example, the skills can be recognizing the entities, relations, and dependencies between them. We can automatically construct training data for learning these skills by modifying the input prompt and decomposing the target structure of the main task. For the hard stage, we combine two training instances of the main task to build a harder training instance by concatenating their input texts to form the new text and their targets to build the new target. The new instance contains more entities, relations, and complicated contexts, making it harder than the original instances. Through these two novel construction strategies, we can reduce much human effort to obtain the data for different stages.
To summarize, our contributions are three-fold: (1) We propose a unified easy-to-hard (E2H) learning framework for IE tasks by imitating the human learning process; (2) We develop two novel strategies to build the easy and hard stages of our framework without using any additional resources; (3) We conduct comprehensive evaluations on 17 datasets across four IE tasks and achieve state-ofthe-art results on 13 datasets. Notably, our E2H method consistently outperforms the one-stage learning counterpart by introducing two extra learning stages with an average increase of 0.38, 2.96, 1.33, and 1.39 absolute points on the NER, RE, EE, and ABSA tasks, respectively.

heading: 2 Task Definition
text: This paper investigates four common IE tasks, i.e., NER, RE, EE, and ABSA. In this section, we provide formal definitions of these tasks. Detailed examples of these tasks are in Appendix A.3.
Named Entity Recognition (NER) Given an input text T , the task is to identify and classify entities in T into predefined categories, i.e., extract {(ei, ci)}, where ei is the i-th entity, which is a continuous text span in T , ci ∈ C is its category, and C is the entity category set.
Relation Extraction (RE) Given an input text T , RE is to identify a set of (head entity, relation, tail entity) triplets, i.e., extract {((ehi , chi ), ri, (eti, cti))}, where the superscripts h and t denote the head and tail entities, ri ∈ R is the i-th relation, and R is the relation set.
Event Extraction (EE) Given an input text T , the task is to identify a set of events where each event consists of an event trigger and a set of corresponding arguments, i.e., extract {((etrii , ctrii ), (earg1i , carg1i ), · · · , (eargmi , cargmi )
)}, where etrii is the i-th trigger, which is a continuous text span in T , ctrii ∈ Cevent is its category, e argj i is the j-th argument of the i-th event, which is also a continuous text span in T , cargji ∈ Cevent is its category, and Cevent consists of all event and argument categories.
Aspect-based Sentiment Analysis (ABSA) There are four essential elements in ABSA, namely aspect category c, aspect term a, opinion term o, and sentiment polarity p. We focus on the aspect sentiment triplet extraction (ASTE) task (Peng et al., 2020) and the aspect sentiment quad
prediction (ASQP) task (Zhang et al., 2021a) given their popularity. Given an input text T , the ASTE task is to identify a set of {(ai, oi, pi)} triplets, and the ASQP task is to identify a set of {(ci, ai, oi, pi)} quadruplets, where ci ∈ Cabsa is i-th aspect category, ai is i-th aspect term, oi is i-th opinion term, both ai and oi are continuous spans in T , pi ∈ {positive, negative, neutral} is i-th sentiment polarity, and Cabsa is the aspect category set.

heading: 3 Our E2H Framework
text: Our proposed easy-to-hard (E2H) framework consists of three sequential stages: the easy stage, the hard stage, and the main stage. In this section, we first introduce our text-to-structure formulation for facilitating three-stage learning in a unified framework. Next, we will describe how to realize the easy and hard stages. Finally, we will discuss the main stage as well as the detailed training and inference process of our framework.

heading: 3.1 Unified Text-to-Structure Formulation
text: Similar to UIE (Lu et al., 2022), we formulate NER, RE, EE, and ABSA as text-to-structure generation problems, which allows us to use a single model to tackle multiple tasks. Given a text T and its corresponding prompt P , we aim to generate the target IE structure S with an encoder-decoder model M : (P, T ) → S. To facilitate the learning of dif-
ferent stages, we design the prompt P containing three types of information: Hint, Constraint, and Schema. Hint guides the model on what elements should be extracted, Constraint indicates specific constraints for the task, and Schema provides necessary information such as the possible relation set for the extraction. With these three types of information, the prompt is able to connect the learning process in different stages.
Taking the RE task as an example, as depicted in Figure 1, Hint consists of one or both of an entity hint and a relation hint. The entity hint, represented by the special token [HE], guides the model to extract entities, and the relation hint, represented by the special token [HR], guides the model to extract relations. The use of both hints guides the model to extract both entity and relation information, in the form of (head entity, relation, tail entity) triplets. Constraint is a specific entity or relation, which limits the target structure to be related to that entity or relation. Lastly, Schema contains pre-defined entity categories or relations or both of them, depending on the information that needs to be extracted. It provides essential information for identifying entities and relations in a text.

heading: 3.2 The Easy Stage
text: The goal of the easy stage is to enable the model to learn basic skills that will aid in tackling the main task. To achieve this, we identify several skills for each task and automatically construct the training
Task Basic Skills
data for them based on the data of the main task. Table 1 presents the basic skills of NER, RE, EE, ASTE, and ASQP. We design each skill to be a subtask of the main task according to its target structure. These skills are more fundamental and welldefined. Combining these skills gives the model a whole picture of how to tackle the main task. For example, the RE task has four skills. Skill1 and Skill3 help the model recognize substructures of the relational triplet, i.e., the entity and relation, respectively, and Skill2 and Skill4 help the model learn the dependencies between these substructures.
To construct the training data for each skill, we modify the input and target of the main task’s training data. Specifically, the input text is the same for the skills and the main task, but the prompt is different. As shown in Figure 1, for the RE task, there is only [HE] in the hint of Skill1 as it only extracts entities and only [HR] in the hint of Skill3 as it only extracts relations. Both [HE] and [HR] are in the hints of Skill2, Skill4, and the main task because they extract (head entity, relation, tail entity) triplets. For Skill2 and Skill4, there is also a Constraint, i.e., a head entity or relation, which requires their targets to be triplets related to a specific head entity or relation. The schema of the RE task consists of both entity categories and relations. For a specific skill of RE, the schema only contains entity categories or relations. The target of each skill is a part of the target of the RE task. For Skill1
and Skill3, which extract a substructure of the relational triplet, we use the substructure as the target. For Skill2 and Skill4, we use the corresponding subset of triplets of the RE task as the target.

heading: 3.3 The Hard Stage
text: The hard stage aims to construct training examples that are harder than the original training examples of the main task to train the model. Intuitively, the training instance is harder if the input text contains more structural elements and more complicated contexts. To this end, we combine two training instances of the original task to construct a harder instance. Formally, given two training instances (P, T1, S1) and (P, T2, S2), we can construct a harder training instance (P, T1◦T2, S1◦S2), where P is the prompt, Ti is the i-th text, Si is the i-th target structure, and ◦ denotes concatenation. An example is shown in the hard stage part of the RE task in Figure 1. The model has to process and understand the combined information from both instances, making it more challenging for the model to correctly extract the target structure.
Let N denote the number of training examples of the original task. For each training example, we randomly sample M training examples whose target structures are not empty to construct M hard instances. This results in a total of N ∗M hard instances. This approach allows us to easily construct a large amount of diverse hard training data.

heading: 3.4 The Main Stage
text: After training the model in the easy and hard stages, we train the model with the main task in this stage.
Training We adopt the pre-trained sequence-tosequence model T5 (Raffel et al., 2020) as the backbone of E2H. The model is trained with a maximum likelihood objective. Given the training example (P, T, S), the loss function Lθ is defined as
Lθ = − n∑
i=1
logPθ (Si | S<i, P, T ) (1)
where θ is the model parameters, P is the prompt, T is the text, S is the target structure, and n is the length of S. We train the model in the easy, hard, and main stages sequentially. For the easy stage, we adopt the weights of pre-trained T5 to initialize the model. For the hard and main stages, we initialize the model with the weights of the model trained in the previous stage.
Inference Once the training process is complete, we use the model trained in the main stage to generate the target structure S for any given tuple of the prompt and text (P, T ). Although our training process has three stages, the inference is a one-stage process. The computational load is the same as that of the one-stage learning counterpart.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets We conduct experiments on 17 datasets across four IE tasks, i.e., NER, RE, EE, and ABSA. We evaluate the flat NER task with CoNLL03 (Tjong Kim Sang and De Meulder, 2003), and the nested NER task with ACE04-Ent (Mitchell et al., 2005) and ACE05-Ent (Walker et al., 2006). For RE, we experiment on CoNLL04 (Roth and Yih, 2004), ACE05-Rel (Walker et al., 2006), and SciERC (Luan et al., 2018). Regarding to EE, we use ACE05E, ACE05E+ (Walker et al., 2006), and CASIE (Satyapanich et al., 2020). As for ABSA, we consider the ASTE and ASQP tasks. For ASTE, we adopt four popular datasets, including Rest14, Laptop14, Rest15, and Rest16 provided by Xu et al. (2020). For ASQP, we use R-ACOS and L-ACOS provided by Cai et al. (2021), and Rest15 and Rest16 provided by Zhang et al. (2021a). These ABSA datasets are derived from the datasets provided by the SemEval ABSA challenges (Pontiki et al., 2014, 2015, 2016), except L-ACOS which is
collected from the Amazon Laptop domain. Statistics of these datasets are provided in Appendix A.1.
Evaluation We use Micro-F1 as the primary evaluation metric. For each experimental result, we report the average performance on three random seeds. For NER, RE, EE, and ASTE, we follow Lu et al. (2022) to use Entity F1, Relation Strict F1, Event Trigger F1 and Argument F1, and Sentiment Triplet F1 as the evaluation metrics and map the generated string-level extraction results to offsetlevel for evaluation. For ASQP, we follow Zhang et al. (2021a) to use Sentiment Quad F1 to evaluate the model. A sentiment quad is correct if and only if the four elements are exactly the same as those in the gold sentiment quad.
Baselines We divide our baselines into two categories: specialized models and unified models. Specialized models are designed for a particular IE task, while unified models are designed for general IE. For specialized models, we use state-of-theart methods such as BARTNER (Yan et al., 2021) and DeBias (Zhang et al., 2022a) for NER, UniRE (Wang et al., 2021) and PURE (Zhong and Chen, 2021) for RE, Text2Event (Lu et al., 2021) and DEGREE (Hsu et al., 2022) for EE, and PARAPHRASE (Zhang et al., 2021a) and Seq2Path (Mao et al., 2022) for ABSA. For unified models, we use TANL (Paolini et al., 2021), UIE (Lu et al., 2022), and LasUIE (Fei et al., 2022) as baselines. To make a fair comparison with one-stage learning methods, we also build T5-base and T5-large baselines. We set their inputs and outputs the same as those of E2H and only train them in the main stage.
Implementation Details E2H has two model sizes: E2H-base and E2H-large, which are initialized with pre-trained T5-base and T5-large models (Raffel et al., 2020), respectively. Other details are reported in Appendix A.2.

heading: 4.2 Main Results
text: We compare E2H with state-of-the-art specialized and unified models. Tables 2-4 report the experimental results on 17 datasets across four IE tasks. We have the following observations: (1) E2H is an effective framework for various IE tasks. E2Hlarge achieves new state-of-the-art results on 13 out of 17 datasets. (2) The proposed easy-to-hard threestage learning method consistently outperforms the one-stage learning counterpart. E2H performs better than T5 on all the datasets for two model sizes,
and E2H-large obtains an average improvement of 0.38, 2.96, 1.33, and 1.39 absolute points over T5-large on the NER, RE, EE, and ABSA tasks, respectively. This demonstrates the strong generalization ability of our framework. (3) Without using any external resources, our method exhibits comparable or stronger performance than models with large-scale continued pre-training. Compared with UIE (Lu et al., 2022), which is pre-trained with large-scale structured, unstructured, and parallel data, E2H-large achieves better performance on the RE, EE, and ASTE tasks and obtains comparable results on the NER task. (4) Easy-to-hard learning brings more benefits to complex tasks than simple tasks. Specifically, compared with the improvement on the NER task, which only extracts entities, the improvements of E2H over T5 are more significant on the other three tasks, which extract tuples with multiple elements. This shows that our method can help the model effectively capture the structural dependency of complex structures.

heading: 4.3 Low-Resource Results
text: Our experiments in low-resource scenarios show that E2H is particularly effective in situations where there is limited training data. As shown in Figure 2, by training on a fraction (1%, 5%, and 10%) of the original data1, we observe that
1We repeat each experiment three times with different samples and report their averaged results.
E2H-base significantly outperforms T5-base on all datasets. For example, when there is only 5% of the training data, E2H-base obtains an average of 7.1, 12.0, 6.4, and 8.2 absolute points of improvement over T5-base on ACE04-Ent, ACE05-Rel, ACE05E, and Rest14 respectively. This highlights the effectiveness of our easy-to-hard learning framework when data is scarce. On one hand, the easy stage facilitates the model to identify the substructures of the target structure and capture the dependencies among them, which are difficult when there is limited data. On the other hand, the hard stage provides diverse and harder data to help the model tackle broad-range variations of the task, which is especially important in low-source scenarios.

heading: 5 More Analysis
text: Analysis on different learning strategies In the main result table, we report the results of E2H trained with the easy→hard→main strategy, i.e., training the model in the easy, hard, and main stages sequentially. In this section, we investigate alternative learning strategies. Table 6 reports the results of T5-base models trained with different learning strategies on four datasets across four tasks. We have the following observations: (1) The easy→hard→main strategy is the best among the seven concerned strategies. It performs better than other strategies on all datasets. (2) Easy-to-hard multi-stage learning outperforms multi-task learning (i.e., easy+main+hard). When the easy, main, and hard parts of the training data are used, the easy→hard→main and easy→main→hard strategies show superiority over the easy+main+hard strategy on all datasets. This indicates that easy-tohard multi-stage learning is essential to the model’s performance. (3) Each stage is critical to our E2H framework. Removing any of the stages will reduce the performance of E2H. (4) In general, three-stage learning is better than two-stage learning, and they are better than one-stage learning.

heading: Is each skill necessary in the easy stage? To
text: quantify the contribution of each skill, we examine the performance of E2H-base after removing a basic skill for training in the easy stage. Ablation results on four datasets across four tasks are shown in Table 5. Removing any skill degrades the performance of E2H on the main task, indicating that recognizing substructures and the dependency between them is crucial to the model’s performance.
Does easy-to-hard learning improve the model’s cross-domain generalization ability? To answer this question, we compare the performance of the E2H-base model and the T5-base model trained on a dataset on another dataset in a different domain of the same task. Table 7 reports the results of the cross-domain generalization performance of different models on two dataset pairs: CoNLL03↔ACE04-Ent of the NER task and Rest16↔Laptop14 of the ASTE task. E2H-base performs better than T5-base in all scenarios. This indicates that easy-to-hard learning can enhance the model’s cross-domain generalization ability.

heading: 6 Related Work
text: IE is a long-standing research area in natural language processing. Over the years, the paradigm
for IE has undergone several transitions. Early approaches to IE focus on sequence labeling techniques (McCallum and Li, 2003; Ma and Hovy, 2016; Zhang et al., 2018; Li et al., 2019; Zhang et al., 2021b), in which each word in a text is assigned a label indicating its role in the extraction task. Span-based approaches (Luan et al., 2019; Wang et al., 2020; Zhao et al., 2020; Xu et al., 2021a; Zhou et al., 2022, 2023), which involve identifying spans in the text that correspond to the desired information, are later introduced for IE. MRC-based methods (Du and Cardie, 2020; Li et al., 2020; Mao et al., 2021; Xu et al., 2023) that frame the extraction task as a reading comprehension problem and generation-based methods (Yan et al., 2021; Lu et al., 2021; Zhang et al., 2021c) that generate the extracted information directly from the text have gained popularity in recent years for IE. They have been shown to be more effective and flexible. Most of these methods target a specific IE task. There have been some efforts to develop unified IE methods (Paolini et al., 2021; Lu et al., 2022; Fei et al., 2022), which can unify various IE tasks with one framework. Our E2H framework, a unified IE framework, introduces a novel easy-to-hard learning paradigm for IE to reduce the gap between model and human learning.
From the perspective of improving the learning process, E2H shares similar spirits with transfer learning (Pan and Yang, 2010), which uses the knowledge gained from solving one task to help solve another related task. By comparison, E2H learns basic skills specifically designed to assist with the target task. E2H is also related to curriculum learning (Bengio et al., 2009; Wang et al., 2022) in its fundamental motivation of learning from easy to hard. Curriculum learning, inspired
by the human learning process, presents examples starting from the easiest samples, then gradually introducing more complex ones. However, curriculum learning involves the intricate task of ordering instances based on their difficulty. This requires a reliable difficulty criterion or a ranking system, which can be challenging to define and often necessitates substantial human effort. In contrast, E2H emphasizes on mastering certain fundamental skills prior to tackling more intricate tasks, eliminating the requirement for a difficulty criterion. This approach can be particularly beneficial in scenarios where the target task requires a distinct set of skills, or when the learning setting does not naturally provide a straightforward measure of difficulty.

heading: 7 Conclusion
text: This paper proposes an easy-to-hard learning framework consisting of the easy stage, the hard stage, and the main stage for IE. Two novel strategies are proposed to build the easy and hard parts of the framework to enable the learning process. Experimental results in both full and low-resource scenarios demonstrate the effectiveness of our framework and its superiority over one-stage learning methods.
Limitations
While the results have shown the effectiveness of our framework in IE without using any additional resources, we did not explore the potential enhancement by utilizing existing resources in the easy-tohard learning process. On one hand, we can build the easy stage with the help of existing data of simpler tasks. On the other hand, the data of harder tasks can be used for the hard stage. To enhance the E2H framework via effectively using existing resources is an interesting and promising direction. Another limitation is that we did not extensively explore the possible skill sets for each task. Exploring more approaches to obtain the skill sets is also open for future research. We plan to investigate these possibilities in our future work.

heading: In Proceedings of the 61th Annual Meeting of the
text: Association for Computational Linguistics.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808–5822, Online. Association for Computational Linguistics.
Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring pre-trained language models for event extraction and generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5284– 5294, Florence, Italy. Association for Computational Linguistics.
Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, and Weiming Lu. 2022a. De-bias for generative extraction in unified NER task. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 808–818, Dublin, Ireland. Association for Computational Linguistics.
Wenxuan Zhang, Yang Deng, Xin Li, Yifei Yuan, Lidong Bing, and Wai Lam. 2021a. Aspect sentiment quad prediction as paraphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9209– 9219, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenxuan Zhang, Ruidan He, Haiyun Peng, Lidong Bing, and Wai Lam. 2021b. Cross-lingual aspectbased sentiment analysis with aspect term codeswitching. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9220–9230, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021c. Towards generative aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 504–510, Online. Association for Computational Linguistics.
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022b. A survey on aspect-based sentiment analysis: Tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering, pages 1–20.
Yuan Zhang, Hongshen Chen, Yihong Zhao, Qun Liu, and Dawei Yin. 2018. Learning tag dependencies for sequence tagging. In Proceedings of the TwentySeventh International Joint Conference on Artificial
Intelligence, IJCAI-18, pages 4581–4587. International Joint Conferences on Artificial Intelligence Organization.
He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020. SpanMlt: A span-based multi-task learning framework for pair-wise aspect and opinion terms extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239–3248, Online. Association for Computational Linguistics.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50–61, Online. Association for Computational Linguistics.
Ran Zhou, Xin Li, Lidong Bing, Erik Cambria, and Chunyan Miao. 2023. Improving self-training for cross-lingual named entity recognition with contrastive and prototype learning. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics.
Ran Zhou, Xin Li, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. ConNER: Consistency training for cross-lingual named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8438–8449, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

heading: A Appendix
text: A.1 Statistics of Datasets
Statistics of datasets are reported in Table 8.
A.2 Implementation Details
We set the maximum input length to 384 and the maximum target length to 256. Following the practices of Lu et al. (2022), we use a batch size of 64 for E2H-base and 32 for E2H-large. The learning rate is chosen from {1e-4, 3e-4} for E2H-base and {5e-5, 1e-4} for E2H-large, and we use the AdamW optimizer (Loshchilov and Hutter, 2019) with linear learning rate decay. The number of training epochs for the easy, hard, and main stages are set to [15, 30, 30] or [25, 50, 50], with the easy stage having fewer epochs as it typically has more data. For the hard stage, we choose M from {1, 2} for the datasets of the NER, RE, and EE tasks and from {1, 2, 3} for the datasets of the ABSA task. The parameters are chosen based on the model’s performance on the development set. Generally, for large datasets such as ACE05-E, a smaller value of M like 1 is more appropriate, while for smaller datasets such as Laptop14, a larger value of M such as 3 is preferred. All experiments are conducted on NVIDIA Tesla A100.
A.3 Examples of IE tasks Detailed examples of different IE tasks are shown in Tables 9-13. We use the structural extraction language proposed by Lu et al. (2022) to encode the target structure.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The last section "Limitation"
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 0 "Abstract" and Section 1 "Introduction"
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 "Experiments"

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 "Experiments"
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 "Experiments"
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.1 "Statistics of Datasets"
C 3 Did you run computational experiments? Section 4 "Experiments"

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 "Experiments" and Appendix A.2 "Implementation Details"
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 "Experiments" and Appendix A.2 "Implementation Details"

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 "Experiments"
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Exploring more approaches to obtain the skill sets is also open for future research. We plan to investigate these possibilities in our future work