NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing
Year: 2023

Abstract:
Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade model generalization. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance. Due to the lack of suitable datasets, previous studies have frequently employed synthetic label noise to mimic realworld label noise. However, synthetic noise is not instance-dependent, making this approximation not always effective in practice. Recent research has proposed benchmarks for learning with real-world noisy labels. However, the noise sources within may be single or fuzzy, making benchmarks different from data with heterogeneous label noises in the real world. To tackle these issues, we contribute NoisywikiHow, the largest NLP benchmark built with minimal supervision. Specifically, inspired by human cognition, we explicitly construct multiple sources of label noise to imitate human errors throughout the annotation, replicating realworld noise, whose corruption is affected by both ground-truth labels and instances. Moreover, we provide a variety of noise levels to support controlled experiments on noisy data, enabling us to evaluate LNL methods systematically and comprehensively. After that, we conduct extensive multi-dimensional experiments on a broad range of LNL methods, obtaining new and intriguing findings.1

heading: 1 Introduction
text: Large-scale labeled data has become indispensable in the notable success of deep neural networks (DNNs) in various domains and tasks (Russakovsky et al., 2015; Wang et al., 2019). Due to imperfect sources like crowd-sourcing and web crawling (Xiao et al., 2015; Zhang et al., 2017b; Lee
∗Corresponding author. 1The dataset is publicly available at https://github.
com/tangminji/NoisywikiHow.
et al., 2018), datasets frequently include real-world label noise (Chen et al., 2021), which may induce model overfitting to noisy labels and hurt the generalization of deep models (Zhang et al., 2017a; Wu et al., 2022a,b). To alleviate this issue, learning with noisy labels (LNL) methods for robustly training deep models have been studied extensively.
Due to the lack of appropriate benchmarks, previous research often studied synthetic label noise to simulate real-world label noise (Zhang et al., 2018; Lukasik et al., 2020). As a general and realistic noise, real-world noise may have several noise sources (i.e., be heterogeneous) (Northcutt et al., 2021) and be instance-dependent (i.e., P (ỹ|y, x), where the probability of an instance being assigned to the incorrect label ỹ depends on the original ground-truth label y and data x) (Han et al., 2021; Song et al., 2022). However, synthetic noise is generated from an artificial distribution and is thus instance-independent (i.e., P (ỹ|y)), which may not always work well in practice.
Recently, various benchmarks for learning with real-world noisy labels have been proposed across fields like computer vision (CV) (Li et al., 2017), audio signal processing (ASP) (Gemmeke et al., 2017), and natural language processing (NLP) (Hedderich et al., 2021). To fully evaluate robust learning methods with real-world label noise, benchmarks should be as close to real-world scenarios as possible. Meanwhile, controlled ex-
4856
periments are encouraged to verify whether LNL methods can remain effective over a wide range of noise levels (Jiang et al., 2020). Nevertheless, the noise levels in most datasets are fixed and unknown, resulting in uncontrolled label noise (Fonseca et al., 2019a; Song et al., 2019). Moreover, the noise therein often comes from the same or ambiguous sources (Li et al., 2017; Jiang et al., 2020), which conflicts with the heterogeneous characteristics of real-world noise. These problems prevent a better understanding of LNL methods.
To bridge this gap, we present NoisywikiHow, a new NLP benchmark for evaluating LNL methods focusing on the intention identification task. Intention identification promotes numerous downstream natural language understanding tasks, from commonsense reasoning (Sap et al., 2019) to dialogue systems (Pepe et al., 2022). Additionally, the complexity of the task (total of 158 categories) facilitates a deeper investigation of the efficacy of LNL approaches. The task form is shown in Table 1.
To make the benchmark more representative of real-world scenarios, we propose a practical assumption: Real-world label noise in a dataset is mainly induced by human errors, regardless of whether the dataset’s construction is automated or crowd-sourced. Existing psychological and cognitive evidence further supports our hypothesis. It shows that different annotators have different preferences and biases (Beigman and Klebanov, 2009; Burghardt et al., 2018), which means human labeling errors typically result from multiple noise sources. Furthermore, humans may make random labeling errors due to random attention slips. But they are more likely to produce label noise when labeling hard cases (Klebanov et al., 2008) (i.e., noise is instance-dependent), such as instance (c) in Table 1.
Motivated by this human cognition, we first collect data from the wikiHow website,2 which contains a collection of professionally edited how-to guideline articles, providing a vast quantity of clean scripts and corresponding categories for free to help achieve controlled experiments and ensure benchmark quality. After that, we explicitly inject a variety of noise sources into clean data to replicate human annotation errors, thus introducing real-world label noise into the benchmark. Notably, training samples in our benchmark exhibit a long-
2https://www.wikihow.com
tailed class distribution, which is in line with the facts, i.e., data in real-world applications is heavily imbalanced (Van Horn et al., 2018; Liu et al., 2019b). Besides, we achieve minimal human supervision by using a series of automated labeling procedures, saving lots of time and human effort.
To evaluate NoisywikiHow, we carry out extensive experimentation across various model architectures and noise sources, execute plentiful LNL methods on our benchmark, compare the more realistic real-world noise with the extensively studied synthetic noise, and investigate a case study and long-tailed distribution characteristics.

heading: 2 Related Work

heading: 2.1 Datasets with real-world noisy labels
text: In early studies of the LNL problem, due to a lack of appropriate benchmarks, synthetic noise was often used to reflect noise in the real world and assess the effectiveness of methods (Han et al., 2018b; Zhang et al., 2018). However, unlike real-world noise, synthetic noise follows an idealized artificial distribution, which leads to inaccurate approximations and inadequate evaluations.
Recent studies have proposed numerous datasets with real-world noisy labels. Table 2 depicts a comparison of existing real-world noisy datasets for evaluating LNL methods in CV, ASP, and NLP. As shown in Table 2, most datasets fail to perform controlled experiments on real-world label noise and cannot be used to study DNNs across different noise levels (Fonseca et al., 2019a,b).
A few benchmarks with controlled label noise, such as NoisyNER (Hedderich et al., 2021) and Red MiniImageNet (Jiang et al., 2020), were produced. However, the noise source in their datasets may be vague. Furthermore, NoisyNER focuses on the named entity recognition task in NLP. Though seven noisy label sets are provided, it is challenging to determine the precise noise level of each label set because a sentence-level instance has numerous word-level labels. Besides, Red MiniImageNet relies heavily on careful human annotation and follows a balanced class distribution, which diverges from real-world application scenarios. In this paper, we publish NoisywikiHow to solve the above limitations. As shown in Table 2, to the best of our knowledge, NoisywikiHow is the largest NLP benchmark for assessing LNL methods.
Dataset Classes Distribution Controlled HumanAnnotation Size

heading: 2.2 Intention identification
text: Intention identification is critical to many applications (Huang et al., 2016; Sap et al., 2019). Therefore, ensuring task reliability is essential. Some previous work formulates intention identification as an event process typing task. Given a sequence of events, the model is designed to understand the overall goal of the event process in terms of an action and an object (Chen et al., 2020; Pepe et al., 2022). In other studies, intention identification is modeled as a sentence classification task (Zhang et al., 2020a,b). When given a procedural event, the system predicts its intention in a 4-choose-1 multiple-choice format. However, none of these studies deal with task reliability. By building NoisywikiHow, we make a preliminary exploration of task reliability (i.e., model performance under label noise). Following Zhang et al. (2020b), we model intention identification as a sentence classification task. The difference is that our benchmark (including 158 labels) is analogous to the retrieval task in a more practical and challenging way.

heading: 3 NoisywikiHow Dataset

heading: 3.1 Data Collection
text: We construct NoisywikiHow by crawling how-to articles from the wikiHow website. Detailed crawling strategies and related statistics are in Appendix A.1. We define the input as a procedural event, i.e., the header of a paragraph in a wikiHow article (e.g., Talk about food differently in Table 1), and the output as the intention of the event, namely the category of this article (e.g., Losing Weight in Table 1). Note that categories present a hierar-
chy (e.g., Health ≫ Nutrition and Food Health ≫ Weight Management ≫ Losing Weight), and we select the category with the finest granularity as the label.

heading: 3.2 Data Cleaning
text: Similar to Jiang et al. (2020) and Hedderich et al. (2021), we realize controlled label noise by injecting various amounts of noise into clean data. However, the data collection process introduces a lot of low-quality or irrelevant data. As a result, we develop a data cleaning procedure to remove bad data and facilitate the target task from two aspects: (1) input filtering and (2) label filtering. Regarding input filtering, we first devise four automatic filters and execute them sequentially to remove low-quality or ambiguous data.
• Sample Length Filter intends to retain instances with more informative and complete semantic information by filtering excessively short or long data.
• Format Normalization is to standardize in-
stances (e.g., unifying the description of “Click Defragment Your Hard Drive.” and “Click Defragment your hard drive.”), ensuring the effectiveness of subsequent strategies.
• Deduplication tries to eliminate redundant or ambiguous data (e.g., a procedural event corresponds to multiple intents).
• TF-IDF Filter attempts to preclude overly uninformative instances by calculating the TFIDF for each token.
After that, we receive high-quality data Dh, which follows a long-tailed class distribution with limited data on tail classes, resembling the distribution in Fig. 1. We create a Sample Size Filter to exclude the categories with too few samples (≤ 300), ensuring an appropriate split of training, validation, and test sets.
We observe that the labels have two types, i.e., concepts defined as nominal phrases (e.g., Nutrition and Food Health), and event mentions defined as nominal or verbal phrases that refer to events (e.g., Losing Weight) (Min et al., 2020; Yu et al., 2021). Therefore, label filtering is required to retain only events, ensuring the effectiveness of intention identification. Specifically, each category is annotated by three graduate students from the NLP field and is regarded as an event if more than two annotators agree. Human annotators are asked to label 736 categories and achieve a high agreement (Fleiss-κ = 0.84) (Fleiss, 1971).
After data cleaning, we obtain clean data D involving 89,143 instances in 158 classes. Due to the limited space, complete filtering strategies and more details are in Appendix A.2.

heading: 3.3 Label Noise Injection
text: To create a benchmark of real-world noisy labels, we introduce various sources of controlled label noise into the clean data. Prior to this, we assume that human mistake is the primary cause of real-world label noise in a dataset. Psychological and cognitive findings further corroborate the rationality of the assumption. It demonstrates that: (1) apparent differences between annotators result from different preferences and biases (Reidsma and op den Akker, 2008; Beigman and Klebanov, 2009; Burghardt et al., 2018), suggesting that human errors are heterogeneous; (2) label noise from humans regularly affects hard cases (Klebanov et al., 2008; Klebanov and Beigman, 2009), proving that noise is instance-dependent.
Heterogeneous noise sources. Based on the above preliminaries, we simulate various mistakes committed by annotators to produce real-world noise containing heterogeneous noise sources. Specifically, human errors are often induced by ambiguity, insufficient annotator expertise, and random attention slips (Beigman and Klebanov, 2009; Hollenstein et al., 2016). Motivated by this, we develop three noise sources as follows:
• Sub-categories (SC) under the same category (e.g., Starting a Business and Running a Business) tend to have higher semantic similarities and can be easily confused. SC depicts the noise caused by labeling ambiguous instances.
• Intents beyond the commonsense categories (BCC) are hard to identify (e.g., Dog Grooming), readily inducing noisy labels. BCC portrays a scenario annotated by a human lack of expert knowledge.
• Considering the long-tailed distribution, even a few labeling errors on tail classes can seriously affect learning of these categories. Therefore, achieving robust training on tail classes is critical. We concentrate on intents under the tail categories (TC), which describe the noise generated by humans randomly shifting their attention.
Then, we design a simple mapping from noise sources to classes to facilitate the subsequent injection of noise from different sources and categories. Specifically, each class is associated with a noise source, and classes under various noise sources do not overlap. This mapping can cover all categories during noise injection and determine the potential noise source for each class. Finally, we divide 158 categories into 68, 36, and 54 to correspond to the sources SC, BCC, and TC, respectively. More details about the mapping can be found in Appendix A.3. Injecting instance-dependent label noise. Since each noise source contains a set of categories, each of which may involve hard cases, instancedependent label noise exists in each noise source. Note that real-world label noise always comes from an open rather than a finite category set (Wang et al., 2018). We therefore enable label noise to derive from categories other than the current label set. However, this operation changes the number of labels and impacts the target classification task. To solve this problem, when injecting label noise
into an instance (x, y), we leave the label y (output) unchanged like Jiang et al. (2020) but replace the procedural event x (input) with the one (x̃) under the other category (ỹ), which may not be in the existing 158 classes. Moreover, NoisywikiHow supports five noise levels (i.e., 0%, 10%, 20%, 40%, and 60%). Like Li et al. (2017) and Saxena et al. (2019), we assume that, given a specified noise level t, t is uniform across noise sources. For example, t = 10% represents that each source has roughly 10% label noise.
We further identify hard cases and inject instance-dependent noise for each noise source. Intuitively, when we mislabel an instance from (x, y) into (x̃, y), if (x, y) is a hard case, the semantic representations of events x and x̃ should be very similar. As a result, for any (x, y), we can assess its difficulty by finding an (x̃, ỹ) whose x̃ has the maximum semantic similarity with x. To identify (x̃, ỹ), we take the following steps: (1) Determine Dn: the candidate set of (x̃, ỹ). To avoid introducing bad data or duplicate data after noise injection, we construct Dn as follows:
• For the sources BCC and TC, Dn = Dh −D. • For the source SC, let Ds be the sample set
of all other sub-categories except y under the same category, and Dn = (Dh −D) ∩ Ds.
(2) Locate x̃ in Dn. Following Zhang et al. (2020b), we map each event to a vector representation by taking the average of the BART embeddings (Lewis et al., 2020) of the verbs. x̃ thus can be calculated as:
x̃ = argmax vx′
cosine(vx, vx′), (x ′, y′) ∈ Dn, (1)
where v(·) is the vector representation of an event, and cosine(·) denotes the cosine similarity of two vectors. For any (x, y), its difficulty can be obtained by calculating a score sx:
sx = cosine(vx, vx̃). (2)
The larger the sx, the harder the instance (x, y). We inject noise into the training set Dtr ⊂ D. Given a specified noise level t (e.g., 10%), all instances in Dtr are arranged in decreasing order of sx, with the top t of the samples in each source considered hard cases. We inject instance-dependent noise by replacing x for each hard case with x̃.

heading: 4 Experiments
text: We first present the general settings for experiments (Section 4.1). Further, we systematically evalu-
ate our benchmark with varied model architectures (Section 4.2) and noise sources (Section 4.3). Also, we assess a broad range of LNL methods on NoisywikiHow (Section 4.4) and compare real-world noise with synthetic noise (Section 4.5). Finally, we conduct a case study (Section 4.6). In addition, we discuss the long-tailed distribution characteristics of NoisywikiHow in Appendix B.2.

heading: 4.1 Experimental settings
text: On our benchmark, all methods are trained on the noisy training sets3 and evaluated on the same clean validation set to verify whether these approaches can resist label noise during training and achieve good generalization on the noise-free data. Before adding label noise, we randomly split out 15,800 instances from clean data and then equally divide them into two sets: a validation set and a test set. The remaining 73,343 instances serve as the training set, which follows a typical long-tailed class distribution and is analogous to heavily imbalanced data in real-world applications, as shown in Fig. 1. The statistics of NoisywikiHow are shown in Table 3. We cast intention identification as a classification problem. We exploit the cross-entropy loss for training models and use Top-1 accuracy and Top-5 accuracy as the evaluation metrics.

heading: 4.2 Comparison of Model Architectures
text: Baselines: We first evaluate the performance of different model architectures under varying levels of real-world label noise. Regarding the model architectures, we use seven state-of-the-art (SOTA) pre-trained language models, including BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019a), GPT2 (Radford et al., 2019), ALBERT (Lan et al., 2020), T5 (Raffel et al.,
3Synthetic noise and various noise sources under realworld noise correspond to diverse noisy training sets.
2020), and BART (Lewis et al., 2020). We finetune each model for 10 epochs with batch size 64, learning rate 3e-5. These hyperparameters remain unchanged in subsequent experiments unless indicated otherwise. In this paper, we conduct all experiments utilizing the base-sized version of the pre-trained language models. Besides, due to long output sequences in partial categories, we adopt beam search (Sutskever et al., 2014) in T5, with a beam width of 5 and a length penalty of α = 1.0.
Results: As shown in Table 4, the Top-1 accuracies of SOTA pre-trained language models on our benchmark are generally not high, and an increase in noise levels can lead to considerable performance degradation for a given model, demonstrating the challenge of the NoisywikiHow dataset.
In Table 4, different architectures are representative of diverse capacities. For example, RoBERTa and XLNet consistently outperform ALBERT under different noise levels. In addition, we observe that BART achieves the best performance among these SOTA models under a majority of noise levels, regardless of Top-1 or Top-5 classification accuracy. This is mainly because a better denoising objective (i.e., masked language modeling) is used during pre-training of BART. In pre-training, BART gains better denoising ability by corrupting text with an arbitrary noise function (thus making the noise more flexible) and learning to reconstruct the original text. In the following, we use the BART model as the base model.

heading: 4.3 Effects of Distinct Noise Sources
text: We further explore the characteristics of different noise sources. To this end, we pick the same model (i.e., the base model) and separately validate the performances on individual noise sources under the same noise level. For convenience, we denote
noise-free data by correct samples and data with label noise by incorrect samples.
Results: Table 5 shows the results of the base model under four different noise sources with 10% label noise. As shown in Table 5, there exists an evident gap between the results under noise source TC and those in other conditions. The label noise from noise source TC is more difficult to mitigate than others at the same noise level, mainly due to the limited data on tail categories. When all noisy labels are derived from TC, fewer correct samples are left, leading to inadequate model training and degradation of model performance. It indicates that resisting label noise from different sources may have varying difficulty levels, although the noises in these sources are all real-world label noise. Additional details are provided in Appendix B.1.

heading: 4.4 Effectiveness of Different LNL methods
text: Baselines: We perform an extensive evaluation of the existing LNL methods on our benchmark. Seven representative baselines are involved for comparison: (1) Base model, which finetunes the BART model with no extra LNL methods; (2) Mixup (Zhang et al., 2018), which mitigates memorization of noisy labels by DNNs regularization, i.e., introducing a data-agnostic data augmentation routine; (3) Data Parameter (Saxena et al., 2019),
which equips learnable parameters to help DNNs generalize better via learning from easier instances first; (4) SR (Zhou et al., 2021), which introduces the sparse regularization strategy, making any loss robust to noisy labels conforming to the specified assumption; (5) Co-teaching (Han et al., 2018b), which combats noisy labels by training two networks, and each network aims to teach the other one with clean data, i.e., the instances with smallloss; (6) CNLCU (Xia et al., 2022), which considers the uncertainty of loss estimation to refine correct sample selection; (7) SEAL (Chen et al., 2021), which provides instance-dependent label correction to resist real-world noise. Complete experimental results and unique hyperparameters for each noise level for each baseline are in Tables 8 and 9 in the Appendix.
Results: As Fig. 2(a) shows, Mixup outperforms the base model with limited performance improvement. It is because Mixup fails to consider the specialty of real-world label noise and improves generalization with a generic regularizationbased method. The performance of Data Parameter is comparable to or slightly better than the base model under different noise levels. Although Data Parameter models the situation that instances within a class have different difficulty levels, it assumes small-loss training samples as correct samples and splits correct and incorrect samples via a loss-based separation. However, as shown in Fig. 3(a), loss distributions of correct and incorrect data overlap closely in the real-world label noise, making Data Parameter has no advantage under real-world label noise. Similarly, Co-teaching and CNLCU fulfill sample selection following the same assumptions. They perform worse than the base model, with the exception of individual noise levels. It implies that Co-teaching and CNLCU are inapplicable to the heterogeneous and instancedependent label noise. SR precedes the base model only at certain noise levels. This is because SR guarantees noise tolerance if and only if the label noise satisfies the instance-independent condition, which is inconsistent with noise in the real world. Hence, the validity of SR is not ensured on our benchmark. SEAL consistently outperforms the base model by a large margin on all noise levels, as SEAL provides instance-dependent label correction to combat real-world noise. However, during the correction, SEAL retrains the classifier using

heading: 60% noise

heading: 40% noise
text: the averaged soft labels, introducing excessive computational overhead.

heading: 4.5 Real-world Noise vs. Synthetic Noise
text: Aside from the real-world label noise, synthetic label noise is one of the most widely studied label noises (Patrini et al., 2017; Wang et al., 2018; Reeve and Kabán, 2019). Unlike real-world noise, which is widespread in real applications, synthetic noise does not exist but is generated from artificial distributions. We further examine the differences between the two label noises. In this paper, synthetic label noise is implemented with symmetric label noise (Han et al., 2018a; Charoenphakdee et al., 2019) (the most common synthetic noise), assuming each label has the same probability of flipping to any other class. We build the dataset of controlled synthetic label noise by injecting a series of synthetic label noises into clean data in a controlled manner (i.e., 10%, 20%, 40%, and 60% noise levels). We pick the same baselines as in Section 4.4. More details are in Tables 10 and 11 in the Appendix.
Results: As shown in Fig. 2(b), SEAL and Mixup consistently outperform the base model, showing their advantages in combating synthetic label noise. Unlike the real-world label noise, SR is effective for the synthetic label noise and achieves improvement over the base model regardless of the
noise levels since the synthetic label noise meets the instance-independent condition. Besides, Coteaching, Data Parameter, and CNLCU improve the base model by an apparent margin under the synthetic label noise. In this case, as shown in Fig. 3(b), the loss distributions of correct and incorrect samples can be well split, allowing loss-based separation to work well.
We discover that few LNL methods can effectively resist both real-world and synthetic noises simultaneously, highlighting the imperative of benchmark construction. Many LNL approaches can mitigate the synthetic but not real-world label noise. It is because synthetic noise is generated from artificial distributions to approximate real-world noise. The mislabeled probability is independent of each instance under synthetic noise but dependent on distinct instances under real-world noise, which makes complex modeling of the latter. Thus, our benchmark contributes to a more systematic and comprehensive assessment of LNL methods. Further, since most LNL method evaluation datasets focus on the CV and ASP, our NLP benchmark facilitates the modal integrity of the existing datasets.
We also contrast the performance of the base model trained for 20 epochs under real-world label noise and synthetic label noise. In Fig. 4, as the running epochs and noise levels increase, the test accuracy curve with the real-world noise (Fig. 4(a))
is much flatter than that with the synthetic noise (Fig. 4(b)) at the same noise level (e.g., with 40% and 60% noise). It demonstrates that the model generalizes much better under real-world noise than synthetic noise of the same noise level.

heading: 4.6 Case Study
text: We construct a benchmark encompassing realworld noise involving multiple noise sources with minimal human supervision, which is analogous to human errors during annotation. To observe the dataset more clearly and intuitively, we randomly select five incorrect instances (i.e., samples with noisy labels) across multiple noise sources. As indicated in Table 6, we find it difficult to determine whether the sample contains noise. On the other hand, for any sample, the noise label and the respective ground-truth label are overly similar, making it challenging to distinguish one from another.

heading: 5 Conclusion
text: In this paper, we study the problem of learning with noisy labels and establish an NLP benchmark called NoisywikiHow with minimal human supervision, which contains more than 89K procedural events with heterogeneous and controlled realworld label noise. Experimental results reveal several new findings. (1) Some widely accepted LNL methods are not always impactful, especially with real-world label noise. (2) Different noise sources may have varying difficulties resisting label noise, although they are all from real-world noise. (3) Few LNL methods can effectively combat real-world noise and synthetic noise at the same time. (4) The model trained under the real-world label noise has better generalization performance.
Limitations
In this paper, we simplify intention identification into a sentence classification task, i.e., exploiting a specific procedural event in an event process to predict the intention of the whole event process. A more realistic way to model this task is to enter the entire event process rather than a single event. We will go into more detail about this type of task in future work.
Ethics Statement
This work presents NoisywikiHow, a free and open dataset for the research community to study learning with noisy labels. Since the data in Noisywiki-
How is constructed based on the wikiHow website, which is free and open for academic usage, there is no privacy issue. We declare that all information in this paper has been obtained and presented following the ACL Ethics Policy. As required by these rules and conduct, we have fully cited and referenced all material and results that are not original to this work.

heading: Acknowledgements
text: We thank the anonymous reviewers for their constructive comments, and gratefully acknowledge the support of the Technological Innovation “2030 Megaproject” - New Generation Artificial Intelligence of China (2020AAA0106501), the National Natural Science Foundation of China (U22B2059, 62176079, 62106061), and the Natural Science Foundation of Heilongjiang Province (YQ2022F005).

heading: A Details of Dataset Construction
text: A.1 Crawling Strategy
According to wikiHow’s crawler rules,4 we use the crawling platform Scrapy (Kouzis-Loukas, 2016) to crawl all the articles in the 19 top-level categories (e.g., Arts and Entertainment, Computers and Electronics, etc.) of the latest wikiHow website, with a total of 100,623 pages (how-to articles), including 1,407,306 samples in 3,334 categories, as shown in Table 7.
A.2 Filtering Strategies
In the main paper, we apply a collection of filters to ensure low-quality instances removal, better dataset division, and task effectiveness. The details of each filter are as follows:
Sample Length Filter: We remove instances with overly short or long event descriptions or with icon information. As too-short events may be less informative, too-long depictions may exceed the length restriction of the pre-trained language model. Icons in events present rich text starting with “smallUrl” without specific semantic information and may interfere with the understanding of procedural events.
Format Normalization: We observe that some identical event descriptions would be slightly different in distinct articles (e.g., “Click Defragment Your Hard Drive.” and “Click Defragment your hard drive.”). Prior to the deduplication procedure, we devise format standardization operations. The manipulations involve standardizing varied languages and symbols with Unidecode, stopword exclusion and lemmatization with spaCy (Honnibal and Montani, 2017), word segmentation & POS tagging by applying the model “en_core_web_sm” in spaCy and reserving events containing verbs.
Deduplication: We first apply inter-class deduplication to remove instances with labels of multiple categories. Then, we filter out repeated samples to achieve in-class deduplication. After the deduplication operation, each procedural event (i.e., event) corresponds to a unique event intent (i.e., category).
TF-IDF Filter: We exploit the TF-IDF filter to preclude events from being overly uninformative when identifying the corresponding event intent and guarantee the instances are representative. Specifically, each wikiHow article is considered a document. We calculate the TF-IDF for each token
4https://www.wikihow.com/robots.txt
and preserve only the events containing keywords. In this context, keywords refer to tokens whose TF-IDF values are in the top 10% in decreasing order. Each article includes a minimum of 3 and a maximum of 10 keywords.
Label Filter: We filter labels by manual annotation to retain categories depicting only events. For human labeling, we used three graduate students from the NLP field. They were educated for two hours about annotation strategy before the labeling process. Specifically, we use Min et al. (2020)’s and Yu et al. (2021)’s definitions of event mention (i.e., an event with surrounding context (text)) as guidelines for annotating events. In addition, categories exhibit a hierarchical structure. Typically, the descriptions of the upper categories are relatively general and vague (e.g., Cleaning), while the more fine-grained categories have more specific intentions (e.g., Kitchen Cleaning, Cleaning Metals). Accordingly, we label the category with the finest granularity as an event except for two cases.
• If a candidate category has a broad intent meaning (e.g., Selling in Finance and Business ≫ Managing Your Money ≫ Making Money ≫ Selling), it will not be considered an event.
• If it is difficult to distinguish semantically between two candidate categories, the category with the larger sample size is designated as an event. For example, in hierarchical categories (Hobbies and Crafts ≫ Crafts ≫ Needlework ≫ Knitting and Crochet ≫ Crochet ≫ Crochet Stitches), we label Crochet (with 1,263 samples) as an event rather than Crocheet Stitches (with 445 samples).
This annotation strategy facilitates the balance between definite event intent and sample size. Sample size and class info reserved after data cleaning are provided in Table 7.
A.3 Mapping from Noise Sources to Classes
In the main paper, we briefly present the correspondence between noise sources and task categories. In particular, we first define 54 tail categories, each containing no more than 400 samples. Following that, we draw on the discussion of commonsense knowledge in Liu and Singh (2004)5 and use it as a guideline for labeling categories beyond commonsense. We define the overall 45 categories beyond
5See Section 1.1 for more details.
commonsense by asking three annotators to label 158 categories as commonsense or not, fulfilling a high agreement (Fleiss-κ = 0.88). To ensure that the label sets under different noise sources do not overlap, we remove 9 categories also appearing in tail categories from the 45 categories and eventually receive 36 categories beyond commonsense. Lastly, the remaining 68 of the 158 classes are designated as the noise source SC.

heading: B Experiments Details
text: For each experimental dimension, we refine the hyperparameters for every baseline across different noise levels. Optimal hyperparameters are obtained by using a popular hyperparameter optimization tool Hyperopt (Bergstra et al., 2013).
B.1 Effects of Distinct Noise Sources
We examine the base model’s performance under four different noise sources. In addition, Fig. 5 further compares the efficacy of typical LNL methods under various noise sources. We discover that regardless of the method employed, they are all less effective in reducing the effect of the noise source TC, further confirming our point of view.
B.2 Long-tailed Distribution Properties
Baselines: Our training set follows a typical longtailed class distribution akin to that in the real world. However, DNNs can be readily biased towards dominant classes with massive training data, triggering poor model performance on tail classes with limited data. This problem inspires large numbers of long-tailed learning studies. To fully explore the characteristics of the NoisywikiHow dataset, we select five long-tailed learning methods in three classical categories as baselines: (1) BBN (Zhou et al., 2020), which applies a resampling strategy to sample more tail-class samples for improving tailclass performance; (2) LDAM (Cao et al., 2019), which rebalances classes by designing an effective loss and training schedule; (3) Decoupling (Kang et al., 2020), which decouples the learning procedure (including three baselines: Decoupling-NCM, Decoupling-cRT, and Decoupling-LWS) to understand how the long-tailed recognition ability is achieved. Complete experimental results of longtailed learning methods are shown in Table 12. We also demonstrate the settings of optimal hyperparameters in Table 13.
Results: We focus on the relative performance boost with various baselines in original papers and that on Noisywikihow. In Fig. 6, we find that all baselines evaluated on the CV datasets can address the long-tailed problem properly and achieve a significant test accuracy boost (7.37%–20.9%) in the original papers. However, as shown in Fig. 7, the performance improvements across varied noise levels on our NLP benchmark are limited, with some methods not exceeding the base model (- 0.07%–2.56%).
Experimental results indicate that the effectiveness of long-tailed learning methods needs to be examined on datasets with different modals. Moreover, although the base model obtains performance degradation with the increase in the noise level, the
effectiveness of each long-tailed learning method is not significantly affected by the noise level variation. The main reason is that the test accuracy we report is the best peak accuracy, producing an effect similar to the early stop and thus preventing the model from overfitting label noise.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: See section Limitations.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: See abstract and introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? See Section 3.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? See Section 2.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. See Section 4.1
C 3 Did you run computational experiments? See Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? See Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? See Section 4.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report results from another dimension, i.e., we simultaneously use Top-1 accuracy and Top-5 accuracy as the evaluation metrics.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? See Appendix A.2.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? See Section 3.2 and Appendix A.3.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? See Appendix A.2 and A.3.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? See Section 3.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? See Appendix A.1.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: See Section 3.

heading: Future Work
text: We will go into more detail about this type of task in future work