EconBERTa: Towards Robust Extraction of Named Entities in Economics
Authors: Karim Lasri
Pedro Vitor Quinta de Castro
Mona Schirmer
Luis Eduardo San Martin
Linxi Wang
Tomáš Dulka
Haaya Naushan
John Pougué-Biyong
Arianna Legovini
Samuel Fraiberger
Venue: EMNLP 2023 Findings (2023)
Keywords: Named Entity Recognition
Large Language Model
Domain Adaptation
Generalization

Abstract: Adapting general-purpose language models has proven to be effective in tackling downstream tasks within specific domains. In this paper, we address the task of extracting entities from the economics literature on impact evaluation. To this end, we release EconBERTa, a large language model pretrained on scientific publications in economics, and ECON-IE, a new expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). We find that EconBERTa reaches state-of-the-art performance on our downstream NER task. Additionally, we extensively analyze the model's generalization capacities, finding that most errors correspond to detecting only a subspan of an entity or failure to extrapolate to longer sequences. This limitation is primarily due to an inability to detect part-of-speech sequences unseen during training, and this effect diminishes when the number of unique instances in the training set increases. Examining the generalization abilities of domain-specific language models paves the way towards improving the robustness of NER models for causal knowledge extraction.

EconBERTa: Towards Robust Extraction of Named Entities in Economics
Karim Lasri Pedro Vitor Quinta Da Castro Mona Schirmer
Luis Eduardo San Martin Linxi Wang Thomáš D Haaya Naushan"
John Pougue Biyong** Arianna Legovini" Samuel P. Fraiberger
oThe World Bank "Ecole Normale Superieure "Universidade Federal de Goiás
University of Amsterdam 'Massachusetts Institute of Technology
University of Oxford "New York University
(klasri, pquintadecastro, mschirmer, lsanmartin, 1wang34, tdulka, hnaushan, jpouguebiyong, alegovini, sfraiberger)@worlddamm org
Abstract
mark datasets have enabled researchers to com- pare and improve the performances of NER models within specific domains such as science (Luan al., 2018), medicine (Jin and Szolovits, 2018), law (Au et al., 2022), finance (Salinas Alvarado et al., 2015), and social media (Ushio et al., 2022); in some cases, these datasets have spanned multiple domains (Liu et al., 2020b) or languages (Tjong Kim Sang and De Meulder, 2003). Such datasets are crucial for building models capable of handling a wide range of downstream applications. In medicine for in- stance, information extraction systems can allow clinician and medical researchers to easily deter- mine which medication is an effective treatment for specific disease. Analogously, knowing which policy intervention produces certain economic outcome is imperative for evidence-based policy- making. However, while much work has been done on identifying relevant causal entities in biomedi- cal research (Chang et al., 2022), this question has been unexplored for impact evaluation (IE) in eco- nomics. To fill this gap, we introduce ECON-IE, a NER dataset of 1,000 abstracts from economic re- search papers annotated for entities describing the causal effects of policy interventions: intervention, outcome, population, effect size, and coreference. To the best of our knowledge, this is the first dataset of this kind in economics, thereby laying founda- tions for causal knowledge extraction in the field.
Adapting general-purpose language models has proven be effective in tackling downstream tasks within specific domains. In this paper, we address the task of extracting entities from the economics literature on impact evaluation. To this end, we release EconBERTa, a large language model pretrained on scientific pub- lications in economics, and ECON-IE, new expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). We find that EconBERTa reaches state-of sate-of-thhearr perfor- mance on our downstream NER task. Addi- tionally, extensively analyze the model's generalization capacities, finding that most cr- rors correspond to detecting only a subspan of entity or failure to extrapolate longer se- quences. This limitation is primarily due to inability to detect part-of-speech sequences unseen during training, and this effect dimin- ishes when the number of unique instances in the training set increases. Examining the gener- alization abilities of domain-specific language models paves the way towards improving the robustness of NER models for causal knowl- edge extraction.
1 Introduction
Implementing robust systems capable of automat- ically extracting structured information from un- structured text is critical for a variety of applica- tions, ranging from event detection (Nguyen and Grishman, 2015; Bekoulis et al., 2019; Tong et al., 2020; Pouran Ben Veyseh et al., 2021; Liang et al., 2022) to building knowledge databases (Khoo et al., 2000; Kim et al., 2020; Harnoune et al., 2021; Wang et al., 2022). In particular, Named Entity Recognition (NER) is canonical information ex- traction task which consists of detecting text spans and classifying them into a predetermined set of en- tity types (Tjong Kim Sang and De Meulder, 2003; Lample et al., 2016a; Chiu and Nichols, 2016; Ni et al., 2017). In the past decade, numerous bench-
The introduction of pretrained language mod- els, sometimes referred to as foundation models (Bommasani et al., 2022), had a profound impact on the field of information extraction, enabling substantial gains in performance across domains by fine-tuning these -enerol-purpose systems on downstream tasks (Alt et al., 2019; Papanikolaou et al., 2022). In this paradigm, such models are often fine-tuned and deployed to perform a domain- specific task, introducing a distribution shift be- tween the pretraining data and examples seen at inference time. Existing work shows that pretrain-

---

Pre-training
11
Fine-tuning
III Probing / Diagnosis
NER Annotations
Fine-tuned model
General Domain
Memorization
Research limited about Intervention how reading digital texts affects reading skills of
In Domain (Economics)
Error Types
General + In Domain
primary students) population
Dependence on Length
Figure Illustration of the pipeline for the models under investigation, from modeling to diagnosis
ing a language model documents from the same domain the downstream task improves perfor- mance (Beltagy et al., 2019), where this domain adaptation can either be implemented by in-domain pretraining from scratch (Chu and Wang. 2018; Ri- etzler al., 2020; Grangier and Iter, 2022; Huang et al., 2019), or by further in-domain pretraining starting from the weights of an existing pretrained general-purre purpose model (Lee et al., 2019; Alsentzer et al., 2019). While pretrained language models are available across wide range of domains (Belt- agy et al., 2019; Lee et al., 2019; Chalkidis al., 2020), economics has yet to follow suit. There- fore we contribute EconBERTa, a language model pretrained on 1.5 million scientific papers from economics, demonstrating that outperforms open- source general-purpose pretrained models on the ECON-IE benchmark dataset.
entity but fails to properly delimit its boundaries. We further find that, as the number of tokens in a span increases, the model is more likely to incor- rectly predict boundaries. Lastly, our model extrap- olates more easily to part-of-speech sequences seen during training, tendency exacerbated by a low number of unique spans for a given entity type the training dataset.
Taken together, our contributions can be summa- rized follows:
In $3.1, we introduce ECON-IE, new NER dataset of 1, 000 abstracts from economic re- search annotated for entities describing the causal effects of policy interventions.
In $3.2, we present EconBERTa, a language model pretrained on economics research pa- pers, which demonstrates state-of offte--f-theee tfe-the-rrt per- formance on in-domain NER.
While assessing the quality of a model of- ten achieved by measuring performance after fine- tuming, the knowledge captured by such transfer learning approaches is opaque. Therefore, it re- mains unclear, which aspects of model require improvement in order to increase robustness dur- ing deployment. Understanding the weaknesses of state-of-the ofee----t-tt-tt---e models on downstream tasks re- quires operationalizing the notion generalization. However, we lack a precise definition of general- ization in spite of recent attempts to re-explore the concept (Hupkes et al., 2023). For instance, high performance on held-out test set does not necessarily imply that model is achieving robust generalization (McCoy et al., 2019). In this con- text, we challenge the surface performance of pretrained model, and seek cases where fails to generalize on the NER task. Our examination re- veals three common error patterns. First, we show that most errors occur when the model detects am
$4, we evaluate the generalization capabili- ties of our fine-tuned NER-model by perform- ing a series of diagnostic tests.
The pretrained EconBERTa models, the ECON-IE dataset, the metadata of the economic pretraining corpus and the source code to replicate the ex- periments are available via https: //gi wor worrlbbank/econberaa-cccoo
Related Work
2.1 NLP for economics
Existing work in NLP for economics have focused developing models and datasets for the finance domain. Notably, several studies investigate the pretraining of BERT-based (Devlin al., 2018) architectures on financial data (Araci, 2019; Yang et al., 2020; Peng et al., 2021; Sakaji et al., 2022).

---

Previously released datasets typically comprise news articles or press releases annotated for senti- ment analysis (Malo et al., 2014), event extraction (Jacobs and Hoste, 2022; Lee et al., 2022: Han et al., 2022), opinion analysis (Hu and Paroubek, 2021) or causality detection in finance (Mariko et al., 2020). In addition to news articles, corpo- rate reports (Loukas et al., 2021; Händschke et al., 2018) are text corpora from economics and busi- ness, but lack token-level annotations. Our work fills two gaps by (i) being the first to address infor- mation extraction from scientific economic content and releasing a neural language model pretrained in that domain and (ii) defining a NER annota- tion scheme and releasing an annotated dataset for causal entities of economic impact evaluation.
Colborne and Langlais, 2020; de Araujo and Roth, 2023). In this study, we also propose novel fine- grained analyses aimed at gaining deeper under- standing of our model's performing abilities.
3 Experimental Setup
In this section, we introduce the ECON-IE dataset for NER ($3.1), present the training procedure for EconBERTa, and describe the baseline mod- els ($3.2).
3.1 ECON-IE dataset
Data collection ECON-IE consists of 1,0 ab- stracts from economics research papers, totalling more than 7, 000 sentences." I The abstracts summa- rize impact evaluation (IE) studies, aiming to mea- sure the causal effects of interventions on outcomes by using suitable statistical methods for causal in- ference. The dataset is sampled from 10, 000 stud- ies curated by 3ie² (White and Gaarder, 2009), pub- lished between 1990 and 2022, and covering all 11 sectors defined by the World Bank Sector Taxon- omy (Bumgarner, 2017). To maximize the diversity of our annotated sample, we perform stratified sam- pling, i.c. we assign higher sampling weights to sectors and year ranges under-represented in the 3ie database. 3 We then generate fixed heldout test set by sampling 20% of the abstracts; the remaining 80% are split to perform 5-fold cross-vasd vross-validation set.
2.2 Domain adaptation
Previous work has also highlighted the utility of do- main adaptation when developing models they can handle tasks in specific domains, as the distri- bution of input texts can different in such cases. This has given rise to a number of adaptation tech- niques (Daumé III, 2007; Wiese et al., 2017; Ma et al., 2019; Cooper Stickland et al., 2021; Grang- ier and Iter, 2022; Ludwig et al., 2022). In the pretrain-fine-tune paradigm, for pretrained models to generalize over task in specific domain, it is advised fine-tune them on domain-specific datasets, which requires domain-specific annotated resources (Tsatsaronis et al., 2015; Zhu et al., 2022; Au et al., 2022; Li et al., 2021). In this paper, we test whether in-domain pretraining improves per- formance on a tionally try to gain a better understanding on these models' weaknesses by examining their generaliza- tion abilities.
Annotation procedure As our goal to extract information relevant to describe causal analyses performed in impact evaluation studies, we draw from annotation schemes for clinical randomized control trials. In evidence-based medicine, the PICO elements are common knowledge repre- sentation structure and consists of population, in- tervention, comparators and outcome (Huang et al., 2006). We further adjust the PICO scheme to rep- resent the causal knowledge found in impact eval- uation studies. We first follow Nye et al. (2018) and collapse comparators and intervention into a single category. We further extract two additional entity types, namely effect size as this information key to compare the effectiveness of various inter- ventions on an outcome, and coreference as spans referring to previously mentioned entity can be useful for a downstream relation extraction model.
task, but we addi-
2.3 Diagnosing fine-tuned models
Since the rise of general purpose pretrained model, examining the generalization abilities of fine-tuned models has sparked interest the NLP commu- nity. Early work has shown that is possible to achieve seemingly high performance without learn- ing a given task, and by relying on heuristics and spurrious correlations (McCoy et al., 2019; Niven and Kao, 2019; Zellers et al., 2019; Xu et al., 2022; Serrano et al., 2023). This has pushed recent efforts complement coarse-grained metrics such ac- curacy and Fi-score with new evaluation sets and metrics (Ribeiro et al., 2020), or with novel ways to diagnose a model's ' errors (Wu et al., 2019; Bernier-
SSe Fig. 1 for example of sentence.
PInternational Initiative for Impact Evaluation
*Sec Fig. 6 in App. A.2.6

---

First
gemmes
find improvess
abstracts sentences
1,000
7,522
(Sal
# intervention
6,157
Figure 2: Example annotation in INCEpTION interface
# outcome
8,628
# effect size
1,445
# population coreference
4,824
Based on these entity types, we designed explicit annotation guidelines paired with examples. 10 annotators with at least graduate-level education in economics and experience reading research pa- pers recruited perform the annotation task. They received training on the annotation guidelines including videos on how use the annotation plat- form INCEpTION (Klie et al., 2018). Each annota- tor performed a test round on 10 abstracts and re- ceived personal feedback from the domain experts who defined the annotation guidelines. We required two rounds of annotation per abstract to ensure an acceptable level of consistency for a reasonable budget. The corpus of abstracts
2,518
Total annotations
23,572
Table Summary statistics of the Econ-IE IE dataset
suited than their bidirectional counterparts for se- quence labeling tasks such as NER (Devlin et al., 2018; Schweter and Akbik, 2020; Yamada et al., 2020). The DeBERTa-v3 architecture is one of the current state-of-the-art bidirectional archi- tectures for NER tasks among encoder-based mod- cls of the same size (He al., 2023). Our initial choice was further pretrain the -eglisc-specffic DeBERTa-V3-base model, however its generator model is not publicly available. We therefore pre- train our economies-specific models based on the mDeBERTa-v2 architecture by following the ELEC- TRA pretraining approach (Clark al., 2020) Our first model, EconBERTa-FS, is pretrained from scratch using vocabulary tailored to the eco- nomics domain. The second one, EconBERTa-FC. relies on mDeBERTa-V3-base generator and dis- criminator checkpoints as initial weights.
divided into 20 subsets of 50 abstracts; each annotator was as- signed four subsets of 50 abstracts, such that all 1,000 abstracts were annotated by two annotators. Fig. 2 shows an example sentence of resulting an- notations.
Given the complexity of the documents and of the annotation task, we expected some level of dis- agreement between human annotators. We indeed found the anter-annotator agreement measured by the Fl-score and the Cohen's Kappa be equal to F1 = 0.87 and = 0.71 respectively. Start- ing from the annotated abstracts, the experts who defined the guidelines then performed a curation phase to resolve any disagreement between annota- tors. When a disagreement was not due to a mistake in applying the rules but rather multiple plausi- ble annotations given existing rulesš, the case was raised and discussed between the experts who came up with new rule, e.g. "Annotate the shortest pos- sible span as long as no information is lost". The curation phase led to a final annotation per abstract and Table summarizes our dataset statistics after the task was completed.
Pretraining dataset We pretrain the two EconBERTa models corpus consisting of 1.5 million economics papers 800, 000 full articles and 700, 000 abstracts. The documents were col- lected from the leading digital libraries in the field. A breakdown of the data sources can be found Table 2. The resulting dataset consists of 9.4 billion tokens, i.c. almost three times more than the 3.3 bil- lion tokens used to train BERT (Devlin et al.. 2018), For more details about the pretraining procedure. see App. B. and App. B.2.
Baseline models training the EconBERTa models in domain, we com- pare them against mDeBERTa-v3 bbse (He et al., 2023), which has the same architecture. Addi- tionally, also include BERT-base- vlin al., 2018) and RoBERTa (Liu al., 2019) as baselines, as they were extensively stud- ied to analyse the role of domain-specific pretrain- ing (Beltagy et al., 2019; Chalkidis et al., 2020; Nguyen et al., 2020; Carrino et al., 2022).
To evaluate the effects of pre-
3.2 Models
Domain-specific language models for economics While the past few years have seen explo- sion of the size of generative language models (Brown al., 2020; Zhang et al., 2022; OpenAI, 2023), their autoregressive nature makes them less
(De-
'For more details, see App. A.2.6
FExamples of sentences where multiple annotations are plausible are presented in App. A.3.
hhttps: thub. com/m wowooorr

---

Source
Documents
Tokens
Type
Model Name
Fl-score
RePEc
399,708
5.0B (53%)
full articles
BeonBERTa-FC FconBERTa-FS
0.687 (=0.003)
SSRN
355,881
3.98 (42%)
full articles
0.684 (00005)
NBER
28,456
0.3B (3%)
full articles
mDeBERTa-Y. (He al., 2023)
0.670 (0.004)
EconLit
16.517
0.1B (1%)
full articles
RoBERTa (Liu al., 2020a)
0.659 (00.004
Scopus
700,340
0.1B (1%)
abstracts
BERT (Devlin et al., 2018)
0.649 (0.003)
Total
1,500,902
9.4B
Table Comparison of models' performance the NER task the ECON-IE dataset.
Table 2: Summary statistics for each source of eco- nomics research papers included the pretraining cor- pus
4.2 Entity-level evaluation
NER fine-tuning Finally, all our fine-tuned NER models rely a conditional random field (CRF) layer for classification (Lafferty al., 2001), as has been shown to improve results for sequential classification tasks (Lample et al., 2016b; Souza et al., 2020). All the trainings were conducted using the crf _af_eag model from the AllenNLP library' (Gardner et al., 2017). The hyperparameters used for finentuning are presented in table App. B.3.
Previous work has argued that comparing Fl -scores on held-out test set fails to highlight the strengths and weaknesses of the models being tested (Fu et al., 2020). In the context of NER, the Fi-score is typically computed the token-level, which makes difficult know if a model is capturing entity boundaries accurately. Instead of considering each token separately, a solution is to measure whether the entire span of an entity is accurately predicted. However, only considering an entity to be correctly predicted when the model detects both the correct label and its boundaries would be too restrictive, as it would fail distinguish between cases when model completely misses an entity, when is confuses an entity type for another, or when fails to capture the correct boundaries.
4 Model Diagnosis and Results
this section, we present the performance of the different pretrained models on the NER task, and analyze their generalization capabilities.
4.1 Downstream performance
To address these issues, we draw inspiration from alternative metrics to evaluate information extraction systems and adapt them to our NER task (Chinchor and Sundheim, 1993). We introduce six fine- the-grained prediction metrics which allow us to analyze the models' successes and failures at the entity-level. In addition to exact matches, three of the metries account for answers that are partially correct, as described in Table Finally, we count as Missed Labels (ML) entities that were labeled but not predicted at all by a model, which could reflect undergeneralization. and False Alarms (FA) cases when an entity is predicted by model but no entity was labeled, which could indicate over- generalization.
To characterize the models' ability to generalize on the NER task, we start by measuring their aggre- gate performance on the ECON-IE dataset. Each model's F1 F1-score is reported in Table 3. In line with previous studies, find that EcomBERTa-FS, which is pretrained from scratch economics re- search papers, outperforms the general-purpose mDeBERTa-v3-base model. This result confirms that in-domain pretraining offers a key advan- tage for maximizing a model's robustness when deployed on downstream task. Additionally. in-domain pretraining of EconBERTa-FC from am existing pretrained model also improves perfor- mance on the task and produces small but in- significant gain relative pretraining from scratch. Finally, we find that mDeBERTa-V3-ban outper- forms LaBERTa-base, which itself outperforms BERT-base, which confirms the findings from pre- vious studies (Liu et al., 2020a: He et al., 2023) and supports our choice of pretraining from DeBERTa-based architecture.
Metric
Entity Type
Boundaries
Exact Match (EM)
Match
Match
Exact Boundary (EB)
Mismateh
Match
Partial Match (PM)
Match
Overlap
Partial Boundaries (PB)
Mismateh
Overlap
Table 4: Attributes of various prediction types based on the comparison of the predicted and labeled entity types and span boundaries.
com.t/uu.coom//aaaaaaaaaaaaaaaaa wwtmm///lea//aallemlllllll models/ httb//ainn/aleennp_mmeoe tagger.py
htt.am///e/g/ggg//mne/eeo/eeoooe ing/modess/cff_

---

These fine- fine-grained evaluation metrics are pre- sented in Fig. We find consistent pattern across models: most predictions are either matched ex- actly or with the right label but inexact boundaries (Partial Match); on the contrary, label confusions (Exact Boundary and Partial Boundary) are very rare. We also find that the models miss less than 10% of entities (Missed Label), which indicates that they could be undergeneraliizigg especially when presented with entities that were not seen during training. Finally, we find that they mistak- enly predict annotations that weren't labeled about 10% of the time (False Alarm), which suggests that they could also be overgeneralizing, in particular for entities that could systematically be mapped into a label regardless of the context. In the next subsections, we will test alternative hypotheses to better characterize these generalization failures.
training data. 9 We therefore examine the impact of lengths on EconBERTa-FC" ability to correctly detect entities Fig. 4. We find that the model's 's robustness decreases with entity length. This sug- gests that the model is failing to generalize properly, as it fails to extrapolate to longer sequences. This failure is relative however, as the reduction in ex- act matches is mirrored by an increase in partial matches, indicating that the model is still able capture sequence which overlaps with the original annotation.
1.2
Progortion of entites
1.0
0.8
Bost Match
0.6
Part Maich
Exact Boundary
0.4
Port al Boundary
Maseed Label
Taken Alym
0.8
0.2
EconBERTa-FC SconBERTa-FS
0.7
0.0
Proportion entities
Entity Length
0.6
mDeBERTa-v3-baal
RoBERTa-base BERT-base-uncase
Figure Proportion exact matches and of the differ- ent error types for EconBERTa-FC as function of entity length.
0.5
0.4
0.3
0.2
4.4 Inference scheme: reliance on memorization
0.1
0.0
Exact Match
Partial Match
Exact Boundary
Partial Boundary
Missed Label
False Alarm
As our task consists in tagging sub-strings with the correct entity type, model could rely a variety of implicit heuristies to achieve good per- formance without having truly generalized. For in- stance, we previously identified that our best model was increasingly struggling to correctly capture the boundaries of an entity as span length increases. this section we test various hypotheses to uncover heuristics causing the model's 's failures.
Figure 3: Proportion of exact matches and of the differ-
ent error types for each model.
4.3 Generalization to longer spans
Lexical memorization We first test the assump- tion that our model could be memorizing strings that were seen during training to successfully detect entities at test time. In other words, we test whether the model is more successful at detecting in-train entities. We compare the predictions at test time in two settings: in-traintea. in which we isolate entities also present in the training set in the exact same lexical form, and out-of-traineess in which we isolate the remaining entities that were not seen
For conciseness purposes, we now focus on EconBERTa-FC for the rest of the analysis. as we demonstrated its ability outperform other fine- tuned architectures. 8 To test the model's capacity to generalize over any entity, we first explore whether the length of an entity affects its ability to detect accurately. We expect that a longer entity would be harder to predict, as (i) accurately matching bound- aries should be more difficult for longer spans, and (ii) longer entities should be less frequent in the
Another factor to consider that the annotation decision on how to set the boundaries of a span hecomes increasingly more complex as span length increases.
"The results for rest of the analysis were found to be quali- tatively similar across architectures, consistently with Fig. 3.

---

1.0
Exact
trainros of
out-aftrain
0.8
Partial Match
0.8
Partial Match
Exact Boundary
Partial Boundary
Parb al Bourdary
0.4
Missed Label
0.4
Missed Label
0.2
0.2
0.0
0.0
0.2
-02
Intervention
Oulceme
Population
Carefsrence e
Effect Size
Total
Intervention
Outcome
Population
Careferance
Effect Size
Total
(a) Performance gain for entities seen during training
(b) Performance gain for POS sequences seen during training
Ween oeruu www.eeeeeaaggg we wn untur uoburntt
20
15
10-
Intervention
Outcome
Population
Coreference
Effest Size
Total
Intervention
Outcome
Population
Careference
Effect Size
Total
(c) Mean count across unique entities in the train set
(d) Mean count across unique POS sequences in the train set
Figure 5: Difference in prediction types for entities and POS sequences that present and absent from the training set (top) as proxies to measure memorization effects, along with number of unique entities and number of POS sequences (bottom)
during training. We then compute the difference in the proportion of each prediction type between in-trainter and are displayed in Fig. 5a.
entity present in the training set.
Lexicosyntactic memorization In addition lexical memorization, alternative hypothesis addressing milder form of memorization could be that the model learns to associate specific lexi- cosyntactic sequences seen during training to spe- cific entity types. To test this hypothesis, we first map each entity to its part-of-speech (POS) se- quence. We then partition entities at test time be- tween those whose POS is present in the training set (in-trainpos) and those whose POS is absent from the training set cout-of-train/ee stance, the POS sequence NNS-NNS for the out- come entity "household income" is present the training set, therefore oulcome entity "school attendance" which has the same POS belongs to the in-train set. The results of this analysis are presented in Fig. 5b.
entities. The results
We find that, for all entity types, the model pro- duces higher proportion of exact matches on in- traintex entities, which suggests a tendency to rely on memorization. This particularly true for coref- erences, where the difference is near 0.6. We also find that the model has fewer partial matches on in- traintes entities, confirming this observation. For coreferences, we also observe reduction of mis- matched entities, captured by the "Exact Bownd- ary" metric, and more false alarms, thus stronger tendency to overgeneralize by memorizing for this entity class.
For in-
We further hypothesize that the mean number of occurrences across unique entities during fine- tuning could explain the tendency to rely on mem- orization. As indicated in Fig. 5c, this metric is significantly higher for the coreference entity type compared to the other entity types, which suggests a direct link between the tendency for a model to rely on memorization and the mean frequency of occurrence, i.e. the mean count, for each unique
We discover that the effect of prior exposure to pfrt-of-speech is higher than prior exposure lexical content. This suggests that seeing given POS sequence during training plays key role in the model's successes and failures. In line with the results on lexical memorization, we find a dif- ference of 0.7 for coreferences on exact matches

---

(see Fig. 5d). Once again, the differences between entity types in the mean number of occurrences for each unique part-of-speech seems to explain the difference in memorization patterns observed for coreferences. The number of entities for each entity type alone could not explain such effect as Table I shows that there are more annotated coreferences than effect sizes in our dataset. Finally, we note that the shares of False Alarms and Missed Labels not significantly different for in-trainpos and in- pos entities, except for coreferences. Again in this case, Missed Labels are fewer for in-trainpos: suggesting that the model undergenerraizes and misses entities that are too dissimilar from those seen during training. For other entities, the increase in Exact Match is mirrored by a decrease in Partial Match, which echoes the difference in prediction types for different lengths found in section 4.3.
We report the results of our substitution tests in Table 5. It disentangles the two forms of mem- orization (lexical and lexicosyntactie) previously observed. We observe that replacing entities which are identified by a POS sequence present in the training data has little effect over the model's per- formance overall. This holds true both when the new entity's 's lexical content is sampled from the training set out of it. This confirms that the model has reached a degree of generalization that extends beyond the shallow memorization of spans during training. Additionally, the drop in perfor- mance is low when performing random replace- ments, therefore the model's performance does not depend on the mutual semantic constraints between the substituted entity and other parts of the original context perform the task.
5 Discussion
4.5 Disentangling lexical and lexicosyntactic memorization
Memorization: a core component of learning or sign of undergenerallizatoon? Throughout this study, we tested several memorization effects ranging from the shallow memorization of lexical patterns to that of lexicosyntactic sequences. While memorization often opposed to generalization and considered to be a weakness of models (Elango- van et al., 2021; Mireshghallah et al., 2022), its role in the learning process is a source of debate. For instance, some authors have argued that memoriza- tion and in particular lexical memorization is a key component of language acquisition and linguistic knowledge (Kess, 1976; Carey, 1978; Nooteboom et al., 2002). While relying too much on memo- rized associations might be detrimental to learning a task properly (Tänzer et al., 2022) (see results on coreferences in section 4.4), penalizing this phenomenon entirely could also be too prescriptive. Memorization could also play a crucial role either in early or in late stages of the learning process, es- pecially if the task is characterized by multiplicity of implicit rules and exceptions.
Finally, we complement the two tests in section 4.4 with a replacement experiment. As we measured a similar effect for both forms of memorization, the root cause for failures remains unknown as both factors are entangled. Behavior tests have been suc- cessfully implemented in previous studies to test targeted hypotheses on fine-tuned models (Vajjala and Balasubramaniam, 2022). In our substitution test, we generate random combinations by sam- pling entities and placing them in contexts where the -frt-hf-speech is preserved. We perform this generation by sampling entities from the training set on the one hand (in-train), and from the test set on the other hand (out-of-train). If the perfor- mance drop is higher in the second case, it shows the model does rely on lexical memorization. If they are similar, the model does not significantly rely on lexical memorization. If the perfomance drop is lower, it is a sign that the model relies on the POS sequence to generalize, but not significantly on the mutual semantic constraints between the con- text and given entity, as these factors define the generative procedure for the resulting sentences.
Learning from mistakes Uncovering how a model reaches a decision is helpful to understand why it makes certain mistakes and implement mod- eling strategies to improve its performance. Re- cent work proposed to teach fine-tuned models to learn from their mistakes on NLU tasks (Malon et al., 2022). Similarly, Xu et al. (2022) attempted to debias NER models by using a counterfactual duta-augmennattion technique. Together with the diagnoses presented in this paper, these studies pro-
Condition
A Acc.
A Prec.
Á Rec.
A F1
in-train -0.38%
-1.28%
-2.24%
-1.75%
out-of-train
-0.39%
-1.28%
-2.56%
-1.92%
Table 5: Variation in performance when randomly sub- stituting entities based on their lexical category, given that they are sampled randomly from the train set (top) or from the test set (bottom).

---

vide a blueprint how improve models used for deployment, which constitute interesting directions for future work. To improve models' ability to ex- trapolate to longer sequences, future work could also explore the impact of the splitting strategy or of augmenting the training data with difficult examples.
coverage, exceptions that are not well captured by our guidelines remain. This problem recurring for certain annotation tasks documents, limiting the interpretation of performance as the learning task is not suited for the ground truth reference to capture several equally plausible answers.
Testing generalization abilities requires forming hypotheses. In this work, we argued that diag- nosing a model's errors is key to improving its robustness given task. This evaluation can only be achieved by making assumptions on the mechanisms leading to the errors being observed. Here, we hypothesized that an entity's 's length and its presence in the training corpus either in its exact lexical form or a syntactically similar form could affect the ability of the model to detect it. In the general case, forming an hypothesis to diagnose model's s errors is not straightforward, reflecting the fact that there is no consensus on the notion of generalization. While future work could draw inspiration from previous studies including ours, there is long way to go before we obtain a precise characterization what model generalization truly means and how to operationalize The complexity arises from the variety of parameters encountered in real-world scenario that are affecting a model's ability to extrapolate from its training data.
6 Conclusion
In this work, we presented EconBERTa, a novel language model adapted to the economic domain. We demonstrated that it achieves state-of-the-art performance Econ-IE, novel NER dataset of 1,000 abstracts from economic research annotated for entities describing the causal effects of policy interventions. In doing so, we demonstrated the advantages of domain adaptation, specifically fur- ther pretraining from cxisting checkpoints. Addi- tionally, we diagnosed the model's generalization abilities in series of fine-grained tests. We found that most mistakes stem from its inability to elicit the exact boundaries of detected entities, and that this phenomenon is due to the model relying on memorization. The memorization effect is mild and extends beyond lexical memorization, as the model is able to extrapolate to entities which have the same part-of-speech sequence as entities seen during training. Finally, we found that the effect of memorization is more pronounced for entities which are less diverse in the training data. Taken together, these findings pave the way towards de- veloping more robust downstream information ex- traction systems.
Testing wide range of approaches for in- domain pretraining is computationally expen- sive. During pretraining, each choice of architec- ture, data selection preprocessing, and hyperpa- rameters leads to running the entire pipeline, which requires substantial computing resources. Choos- ing configurations with parsimony reduces the com- putational load, but does so at the expense of the range of configurations that can be explored.
Limitations
The results and methods presented throughout this work present certain limitations which should be taken into account future work.
Ethics Statement
Some of the annotations can be debatable. As- signing labels can be straightforward for simple learning tasks, but can become increasingly com- plex with natural languages due to the multiplicity of strings encountered in real-world corpora and the ambiguity which lies in certain sentences. This could lead to annotations resulting from subjec- tive or even arbitrary judgements from annotators. While we tried to reduce this effect by relying on multiple parallel annotations, arbitrating disagree- ments, writing precise annotation rules and iter- ating over the rules to improve their clarity and
The authors foresce no ethical concerns with the work presented in this paper.
Acknowledgements
We would like kindly thank Saqib Hussain. Sab- rina Yusoff, Leonardo Timperi, Nihaa Sajid, Rohit Tripathi, Nadir Khan, Pierina Forastieri, Morgan Routh for their excellent research assistance.

---

References
Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma. Ali Malik, Christopher D. Manning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayaman, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potns, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang. Tianyi Zhang. Xikun Zhang, Yuhui Zhang. Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the opportunities and risks of foundation models.
Emily Alsentzer, John Murphy, William Boag, Wei- Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermou,. 2019. Publicly available clin- ical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Associ- ation for Computational Linguistics.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1388- 1398, Florence, Italy. Association for Computational Linguistics.
Dogu Araci. 2019. Finbert: Financial sentiment analy- sis with pre-trained language models. arXiv preprint arXiv:1908. 10063.
Ting Wai Terence Au, Vasileios Lampos, and Ingemar Cox. 2022. E-NER an annotated named entity recognition corpus of legal text. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 246-255, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2019. Sub-event detection from twitter streams as sequence labeling problem. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- fional Linguislics: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 745-750, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Tz. Beltagy, Kyle Lo, and Arman Cohan. 2019. SeiB- ERT: pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing DEMNLP-IJCNLA pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics.
Kimberly Marie Bumgarner. 2017. New sector taxon- omy and definitions.
Susan Carey. 1978. The child word learner.
Casimiro Pio Carrino, Joan Llop, Marc Pàmies, Asier Gutiérrez-Fnddñño, Jordi Armengol- Joaquín Celveira-Ocampo Alfonso Valencia, Aitor Gonzalez- Agirre, and Marta Villegas. 2022. Pretrained biomed- ical language models for clinical NLP in Spanish. In Proceedings of the 21st Workshop Biomedi- cal Language Processing, pages 193-199, Dublin, Ireland. Association for Computational Linguistics.
Gabriel Bernier- Colbbree and Phillippe Langlais. 2020. HardEval: Focusing on challenging tokens to assess robustness of NER. In Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 1704-1711. Marseille, France. European Lan- guage Resources Association.
Tlass Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and lon Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji. Annie Chen, Kathleen Creel. Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman,
Lu Chang, Ruihuan Zhang, Jia Lv, Weiguang Zhou, and Yunli Bai. 2022. A review of biomedical named en- tity recognition. Journal of Computational Methods in Sciences and Engineering, 22(3):893-900.

---

Nancy Chinchor and Beth Sundheim. 1993, MUC-5 evaluation metrics. In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993.
Borja González-Pereira, Vicente Guerrero-Bote and Felix Moya-Anegon. 2009. The sjr indicator: A new indicator of journals' scientific prestige. arXiv preprint ar11::091..4141.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional I. LSTM-CNNs. Trans- actions of the Association for Computational Linguis- lics, 4:357-370.
David Grangier and Dan Iter. 2022. The trade-offs of domain adaptation for neural language models. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pages 3802-3813, Dublin, Ireland. Association for Computational Linguistics.
Chenhui Chu and Rui Wang. 2018. A survey of do-
main adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa Fe. New Mexico, USA. Association for Computa- tional Linguisties.
Cuiyun Han, Jinchuan Zhang, Xinyu Li, Guojin Xu, Weihua Peng, and Zengfeng Zeng. 2022. Duce-fin: A large-scale dataset for document-level event extrac- tion. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 172- 183. Springer.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. CoRR, abs/2003. 10555.
Sebastian GM Händschke, Sven Buechel, Jan Gold- enstein, Philipp Poschmann, Tinghui Duan, Peter Walgenbach, and Udo Hahn. 2018. A corpus of cor- porate annual and social responsibility reports: 280 million tokens of balanced organizational writing. In Proceedings of the first workshop economics and natural language processing, pages 20-31.
Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. 2021. Multilingual domain adaptation for NMT: Decoupling language and domain infor- mation with adapters. In Proceedings of the Sixth Conference on Machine Translation, pages 578-598, Online. Association for Computational Linguisties.
Ayoub Harnoume, Maryem Rhanoui, Mounia Mikram, Siham Yousfi, Zineb Elkaimbillah, and Bouchra Asri. 2021. BERT based clinical knowledge extrac- tion for biomedical knowledge graph construction and analysis. Computer Methods and Programs in Biomedicine Update, 1:100042.
Hal Daumé III. 2007. Frustratingly easy domain adap- tation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023, DeBERTav3: Improving deBERTa using ELECTRA- style pre-training with ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Pedro Henrique Luz de Araujo and Benjamin Roth. 2023. Cross-functional analysis of generalisation behavioural learning.
embed-
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.04800
Jiahui Hu and Patrick Paroubek. 2021. A fine-grained annotated corpus for target-based opinion analysis of economic and financial narratives. I Proceedings of the Third Workshop on Economics and Natural Language Processing, pages 1-12.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1325-1335, Online. Association for Computational Linguistics.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and pre- dicting hospital readmission. ArXiv, abs/1904.05342.
Xiaoli Huang, Jimmy Lin, and Dina Demner- Fushman. 2006. Evaluation of pico as a knowledge representa- tion for clinical questions. In AMIA annual sympo- sium proceedings, volume 2006, page 359. American Medical Informatics Association.
Jinlan Fu, Pengfei Liu, and Graham Neubig. 2020. In- terpretable multi-dataset evaluation for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- Online. Association for Computational Linguistics.
ing (EMNLP), pages
Dieuwke Hupkes. Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christocoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann. Khuyaghaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023. State-of-the-ar gen- eralisation research in nip: A taxonomy and review.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: deep semantic natural language processing platform.

---

Gilles Jacobs and Véronique Hoste. 2022. Sentivent: en- abling supervised information extraction of company- specific events in economic and financial news. Lan- guage Resources and Evaluation, 56(1):225-257.
pages 260-270, San Diego, California. Association for Computational Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1244)
Di Jin and Peter Szolovits. 2018. PICO element de- tection in medical text via long short-term memory neural networks. In Proceedings of the BioNLP 2018 workshop. pages 67-75, Melbourne, Australia, Asso- ciation for Computational Linguistics.
Meisin Lee, Lay-Ki Soon, Eu-Gene Siew, and Ly Fie Sugianto. 2022. Crudeoilnews: An annotated crude oil news corpus for event extraction. arXiv preprint 12Xi::2044008771
Joseph F. Kess. 1976. The psychology of language: An introduction to psycholinguistics and generative grammar, by j. a. fodor, t. g. bever, and m. f. gar- rett. new york: Megraw-hill, 1974. pp. xvii 537. Canadian Journal of Linguistics/Revie canadienne de linguistique, 21(1):126 131.
Yingjie Li. Tiberiu Sosea, Aditya Sawant, Ajith Jayara- man Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in polit- ical domain. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2355-2365, Online. Association for Computational Linguistics.
Christopher S. G. Khoo, Syin Chan, and Yun Niu. 2000.
Extracting causal knowledge from a medical database using graphical patterns. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 336-343, Hong Kong. Association for Computational I Linguistics.
Zhengzhong Liang, Enrique Noriega-Atala, Clayton Morrison, and Mihai Surdeanu. 2022. Low resource causal event detection from biomedical literature. Proceedings of the 21st Workshop on Biomedical Lon- guage Processing, pages 252 263, Dublin, Ireland. Association for Computational Linguistics.
Bosung Kim, Taesuk Hong, Youngjoong Ko, and Jungyun Seo. 2020. Multi-task learning for knowl- edge graph completion with pre-trained language models. In Proceedings of the 28th International Conference Computational Linguistics, pages 1737-1743, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1199911
Jan-Chrisse Chn-Chrittoph Klie. Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INOEPTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Ro (orbet)):: a: A robustly optimized (bert) pretraining approach.
Zihan Liu, Xu Yan, Tiehzheng Yu, Wenliang Dai, Zi- wei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020b. Crossner: Evaluating cross- domain named entity recognition.
John Lafferty. Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling se- quence data, In Proceedings of the Eighteenth In- ternational Conference on Machine Learning, ICML. pages pages 282-289, San Francisco, CA, USA. Mor- gan Kaufmann Publishers Inc.
Lefteris Loukas, Manos Fergadiotis, Ion Androutsopou- los, and Prodromos Malakasiotis. 2021. Edgar- corpus: Billions of tokens make the world go round. arXiv preprint 1rXiv:2109.1453
Guillaume Lample, Miguel Ballesteros, Sandeep Subra- manian, Kazuya Kawakami, and Chris Dyer. 2016a. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Florian Ludwig, Klara Dolos, Torsten Zesch, and Eleanor Hobley. 2022. Improving generalization of hate speech detection systems to novel target groups via domain adaptation. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 29-39, Seattle, Washington (Hybrid). Asso- ciation for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subra- manian, Kazuya Kawakami, and Chris Dyer. 2016b. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies,

---

Xiaofei Ma. Peng Xu, Zhiguo Wang. Ramesh Nalla- pati, and Bing Xiang. 2019. Domain adaptation with BERT-based domain classification and data se- lection. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (Deeplo 2019), pages 76-83, Hong Kong, China. Association for Computational Linguistics.
of Association for Computational Linguisties (Vol- ume 1: Long Papers), pages 1470-1480, Vancouver. Canada. Association for Computational Linguisties.
Timothy Niven and Hung- Yu Kao. 2019, Probing neu- ral network comprehension of natural language argu- ments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658 4664, Florence, Italy. Association for Compu- tational Linguistics.
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wal- lenius, and Pyry Takala. 2014. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796
Sieb Nooteboom, Fred Weerman, and Frank Wijnen. 2002. Storage and Computation in the Language Faculty.
Christopher Malon, Kai Li, and Erik Kruus. 2022. Fast few-shot debugging for NLU test suites. In Proceed- ings of Deep Learning Inside Out (DeelJO 2022): The 3rd Workshop on Knowledge Extraction and In- legration for Deep Learning Architectures, pages 79- 86, Dublin, Ireland and Online. Association for Com- putational Linguistics.
Benjamin Nye, Junyi Jessy Li. Roma Patel, Yinfei Yang. Tain Marshall, Ani Nenkova, and Byron C Wal- lace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes support language processing for medical literature. In Pro- ceedings of the conference. Association for Computa- tional Linguistics. Meeting, volume 2018, page 197. NIH Public Access.
Dominique Mariko, Hanna Abi Akl, Estelle Labidurie, Stephame Durfort, Hugues De Mazancourt, and Mah- moud El-Haj. 2020. Financial document causality detection shared task (fincausal 2020), arXiv preprint arXiv:2012.02500
OpenAL 2023. Gpt-4 technical report.
Yannis Papanikolaou, Marlene Staib, Justin Joshua Grace, and Francine Bennett. 2022. Slot filling for biomedical information extraction. In Proceedings of the 21 Workshop on Biomedical Language Pro- cessing, pages 82-90, Dublin, Ireland, Association for Computational Linguistics.
Tom McCoy, Ellic Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagmosing syntactic heuris- tics in natural language inference. In P Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428-344 -348,, Florence, Italy, Association for Computational Linguistics.
Bo Peng. Emmanuele Chersoni, Yu-Yin Hsu, and Chu- Ren Huang. 2021. Is domain adaptation worth your investment? comparing bert and finbert on finan- cial tasks. In Proceedings of the Third Workshop on Economics and Natural Language Processing, pages 37-44.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatriee 2022. An empirical analysis of memorization fine- tuned autoregressive language models. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1816-1826, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Amir Pouran Ben Veyseh, Viet Lai, Franck Dernon- court, and Thien Huu Nguyen. 2021. Unleash GPT-2 power for event detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 6271-6282, Online. Association for Computational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre- pre-triied language model for English tweets. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, On- line. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 365-371, Beijing, China. Association for Computational Linguistics.
A lexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020. Adapt get left behind: Do- main adaptation through BERT language model fine- tuning for aspect-target sentiment classification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4933-4941, Marseille, France. European Language Resources Association.
Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recog- nition via effective annotation and representation pro- jection, In Proceedings of the 55th Annual Meeting

---

Hiroki Sakaji. Masahiro Suzuki, Kiyoshi Izumi, and Hiroyuki Mitsugi. 2022. Gradual further pre-training architecture for economies/fiance domain adapta- tion of language model. In 2022 IEEE International Conference on Big Data (Big Data), pages 2352- 2355. IEEE.
International Joint Conference on Natural Language Processing ( (Volame 1: Long Papers), pages 009-319. Online only, Association for Computational Linguis- tics.
Sowmya Vajjala and Ramya Balasubramaniam. 2022. What do we really know about state of the art NER? In Proceedings of the Thörtennth Language Resources and Evaluation Conference, pages 5983-5993 Mar- seille, France. European Language Resources Asso- ciation.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Tim- othy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Tech- nology Association Workshop 2015, pages 84-90, Parramatta, Australia.
Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple contrastive knowledge graph completion with pre-trained language models, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 4281 14244, Dublin, Ireland. Association for Computational Linguisties.
Stefan Schweter and A. Akbik. 2020. Flert: Document- level features for named entity recognition. ArXiv, 0oss2011106999)
Sofia Serrano, Jesse Dodge, and Noah A. Smith. 2023. Stubborn lexical bias in data and models.
Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020, Bertimbau: Pretrained bert models for brazil- ian portuguese. In Intelligent Systems, pages 403- 417, Cham. Springer International Publishing.
Howard White and Marie Gaarder. 2009. The interna- tional initiative for impact evaluation (3ie): intro- duction. The Journal of Development Effectiveness. 1:378-386.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7564-7578, Dublin, Ireland. Association for Computational Lin- guistics.
Georg Wiese, Dirk Weissenboru, and Mariana Neves. 2017. Neural domain adaptation for biomedical ques- tion answering. In Proceedings of the 21st Confer- ence on Computational Natural Language Learning (CoNLL 2017), pages 281-289, Vancouver, Canada. Association for Computational Linguisties.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 747-763, Florence, Italy. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction the CoNLL-2003 shared task: Language-inde ienguee-indeeennentt named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning HLT-NAACL 2003, pages 142- 147.
Nan Xu, Fei Wang. Bangzheng Li, Mingtao Dong, and Muhao Chen. 2022. Does your model classify en- tities reasonably? diagnosing and mitigating spuri- ous correlations in entity typing. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 8642-8658, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Meihan Tong, Bin Xu, Shuai Wang. Yixin Cao, Lei Hou, Juanzi Li, and Jun Xie. 2020. Improving event detec- tion via open- domain trigger knowledge. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5887-5897, On- line. Association for Computational Linguistics.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael Alvers, Dirk Weißenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, Yannis Almirantis, John Pavlopoulos, Nico- las Baskiotis, Patrick Gallinari, Thierry Artieres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, and Georgios Paliouras. 2015. An overview of the bioasq large- scale biomedical semantic indexing and question an- swering competition. BMC Bioinformatics, 16: 138.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo. Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextuaizzd entity representations with entity- aware self eelf-attention In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4442-6454, On- line. Association for Computational Linguistics.
Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint 2XXi:2000000097
Asahi Ushio, Francesco Barbieri, Vitor Sousa. Leonardo Neves, and Jose Camaaco-Collados. 2022. Named entity recognition in Twitter: A dataset and analysis on short-term temporal shifts. In Proceedings of the 2nd Conference of the sic-ffecfiic Chapter of the As- sociation for Computational Linguistics and the 12th
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of

---

the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791-4800 Florence, Italy, Association for Computational Linguisties.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Wenhao Zhu, Shujian Huang, Tong Pu, Pingxuan Huang, Xu Zhang, Jian Yu. Wei Chen, Yanfeng Wang, and Jiajun Chen. 2022. FGraDA: A dataset and benchmark for fine-grained domain adaptation machine translation. Proceedings of the Thir- leenth Language Resources and Evaluation Confer- ence. pages 6719-6723 Marseille, France. European Language Resources Association.

---

A ECON-IE Dataset
An entity cannot span across sentence bound- aries.
A.1 Article Selection
Since our focus lies on causal entity recognition, we limit ECON-IE to abstracts from studies dedi- cated to reveal causal relationships. We keep stud- ies using any of four standard causal inference methods: randomized control trials, difference-in difference, instrumental variable estimation and regression discontinuity design. The 3ie Develop- ment Evidence Portal highly imbalanced across scctors as well as years. By attributing higher weights to under-represented sectors and year ranges, we ensure ECON-IE covers different writ- ing styles and sector-specific vocabulary. Fig. 6 displays relative share per sector for the source cor- pus from 3ie and ECON-IE. The latter gives more balanced coverage of different economic activities.
Entity spans cannot overlap.
Try not to include determiners (the, a), or ad- jective pronouns (this, its, these, such) in the span.
Label distributive conjunctions and enumera- tions as a single span, otherwise as separated spans (e.g. "vitamins A, B, C and D as a sin- gle span but "children and parents" as two).
Annotate mentions of entities, including sentences providing background, conclusions, summaries, or with claims of external validity,
A.2.2 Intervention
Intervention describes deliberate involvement in process or system intended to influence events and/or consequences. In our context, intervention refers the activities of project, program, pol- icy, or instrument in the field of development that aims to bring about change an outcome thereby improving the conditions of a target population. Ex- amples: conditional cash transfer, leadership pro- grams, household rainwater harvesting, research fundiig, access to microcredit.
200t
Erancci MACEE
MA68
trccry and eOTUCTIve
Educction
Entemattor techniloges
Annotate program names and acronyms of in- terventions as interventions (and not as coref- erences), e.g. Start-and- crrr-aan-Immpooeee Your Busi- ness (SIYB) Program, No Child Left Behind, Progresa
Figure 6: Sector share before (3ie database) and after (ECON-IE) article selection process
A.2 Annotation Guidelines
Include generic terms the intervention span when they describe the entity, e.g. program, courses, lectures. For example, in the follow- ing sentence, the generic term courses should also be part of the intervention span (business training courses):
Our annotation guidelines consist a range of rules, which comprise seven generic rules common to all entity types, and five to eight rules specific to each entity type. In the following, we list the annotation guidelines as presented to the annotators.
A.2.1 General rules
We used randomized experiment to measure the impact of business training courses.
Span annotation comprises of (1) selecting span of tokens (determining boundaries of an entity) and (2) tagging it with an entity label. The following rules apply to all entities regardless of their type.
The same applies other generic terms next intervention descriptions such as program or intervention.
Use only the context of a sentence for deter- mining entities. The only exception is the coreference, which can refer to interventions or outcomes in other sentences.
Do not include delivery details or attributes of the interventions. For example, in the follow- ing sentence the intervention span should be Carrying of firearms was banned excluding the details on which day the ban took place.
An entity can span over one or more consecu- tive tokens (words).

---

Carrying of firearms was banned on weekends after paydays, on holi- days and on election days.
intervention, do annotate that. We only interested in the populations that received an intervention.
Annotate as interventions mentions of treatment group by a program description, acronym, or program name; but do not include the term group in the span.
Do not annotate standalone generic references such as the word population with the popula- tion tag.
Do not label standalone geographical names such countries, regions cities. For ample, low-income households in Nairobi. Label geographical names only when part of greater entity or when otherwise the pop- ulation wouldn't have a noun. For exam- ple, malnourished Indian children and rural Bangladesh
Do not label the standard treatment that the control group receives as an intervention.
A.2.3 Outcome
An outcome is the likely or achieved change and effects of an intervention. Examples: adoption of recommended animal breeding practices, institu- tional trust, math test scores, air pollution, wage levels, HIV treatment success, knowledge of water contanination
Do not label standalone unit of intervention, e.g. individuals, households, villages. Label them when part of greater entity, e.g. in- dividuals aged 70 years or more, remittance- receiving households.
Include the span generic references such as the word effects or outcomes when they are next to an outcome. For example:
Do not label treated group as population, c.g. impregnated chaddar users (impregnated chaddar is the intervention; user is generic term and not labeled).
The outcome we consider is con- sumer behavior. Consumer behav- ior effects are measured by house- hold survey.
Words defining the status of the population should be part of the population span, e.g. child with acute diarrhea and mothers suffer- ing from anemia the whole spans are popula- tion.
Include words that make the outcome quan- tity (e.g, incidence of HIV and not HIV) when- ever possible.
Very important: mentions of effect sizes should not be annotated outcomes.
A.2.5 Effect size
Do not include population in the outcome, e.g. in teenage fertility, child mortality tag teenage and child as population. fertility and mortality as outcome.
Effect sizes the mentions of quantitative mea- sures of the magnitude of the intervention' 's effect in an outcome. Examples: 28 percentuge points, 4.7%, 0.37 births, 102 μ/m3
A.2.4 Population
Whenever possible, the effeet size span should include only the numeric measure but also the unit of measure, long as this is not part of an outcome span. For example,
The population, sometimes referred to as popula- tion target, is the group of people on which the intervention is implemented or in which the out- come is measured. Examples: women and girls, farmers, civil servants, teachers, women in rural areas
PM2.5 exposure means were 266 and 102 p/m3 during the trial riod in the control and intervention groups, respectively.
Include mentions of subgroups of the popula- tion where the intervention was applied or on whom an outcome was measured.
Tag all the mentions of effect sizes, including mentions for different treatment groups, sub- groups, control groups, or about the difference between treatment and control groups
If the text mentions a control group that had different population and did not receive an

---

Do not include in the span words indicating the direction of the effect size (example: in- crement, decrease, increase).
B EconBERTa Models
B.1 Pretraining data collection
We sourced economic articles for pretraining from variety of online libraries. To ensure economic- specific content in the pretraining corpus, we im- plemented two-step article selection process. In the first step, we collect articles from both (i) econ- specific libraries (RePEc, Econlit and NBER) and (ii) the economic category of general science li- braries (Scopus and SSRN). Despite this first fil- tering step, the resulting corpus from the initial stage turned out to be diverse due to the inclusion of articles from neighboring fields such law and political science.
The results indicate that primary school completion reduces teenage fertility by 0.37 births and the in- cidence of teenage childbearing by around 28 percenlage points.
Statistical measures such as t-stats, p-values, or confidence intervals should not be consid- ered effect sizes.
Odds ratios and incidence rate ratios should be considered effect sizes.
To ensure isolate content specifically rooted in economies, we implemented second filtering process based on the journal in which the article was published. An article is included the pre- training corpus if its publication venue meets any of the following criteria:
A.2.6 Coreference
A coreference accounts for all generic expressions that relate the same intervention or outcome entity. Examples: intervention, project, program, outcome, effect, results, it, they
Tag only coreferences of interventions and outcomes.
is listed among the most influential eco- nomics journals by the SCImago Journal and Country Rank (González-Pereinn et al., 2009), containing 1 1166 journals.
Mentions of intervention by its name (e.g. Progresa, Sesame Street) should not be tagged as coreferences but as interventions.
The journal has been classified as economic- specific in manual annotation process of 1500 journals that we carried.
Words trealment, intervention, experiment in treatment group, intervention group, experi- ment group should also be annotated as coref- erences excluding the word group in the span.
The journal name contains the string "econ".
The paper is working paper from RePEc.
The second filtering step reduced the corpus size from 2.5 million to 1.5 million documents.
A.3 Multiple Plausible Annotations
Multiple annotations can be plausible for the same sentence. We illustrate this using examples for which (non-exhaustive) competing options dis- played in Table 6.
B.2 Pretraining procedure
The two EconBERTa models developed in this work were pretrained using the ELECTRA-Style training (Clark et al., 2020). We display pretraining hyperparameters Table 7. For EconBERTa-FS. our model pretrained from scratch, we used a Sen- tencePiece vocabulary containing 128.000 tokens, trained on the same pretraining corpus. As for EconBERTA-FC, the mDeBERTa based approach, we used the same vocabulary as DeBERTa-V3-bast containing the same amount of tokens. We opted for using vocabulary size matching DeBERTa's instead of its multilingual counterpart due its larger dimension, and adjusted the dimensions the embeddings layer. The resulting weights were used as an initial checkpoint for EconBERTa-FC. Both models were trained on a cluster composed
Our examples comprise candidate nested spans (1, 2), conjunctions of multiple spans (3), auxiliary details (4), enumerations (4, 5, 6), potential risk factors rather than outcomes (6), vague entities (7), multiple populations (8), and nested entities (9).

---

Number
First annotation option
Second annotation option
We find that participating CSA increased families' likelihnod report having saved money (outcome).
We find that participating in CSA increased families' likelihnod report having saved moncy (outcome).
2 We evaluate one attempt make local institutions more democratic and egalitarian by imposing participation requirements (intervention) for marginalized groups (in- cluding women) and test for learning seaming-by-dooine bb--doin effects.
We evaluate one attempt make local institutions more democratic and egalitarian by imposing participation requirements (intervention) for marginalized groups (in- cluding women) and test for learning-by-don bo-iiiggg effects.
3 Neither program affected progression to secondary school, but children in grade 6 in SFP schools at base- line were significantly more likely to remain in primary school and repeat a grade (outcome) than drop out.
Neither program affected progression to secondary school, but children in grade in SFP schools at base- line were significantly more likely to remain in primary school (outcome) and repeat grade (outcome) than drop out.
The nutrition education intervention programme (NEIP) (intervention) comprised ten topics emphasising healthy cating, hygiene and sanitation.
The nutrition education intervention programme (NEIP) (intervention) comprised ten topics emphasising healthy cating (intervention), hygiene (intervention) and sanita- tion (intervention).
5
This study has brought about positive changes in the knowledge, attitude and practice of mothers and their children towards disposal of children's 's facces (out- come) in rural communities.
This study has brought about positive changes in the knowledge (outcome), attitude (outcome) and practice (outcome) of mothers and their children towards the disposal of children's facces in rural communities.
Mothers giving birth to low birth weight babies (LB- WBs) have low contidence in caring for their habies because they are often still young and may lack the knowledge (outcome). experience (outcome), and ahility care (outcome) for the baby.
Mothers giving birth to low birth weight babies (LB- WBs) have low confidence in caring for their babies because they are often still young and may lack the knowledge, experience, and ability to care for the haby.
7
We find large positive impacts on school enrolment (out- come), number of teachers (outcome), and other inputs (outcome) for programme schools near the minimum pass rate.
We find large positive impacts on school enrolment fout- come), number of teachers (outcome), and other inputs for programme schools near the minimum pass rate.
implement randomized experiment offering Sal- vadoran migrants (population) matching funds for edu- cational remittances, which are channeled directly to a beneficiary student (population) in El Salvador chosen by the migrant (population).
implement a randomized experiment offering Sal- vadoran migrants (population) matching funds for cational remittances, which are channeled directly a beneficiary student EI Salvador chosen by the migrant (population).
We find that girls more likely refuse free eye- glasses, and that parental lack of awareness of vision problems, mothers' education, (expenditures per capita and price) significantly affect whether children (population) wear eyeglasses (out- come) in absence the intervention.
We find that girls glasses, and that parental lack awarcness of vision problems, mothers' education, and economic factors (expenditures per capita and price) significantly affect whether children wear eyeglasses (outcome) the ab- sence the intervention.
likely refuse free eye-
economic factors
Table Examples where multiple annotations are plausible

---

of 16 nodes, each one comprising 4 RTX 8000 GPUs, with 48GB of RAM each. We used the pre- processing available DeBERTa's github reposi- tory.". 10
of exact match among entities and part-of-speech not seen during training also stabilizes early during finetuning overall. This seems especially true for EconBERTa-FC, our best performing model, while takes more epoch for BERT, the worst perform- ing model. Note however that the observation is qualitative at this stage and the evidence remains weak.
pper-parameter
EooBERRIa+ES
BooBERRTa-FC
Number Layess
12
Hidden Size
768
768
Hidden
3072
3072
Attention Heads
12
12
Heads
Dropour
0.1
Warmup Stepe
10% of total
10% intal
Learning Raie
7 x 10
6 x 10
Size
3600
3600
Weight Decay
0.01
0.01
Max. Steps
65,388
68,650
Epochs
Learning Rate Decay
Linear
Linear
Adann
1 x
x 10
Adam $1
0.9
0.9
Pa
Grothent Clipping
1.0
1.0
Table 7: paremetasseeters for pre-training
B.3 Fine-tuning Parameters
We display in Table 8 the hyperparameters used to fine-tune each of models.
Hyper-parameter
Value
Dropout of Task Layer
0.2
Learning Rate Batch size
[5e-5, 6e-5, 7e-5]
12
Gradient Ace. Steps
Weight Decay
Maximum Training Epochs
10
Learning Rate Decay
Slanted Triangular
Fraction of steps
6%
Adam
1e-8
Adam B1
0.9
Adam /3
0.999
Table Hyper-parameters for fine-tuning
C Training Dynamies
In order to test whether domain adaptation im- pacted significantly the difficulty to learn the task for our pretrained models, measured how the er- ror types and memorization--elltee metrics evolved with the number of epochs. We display such results in Fig. While we pretrained our models with 10 epochs (see Table we see that most models almost reach their final share exact matches after only few epochs. We further show that the share
10 masteer/experiients///ngguus heree///eeraeggeeeeoeeerrrerrrre

---

Properton Exort Marches
1.0
Propeton of Foart
6.6
0.6
0.6-
Prgpntoon
0.4
0.4
0.2
Drit en tratn 1aneet
a05assuerce zarast
10
10
Proporion of Eoeect Matches
of Exaect Matches
2.0
Prpprtion eren
0.8-
0.6
0.6
0.6
0.4
0.4
0.2
brt cs
H0s senserceie sarset socuecce
Iroportion Iract Mettthee
1.0
Propurtonn bon Deart Matchee
1.0
3.00-
0.75
DaCL excc 200 1ary
6.4
0.4
0.25
0.2
0.2
encLurce
Cowch
Coach
Proporton Exect Matches
1.0
Propertion Exauct Matches
1.0
5.00
mmth 2017 aod
0.4
0.2
0.2
sezuencein
Soarh
Ceach
Proportion of taact Matchiss
2.0
Propattinn Eawert Mattcher
3.0
Prgpomta arror tygee
1.00
0.8
0.0
0.8
0.6
0.6
0.4
02
0.2
0.0
0.0
Figure 7: Training dynamics for EconBERTa-FCC, EconBERTa-FS. mDeBERTa-v3,, RoBERTa and BERT (from top to bottom). In all figures, the x-axis shows epochs. We display the evolution of shares of each prediction type (left column), as well as the rate of Exact Match for entities present/absent from the train set (middle column) and part-of-speecl sequences present/absent from the training set (right column).


Reviews:
[{"review_id": "GPUoE5ENrw", "reviewer": "Reviewer_GfJT", "summary": "", "questions": "Will you share any of your models, code and datasets? \nCan you contextualize the analysis from figure 4 with the general class performance ?\nI assume that the number of test instances of some combination of entity classes on the one hand and the occurrence of an entity in the training data on the other hand varies a lot. \nIs it possible to provide a breakdown of the absolute numbers?\nIs it possible to supplement figures 2 (c) and (d) with information on how many unique entities exist in each type (in-train/out-of-train)?\nThe fact that the distribution of the absolute number of unique entities, at least for entity types like coreference, is very uneven (some concrete entities occur very often, most of them less) is little taken into account. Is it possible to quantify and possibly discuss this fact?\nCan the results for table 5 be split by entities in the appendix?\nDoes the discussed weaker performance of longer entities have a significant impact on the results in Figure 4?\nThe Figure 6 seems to imply, that smaller models like BERT perform equally well on entities in train set and the performance gain can be explained by better performance on unseen examples. Have you investigated this assumption more closely?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their appreciation of the paper. \nWe plan to make all of our results fully reproducible by releasing the pre-trained EconBERTa models, the fine-tuned versions along with the ECON-IE dataset, as well as all of the code used for the experiments. While we do not have a license to redistribute the raw text of the research publications used for pre-training, we will share a file containing the article names and metadata. All of this material will be openly available on GitHub as indicated in the footnote at the end of page 2. \nAdditionally, we computed inter-annotator agreement and found f1=0.87 and Cohen’s Kappa=0.71 prior to curation.\n\nWe will also take into account the suggestions to display additional figures. As suggested, we computed the gain averaged across classes in figure 4, and found that it is very similar to all classes except coreference, which also supports our claim that the effect of memorization is stronger for this class.\nPrompted by the reviewer’s suggestions, we also computed two new quantities to account for the memorization effect displayed in figure 4: (i) the proportion of unique in-train entities relative to all unique entities in the test set, and (ii) the mean number of occurrences for each unique entity in the train set only – a refined version of the metric displayed in fig.4c-d normalized by the total number of occurrences for each class. Both metrics more accurately match the observed memorization effect, especially the latter, which also explains the slightly stronger effect measured for population. \n\nWe also checked the distribution of length for each entity type to see whether the length of entities (as measured in fig. 3) could have an effect on memorization. While the mean length of coreferences is lower, all entity types have a mean entity length close to 1. The distribution of entity length for each entity type therefore doesn’t seem to account for the memorization effects measured in fig. 4, but it will be an interesting piece of information to display in the appendix.\n\nFinally, we will improve the manuscript structure by moving the “Related Work” section after the introduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Craft soliede implementation of an approach to tackle a domain specific NER-Task incl. domain specific task independend pretraining, annotation, finetuning, model comparison and research on memorzation tendencies while finetuning foundation models.\nThe methods for analysing NER models can serve as a blueprint for a better understanding how finetuned foundation models learn to tackle NER tasks in various domains.", "reasons_to_reject": "No statement about sharing the pretrained model (EconBERTa), or the finetuned versions, or the used code base.\nNo Interrator agreement reported.\nMinor reason: The structure is not matching the typicall structure well: Short related work at the end of the paper, bigger parts of the larged number of references introduced in the introduction.", "questions_to_authors": "Will you share any of your models, code and datasets? \nCan you contextualize the analysis from figure 4 with the general class performance ?\nI assume that the number of test instances of some combination of entity classes on the one hand and the occurrence of an entity in the training data on the other hand varies a lot. \nIs it possible to provide a breakdown of the absolute numbers?\nIs it possible to supplement figures 2 (c) and (d) with information on how many unique entities exist in each type (in-train/out-of-train)?\nThe fact that the distribution of the absolute number of unique entities, at least for entity types like coreference, is very uneven (some concrete entities occur very often, most of them less) is little taken into account. Is it possible to quantify and possibly discuss this fact?\nCan the results for table 5 be split by entities in the appendix?\nDoes the discussed weaker performance of longer entities have a significant impact on the results in Figure 4?\nThe Figure 6 seems to imply, that smaller models like BERT perform equally well on entities in train set and the performance gain can be explained by better performance on unseen examples. Have you investigated this assumption more closely?", "paper_topic_and_main_contributions": "The paper descibes data collection, model training (inlc. pretraining) and deep analysis of a domain specific NER task.\nThe domain dependend NER task is the detection of  valuable entity types to monitor the impact of policy decisions based on abstracts of economic resarch publications using six enity types. \nThe modeling not only includes finetunig of three open available Pretrained Language Models, but two pretrained Models by the authors.\nThese models pretrain a mDeBERTa-V3 model on a corpus of research publications (10.8 B Token).\nThe authors not only show the performance gain of 3% of the NER-fintuned version of there pre-trained model, but also share insghtful analysis on the NER model peformance reflecting the influence of span length on the model performance. \nAdditionaly they research the possible influence of lexical and lexicosyntactical model memorization during train time.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ITIfS7hxLC", "reviewer": "Reviewer_XXaT", "summary": "", "questions": "Question 1: Who prepared the annotation guidelines?\n\nQuestion 2: When the annotations are different, either overlapping or not, how do you resolve them? \n\nQuestion 3: Did you compute the inter-annotator agreement at any point of annotation process?\n\nQuestion 4: Do you do any pre-processing of the 1.7 million papers?\n\nQuestion 5: Isn't the EconBERTa-FS trained from scratch on economics domain? Isn't its performance worse than EconBERTa-FC? \nThe description in the text is not consistent with the results. lines 242 - 243 and 249 -250\n\nQuestion 6: How do you achieve the results in Table 3? What data do you train the models on? 80% of 1000 abstracts? what data do you use to validate your model on? What is the distribution of entities in each set?\n\nQuestion 7: Are the results in Table 3 exact match?\n\nQuestion 8: What does entity length of \"0\" mean?\n\nQuestion 9:  What you exactly mean by isolating entities in section lexical memorization? What happens if in one sentence one entity is in -train and another not?\n\nQuestion 10: Can you show the performance of the models for each entity?\n\nQuestion 11: How long did the training of models take?\n\nQuestion 12: Wasn't EconBiz data (https://www.econbiz.de/) appropriate for the paper's analysis?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for his questions and suggestions.\n\nThe final paper will include clarifications on the annotation process: \n* The annotation guidelines were produced by experts in the field of economics who decided on the key entities of interest, and experts in NLP and linguistics who designed explicit and unambiguous annotation rules paired with examples (Q1). We will clarify this in the final version of the article. \n* We also computed the inter-annotator agreement and found f1=0.87 and Cohen’s Kappa=0.71 prior to curation (Q3).\n* Annotations underwent a curation phase, where discrepancies between annotators were resolved by a curator who contributed to the guidelines. When the disagreement was due to multiple plausible annotations, a new rule was set and discussed with the team to arbitrate all such cases (Q2). \n\nAdditionally, we will add further clarifications following some of the reviewer’s questions:\n* (Q6) the procedure for Table 3  is described in the “NER fine-tuning” portion of the “Models” subsection (2.2, l.222-230), and the splitting strategy (80/20 train-test split and 5-fold cross-validation) is described in the “Sample selection” paragraph of the “Econ-IE dataset” subsection (2.1, l.152-155). We will add cross-references to improve clarity throughout the article.\n* The results in table 3 are F1-scores as mentioned in the second column. The F1-score is computed at the token-level (Q7) as mentioned in section 3.2 (l.261). We will clarify this point in 3.1 in the final version of the draft. \n* We will also clarify what was meant by “isolate entities [X]”, as we meant “compute scores over entities [X]” (Q9). The reviewer asked if having an in-train and an out-of-train entity in the same sentence would be problematic in this case. As our metric is defined at the entity-level and not the sentence-level, the rest of the sentence, including other entities, does not matter.\n* We will clarify that EconBERTa-FS and EconBERTa-FC both outperforms publicly available models (l. 242-243). When comparing EconBERTa-FS to EconBERTa-FC (lines 249-250), we find that the difference is not significant as indicated by the confidence intervals in table 3.\n\nWe will also add more information on pre-training:\n* (Q4) Our pre-training data first required extracting raw text from pdfs. We then performed deduplication and noise removal, the quality of which we controlled by checking chunks that were randomly sampled from the clean text. We will detail this process in the final draft. \n* (Q11) The pre-training of the DeBERTa language models took around 60 hours (hardware and hyperparameters in App. A.2 and Table 6), and fine-tuning of the NER models took around 9 minutes (hyperparameters in App. B.4, Table 8). \n* As for the suggestion to use EconBiz papers, it is indeed relevant but greatly overlaps with the data sources we already collected (RePEC, SSRN). We will explore incorporating additional corpora for subsequent releases of the model.\n\nWe will also incorporate the reviewer’s suggestion by displaying performance by entity (Q10) and fixing the indexing in figure 3 by starting at an entity length of 1 (Q8). \n\nTo conclude, we kindly wish to draw the reviewer’s attention to important aspects of our paper: \n* As we mention in the introduction (l.50-53), to our knowledge there exists no NER dataset in economics, which underlines one of the novelties of our work. \n* Additionally, our main scientific findings (i.e. the results we derived from the series of diagnostic tasks in section 3) are of interest to the community, in addition to the public release of the models and dataset. \n* Finally, we will release the code, model weights, annotated dataset and annotation  guidelines to make all of our results reproducible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a new dataset for NER in economics.\n- The paper provides a model for NER in economics.", "reasons_to_reject": "- The manual annotation needs to be clear.\n- The method, experimental setting and evaluation sections require clarification and more details.\n- Literature review needs improvement; no literature review for economics data and NER in economics domain is presented. It would be helpful to mention what exists and why it is not appropriate for this paper's analysis.", "questions_to_authors": "Question 1: Who prepared the annotation guidelines?\n\nQuestion 2: When the annotations are different, either overlapping or not, how do you resolve them? \n\nQuestion 3: Did you compute the inter-annotator agreement at any point of annotation process?\n\nQuestion 4: Do you do any pre-processing of the 1.7 million papers?\n\nQuestion 5: Isn't the EconBERTa-FS trained from scratch on economics domain? Isn't its performance worse than EconBERTa-FC? \nThe description in the text is not consistent with the results. lines 242 - 243 and 249 -250\n\nQuestion 6: How do you achieve the results in Table 3? What data do you train the models on? 80% of 1000 abstracts? what data do you use to validate your model on? What is the distribution of entities in each set?\n\nQuestion 7: Are the results in Table 3 exact match?\n\nQuestion 8: What does entity length of \"0\" mean?\n\nQuestion 9:  What you exactly mean by isolating entities in section lexical memorization? What happens if in one sentence one entity is in -train and another not?\n\nQuestion 10: Can you show the performance of the models for each entity?\n\nQuestion 11: How long did the training of models take?\n\nQuestion 12: Wasn't EconBiz data (https://www.econbiz.de/) appropriate for the paper's analysis?", "paper_topic_and_main_contributions": "The paper presents a dataset of 1000 economics abstracts for named entity recognition and models trained on economics literature for five types of entities relevant to impact evaluation studies in economics. The paper further analyzes the models' performances based on a set of entity characteristics, such as the entity length and POS tags.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 322 remove \"label\": it is not label, it is the length of the entity itself."}, "extra_text": {}}, {"review_id": "Hlx0k49uCl", "reviewer": "Reviewer_iiaL", "summary": "", "questions": "- Out of curiosity, why is it important that we have a language model for the economy domain? Specifically, why do we annotate for these specific named entities as proposed in the paper and nb economic research papers? I'm missing this connection in the paper.\n- Is the pre-training data also consisting of Abstracts from Economics papers? If not, how did you obtain the full text from the PDF e.g., SSRN?\n- Is there any chance that you release the full annotations before they were merged? This is also valuable information.\n- In the results, the best-performing model has an F1-score of 0.736. In the analysis, we can see a gain of 0.5 (what?) when we measure between in-train and out-train. Is this 50 F1 points or just 0.5 F1 points?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first wish to thank the reviewer for the insightful comments and questions.\n\nWe will clarify the annotation process by incorporating the reviewer’s suggestions:\n* The corpus of abstracts was divided into 20 subsets of 50 abstracts. Each annotator was assigned four subsets of 50 abstracts, such that all 1,000 abstracts were annotated by two annotators. Before producing the final annotations, the annotators first performed a practice round on 10 abstracts each and received feedback from the NLP researchers who designed the guidelines. \n* As mentioned in l. 170-175, there were discrepancies between annotators. We computed IAA and found f1=0.87 and Cohen’s Kappa=0.71 prior to curation.\n* In some cases, the disagreement was due to at least one annotator not properly applying the annotation rules. In other cases where multiple annotations are plausible, we added a new rule after discussion between our NLP and economics experts, e.g. “annotate the shortest span as long as no information is lost”.\n* Our annotation process requires competent annotators (“at least graduate-level education in economics” l. 161-164) which are a costly human resource. We chose to perform two rounds of annotation per abstract to ensure an acceptable level of consistency for a reasonable budget, and included a curator to resolve disagreements. \n\nAdditionally, we will strengthen the motivation for our analysis: \n* Memorization (l.331-339) is often opposed to true generalization and the goal with our in-depth analyses is to test if the model truly generalizes (e.g. to long sequences, or spans not seen during training). Assessing generalization in NLP beyond coarse-grained metrics has been a hot topic in NLP over the past years (l. 92-109, l. 456-470, l.486-507).\n* We motivate the importance of an in-domain model in l. 73-91: they allow performance gains on tasks performed in a given domain. Our model is valuable for a wide range of downstream applications in economics including information extraction, and could become as useful as successful counterparts in other domains, such as BioBERT, or BERTTweet. cited in l.85 and l.220-221.\n* We also specify that causal inference analysis is almost ubiquitous in economics literature and the five entities we extract are commonly used to summarize impact evaluation research. This specificity makes the target classes unlikely to be found in the existing NER literature. We already mention that our aim is to extract causal knowledge (l.24, 60, 141, 1183) but will extend on their utility for practitioners in the field of economics and impact evaluation. \n* Additionally, we will clarify that we chose to identify spans using POS sequence as it can be restricted to a subspan for a given entity, while dependency parsing would have arrows pointing to the sentence and is less handy to identify a subsequence syntactically. Another candidate was constituency parsing, which we also discarded as the boundaries of our annotated entities do not necessarily match with those of syntactic constituents. \n* We also shared the intuition that overlap between entities found train/test sets alone could account for memorization effects displayed in fig. 4, but it is not as informative as the mean number of occurrences for unique instances found in the train set: it varies between 61% and 75% for all classes. We can add this in the appendix. \n\nAs for the other questions asked:\n* All scores are displayed in a range from 0 to 1 (0.x F1), except from table 5 where the gains displayed are F1 points (xx.0 F1), due to their small magnitude (we report a “little effect” in l.442). We will display them as 0.x F1 in the final version of the manuscript to avoid any confusion and mention it in the text. \n* We fully agree that the test set shouldn’t interfere with model selection. We carry the memorization analysis only after the best performing model was chosen, a process that did not influence model selection. In doing so, we separate this diagnosis analysis from the model development phase. For the final version of the paper, we can display this analysis for the dev set in the appendix.\n* For pretraining, we did use the full text of articles which we obtained from pdfs by performing text extraction using a pdf2txt python library, and further preprocessed the text to remove noise. We then performed a random sampling across sources to control for the quality of our preprocessing. We will add more details on this full preprocessing pipeline in the final draft. \n\nFinally, we thank the reviewer for the missing references and will also add more information about our annotators in the appendix as suggested in Bender and Friedman’s paper, as this can lead to improved transparency and clarity regarding the annotation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-structured, clearly written, and an interesting domain. In addition, the authors release a novel dataset for the economy domain.", "reasons_to_reject": "- The annotation process is rather handwavy, there's a lot of information missing: \n  - How many rounds did the annotators annotate for? \n  - Was there any discrepancy between annotators? \n  - What did the annotators get paid?\n  - Did they uniformly annotate as much as others? \n  - Why two annotators per abstract and not more? \n  - In Appendix B3, the authors mention multiple plausible annotations, how did the final annotator decide on which one to choose? \n  - What was the overall agreement between annotators?\n- The analysis generally could be improved. \n  - The sections on memorization are verbose and not motivated (why memorization?). This analysis could be much shorter by calculating the overlap between unique entities of train/test. This already could explain the performance differences between train and test. Given the low number of unique entities in, e.g., coreference, it could already be that the in-train split, could be much larger than the out-of-train split.\n  - The authors evaluated with an in-train and out-train split where the entities overlap or do not overlap with test. It is unclear what the performance gains are on the y-axis, is this F1? If so, are these gains of 0.x F1 or xx.0 F1?\n  -For the lexicosyntactic memorization, why exactly POS tagging? Why not dependency parsing? Would this have the same results? This is not mentioned.\n  - I personally would not suggest doing an analysis on the test set. Insights on the test set can influence model development and thus inflate performance metrics. Therefore, I suggest using a dev. set next time.", "questions_to_authors": "- Out of curiosity, why is it important that we have a language model for the economy domain? Specifically, why do we annotate for these specific named entities as proposed in the paper and nb economic research papers? I'm missing this connection in the paper.\n- Is the pre-training data also consisting of Abstracts from Economics papers? If not, how did you obtain the full text from the PDF e.g., SSRN?\n- Is there any chance that you release the full annotations before they were merged? This is also valuable information.\n- In the results, the best-performing model has an F1-score of 0.736. In the analysis, we can see a gain of 0.5 (what?) when we measure between in-train and out-train. Is this 50 F1 points or just 0.5 F1 points?", "paper_topic_and_main_contributions": "This paper is about a new language model for the Economy domain: EconBERTa, two models EconBERTa-FC further pre-trained from a checkpoint and EconBERTa-FS pre-trained from scratch on 10.8B tokens with a (m)deBERTa-v3 architecture. They furthermore release a complementary NER dataset ECON-IE to extract named entities from scientific publications of economics. They annotated for 5 classes: intervention, outcome, effect_size, population, coreference. They show that their model EconBERTa-FC works best. They further do analysis on memorization of training, performance on span length, and analyze types of errors.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Related Work: There is a lot of work on domain-adaptive pre-training similar to what is introduced here in the paper. There is more work on domain-adaptive pre-training:\n  - https://aclanthology.org/D19-1433 (Han and Eisenstein, 2019)\n  - https://aclanthology.org/2020.acl-main.740 (Gururangan et al., 2020)\n  - https://aclanthology.org/W19-1909/ (Alsentzer et al., 2019), ClinicalBioBERT\n  - https://aclanthology.org/2022.naacl-main.366 (Zhang et al., 2022), JobBERT\n\nThe evaluation suite is very similar to SemEval 2013, task 9: https://aclanthology.org/S13-2056.pdf. Possibly reference to this.", "Typos_Grammar_Style_and_Presentation_Improvements": "- There is some inconsistency in usage of a period after a paragraph header;\n- In the discussion section, you have a paragraph on \"learning from mistakes\". Even though you pinpointed the mistakes the model makes in the analysis, you did not leverage this information to improve the model. Personally, a bit of a handwavy paragraph that could be removed.\n- I suggest creating a data statement (Bender and Friedman, 2018): https://aclanthology.org/Q18-1041\n- The Conclusion section is missing a number."}}, {"review_id": "7wmChWPQlD", "reviewer": "Reviewer_Cyad", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first wish to thank the reviewer for the insightful comments, as well as the comprehensive acknowledgement of our paper’s strengths.\n\nWe agree that the novelty in our paper does not lie in the model development pipeline. That being said, we produce a novel dataset which can be a precious resource for information extraction in the fields of economics, and a novel model which outperforms publicly available architectures for NER extraction in that domain. We also introduce novel experiments to diagnose how models generalize after they have been trained, which can inspire future diagnoses of fine-tuned models in a variety of settings for downstream deployment. \n\nRegarding the suggested length of the paper, we agree that our results and contributions would fit in a short paper if we planned on releasing only the dataset and best performing model. However, we believe that our experiments are crucial as diagnosing models is an important quest in NLP, given that surface performance can be deceitful. This has been shown for instance in previous work which we cite, including McCoy, Pavlick and Linzen (2019) along with a large body of research cited in l.462-464.\n\nWe also fully agree that it would be very useful to present the specificities of our NER task, as detecting our entities can be less straightforward than detecting other entity types that are more frequent in the literature. For example, disagreement can be rarer for the boundaries of a span describing a person, a location or a quantity. However, for populations or interventions, we fall into some difficulties regarding e.g. the level of details that we want to keep (e.g. “children aged 4-10”). We formulated precise annotation rules to handle such cases, in order to avoid any confusions, for annotators and for the model. However the obtained set of entities are heterogeneous in length and complexity, which makes the task harder for the model than NER addressing simpler entity types. This specificity is not exclusive to economics however, but to the type of entities that we are extracting in this context. We plan on clarifying this in the final version of the paper, and are thankful for this suggestion as it will make the paper stronger, and can be beneficial to a broader audience. Also, please note that the annotation guidelines will be made public along with the ECON-IE dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper was well written.  I found it very easy to follow!\n2) The authors focus on a less-studied domain (economics), which expands the reach of NLP beyond standard datasets. The authors created a new IE dataset for the domain and a new foundation model.\n3) The authors show a performance increase for their target dataset, with statistical testing and reasonable baselines.\n4) The paper includes an extensive analysis of where the model performs well and does not perform well. ", "reasons_to_reject": "1) Without diminishing the impact of new datasets within the field, the authors do not introduce any new modeling techniques for the EconBERTa model. They appear to use a standard fine-tuning technique, and the fact that this improves performance on the in-domain NER task is predicted by other work.\n2) I'm not sure the content of this paper is sufficient for a long paper.  The model analysis is a great addition to the paper, but I wonder if reconfigured (specifically sections 3.2 - 3.5, the highlights included in the main paper, the details in the appendix), if it isn't better suited for a short paper contribution.  This is especially true since the analysis generally meets expectations of other models.  I think this might make the paper stronger. \n3) I think another path to making this paper stronger is delving deeper into the specifics of the domain.  I don't see much discussion of what makes Economics NER distinct in the evaluation section - is this just the same as any other NER dataset?  Even better would be an exploration of modeling changes that are specific to Economics can improve performance, but I understand this is a more difficult task.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper focus on training a LLM (based on DeBERTa) for the economics domain.  They train their new model, EconBERTa, on 1.7 million economic papers.  The authors apply this model to the information extraction task, for which they annotate a new economics-specific dataset.  To test the performance of EconBERTa, they fine-tune a CRF layer for NER classification for EconBERTa and several baseline models.  They find that EconBERTa has a small but significant performance increase over baselines.  \n\nThe authors then proceed to perform a large evaluation of the model's performance.  This includes finer-grained metrics to evaluate where errors are occurring, including boundary errors or entity type errors.  They find that most errors consist of boundary errors with the correct entity type.  Further, they find that entity length has an effect on performance.  Finally, they probe the model's tendency to memorize, and explore whether it memorizes via lexical memorization or via lexicosyntatic memorization (i.e. part of speech).  \n\nThe authors promise to release their dataset and code.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers appreciated the well-written presentation of a new pretrained model and NER dataset for the field of economics. Several reviewers found issues with the level of detail provided about how the dataset was created, as well as the specific protocols for annotation which may impact the reproducibility and soundness of the approach. Annotations were shared with reviewers during the evaluation period, but the pretrained model was not made available. Reviewers also suggested a deeper description of the motivations of targeting the economics domain, domain-specific considerations that express the unique challenges in economics, and additional or revised analysis. Reviewers also suggest a discussion of related work, both for comparing their approach to other domain-specific pretrained LMs, existing, widely-used NER models, and the core tasks in the economics domain. This paper is unlikely to have an impact in NLP communities due to a lack of novel methods or analysis, but may be a valuable contribution for research in economics.
Decision: Accept-Findings