GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets
Authors: Wolfgang Otto
Matthäus Zloch
Lu Gan
Saurav Karmakar
Stefan Dietze
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction (IE)
ML models
datasets
Named Entity Recognition (NER)

Abstract: Named Entity Recognition (NER) models play a crucial role in various NLP tasks, including information extraction (IE) and text understanding. In academic writing, references to machine learning models and datasets are fundamental components of various computer science publications and necessitate accurate models for identification. Despite the advancements in NER, existing ground truth datasets do not treat fine-grained types like ML model and model architecture as separate entity types, and consequently, baseline models cannot recognize them as such. In this paper, we release a corpus of 100 manually annotated full-text scientific publications and a first baseline model for 10 entity types centered around ML models and datasets. In order to provide a nuanced understanding of how ML models and datasets are mentioned and utilized, our dataset also contains annotations for informal mentions like "our BERT-based model" or "an image CNN". You can find the ground truth dataset and code to replicate model training at https://data.gesis.org/gsap/gsap-ner.

GSAP-NER:
A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets
Wolfgang Otto', Matthäus Zloch 1.2 Lu Gan 1.2 Saurav Karmakar', and Stefan Dietze 1.2
GESIS Leibniz Institute for the Social Sciences, Cologne, Germany Ceetrrih--Hiin-Univeee Düsseldorf, Germany
(wolfgang.otie lu.gan, saurav. saurav.karmakar, karmakar, stefan (lefan.dietze] ) |@ @gesis.org (matthaeus.zloch, lu.gan, stefan.dietze] @ hhu.de
Abstract
MI.Mod:\Generic
MLModel
Our model
dubbed
XLM-R
significantly
Named Entity Recognition (NER) models play a crucial role in various NLP tasks, includ- ing information extraction (IE) and text un- derstanding. In academic writing, references to machine learning models and datasets are fundamental components of various computer science publications and necessitate accurate models for identification. Despite the advance- ments NER, existing ground truth datasets do not treat fine-grained types like ML model and model architecture as separate entity types, and consequently, baseline models cannot rec- ognize them as such. In this paper, we re- lease a corpus 100 manually annotated full- text scientific publications and first baseline model for 10 entity types centered around ML models and datasets. In order to provide nuanced understanding of how ML models and datasets are mentioned and utilized, our dataset also contains annotations for informal mentions like "our BERT-based model" or "an image CNN", You can find the ground truth dataset and code replicate model training https://data, gesis. wwrggggap/gssat happ/ggappneeeee ner.
MLModel
MI.Mode
outperforms multilingual BERT BBER
DatasetGeneric
on a variety of cross-lingual benchmarks in-
Datasct
cluding + +14.6% average accuracy on XNLI
Datasct
+13% average Fl score on MLQA and
Dataset
+2.4% Fl score on NER
Figure 1:
Example of machine learning models and dataset related mentions annotated according tag set. "Our model" and "cross-lingual benchmarks" are considered informal mentions, whereas the others are noun phrases mentioning named entities.
models (LLM). Consequently, effectively com- prehend scientific articles related to ML, AI, or data science, becomes crucial to identify and comprehend these emerging entity types.
1 Introduction
Developing effective NER models for these enti- ties requires annotation guidelines and ground truth datasets train robust language models (Qasem- iZadeh and Schumann, 2016; Luan et al., 2018). However, existing guidelines and ground truth datasets for scholarly entities have not adequately addressed the finer-grainn entity types, such MI Models and Datasets distinct entities. In- stead, state-of-the toe-thhearttt works treat ML models Methods (Färber et al., 2021), failing to differenti- ate between the model instance, type, and under- lying architecture. Similarly, dataset mentions are typically categorized as Muterial, overlooking the fact that this can also encompass knowledge hases, resources, or other corpora.
Throughout various disciplines, the scientific pro- cess constantly produces new knowledge, innova- tive discoveries, and valuable insights, which typi- cally published in conference proceedings and journal articles. The increasing volume of scholarly artifacts underscores the importance for scientists to efficiently locate, comprehend, and utilize these resources in their daily work. Consequently, the NLP community is constantly creating methods to extract named entities from the scholarly domain, recognizing its significance in facilitating scientific understanding.
With the advent of artificial intelligence (Al), the landscape of machine learning (ML.) approaches has evolved, incorporating techniques such as deep learning (DL) and fine-tuning of large language
This paper presents GSAP-NER ground truth
Acronym stands for: GESIS Scholarly Annotation Project

---

dataset specifically designed enable the develop- ment of language models tailored for identifying named entities associated with the interplay be- tween machine learning models and datasets. benefits from a detailed annotation scheme that is customized for the discussion and use of machine learning models and the data used. We address the limitation of existing datasets by emphasizing comprehensive annotation of full scientific paper annotation rather than solely focusing on annotated abstracts or pre-selected sections. Our dataset of- fers two significant advantages. Firstly, we place particular emphasis on capturing informal mentions of named entities. These unnamed, descriptive mentions indirectly relate to named entities (e.g.. "cross-lingual benchmarks" in Figure 1), providing valuable training data for co-reference resolution tasks. Secondly, our dataset features nested entity annotations (Finkel and Manning, 2009; Katiyar and Cardie, 2018), enabling the annotation of mul- tiple sub-parts of text span within a single noun phrase.
SciERC SciREX TDMSci Heddes
GSAP
NER
Ann.unit
# pub.
500
438
n/a
n/a
100
pos, sent.
2,558
76,223
2,000
2,911
25,857
# neg. sent.
214
44,209
3,089
7,530
# ent.types # mentions # ment/pub.
10
8,089
156,931
2,937
3,729
54,598
16.2
358.3
n/a
1/a
546.0
Material
Datasct
DataSource
Metric
Method
Model
ModelAreh.
Task
Table 1: Comparison ground truth datasets for schol- arly NER Tasks including annotated entity types. Annotation units: =absttrctt, =seentence, =ffull-text.
We introduce fine-grained tag set designed for detecting scholarly entities and concepts, customized to reflect the use and presentation of machine learning models and datasets scientific publications (Section 3.1).
Based on our ground truth we fine-tune first baseline model for our task of ML model and dataset named entity recognition. We employ three state-of-the-art baseline models for that, SciB- ERT (Beltagy et al., 2019), RoBERTa (Liu al., 2019), and more recent pre-trained language model called SciDeBERTa-CS (Jeong and Kim, 2022). We have found that SciDeBERTa-CS per- forms best on the entity types MLModel and Dataset, with Fl score of 0.71 and 0.81, re- spectively.
We conduct a comprehensive performance evaluation of baseline models for our ten de- fined entity types (Section and 5).
We explore the minimum number of amno- tated publications needed to achieve satisfac- tory performance in our fine-grained scholarly NER task, which can guide future annotation projects (Section 6.2).
Creating sizable ground truth dataset like GSAP-NER is costly in terms of effort. There- fore, as a final experiment, we explore the mini- mum quantity of fully-annotated texts required to observe noteworthy improvement in performance by incrementally increasing the size of the training data. Our datasct enables researchers and practi- infor-
All materials, such as the ground truth dataset and the code to replicate model training can be found at httpss:/datte .eriii..gesss............ htsp///ap/ggapppaaeeeee
Related Work
tioners to extract precise and domain-specti mation, contributing fields such as information retrieval, scientific knowledge mining, automated literature analysis, and knowledge graph creation.
Among the works dealing with the task of scholarly information extraction, in this section we focus on those named entity recognition methods which are mmachine-earming-basee and not rule-based ap- proaches. As general overview, Nasar al. (2018) gives a comprehensive list of approaches on information extraction from scientific publications. Multiple datasets serve as ground truth datasets for Named Entity Recognition (NER), each cater-
Our research presents four key contributions, each aimed at advancing the state of scholarly en- tity and concept detection:
We provide a manually annotated dataset con- taining 100 full-text computer science publica- tions with over 54k entity mentions 25,857 sentences (Section 3).
In the literature, it is also referred to as key-insight extrac- tion, (typed) entity recognition, entity extraction, or (scientific) concept extraction.

---

ing to specific tasks. Please consult Table for a comparison of the most related ground truth datasets to ours, GSAP-NER.
nected methods aimed creating knowledge bases or knowledge graphs. Agrawal al. (2019) fo- cused on extracting the aim, method, and result sections from scientific articles, utilizing this infor- mation to construct a scientific knowledge graph. Similarly, Mondal et al. (2021) developed SciNLP- KG, framework designed to extract TDM entities and relations from papers in the NLP domain. Fur- thermore, Dessí al. (2022) presented a computer science knowledge graph (CS-KG) that is automat- ically generated and periodically updated. They achieved this by applying an information extrac- tion pipeline to a vast repository of research papers, offering a comprehensive and up-to-date resource for the computer science domain.
Among datasets working abstracts, SciERC stands out as prominent dataset, comprising 500 abstracts extracted from 12 AI conference and workshop proceedings (Luan et al., 2018). This rich dataset contains annotations for scientific enti- ties, their relationships, and co-reference clusters, which are invaluable for related NLP tasks.
Another dataset, SeiREX, offers comprehen- sive coverage with 438 fully annotated documents, specifically targeting mention identification and re- lationship extraction of entities related to methods, tasks, datasets, and metries (Jain et al., 2020). To prepare their ground truth dataset for the NER task, they combined distant supervision and manual cor- rection of automatically pre- pre-taent full-texts. contrast to their work, we created a fully man- ually annotated corpus. This enables us to define our tag set independently of current approaches and to avoid potential model bias introduced by pre-annotation.
3 Dataset
Machine learning models and datasets are essen- tial scholarly entities that are discussed in various scientific disciplines. For named entities, it is fre- quently observed that, depending on context, iden- tical string spans refer to different semantics. Take "BERT" in natural language processing an exam- ple: it can refer to a particular pre-trained model with fixed parameters like "BERT Base" or its ar- chitecture, depending on the context. In addition to named entities, unnamed or informal mentions of machine learning models or datasets are more com- mon in scientific text. But those informal mentions often contain nested references to other named en- tities and thereby carry extra information linking not only machine learning models or datasets but also to other scholarly entities such as methods, model architectures and tasks. An illustrative exam- ple can be found in Table 2: "For the ResNets we train [...]". The additionally carried information via informal, generic mentions requests a nested an- notation style, meaning both the informal mention and the referenced nested named entity needs to be annotated. Therefore, in order to construct gold standard scholarly entity mentions dataset, we have defined 10 different entity types in categories: MLModel related, Dataset related and miscella- neous. The following gives a brief description of the entity types in our tag set. The more detailed annotation guideline includes further description, examples and figures, and can be found on the projects Web page
Hou et al. (2021) contributed significantly to this area by presenting TDMSci, a corpus contain- ing domain expert annotations for TDM entities in 2000 sentences extracted from NLP papers, along- side dedicated TDM tagger designed for this spe- cific task.
Heddes et al. (2021) developed a ground truth dataset for dataset mention detection, comprising 6,000 annotated sentences selected by the occur- rence of dataset related word patterns that were sourced from four major Al conference publica- tions. Approximately half of them containing one or more named datasets.
An emerging NL. task known as leaderboard ex- traction focuses on extracting Bask-Dataset-Mrr Metric- Score (TDMS) tuples from scholarly papers, en- abling the generation of an aggregated compari- son view of the main entities of interest (Kabongo et al., 2021). Along this direction, Kardas et al. (2020) introduced extraction pipeline, AxCell, for extracting results from tables listed in scientific articles. In 2022, D'Souza and Auer (2022) created CS-NER, a corpus of contribution-centti informa- tion extraction targets, namely rescarch problem, method, solution, dataset, metric, and more.
Recent lines of research have explored end-to- end frameworks based on NLP extraction tasks, such as NER, which involve a series of intercon-
hhttps://data. http://aaaa.ggii....esssssssssss wrw.ggaa/ggaapnnerrrrrrrrr

---

Annotation sentence with identified spans
Justification
For the ResNets
train ResNet-50
ResNet-101
the ResNets is annotated as whole as informal mention of multiple concrete Models, Additionally, it includes the nested annotation ResNets which corresponds to the structural type information ModelArchittettrr. ResNet-50 and ResNet-101 are actual executable MidModels.
and then more
a new large-scale dataset called SearchQA ... (an existing question-an answer pair crawled from !!Archive and augment it with the text snippets re-
We publicly release
Dataset and DatasetGeneric are identified with similar reason to the previous example, while unstable data source informa- tion are reengnized as DataSowre..
trieved by Google
Table 2:
Annotation examples for machine learning model related entities: DatasetGeneric, and DataSource
MLModel
MLModelGeneric
ModelArccitecture, And data related entities: Dataset
3.1 Annotation Tag Set
tion in Table 2, "ResNets" is labeled as a ModelAr- chitecture due to its abstract and categorical nature, rather than denoting a specific resource.
We categorize our annotation tag sets into three cat- egories: (1) MLModel related, including MLModel, ModelAvchitecture, MiMooeeGennrri, Method and Task; (2) dataset related, including Dataset, DatasetGeneric and DataSource; (3) miscella- neous, including ReferenceLink and URL. In par- ticular, the Generics (MLModelGenerice Dataset- Generic) correspond to the informal mentions of named entities.
MIoodelGeneiic corresponds to the informal or unnamed mentions of MLModel entities. These informal mentions use possessive, temporal, quan- titative or qualitative features refer one, multiple or general MLModel entities.
Method corresponds to non-MLModel ap- proach, a scholarly entity produced by MLModel entities and non-MLModel approaches (c.g., "word embedding"), This definition accordance with other annotation guidelines (ScilE, SciERC), which also define "Method" as a broad category of various methodological statements.
3.1.1 MLModel Related
Machine learning model-related entities are tagged with this category of tags. We specifically separate ML pre-trained models from MI. concepts, and map them into MLModel and ModelArchütecture. We explain each single of the entity tags below and illustrate them with a real world example from our annotation work in Table 2.
Task refers to a named entity machine learn- ing task. We note that a task can relate to both ML models and datasets; MLModel can be applied on a Task and Task can be based on a Dataset. For simplification, we assign it under MLModel related category.
MLModel refers to string span that represents a named entity of a machine learning model. For neu- ral network based machine learning models, such a string span should correspond to an executable resource of the model in the context. In the first example of Table 2, "ResNet-50" corresponds to trained executable resource and is therefore anno- tated as MLModel. A MLModel usually is based on some machine learning (ML) architecture, and can be applied to some ML tasks.
3.1.2 Dataset Related
Dataset related entities are tagged with this cate- gory of tags. We explain the entity tags below and illustrate their usage with real examples from our annotation work in Table 2.
Dataset refers to named string span correspond- ing to an explicit dataset object in the text (e.g., "So- cial Bias Inference Corpus", "SBIC", "SQuAD"), DataSource corresponds a named entity of some unstable or unstatic data source information (e.g..
MocteArrcheeectuee refers named entity cor- responding to the conceptual or structural informa- tion of machine learning model. ModelArchitec ture can usually be interpreted as type information of other MLModel entities". In the nested annota-
particularly assign a corresponding name entity as MLModel when it is mentioned for performance comparison, as shown in Figure 1. During the annotation process, we collect and categorize confusing and borderline cases according to the mention patterns, which we give a more detailed domonstra- tion in the additional material.
"We differ a MLMedel from a ModeAArchitectuum for a name entity essentially hy whether the name entity is served as a resource/artiffct or concept/idea in the context. We

---

"Google", "Twitter"). DataSource is unstable due to its time-evolving nature and intractable timestamp. Therefore, knowledge bases and gen- eral mention of Wikipedia are considered as Data- Source. On the contrary, a Wiki dump with a spe- cific timestamp will be annotated as Dataset.
mutual F1 exact-match
mutual Fl
partial-match
MLModel
72.1
74.6
MLModelGenerie
60.7
67.6
ModelArchitecttrc
23.7
34.4
Method
47.0
60.7
DatasetGeneric corresponds to the informal or anonymous mentions of Dataset entities. Similar to MeMModelGeneric for MLModel, DatasetGeneric entities use possessive, temporal, quantitative or qualitative features to refer one, multiple or general Dataset entities.
Task
51.4
55.2
Dataset
84.1
86.7
DatasetGeneric
56.2
65.8
DataSource
55.3
62.7
Referencel.nnk
90.5
94.8
URL
86.1
94.1
3.1.3 Miscellaneous
all
61.4
69.3
URL corresponds to a string span that an URL in the text. ReferenceLink string span that rep- resents reference in the text. A ReferenceLink may present in different style, but requires to be linkable to the bibliography section the end of the scientific article.
Table 3: Interrator agreement measured by the
erage mutual FI of three annotators on the 14% co- annotated publications.
arXiv, including citation hints. This process sults in selection of 50 publications that not only present models available Huggingface, but also discuss datasets, tasks, architectures and methods used.
3.2 Publication Sampling
Selecting relevant and representative publications for the purpose of training and evaluating, which either introduce or harness machine learning mod- els and datasets presents dual challenge. On one hand, it necessitates the inclusion of cutting-edge methodologies and well-established models and datasets to ensure comprehensive overview, On the other hand, it's equally crucial to embrace diver- sity by incorporating publications that might be less recognized or have garnered fewer citations, thus providing broader spectrum of perspectives. For primary source of full-text materials, place trust in arXiv?, the preeminent open-access repository within the domain of computer science. In our quest to curate a selection of 100 publica- tions, we employ two distinct but intertwined strate- gies: one that prioritizes popularity (I), and another that promote diversity (2).
To account for diversity (2), we randomly se- lect out of model-related arXiv publications. To identify those, we filter the arXiv publications by research area (i.e., "cs.LG: Machine Learning)), based keyword match and by time frame (i.e., first upload between 2018 and 2022).² For the keyword- based relevance classification, a heuris- tic is utilized. Publications must mention the term model the title or at the beginning of the abstract (first 20 tokens), and data must be mentioned in the abstract." Finally, we randomly selected 50 pub- lications from the resulting pool of 12,641 arXiv publications. The final collection of publications is subjected validation process to ensure that not part of other datasets such as SciERC or SciREX.
Due its popularity (I), turn Hugging- face', a premier and dominant platform today for showcasing and distributing machine learning mod- els (Jiang et al., 2023). Using the number down- loads as a metric, we compile list of the most frequently used models. We then search the mod- els' README files for links to publications on
3.3 Annotation Strategy
We have three annotators with computer sci- background to conduct the annotation using INCEPTION".
All three annotators had annota-
We used Version 123 of the arXiv Dataset available at https:/ /tt:://wwww kaggle. waggle.com// com/coo//aaaaa/o /ataaatassss////// Cornell Lemell-nnivers//e wwtssstttaaaiiiii http/aaaxw/veessoonss.s 1ons/123.
'https: /ttp::://axxv.orrr/
"Data is considered be mentioned if one of the following present the abstract: "dataset". "datasets", "data". "corpus", "corpora".
"We used Huggingface data as of November 13, 2022, as the basis for our selection. It is available at https:// huggingface co/coo/ wttpmm/dddssseeeeeeeeee wwort=downloss
hhttp::/inncettion-ee itteeecct.gtuu..iii.

---

We calculate interrater agreement to measure the annotation coherence of the 14 common annotated publications. For this, we report the average mutual Fl score. To compute this metric, we compare the annotations for each pair of annotators using the Fl score, where one annotator is the ground truth and the other is the prediction, and then reverse their roles. The Fl score is reported for exact and partial matches. Compared to "exact match", the "par- tial match" setting considers partially overlapping spans matches, enabling us better compre- hend the disparities in annotations as partial-match disregards dissimilar annotation boundaries as er- 5,781
# spans
# unique spans
Method
12,826
6,547
DatasetGeneric
9,838
rors. Our exact match and partial match agreement scores for each entity type are presented in Table 3.
MeMrreeGenerie
8,521
4,238
ReferenceLink
7,172
2,257
MLModel
5,012
944
3.5
Corpus Statisties
Task
4,143
1,478
Table lists some statistics of our corpus annota- tions on 100 publications. We report the number of spans per entity type as well as the corresponding unique number of spans throughout the documents. In total, GSAP-NER contains 54,598 annotated spans out of which 23,359 are unique spans.
Dataset
3,898
883
ModelArchitecture
2,612
985
DataSource
508
185
URL
68
61
Total
54,598
23,359
Table Text span statistics in our GSAP-NER dataset ordered by the number of spans per entity type.
Baseline Model
4.1 Problem Definition
tion training before starting to annotate on target publications. We randomly select 14% of the publi- cations for joint annotation by three annotators. The rest of the publications are split and assigned to a single annotator each. The annotators iden- tify mentions according to our tag set definitions, and nested annotations are allowed. For particular linguistic cases, combine the reuse of ACL RD- TEC Guideline 10 and creation of new rules adapt our annotation schema. The linguistic cases in- clude but are not limited to articles, abbreviations, adjective modifiers, conjunctions and prepositions, and plurals. For articles like "a" or "the", anno- tators are instructed not to include them, except for generic mentions. Abbreviations and adjec- tive modifiers, conjunctions and prepositions are generally requested to be annotated following the ACL RD-TEC Guideline. Most plural forms are considered be of generic type, unless it is named entity.
Our goal is identify named entities in the full text scientific publications. We denote the tag set as T, where 4 E T(i = 10) is tag described in Section 3. For each publication D the goal is to gen- erate list of j entity mentions, identified by a tuple mj = (t,, by. ej), where m, represents the mention span, tę the type of the named entity, b, and ej the start and end of each span in D. Note that since nested annotations are allowed, two identified men- tions m, my, # I, can have overlapping spans. This problem definition is flexible enough to use both transformer-based architectures and genera- tive approaches to NER.
4.2 Pre-trained Model Selection
NER approaches currently follow one of two com- peting NLP paradigms (Liu et al., 2023). The state-of-thh-ar models for NER in the scientific domain follow the "pre-train, fine-tune, predict"- paradigm (Jeong and Kim, 2022). This paradigm involves pre-training with out-of-task goals, such as masked token prediction (MTP), in pervised manner and fine-tuning these pre-trained language models (PLMs) on the downstream NER task. In contrast, recent, popular in-context learn- ing based approaches utilize the "pre-train, prompt, predict" "parict"- and generative large language models are proposed for NER. However, Ye et al. (2022) show that they are not yet competitive for domain-specific downstream extraction tasks, such as NER on scholarly documents. Therefore, we present a baseline comparison based on state-of- the-art fine-tuning approaches of PLMs, which are proven to outperform previous approaches (Heddes et al., 2021; Jeong and Kim, 2022) in this domain.
3.4 Interrater Agreement
10 ACL RD-TEC Guideline: https://doi. http///i..o///////////
org/10.1111 13140/RG. 2. 1939. 1999.144.. 1446.

---

To set strong baseline that accompanies our GSAP-NER corpus, we conducted comparative analysis of three baseline models. The initial model was chosen as a benchmark, reflecting a well-established foundation within the field. In par- ticular, SciBERT (Beltagy ct al., 2019) has consis- tently demonstrated good performance in various scholarly NER tasks, rendering it the default model for pre-training in this specific domain. It is ver- sion of BERT (Devlin al., 2019) additionally pre- trained on scholarly documents using Multi-Task Prediction (MTP) objective and has a parameter count of 109 million. Subsequently, we fine-tuned DeBERTa-CS (Jeong and Kim, 2022), a more re- cent iteration of an in-domain pre-trained Language Model. This choice was motivated by the authors' track record of achieving state-of-the-an art results in NER tasks on datasets such as SciERR (Luan al., 2018). Like SciBERT, SciDeBERTa-CS was also pre-trained within the scientific domain, with a more focused emphasis on the Computer Sci- ence (CS) domain. This model employs a config- uration consisting of 125 million parameters. To enable comparative performance evaluation with pre-trained foundation models not specialized for the scientific domain, we fine-tuned two RoBERTa model versions, "Base" and "Large", which com- prise 125 million and 355 million parameters, re- spectively. These models leveraged a dynamic masking strategy and were pre-trained on a larger training corpus (Liu et al., 2019).
We transform the annotated spans into token wise labels based on the BIO tag scheme. As described in Section 3, our annotation guideline allows nested annotations, which presents chal- lenges for out-of-the-box NER models. This is because the task of classifying each token becomes multi-label classification problem rather than a multi-class classification problem. Analyzing on GSAP-NER corpus showed two major patterns of nested annotations. The first involves named en- tities nested in Generies, while the second relates to benchmark entities that are double annotated for both Dataset and Task. To address the first case, we split the entity tag set into two parts: Generic mentions and all other type of mentions. Addition- ally, we simplify the double annotated benchmark entities by converting them into single Dataset span in the training set. This approach enables us to use two separate models for each of the tag set, resolving the nested entity annotation problem An analysis showed that the simplifications need to be done for generating the training data leads to a upper bound of 98.7% FI score when testing the performance of the simplified tag set on the full annotations without any simplification. While fine- tunig PLMs we final fully connected layer on top of the encoding layer and trained predict one label for each token using a cross entropy loss for each of the two models.
5.2 10-fold Cross Validation
The broader perspective of our model is to solve the NER task on whole documents, even if the unit or processing for our model is one paragraph. We consider 10-fold cross validation where folds are created such that all paragraphs from one publica- tion are present in the same fold. In each cross validation round 80% of publications are used for training, 10% for validation and the remaining 10% for testing. For reasons reproducibility, we pub- lish the publications used in each fold as part of the data set.
5 Experimental Setup
5.1 Preprocessing
For our experiments, we choose paragraphs as the processing unit instead of sentences, providing the models with more contextual information. How- ever, the used PDF-to-tex tool introduced some errors for footnotes, figures or tables, during conver- sion to text. Therefore we asked the annotators to mark these errors as corrupt. Consequently, we ex- clude all paragraphs containing corrupt parts from train, validation and test set. The generated para- graphs exhibit an average length of approximately 4.5 sentences. The average token count per para- graph stands at 109.1, with a median of 88 tokens. Notably, only 16 paragraphs exceed the threshold of 512 tokens, as determined by tokenization using the SciBERT tokenizer. This enables their usability across various language models.
5.3 Metries
We evaluate our models with entity-level Fl score, where each entity annotation is identified by the paragraph id, start index, end index, and label. Gold annotations and predictions are represented sets of entity annotations, and the Fll score calcu- lated based on these sets. For comparison, we also employ partial-matee Fl score, which considers a predicted entity span as a match if overlaps with
grobid: https:/ / htt.bb/..ommeemmttt///rridddd

---

exact-match F1
partial-match Fl
SeiDeBERTa-Cr
ROBERTa- Base
ROBERIa Large
SciDeBERTa-CK
RoBERTa- Ta-Bae Base
RoBERTa-Large
SciBERT
SciBERT
MLModel
60.8
70.1
67.1
69.3
63.5
73.0
70.1
71.7
MeMeoeeGGeneric
68.0
70.1
68.7
68.6
74.4
76.5
75.5
75.5
ModcIArchitectuue
30.9
33.9
30.6
30.2
44.7
48.3
45.6
44.9
Method
44.7
47.6
46.0
47.3
60.2
62.5
61.2
62.2
Task
52.1
55.3
52.8
53.7
59.3
60.8
59.5
60.5
Dataset
72.6
81.7
78.0
80.5
77.4
85.5
81.9
84.0
DatasetGeneric
63.3
63.2
63.8
63.8
73.4
73.6
73.5
74.2
DataSource
41.7
51.6
48.6
49.4
48.8
59.9
56.3
57.6
Referencel.ink
95.9
92.3
92.2
90.4
98.0
98.0
97.8
98.0
URL
68.3
50.5
64.9
32.8
85.0
64.1
77.2
85.2
61.9
64.6
63.0
63.5
70.6
73.4
72.0
72.7
Table 5: Fl performance comparison of four fine-tuned pre-trained language models (PI.Ms) for scholarly entity and concept detection. The metrics are calculated using 10-fold cross-validation As performance measurements, we provide results both exact matches and partial matches. The latter considers any overlap between the predicted outcome and the correct annotation match.
a gold annotation of same label. This metric particularly useful for assessing the correct position of entity annotations, even if the beginning and end of the tag span are not precisely predicted. To mea- sure performance for each entity type separately, we only consider gold annotations and predictions for specific label.
performance enhancements relative to the other models our ten-fold cross-validation setup, we conducted paired (-test. The obtained p-values for all comparisons were found to be below the signifi- cance threshold (commonly set at 0.05), indicating statistically significant difference in performance. To characterize the similar performance differences among all models for different labels, we iden- tified two distinguishing criteria for entity types. Firstly, concrete named entities (e.g., MLModel: 70.1% Fl and Dataset: 81.7% F1) exhibit supe- rior performance in contrast to conceptual enti- ties (e.g., Method: 17.6% F1 or ModelArchiteture: 33.9% FI), irrespective of the quantity of train- ing samples. Notably, the Method entity type is the most prevalent, encompassing over 12,500 an- notated text spans (as shown in Table 4). The second criteria employs the presence of structural anchor points to distinguish hetween entity types. For instance, standardized patterns such as cita- tions can be easily identified with supervised approach. The existing weak URL extraction per- formance be attributed to the limited number of annotations and the fragmented nature of URLS
6 Experimental Results
6.1 Baseline Models
Fine-tumed PLMs proved perform well on NER tasks for scholarly document processing. The Fl score performance overview of the four com- pared models (i..., SciBERT, SeiDeBERTa-CS. RoBERTa-Base, and RoBERTa-Large) show gen- eral applicability on the given task. With exact- match Fl score in the range from 61.9 to 64.4 (Ta- ble 5), the performance is comparable with other annotation approaches (Luan al., 2018). The best performing model SeiDeBERTa-CS outper- forms the much bigger RoBERTa model for nearly every entity type. Nonetheless, the performance varies in terms of entity types and models. To us- sess the significance of SciDeBERTa-CS model's

---

generated during the conversion from PDF to text. Inaccuracies in rare URL annotations can signifi- cantly increase the error rate, whereas fragmented URLs pose challenges in accurately detecting the beginning and end of a URL. Another valuable insight arises from the observed variation in per- formance between exact-match and partial-match metrics. The performance gap ranges from +2.9% ML.Moel to + 14.4% Method, emphasizing the significance of distinguishing between concrete and conceptual entity types. Recognizing the cor- rect text spans for conceptual entity types remains a challenge for the models due to the uncertainties among the annotators. Please refer to Table 3 for comparison.
stantial standard deviation in the case of models with a limited number of publications within the training dataset. Conversely, for models trained with more than 24 documents, resulting in a dataset of greater variability, this phenomenon was markedly mitigated. Subsequently, our investi- gation revealed that the fluctuations in the exact- match Fl score demonstrated diminishing trend, stabilizing after the incorporation of more than 40 publications for training.
7 Conclusion
We introduce GSAP-NER, a manually annotated corpus over full-text scholarly publications from the computer science domain, designed for in- formation extraction of ML models and datasets. By distinguishing ML models from methods and datasets from materials, our dataset enables re- searchers and services to gain deeper insights into the specific methods and materials employed computer science research. We utilized our data and fine-tuned three state-of-the-we baseline mod- els. The experiments showed that SciDeBERTa-CS reaches best performance ties types, with an overall F1 score of 0.61 and 0.73 on exact span matches and partial span matches, respectively.
6.2 Train Size Experiment
In comprehensive full-text annotation initiatives, the quantity of annotated full-texts assumes a piv- otal role owing to the substantial cost advantages linked with annotating a smaller subset publi- cations. To explore this facet, we executed an ex- periment aimed at assessing performance metrics across various training set dimensions. For the sake of efficiency, the SciBERT model was used in this experiment. In our 10-fold setup, conducted fine-tuning for ten distinct models within each fold, every iteration, designated proportion of doc- uments from the training dataset was allocated to individual models. The number of training docu- ments varied in each iteration, ranging from 8 to 80, with increments of eight. The findings from this experiment are visually represented in Figure 2.
the majority of enti-
Despite the challenges involved in its creation, we believe GSAP-NER remains a valuable resource for the development, evaluation, and benchmarking of NER models in the computer science domain. It offers researchers and practitioners comprehen- sive and domain-specific dataset, addressing the limitations of existing datasets that often lack spe- cialized entity differentiation. Furthermore, this dataset can contribute to advancing research in ar- eas such as information retrieval, scientific knowl- edge mining, automated literature analysis, and knowledge graph creation.
When assessing the Fl score, we observed sub-
1.0
exact-match fl
0.6
0.4
As future work, we aim to employ a multi-stage model training approach and leverage additional background knowledge to disambiguate syntactic mentions from their semantic context. Such back- ground knowledge could come from ML ontolo- gies and knowledge graphs, such the ORKG CS-KG. We also envision exploring entity relation- ships, co-reference resolution, and entity attributes as future directions to enhance the value of this dataset.
0.0
8
16
24
32
40
48
56
64
72
80
# publications in train set
Figure 2: Increasing overall exact-match F1 perfor- mance of the SciBERT model trained on varying number publications. The train set size in the 10-fold set up ranges from 8-80 publications in every fold. The box- plot illustrates the performance differences across folds.

---

Limitations
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguisties.
Despite our diligent efforts, developing a gold standard dataset for entity extraction using a fine- grained and comprehensive tag set focused on ma- chine learning models and datasets remains non- trivial undertaking, This leads to the following limitations associated with the creation of our cor- pus. First, our work suffers from low interrater agreement on certain entity types, and thus, the model performs poor on those types. For instance, when frequently used models such as "RoBERTa" are mentioned, it can be difficult to determine whether to classify them as ModelArchitecturee or MLModel, depending strongly on the context. Ef- forts to address ambiguous types in the annotation guideline or increased training time of annotators did not solve this issue. Second, the paper selection is conducted within the machine learning domain and does not include infrequent publication types, such as surveys or reproducibility studies. Further- more, the potential applicability of our approach across various research domains remains a topic for future investigation. Finally, during the model training process, we excluded paragraphs that were identified as erroneous by the annotators. It is es- sential to address and resolve the resulting issues before the model can be effectively used in a pro- ductive real-wo wooldworld setting.
Danilo Dessí, Francesco Osborne, Diego Reforgiato Re- cupero, Davide Buscaldi, and Enrico Motta. 2022. Cs-kg: large-scale knowledge graph of research entities and claims in computer science. In The Se- mantic Web-ISWC 2022: 21 International Seman- tic Web Conference, Virtual Event, October 23-27, 2022, Proceedings, pages
Springer.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. P Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jennifer D' Souza and Sören Auer. 2022. Computer sci- ence named entity recognition in the open research knowledge graph. In From Corn-Physicall to Born- Virtual: Augmenting Intelligence in Digital Libraries: 24th International Conference on Asian Digital Li- braries, ICADI. 2022, Hanoi, Vietnam, November 30 December 2, 2022, Proceedings, page 35-45, Berlin, Heidelberg. Springer-Verlag.
Michael Färber, Alexander Albers, and Felix Schüber. 2021. Identifying used methods and datasets in scien- tific publications. In Proceedings of the Workshop on Scientific Document Understanding: co-located with 35th AAAI Conference Artificial Inteligence (AAAI 2021): : Remote, February 9, 2021. Ed.: A. P. B. Vey- seh, volume 2831 of CEUR Workshop Proceedings. RWTH Aachen.
Ethics Statement
The authors foresee ethical concerns with the work presented in this paper.
Acknowledgements
We thank the anonymous reviewers for their con- structive feedback and intense rebuttal phase. This work has been partially funded by the Deutsche Forschungssemminnccaatt (DFG, German Research Foundation) part of the Projects BERD@ NFDI (grant number 460037581). NEDI4DS (grant num- ber 460234259), well as Unknown Data (grant number 460676019).
Jenny Rose Finkel and Christopher D. Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods Natu- ral Language Processing, pages 141-150, Singapore. Association for Computational Linguisties.
Jenny Heddes, Pim Meerdink. Miguel Pieters, and Maarten Marx. 2021. The automatic detection of dataset names in scientific articles. Data, 6(8).
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A spe- cialized corpus for scientific literature entity tagging of tasks datasets and metries. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 707-714, Online. Association for Computa- tional Linguistics.
References
Kritika Agrawal, Aakash Mittal, and Vikram Pudi. 2019, Scalable, semi-supervised extraction of structured in- formation from scientific literature. In Proceedings of the Workshop on Extracting Structured Knowledge from Scientific Publications, pages 11-20, Minneapo- lis, Minnesota. Association for Computational Lin- guistics.
Sarthak Jain, Madeleine van Zuylen, Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. SciREX; A challenge

---

dataset for document-leve. information extraction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7506- 7516, Online. Association for Computational Lin- guistics.
Ishani Mondal, Yufang Hou, and Charles Jochim. 2021. End-to-end construction of NLP knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1885-1895, Online. Association for Computational Linguistics.
Yuna Jeong and Runhui Kim. 2022. Scideherta: Learn- ing deberta for science technology documents and fine-tuning information extraction tasks. IEEE Ac- cess, 10:60805-60813.
Zara Nasar, Syed Waqar Jaffry, and Muhammad Kam- ran Malik. 2018. Information extraction from scientific articles: survey. Scientometrics. 117(3):131-11990.
Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Tay- lor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis. 2023. An empirical study of pre-trained model reuse in the hugging face deep learning model registry. In Proceedings of the 45th International Conference on Software Engineering, ICSE '23, page 2463-2475. IEEE Press.
Behrang QasemiZadeh and Anne-Kathrin Schumann. 2016. The ACL RD-TEC 2.0: A language resource for evaluating term extraction and entity recognition methods. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC*16), pages 1862-1868, Portorož. Slovenia. European Language Resources Association (ELRA).
Hongbin Ye, Ningyu Zhang, Hui Chen, and Huajun Chen. 2022. Generative knowledge graph construc- tion: A review. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1-17, Abu Dhabi, United Arab Emirates. Association Computational Linguistics.
Salomon Kabongo, Jennifer D'Suzza and Sören Auer. 2021. Automated mining of leaderboards for pirical ai research. Towards Open and Trustwor- thy Digital Societies; 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Vir- taal Event, December 1-3, 2021, Proceedings, page 453-470, Berlin, Heidelberg. Springer-Vertee Verlag.
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebas- tian Ruder, Sebastian Riedel, Ross Taylor, and Robert Stojnic. 2020. AxCell: Automatic extraction of re- sults from machine learning papers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8580- 8594, Online. Association for Computational Lin- guistics.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pa- pers). pages 861-871, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Sure, 55(9).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Yi Luan, Luheng He, Mari Ostendorf, and Hannanch Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.


Reviews:
[{"review_id": "UXtj6Exhzw", "reviewer": "Reviewer_AUcQ", "summary": "", "questions": "1. Why are reference links and URLs annotated?\n2. Did you try experimenting with RoBERTa-Base? Ideally it would help to compare performance across models of the same size if the goal is to check that performance is better when pretraining domain is closer to dataset domain. \n3. Why is the training set size experiment conducted on SciBERT? If SciBERT and SciDeBERTa-CS are the same size, it might be better to take the latter model for these experiments since it has better performance overall.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank  you for your ef fort, your time and your detailed, and constructive comments. We have gone through all the comments in detail and discussed all points among all co-authors. Below you will find our rebuttal for the major issues raised by you. \nOur rebuttal is structured as followed. First, we comment on the major points from the “Reasons to Reject” section of your review.  At the end we answer the question of the “Questions to the authors” section.\n\n## Schema Development (Reasons to Reject 1-5):\n1.\tRE your comment on „the distinction between ML model and ML architecture”\nWe see a model as an executable piece of software that can be applied on a dataset . For instance, the mention \"BERT-base\" is a typical MLModel.\nOn the other hand, there are mentions of models that combine architectural choices, like \"CNN-LSTM\", which we consider as MLModelGeneric with nested ModelArchitecture mentions. We will clarify this in the paper and the guideline. \n2.\tRE your comment on “ML method category is extremely unclear” \nThank you for the suggestion. We agree that \"Method\" is a very broad category, which does not distinguish between ML methods from other methods. We have done this in accordance with other annotation guidelines (SciIE used for SciERC), which defines \"Method\" as a broader category to annotate methodological statements. Introducing a separate entity class \"ML Method\" could capture an even more fine-grained picture of methodological mentions, which we could introduce in a future version of GSAP-NER . We will improve the description of the entity type \"Method\" in the annotation guideline and add more examples.\n3.\tRE your comment on the “justification for finer-grained dataset typing”, esp. Datasource vs. Dataset\nWe agree with the example you gave: KBs are also considered as Datasources.\nThe key distinction (as stated in our guideline) is that Datasource has unstable reproducibility. We will clarify the Datasource definition in the paper and illustrate it with further examples (KG, dynamic Benchm arks). We will also add further examples to the guideline.\n4.\tRE you comment on the missing entity type \"Metric\". \nThank you point this out. We are aware of the missing category metric. For this project Metrics and Measures were not in our focus. We will add the entity type Measurements in future, as Leaderboard extraction is part of our future plans.\n5.\tRE your question whether task entities are considered model-related?\nThank you for your comment. We agree that a task can be related to a model as well as to a dataset. For simplification, we selected the model related category, but we will clarify the connection to a dataset in the entity type description.\n\n### RE your comment on the paper selection method. \nThank you for your suggestion. We paid a lot attention to the paper selection method in the beginning of the project and deliberately made it as described, in particular, to focus on two objectives: popularity (first part of the method) and diversity (second part of the method). For these reasons, we did not choose to re-annotate papers from the SciREX corpus. Another reason is that the authors or SciREX do not clearly define their final selection method in the paper, unfortunately. Furthermore, we think that SciREX is also biased towards publications from PWC. \nHowever, your assumption, that our selection criteria of “popularity” will lead towards having only modeling papers, does not hold. The focus of the popular publications are new models, datasets or focus on specific tasks.\nWe will give a short overview of the selected paper including citation count in the Appendix.  \nWe believe that our paper selection method broadens the set of annotated publications available to researchers in the research domain. Apart from that, the authors of SciREX  do not clearly define their final selection method in the paper, unfortunately.  \n\n### RE your comment on the missing error analysis of annotators disagreement: Hard Cases\nThanks for this suggestion. We have executed an analysis during the annotation process. We will add insights on particular hard cases: i.e., MLModel vs. Modelarchitecture in the final version of the paper.\n\n## Questions for the authors:\n1.\tRE your question “Why are reference links and URLs annotated?“\nFuture method development could benefit from those annotations. E.g. Disambiguation or better models to detect MLModel or Dataset entities using thiattributive information.\n2.\tRE your question “Why did you not compared to RoBERTa-BASE?”\nYou are right, that we did not argue well on the selection of RoBERTa model for comparison. The authors of SciDeBERTa have allready shown the performance gain of using non in domain continual learning of PLMs before finetuning on downsstream tasks. With comparison to RoBERTa-Large we can show, that the finetuned SciDeBERTa-CS model  even perform better than the 3 times bigger RoBERTa model.\n3.\tRE your question “Why not using SciDeBERTa-CS for size experiment?” \nWe take the smallest model SciBERT (109M vs 139M Parameters) for this experiment because of the high number of iterations to perform on the 10-fold setup with 10 different trainset sizes (100 Models to train)\n\nThank you for your review. We look forward to incorporating all your valuable hints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed dataset, GSAP-NER, could be an interesting and challenging new benchmark for the scholarly information extraction community \n2. The authors benchmark the performance of a range of general and science-specific BERT-based language models on their dataset and also evaluate performance on the task under limited data settings, highlighting potential directions for future work on this task\n", "reasons_to_reject": "One of the key contributions is the development and annotation of a finer-grained set of entity types, however some of the choices are unclear and lack justification. Specifically:\n1. It is a little unclear how the distinction between ML model and ML architecture was defined. Based on the examples of GPT, BERT, etc. being considered models, it seems like this distinction is dependent on atomicity, i.e. any entity that involves a combination of multiple architectural choices is considered a model. But it would help to have this notion clarified in the paper as well as the annotation guidelines. \n2. The scope of the ML method category is extremely unclear, which could potentially explain the low agreement on this category. Additionally, it’s not clear why we cannot consider method to be the parent category for all model-related spans, and rename this category to miscellaneous? The distinction between model/architecture and method seems a little artificial.\n3. The justification for finer-grained dataset typing (i.e. types for resources like KBs, etc. being unclear) is reasonable, but this problem seems to be present even with the new categorization. Are knowledge bases considered to be under datasource since they’re not static? What would dynamic benchmarks or pre-training corpora (C4/Pile) be classified under?\n4. Some prior work on entity extraction from CS papers also identify metrics as entities (as mentioned in the paper), however this dataset does not consider that category - what motivated this decision?\n5. Why are task entities considered model-related? It seems like including them under dataset-related would be more suitable (considering that later, the same span being labeled as task and dataset is a common observation)\n\nIn addition to schema development, the process by which full-text papers are selected for annotation also seems to have some flaws:\n1. Instead of sampling a new set of papers, it might have helped to re-annotate a subset of papers from an existing dataset like SciREX.\n2. Currently, popular papers are chosen by looking at the most downloaded models on HuggingFace and selecting papers associated with them. However, that biases the sample towards only modeling papers and disregards papers focused on dataset creation, evaluation testbeds, survey papers and reproducibility studies, all of which could contain information about ML methods and datasets. Instead of this metric, popular papers could have been selected based on the highest number of citations (limited to the past X years for recency, if that was another criterion).\n\nLastly, the paper has very little discussion of what kinds of cases were ambiguous and hard for annotators to agree on. It also does not discuss interesting phenomena in more detail (e.g., spans that models consistently found hard to annotate), which could spark more future work\n", "questions_to_authors": "1. Why are reference links and URLs annotated?\n2. Did you try experimenting with RoBERTa-Base? Ideally it would help to compare performance across models of the same size if the goal is to check that performance is better when pretraining domain is closer to dataset domain. \n3. Why is the training set size experiment conducted on SciBERT? If SciBERT and SciDeBERTa-CS are the same size, it might be better to take the latter model for these experiments since it has better performance overall.\n", "paper_topic_and_main_contributions": "This work focuses on developing a new dataset for the task of identifying machine learning datasets and models in scholarly literature. Unlike prior datasets, they adopt a more fine-grained set of entity types and annotate a corpus of 100 full-text scientific publications called GSAP-NER. Additionally, they also consider informal mentions (such as “our BERT-based model”) as entities, including type annotations for them and allow nested NER annotations. They benchmark the performance of several pretrained models, including both general and science-specific models, on their dataset. Their experiments reveal that while models achieve reasonable performance on this task, some entity types still show a huge performance gap. Finally, they run additional experiments testing model performance on the same task under various low-data regimes to better understand how much data annotation is needed to develop strong models for this purpose. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It might be helpful to include more motivation for why fine-grained typing is useful (e.g., can help more in constructing KGs/leaderboards) to further establish the utility of your dataset in the introduction.\n2. existing guidelines and ground truth datasets for scholarly entities have not adequately addressed the finer-grained entity types, such as ML Models and Datasets as distinct entities. -> not sure this is the right comparison since some datasets do label both separately? The models vs architectures example seems more reasonable.\n3. It’s unclear why informal mentions are called non-descriptive, when that doesn’t necessarily seem to be the case?\n4. “Stacked” annotations seem to be referring to nested annotations: e.g., https://nlp.stanford.edu/pubs/nested-ner.pdf \n5. Line 88: entity types\n6. Line 90: sizable\n7. Line 141: contains\n8. Line 146: tuples\n9. Line 200-201: depending on context\n10. Line 208-210: often contain mentions of other named entities (you can repeat the CNN example here)\n11. Line 255: refer to\n12. are ready at dawn, even if comparisons show the changing of the guard is still in progress -> this statement is a bit unclear\n13. Line 411: aspect is. It might be better to rephrase this to say that you’re exploring different models with more and more continual pretraining in your domain of interest.\n14. Line 445: extra space between . and ,\n15. Line 471-472: Folds are created such that all paragraphs from one publication are present in the same fold\n"}}, {"review_id": "HHmBGhnxLo", "reviewer": "Reviewer_RRze", "summary": "", "questions": "A. Can you please clarify the input length part for the experiments? You are using paragraphs as units, right?\nB. How do you plan to handle corrupt paragraphs in a real-life deployment? Or for that matter, any corruption that comes from pdf to text conversion?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank  you for your effort, your time and your detailed, and constructive comments. We have gone through all the comments in detail and discussed all points among all co-authors. Below you will find our rebuttal for the major issues raised by you. \nOur rebuttal is structured as followed. First, we comment on the major points from the “Reasons to Reject” section of your review.  At the end we answer the question of the “Questions to the authors” section. The enumeration refers to the enumeration of the comments/questions brought up by you.\n## 1. RE your comment on the low contribution of the paper due to that we focus on ML models and datasets only.\nIndeed, with PwC we saw that crowdsourcing  can work well. But, apart from the dataset itself, one of the key contributions of our paper is the annotation guideline and the data model, which both have a detailed granularity than related works. Both are required for a crowdsourcing task for an outcome of certain quality. However, our annotation experiences show that annotation at that level of granularity is hardly crowdsourcable. Moreover, for testing crowdsourced data one will need a very reliable dataset too.\n## 2. RE your comment on prior works such as SciREX that uses PWC as weak supervision and why is that not used in our paper\n\nWhile manual annotation is not scalable it produces a work of high quality. In detail, the weak supervision approach, as was used to generate the SciREX dataset, does not convince us. They pre-annotated the full texts with a model trained on a different dataset (SciERC) that has annotations on abstracts only. The reported number of changed annotations  are not aligned with the state-of-the-art model performance on this dataset (f1: 0.71). The used model for pre-annotating  is introducing a bias which is not discussed or handled appropriately.\nSo we decided to introduce a new dataset to the community, that is manually annotated from scratch on full text. This could be useful for future usage in annotation approaches like SciREX, incorporating a mixture of weak supervision, automatic pre-annotation and final curation by manual annotators, which focus on scalability.\n## 3. RE your comment on methodological issues in our work:\n### a) Stacked vs. Nested annotation\nThank you for your hint, we will change terminology throughout the whole paper.\n### b) Input length of paragraphs longer than 512 token (Compare Qustion A)\nIn fact we cut at 512 tokens, but the paragraph lengths is shorter than you expect. A Paragraph is defined simply by the output of grobid text extraction. Statistics about the paragraph length (number of tokens using scibert tokenizer as an example) show, that the resulting paragraphs contain ~4.5 Sentences in mean. The number of tokens per paragraph are: in mean: 109.1, median 88. Only 16! Paragraphs are longer than 512token (scibert tokenizer) which results in 25 out of 54,598 annotated entities, which are not able to be annotated when cut after 512 tokens. This is negligible.\n### c) 10-fold has no standard train/dev/test split for future comparison\nBecause of that reason we share our data in a fixed 10-fold setup. (Compare data in download link). We will point this out in the publication itself.\n### d) Temporal aspects and generalization on unseen entities.\nThis is a good hint. Because of the 10-fold setup we do not see a good solution to face that concern in general. But a separate analysis of the generalization over unseen entities is a valid point for future work.\n## Question from the authors:\nA.\tRE your question  about the input length … \nB.\tRE your question on how to handle corrupt paragraphs in real-life deployment?\nFor this publication we have no final solution for this issue. It is a valid point to discuss in the Limitations. For future real-life deployment we plan to analyse the impact of annotating corrupt paragraphs and develop automatic detection mechanisms. \n### Comments on missing references:\nThank you for the hints. We will add the two references into the paper.\n\n\nThank you for your Review. We are looking forward to incorporating all your valuable hints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a focused contribution that is well explained. I do think the dataset is going to be interesting for future research. The annotation scheme is well thought and a small-scale multi-annotator experiment is reported. The agreement between the annotators shows the validity of the annotation scheme. Two things I like in particular: a) there is a full-text annotation as opposed to just the abstract; b) the paper selection policy is better than random. Also, I do appreciate the fact that in the experiments, the authors use paragraphs as units but cross-validate over the documents to ensure document-wide generalizability of the models.", "reasons_to_reject": "I believe these are the key weaknesses of the paper:\n1. I do have some issues with the applicability of the work. Scholarly entity extraction is indeed an important problem, but if the entities are just dataset names and model names, the contribution becomes too focused. Moreover, we have previously seen that crowdsourcing is an excellent alternative for this problem (paperswithcode).\n2. The authors mention that manual annotation is not a very scalable process here (L90-L91). I completely agree. But there isn't any remedy prescribed for that, except an experiment in section 6.2 for training size. Prior work such as SciREX uses PWC as weak supervision, why is that not used here? Without any serious attempt at making the process scalable, there is no extensibility of this work, which severely restricts its utility.\n3. Some methodological problems:\na. What authors consider as stacked annotations is a well-known problem called nested named entity recognition (https://aclanthology.org/D09-1015.pdf ) and multi-label classification is not necessary (L449).\nb. The baseline models are BERT based, which handles at most 512 tokens, and I am surprised that paragraphs (L435) can be used as the input unit. It is either a) the paragraphs are uncharacteristically small, or b) they are truncated/broken somewhere. \nc. While the cross-validation setup helps to test for inter-document generalization, there is probably not going to be a standard train/dev/test split: this will make it very hard for future work to compare against the proposed methods.\nd. Scientific entities are emerging, so the test setup needs to capture a temporal aspect: i..e, the models should be tested with documents that are published later than the training ones (hence possibly have some unseen entities).", "questions_to_authors": "A. Can you please clarify the input length part for the experiments? You are using paragraphs as units, right?\nB. How do you plan to handle corrupt paragraphs in a real-life deployment? Or for that matter, any corruption that comes from pdf to text conversion?", "paper_topic_and_main_contributions": "The paper a) creates a new dataset by annotating ML papers with machine learning models and dataset names and b) tests some baseline models on the annotated dataset. The main contribution is a new data resource.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think it's mostly okay, noticed an extra space at L445."}, "extra_text": {"Missing_References": "For the stacked annotation part, I would say:\n1. Finkel, J. R., & Manning, C. D. (2009, August). Nested named entity recognition. In Proceedings of the 2009 conference on empirical methods in natural language processing (pp. 141-150).\n\nAlso, possibly related to the general theme of the paper:\n\n2. Kardas, M., Czapla, P., Stenetorp, P., Ruder, S., Riedel, S., Taylor, R., & Stojnic, R. (2020). Axcell: Automatic extraction of results from machine learning papers. arXiv preprint arXiv:2004.14356."}}, {"review_id": "t5HyVDYW4k", "reviewer": "Reviewer_Z9Pn", "summary": "", "questions": "Question A: What is the data card for this dataset?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and your excellent suggestion. We agree that this dataset should have a data card too. We did not add it, as we do not have so much experience with writing data cards so far. We will create the data card for this dataset in case of acceptance.\n\n### RE your question why we did not introduce a sample analysis illustrating how this dataset can be used?\nWe plan an in-depth study applying our model on a bigger corpus of publications. This study is planned to cover 100k papers from various domains. This paper is just about the resource, i.e., the ground truth dataset itself, which we think is very valuable for the community, for example, as a second source of truth (next to other ground truth datasets like SciREX, e.g.). \nThank you for your Review. We are looking forward to incorporating your valuable hints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper is well-written, and the data and annotation process/reasoning are well-explained. I can imagine many positive use cases for the dataset. For example, researchers could measure the popularity of methods in specific domains, indicating which domains certain approaches work best on. Moreover, the strong baselines the paper introduces will be useful for future researchers.", "reasons_to_reject": "The paper is missing a complete \"Data Card\" for the new corpus [1]. This is vital for new datasets. When should the database used? What use cases should it not be used? What is the distribution with regard to the \"perceived\" author gender in the 100 papers? All this information is important.\n\n(Minor) It would have been useful if a simple analysis was completed of how the dataset could be used for unique analysis. I think this would have been more useful than the model comparison. Overall, I think this dataset will have a large impact on practical analysis research. From a technical perspective, I don't think this dataset will add much to the existing NER datasets in terms of novel methodological development, unless the reasoning is missing (e.g., harder overlapping entites, non-contiguous entites, or something else).\n\n[1] Pushkarna, Mahima, Andrew Zaldivar, and Oddur Kjartansson. \"Data cards: Purposeful and transparent dataset documentation for responsible ai.\" Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022.\n\n", "questions_to_authors": "Question A: What is the data card for this dataset?", "paper_topic_and_main_contributions": "This paper introduces a new dataset GSAP-NER covering unique machine learning-related entities such as Method (e.g., \"word embedding\"), MLModel (e.g., \"XLM-R\"), DatasetGeneric (e.g., \"cross-lingual benchmarks\"), Dataset (e.g., \"MLQA\"), and more. The dataset contains 54,598 entity mentions across 100 publications. Three annotators labeled the dataset 14% with good agreement, and the rest of the dataset was labeled independently. Finally, the paper introduced several baselines on the data that can be used by researchers looking to analyze mentions, or the baselines can be improved on by researchers looking to develop new methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jubNo4Izbb", "reviewer": "Reviewer_CSpT", "summary": "", "questions": "- Can you comment on models exhibiting super-human performance? I.e., they beat inter-annotator agreement results.\n- Can you provide a clear-cut definition of MLModel vs ModelArchitecture, which unambiguously allows annotators to differentiate them? (I found the annotation guidelines only partly satisfying in this respect)\n- Can you please explain the model architecture, i.e., what you put on top of the BERT models, to finally output the NER spans?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your effort, your time, and your detailed, and constructive comments. \nEspecially your detailed line by line comments are very helpful.\nWe have gone through all the comments in detail and discussed all points among all co-authors. Below you will find our rebuttal for the issues raised by you. \nWe comment on all your concerns on our paper based on your “Reasons to Reject” section. Doing so, we face all three questions in your “Questions to the authors” section. We point to the related questions in our comments.\n\n## Resonse to reported key weaknesses:\n### 1. Motivation - Missing novelty compared   to existing work, e.g. SciREX\nWe decide to go for a new annotation corpus for full-text publications, to consider two major shortcomings of existing corpora like SciREX. (a) The lack of expressiveness towards named entity mentions of machine learning models and datasets. Our finegrained entities allow the possibility to detect mentions of MLModels and, for example, be able to capture relations between MLModel and Datasets (futurework). And (b) the shortcomings of an annotation pipeline using automated pre-annotated model suggestions and correct them by the annotator. We wanted to bring a novel source to the community.\nThe fact that human annotators have issues on specific entity types is a sign, that the task is not easy, which leads to further research questions. But it should not lead to cancelation of further research on this topic. The comparable good performance on MLModel adds value to the research community on its own and enables better research for example on the interplay of MLModels and Datasets. \n### 2. Missing Explanations of NER Models (Q3: Model Architecture not explained)\nWe transform the annotated spans into token wise labels based on the BIO tag scheme. As in ll 461 described we split up the tag set and trained two expert models for each of the tag sets. We used a final fully connected layer on top of the encoding layer and trained to predict one label for each token using a cross entropy loss. To be able to reproduce our results we shared our code (link in abstract).\nWe will incorporate a model architecture description of our NER approach into the final version of the paper.\n### 3. Definition: MLModel vs. ModelArchitecture  (Q2: Clear Definition)\nWe see a model as an executable piece of software that can be applied on a dataset . For instance, the mention \"BERT-base\" is a typical MLModel. On the other hand, ModelArchitecture is used for introduced terms or names that designate the architecture of a model. For architectures mentioned by name, there must additionally be a clear indication in the context that the concept of the architecture and its properties are explained and not a concrete instance of the model architecture.\n### 4. No explanations or comparison of low inter-annotator agreements\nThis is a good point to contextualize our inter-annotator agreements and further discuss the resulting limitations  which will be incorporated in the final submission.\n### 5. Super human performance (Q1: super-human performance)\nFirst of all we need to differentiate the two values you are comparing. The first is based on annotations of our three annotators of 14% of the publications. The underlying data is different to the model scores calculated in the 10-fold setup.\nAlso the metric is based on the mean mutual f1 scores between the annotations. Which have potential side effects like a huge influence of annotator outliers with huge differences in the annotations on the f1 means of specific entities. \nAnyway, to better be able to compare those numbers to explain this apparent super human performance we compared the model performance on the 14 commonly annotated publications on all three annotators annotations. We found that the models tends to find a common ground of the annotations for low entity types with low interrater agreement.  \n### 6. Missing report of statistical significance\nThis is a valid point for comparing the different finetuned PLMs. Because of our 10-fold setup we decided not to have multiple training runs because of the immense number of models to train. For the final submission we will report and discuss the standard deviation of the performance between the folds. Even if this has the disadvantage that they are not directly comparable to performance standard deviations based on multiple runs on the same training data.\n### 7. Limitations of dataset and baseline models\nWe will incorporate a limitation section in the final version of the paper at least facing:\n * the shortcomings of the used fine-grained schema\n * limitations of the model concerning specific entity types\n\n## Question to the authors:\nWe have commented on all your Question above:\n * Question 1: Point 5\n * Question 2: Point 3\n * Question 3: Point 2\n\nWe have perceived your extensive comments which enable us to greatly improve the quality of our paper. We will go through them very conscientiously and incorporate your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New dataset, with slightly finer granularity entity types compared to the most related (but 3x larger) SciREX dataset", "reasons_to_reject": "- Motivation for the new dataset -- whose unique feature seems to be that it is specifically targeted at computer science, and particularly machine learning -- is only partly convincing; why would SciREX's 'method' not suffice in practice? Especially since human annotators seem to have difficulty agreeing on correctly labeling MLModel vs ModelArchitecture instances?\n- The NER models are not explained in sufficient detail (and basically I'd be at loss when I'd try and reproduce the author's work based on the paper alone); the only thing that is clear is that they use BERT-based models; but how the BERT token representations eventually lead to (potentially overlapping) span annotations is not explained (not even with a reference to a paper that provides the model architecture)\n- The definitions of the newly defined types, esp. MLModel vs ModelArchitecture are not clearly articulating how exactly they differ from one another (I also did not find a very unambiguous definition in the annotation guidelines); apparently, annotators face the same difficulty, given the low agreement\n- The low inter-annotator agreement (on the newly defined entity types) is just stated as is, without any insight into whether related datasets (as referenced in the paper) have clearly higher/lower/comparable agreements.\n- The fact that the baseline models outperform inter-annotator agreement results is not commented on (I'd expect these agreements to be regarded as upper bounds that a system could reach; wouldn't significantly higher model scores suggest over-fitting of some kind?)\n- No numerical or qualitative information is stated on (statistical) significance of reported performance differences\n- No clear discussion of the work's (potential) limitations (not for the dataset, nor for the baseline models)", "questions_to_authors": "- Can you comment on models exhibiting super-human performance? I.e., they beat inter-annotator agreement results.\n- Can you provide a clear-cut definition of MLModel vs ModelArchitecture, which unambiguously allows annotators to differentiate them? (I found the annotation guidelines only partly satisfying in this respect)\n- Can you please explain the model architecture, i.e., what you put on top of the BERT models, to finally output the NER spans?", "paper_topic_and_main_contributions": "The paper presents a newly constructed dataset for (named) entity recognition task specifically from scientific literature (focused on extracting models, dataset and coreferenced mentions thereof). This new dataset comprises 100 papers, that jointly comprise ~55k mentions, in ~25k sentences. They define their own entity types (adding new types 'Datasource' and distinguishing 'MLModel' and 'MLModelArchitecture', which were not represented in earlier similar efforts in literature).\nThe authors also provide baseline models for the NER task, based on existing pre-trained language models (both general purpose, Roberta, and specifically for scientific literature, SciBERT, and even more tailored to computer science, SciDeBERTa-CS).\nThe dataset is presented in terms of briefly stating the entity types and indicating human inter-annotator (dis)agreement.\nBaseline model results on the defined NER task are presented.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- See my 'Questions for the authors'; such clarifications should be incorporated in the paper itself as well\n- Please use curly quotes; and where you already do, please use the correct style for opening vs. closing ones (e.g., in the abstract)\n- Please use a comma after \"i.e.\" and \"e.g.\"  (this is customary in US English, AFAIK)\n- Please always put footnote superscript markers *after* punctuation (e.g., line 298)\n- Table 1: please use the exact same terminology as where you define the precise entity types (\"ML Arch.\" vs \"ModelArchitecture\")\n- §1:\n  + 'stacked annotations ...' -> maybe rephrase as 'mentions comprising overlapping spans' (this would be more clear to the general readership?)\n  + 'very expensive': please quantify this, if possible (how many hours are we talking about? whether that's 'expensive' may be subjective)\n  + l 110: tailored to around -> tailored to\n  + Why are section references typeset in bold?\n  + Fig 1 caption: rather than cite text parts in bold, just put them between quotes.\n- §2:\n  + footnote 1: may also refer -> also refers\n  + 143: NLP-related tasks -> related NLP tasks\n  + 159: more.. -> more.\n  + it seems you sometimes typeset numbers in math mode, sometimes not. I'd simply never do it in plain text\n- §3\n  + 205: architecture depending -> architecture, depending   [add comma]\n  + Table 2 caption: instead of typesetting \"Pink spans\" in pink, and later mention what it means (MLModel), just write the meaning formatted as in the table (i.e., a pink box around \"MLModel\" in this case); same for other color labels; would be even better to have them as a legend in / next to the table, rather than in the caption\n  + ModelGeneric: \"informal or anonymous\": this is not clear; it seems you mean non-named (or under-specified) entity mentions that (co)refer to a specific named entity (ie. a MLModel, or ModelArchitecture?); I'd rather use 'ModelReference' as type name\n  + Method:  also is not very clear; seems weird to combine (non-ML) models as well as their outputs (e.g., the final layer's token embeddings produced by BERT would also be a \"method\")?\n  + Datasource:  \"unstable or not static\": not 100% clear what this means; what about Wikipedia dumps? The timestamped dump is static, right? But it would still be a web source, so 'datasource' and not 'dataset'?\n  + DatasetGeneric: same comment as for ModelGeneric\n  + On the 'popularity' selection: wouldn't this introduce a bias towards 'easy' (because frequently referenced, i.e., a model could basically memorize the names) instances? \n  + On the paper selection strategy:  is the test set also stratified, i.e., the \"popular\" and \"diverse\" paper proportions therein are also 50/50?\n  + 319: token -> tokens\n  + 329: why exactly 14%? Seems like a random number\n  + 338: not exhaustively -> are not limited to\n  + 341: ... them. However, articles should be considered for generic mentions only. -> ... them, except for generic mentions.\n  + 344: asked to annotate -> requested to be annotated\n  + 358-61: The difference between exact match and partial match is, that partial match also consider the spans that are partial overlapping, which can evaluate more effectively on stacked annotations. --> Compared to ``exact match'', the ``partial match'' setting also considers partially overlapping spans as a match, which allows to more effectively evaluate stacked annotations.\n  + 361: how do you define \"more effectively\"? \n  + 362: And in -> In\n  + 367: report on the -> report the\n- §4\n  + 382: are allowed to be -> can be\n  + 385: \"encourage\": what do you mean exactly? How can a problem definition 'encourage' a model?\n  + 399: \"ready at dawn\":  I don't know what you are trying to say\n  + 400:  \"the changing of the guard\":  what do you mean exactly?\n  + 402: tasks like -> tasks such as\n  + 411: \"three different aspects\":  I'm not sure what you mean with 'aspects'; it seems that you refer to 3 different models (each with their own distinct characteristics), which you have chosen for a particular reason (i.e., there's an associated research question) --> rephrase: state the 3 research questions (or criteria for your model choices) + then list the corresponding models\n  + 425: representing the aspect to compare a state-of-the-art model -> representing a state-of-the-art model [?]\n- §5\n  + 445: start of sentence seems to be missing?\n  + 450: analysing -> analyzing  [in US English at least]\n  + 462: show, that -> showed that\n  + 464: S1 -> F1\n  + 475: rest 10% -> remaining 10%\n- §6\n  + 501: \"comparable with\": please be specific; how much % does the difference amount to?  ('comparable' is subjective)\n  + 513: sheer number -> number\n  + 520 etc.: the explanation for low URL performance suggests it is a dataset issue (i.e., a result of poor PDF to text conversion)? A priori, I'd expect URLs to be very easy to extract, e.g., even just regex based. If it's a dataset issue, shouldn't we focus on correcting the data, rather than trying to have the model deal with it?\n  + Table 5: Please report stdev (or some other indication that allows to assess the observed variation) over the 10 folds; this will also allow to comment on statistical significance of performance differences observed across different models.\n  + Fig.2: What datapoints are the box plots based on? Are these different subsets? Are all boxplots taken over the same number of datasets?\n  + 533: \"the difference\": unclear what difference exactly you're referring to here\n  + 551: \"a reasonable standard deviation\": what is 'reasonable' exactly? Are you talking about the left bars in Fig.2? Or previous results (which did not include any stdev information)?\n- §7\n  + 572: the stated F1 values exceed those for inter-annotator agreement. Does that make sense?\n  + 587: \"multi-stage\": please indicate what these 'stages' would be. Are you talking about a pipeline, e.g., first identifying spans, then classifying their types? Or rather also extract 'relations'? What would these be?\n  + 589: \"background knowledge\":  this is very broad/vague; please be more specific"}}]

Author rebuttal:

Meta-review: Reviewers were enthusiastic about the potential of a new NER resource for scholarly information extraction and appreciated the thoughtful approach to designing fine-grained entity classes for machine learning. Several reviewers questioned the schema of NER tags chosen by the authors, particularly raising questions of distinctions between classes such as ML model and ML architecture or Dataset and Datasource, which may also impact annotator agreement. Reviewers also questioned the efficacy of having a small, human-annotated corpus versus other works that have larger, crowdsourced approaches. Reviewers offered many suggestions for improving the writing by correcting terminology, typographical errors, and improving the clarity of ideas. Ultimately, the limitations of the dataset flagged by the reviewers may limit the impact and adoption of this resource by the community.
Decision: Accept-Findings