S2abEL: A Dataset for Entity Linking from Scientific Tables
Authors: Yuze Lou
Bailey Kuehl
Erin Bransom
Sergey Feldman
Aakanksha Naik
Doug Downey
Venue: EMNLP 2023 Main (2023)
Keywords: Table Entity Linking
Machine Learning
Dataset

Abstract: Entity linking (EL) is the task of linking a textual mention to its corresponding entry in a knowledge base, and is critical for many knowledge-intensive NLP applications.  When applied to tables in scientific papers, EL is a step toward large-scale scientific knowledge bases that could enable advanced scientific question answering and analytics.  We present the first dataset for EL in scientific tables.  EL for scientific tables is especially challenging because scientific knowledge bases can be very incomplete, and disambiguating table mentions typically requires understanding the paper's text in addition to the table. Our dataset, Scientific Table Entity Linking (S2abEL), focuses on EL in machine learning results tables and includes hand-labeled cell types, attributed sources, and entity links from the PaperswithCode taxonomy for 8,429 cells from 732 tables. We introduce a neural baseline method designed for EL on scientific tables containing many out-of-knowledge-base mentions, and show that it significantly outperforms a state-of-the-art generic table EL method. The best baselines fall below human performance, and our analysis highlights avenues for improvement.

S2abEL: A Dataset for Entity Linking from Scientific Tables
Yuze Lou't Bailey Kuehl?, Erin Bransom'
Sergey Feldman', Aakanksha Naik?, Doug Downey² Uniivserty of Michigan, 2 Allen Institute for yuzelou unuch.edu,
@allenai.org
Abstract
from free-form natural language (Gu al., 2021; De Cao et al., 2021; Li et al., 2020; Yamada et al., 2022). Some consider tabular data, but focus on tables from the general domain (Deng et al., 2020; Tang et al., 2021b; lida et al., 2021; Yu et al., 2019), Despite significant research in EL,, there is lack of datasets and methods for EL in scientific tables. Linking entities in scientific tables holds promise for accelerating science in multiple ways: from augmented reading applications that help users un- derstand the meaning of table cells without diving into the document (Head et al., 2021) to automated knowledge base construction that unifies disparate tables, enabling complex question answering or hypothesis generation (Hope et al., 2022).
Entity linking (EL) is the task of linking textual mention to its corresponding entry in a knowledge base, and is critical for many knowledge-intensive NI.P applications. When applied tables in scientific papers, EL is step toward large-scale scientific knowledge bases that could enable advanced scientific question answering and analytics. We present the first dataset for EL scientific tables. EL for scientific tables is especially challeng- ing because scientific knowledge bases can be very incomplete, and disambiguating ta- ble mentions typically requires understand- ing the paper's text in addition to the table. Our dataset, Scientific Table Entity Linking (S2abEL), focuses on EL in machine learning results tables and includes hand-labeled cell types, attributed sources, and entity links from the PaperswithCode taxonomy for 8,429 cells from 732 tables. We introduce a neural base- line method designed for EL scientific ta- base
EL in science is challenging because the set of scientific entities is vast and always growing, and existing knowledge bases highly incomplete. A traditional "closed world" assumption often made in EL systems, whereby all mentions have corre- sponding entities the target KB, not realistic scientific domains. It is important to detect which mentions are entities not yet in the reference KB, re- ferred to as outKB mentions. Even for human anno- tators, accurately identifying whether a rarely-seen surface form actually refers to vurcly-mentions long-tail inKB entity or an outKB entity requires domain expertise and a significant effort inves- tigate the document and the target KB. further challenge is that entity mentions in scientific ta- bles are often abbreviated and opaque, and require examining other context in the caption and paper text for disambiguation. An example is shown in Figure 1.
bles containing many nut-of-knowledd mentions, and show that significantly out- performs mate-of-thh--ee the-art generic table EL method. The best baselines fall below human performance, and our analysis highlights av- enues for improvement. Code and the dataset available at: https:/ / / hitttuu.a.. con/ al aeeeai//aabb.///
/ main.
1 Introduction
Entity Linking (EL) is a longstanding problem in natural language processing and information ex- traction. The goal of the task is to link textual mentions to their corresponding entities in knowl- edge base (KB) (Cucerzan, 2007), and it serves building block for various Knowledge-intensive ap- plications, including search engines (Blanco et al., 2015), question-answering systems (Dubey et al., 2018). and more. However, existing EL methods and datasets primarily focus on linking mentions
In this paper, we make three main contributions. First, we introduce S2abEL, a high-quality human- annotated dataset for EL in machine learning sults tables. The dataset is sufficiently large for training and evaluating models on table EII and rel- evant sub-tasks, including 52,257 annotations of ap- propriate types for table cells (e.g. method, dataset),
Work done during an intersshi at A12.

---

OPT: Open Pre-trained...
[mni]]
Papers With Code
linguage mocels, which trained We Open Transformers IOPTI, deooder only..-................
C2
/dataset/ConvA12
Introduction
Name Conversational Intelligenc Challenge?
-. th technical repart, we
Ferplxity
Entity Linking for Table
Description ComMMZ NeuriPS competition... Usage
present Open Pre-trained
BST
Trarsformera (ORI), sulle of decoder
2.71
Evaluations
WW
/diaaset/Wizard-.....
sodating span sounte
R2C2 Handdrtioo
diologue nlodids including
Tine BlenderBuc ci, 2021 pre-tralinng part RRedd 2018 ... fine tuned R2G2 BlenderBol, parmette
Hame Wilzard Willpedla
Description Witard of Wikipddaa is lange dataset conversations..
Tabll Divlogue Evaluations. OPT-1758: fully unsupervised netting unsupervised setting, parforss competiivvly against fully supervised models.
Usage
model Innired by Shuster at at (1022).
OPT
Out-of-KB
Figure 1: Part of a table OPT: Open Pre-trained Transformer Language Models (Zhang et al., 2022) showing relevant context that must be found for entity mentions in the table, and portion of EL. results mapping table mentions to the PapersWithCode KB.
9,565 annotations of attributed source papers and candidate entities for mentions, and 8,429 annota- tions for entity disambiguation including outKB mentions. To the best of our knowledge, this is the first dataset for table EL in the scientific do- main. Second, we propose a model that serves as a strong baseline for each of the sub-tasks, as well as end-to-end table EL. We conduct a comprehensive comparison between our approach and existing ap- proaches, where applicable, for each sub-task. Our method significantly outperforms TURL (Deng al., 2020), olate-of-the-an method closest to the table EL task, but only designed for general- domain tables. We also provide a detailed error analysis that emphasizes the need for improved methods to address the unique challenges of EL from scientific tables with oB mentions.
general-purpose KBs such as Wikidata (Vran- dečić and Krötzsch, 2014) and DBPedia (Auer et al., 2007) and typically test their approaches with the assumption that the target KB is complete with respect to the mentions being linked (e.g., De Cao et al., 2021; Deng et al., 2020; Hoffart et al., 2011; Tang et al., 2021a; Yamada et al., 2019),
There is a lack of high-quality datasets for table EL in the scientific domain with abundant outKB mentions. Recent work by Ruas and Couto (2022) provides a dataset that artificially mimics an incom- plete KB for biomedical text by removing actual referent entities but linking concepts to the direct ancestor of the referent entities. In contrast, our work provides aanmatted labels of realistic missing entities for scientific tables, without rely- ing the target KB to contain ancestor relations. Our dataset offers two distinct advantages: first, provides context from documents in addition original table mentions, and second, explicitly identifies mentions referring to outKB entities.
2 Related Work
2.1 Entity Linking
In recent years, various approaches have heen pro- posed for entity linking from free-form text, lever- aging large language models (Gu et al., 2021; De Cao et al., 2021; Li et al., 2020; Yamada et al., 2019). Researchers have also attempted to extend EL structured Web tables, but they solely rely on table contents and do not have rich surrounding text (Deng et al., 2020; Zhang et al., 2020; Bhaga- vatula et al., 2015; Mulwad et al., 2023; Tang et al., 2020; Tida et al., 2021). Most of these works focus
2.2 Scientific IE
The field of scientific information extraction (IE) aims to extract structured information from scien- tific documents. Various extraction tasks have been studied in this area, such as detecting and classi- fying semantic relations (Jain et al., 2020; Sahu et al., 2016), concept extraction (Fu al., 2020), automatic leaderboard construction (Kardas et al., 2020; Hou et al., 2019), and citation analysis (Jur-

---

gens et al., 2018; Cohan et al., 2019).
(e.g.. "QNLI (acc)" indicates the accuracy of some method evaluated on the Question Question-answerti answering NLI dataset (Wang et al., 2018)). Therefore, we intro- duce a separate class for these instances, reduc- ing CTC to a single-label classification task with four positive classes: method, dataset, metric, and m
Among these, Kardas et al., 2020; Hou et al., 2019; Yu et al., 2019, 2020 are the closest to ours. Given a set of papers, they begin by manually ex- tracting a taxonomy of tasks, datasets, and metric names from those papers. Whereas our data set maps each entity to an existing canonical external KB (PwC), they target taxonomy manually built from the particular papers and surface forms they extract from. Notably, this taxonomy emphasizes lexical representations, with entities such as "AP" and "Average Precision" treated as distinct entities in the taxonomy despite being identical metrics in reality, due to different surface forms appearing in the papers. Such incomplete and ambiguous entity identification makes it difficult for users interpret the results and limits the practical upplicability of the extracted information. In contrast, we propose a dataset and baselines for the end-to-end table EL. task, beginning with a table in the context of a pa- per and ending with each cell linked to entities in the canonicalized ontology of the target KB (or classified as outKB).
Attributed Source Matching (ASM) is the task of identifying attributed source(s) for a table cell within the context of the document. The attributed source(s) for a concept in a document p is the ref- erence paper mentioned in p to which the authors of p attribute the concept. ASM is a crucial step in distinguishing similar surface forms and finding the correct referent entities. For example, in Figure I, ASM can help clarify which entities "BlenderBot I" and "R2C2 BlenderBot" refer to, as the first mention is attributed to Roller et al., 2021 while the second mention is attributed to Shuster et al., 2022. Identifying these attributions helps a sys- tem uniquely identify these two entities despite their very similar surface forms and the fact that their contexts in the document often overlap. In this work, we consider the documents listed in the reference section and the document itself as poten- tial sources for attribution. The inclusion of the document itself is necessary since concepts may be introduced in the current document for the first time.
3 Entity Linking in Scientific Tables
Our entity linking task takes as input a reference KB (the Papers with Code' taxonomy in our exper- iments), a table in a scientific paper, and the table's surrounding context. The goal is to output tity from the KB for each table cell (or "outKB" if none). We decompose the task into several sub- tasks, discussed below. We then present S2abEL, the dataset we construct for scientific table EL.
Candidate Entity Retrieval (CER) is the process of identifying a small set of entities from the target KB that are most likely to be the referent entity for a table cell within the context the document. The purpose of this step is to exclude unlikely candi- dates and pass only limited number of candidates to the next step, to reduce computational cost.
3.1 Task Definition
Cell Type Classification (CTC) is the task of iden- tifying types of entities contained in table cell, based the document in which the cell appears. This step helpful to focus the later linking task on the correct type of entities from the target KB, and also excludes non-entity cells (e.g. those con- taining numeric values used to report experimental results) from later processing. Such exclusion re- moves substantial fraction of table cells (74% in our dataset), reducing the computational cost.
Entity Disambiguation (ED) with outKB Iden- tification is the final stage. The objective is to determine the referent entity (or report ourKB if none), given a table cell and its candidate entity set. The identification of outKB mentions significantly increases the complexity of the EL task, as it re- quires the method to differentiate between e.g. unusual surface form of an inKB entity versus an outKB mention. However, distinguishing outKB mentions is a critical step rapidly evolving do- mains like science, where existing KBs are highly incomplete.
One approach to CTC is view it a multi- label classification task since a cell may contain multiple entities of different types. However, our initial investigation found that only mentions of datasets and metrics co-appear to a notable degree

---

3.2 Dataset Construction
includes basic relations between entities, such as relevant entities for a paper (denoted Paper- RelatesTo-Eniity in the rest of the paper), the in- troducing paper for an entity, etc. Its data is col- lected from previously curated results and collab- oratively edited by the community. While the KB has good precision, its coverage is not exhaustive in our experiments, 42.8% of our entity mentions are outK and many papers have empty Paper- RelatesTo-Entig relations.
Obtaining high-quality annotations for S2abEL. is non-trivial. Identifying attributed sourees and gold entities requires global understanding of the text and tables in the full document. However, asking annotators to read every paper fully is prohibitively expensive. Presenting the full list of entities in the target KB to link from is also not feasible, while showing annotators short auto-populated candidate entity sets may introduce bias and miss gold enti- ties. We address these challenges by designing a special- annotation interface and pipeline, as detailed below.
Human Annotation We developed web in- terface using the Flask library for the annotation process. It provides annotators with a link to the original paper, an indexed reference section, and an- notation guidelines. The detailed annotation inter- face with instructions can be found at Appendix C. For the CTC sub-task, we asked annotators make necessary modifications to correct errors in SegmentedTables and accommodate the extra dataset&metric class. During this phase, 15% of the original labels were changed. For the ASM sub- task, annotators were asked to read relevant docu- ment sections for each cell and identify attributed sources, if any. This step require a global un- derstanding of the document, but candidate lists are relatively small since reference sections usually contain just tens of papers. For the EL sub-task, the web interface populates each cell with entity candidates that are I) returned from PwC with the cell content as the search string, and/or 2) asso- ciated with the identified attributed paper(s) for this cell via the Caper-RelatesTo- Enniit relation in PwC. Automatic candidate population is designed to be preliminary to prevent annotators from be- lieving that gold entities should always come from the candidate set. Annotators were also asked to search against PwC using different surface forms of the cell content (e.g., full name, part of the cell content) before concluding that cell refers to an outK entity.
In the construction process, we used two in- house annotators with backgrounds in data ana- lytics and data science, both having extensive expe- rience in reading and annotating scientific papers. In addition, one author of the paper (author A) led and initial training phase with the annotators, and another author the paper (author B) was respon- sible for evaluating the inter-annotator agreement (IAA) at the end of the annotation process.
Bootstrapping existing resources We began constructing our dataset by populating it with ta- bles and cell type annotations from SegmentedTa- bles? (Kardas et al., 2020), dataset where table cells have been extracted from papers and stored in an array format. Each cell is annotated according to whether it is a paper, metric, and so on; and each pa- per is classified into one of eleven categories (e.g., NLI and Image Generation). To gather data for the ASM task, we fine-tuned a T5-small (Raffel et al., 2022) model to extract the last name of the first author, year, and title for each paper that appears in the reference section of any papers in our dataset from the raw reference strings. We then used the extracted information to search for matching pa- pers in Semantic Scholar (Kinncy et al., 2023), to obtain their abstracts. Since the search APIs do not always return the matching paper at the top of the results, we manually verified the output for each query.
To ensure consistency and high quality, we con- ducted a training phase led by author A, where the two annotators were given four papers at a time to perform all annotation tasks. We then calculated the IAA between author A and each annotator for the four papers using Cohen's Kappa (McHugh, 2012), followed by disagreement discussion and guideline refinement. This process was repeated un- til the IAA score achieves "substantial agreement" as per (McHugh, 2012). Afterward, the remain-
Target KB- -Papers with Code (PwC)34 4 is a free and open knowledge base in the scientific domain with a total of 304,611 I papers, 6,550 datasets, and 1,942 methods entities as of this writing. PwC
'https://gith.b http:/tttt.......eooeeeeeeeeeeee http:/ghhtt...m/////t/tttttaaaae axcell/reles
'Our corpus based on Papers with Code 2022/07 dump. 'https: //g! thub. com/ paperssi thcode/ paperswit weterssiihhhodeddee data
$flask www.spctssss..oomm

---

CTC
ASM
EL
same row and column usually refer the same type of entities. More details about cell representation features are in Table
# papers
327
316
303
# tables
886
790
732
# cells 52,257
9,564
8,429
Paper Representation - For each referenced pa- per, extract its index in the reference section, the last name of the first author, year, title, and ab- stract. Index, author name, and year are helpful for identifying inline citations (which frequently take the form of the index in brackets or the author and year in parens). Additionally, the title and abstract provide summary of paper which may contain information on new concepts it proposes.
Table 1: Overall statistics of S2abEL. It consists of 52,257 data points for cell types, 9,564 for attributed source matching, and 8,429 for entity linking, with ground truth.
ing set of papers was given to the annotators for annotation.
KB Entity Representation To represent each entity in the target KB, we use its abbreviation, full name, and description from the KB, available. The abbreviation and full name of an entity are crucial for capturing exact mentions in the text, while the description provides additional context for the entity (Logeswaran et al., 2019).
3.3 Dataset and Annotation Statistics
Dataset Statistics - Table provides a summary of the statistics for S2abEL. ASM and EL annota- tions are only available for cells labeled positively in CTC. Metrics only are not linked to entities due to the lack of controlled metric ontology in PwC. It is worth noting that S2abEL contains 3,610 outKB mentions versus 4,819 inKB men- tions, presenting significantly different challenge from prior datasets that mostly handle inKB men- tions. More details are in Appendix A.
Cell Type Classification We concatenate fea- tures of cell representation (separated by special tokens) and input the resulting sequence to the pre- trained language model SciBERT (Beltagy et al., 2019). For each token in the input sequence, we add its word embedding vector with additional trainable embedding vector from a separate em- bedding layer to differentiate whether token is in the cell, from context sentences, etc. (Subsequent mentions of SciBERT in this paper refer to this modified version). We pass the average of the out- put token embeddings at the last layer to linear output layer and optimize for Cross Entropy loss. However, because the majority of cells in scientific tables pertain to experimental statisties, the distri- bution of cell types is highly imbalanced (as shown in Appendix A). To address this issue, we oversam- ple the minority class data by randomly shuffling the context text extracted from the paper for those cells at sentence level.
Post-hoe IAA Evaluation - We conducted a post- hoc evaluation to verify the quality of annotations, where author B, who is researcher with Ph.D. in Computer Science, independently annotated five random tables. The Cohen's Kappa scores show a substantial level of agreement (McHugh, 2012) between author B and the annotations (100% for CTC, 85,5 555 for ASM, and 60.6% for EL). These results demonstrate the quality and reliability of the annotations in S2abEL. A more detailed analysis on why EL agreement is relatively low can be found at Appendix D.
4 Method
In this section, we describe our approach for rep- resenting table cells, papers, and KB entities, as well as our model design for performing each of the sub-tasks defined in Section 3.1.
Attributed Source Matching To enable con- textualization between cell context and a potential source, we combine the representations of each table cell and potential attributed source in the doc- ument as the input to a SciBERT followed by lin- ear output layer. We optimize for the Binary Cross Entropy loss, where all non-aue non-attributed sources in the document are used as negative examples for a cell. The model output measures the likelihood that source should be attributed to given table cell.
Cell Representation - For each table cell in a document, we collect information from both doc- ument text and the surrounding table. Top-ranked sentences were retrieved using BM25 (Robertson and Zaragoza, 2009) as context sentences, which often include explanations and descriptions of the table cell. The surrounding table captures the row and column context of the cell, which can offer valuable hints, such as the fact that mentions in the
Candidate Entity Retrieval We design a method that combines candidates retrieved by two

---

strategies: (i) dense retrieval (DR) (Karpukhin al., 2020) that leverages embeddings repre- sent latent semanties of table cells and entities, and (ii) attributed source retrieval (ASR) which uses the attributed source information to retrieve candidate entities.
work by evaluating their performance on sub-tasks of our dataset (Section 5.1). Additionally, we report the performance of the end-to-end system to pro- vide baseline results for future work (Section 5.2). Finally, to understand the connection and impact of each sub-task on the final EL performance, we conducted a component wise ablation study (Sec- tion 5.3). This study provides valuable insights into the difficulties and bottlenecks in model per- formance.
For DR, we fine-tune bi-encoder architecture (Reimers and Gurevych, 2019) with two separate SciBERTs to optimize triplet objective function, The model only trained on cells whose gold refer- ent entity exists in the KB. Top-ranked most similar entities based on the BM25F algorithm (Robertson and Zaragoza, 2009) in Elasticscarch are used as negative examples. For each table cell i, the top-k nearest entities Og in the embedding space with ranks are returned candidates.
The experiments are designed to evaluate the performance methods in a cross-domain setting (following the setup in Kardas et al., 2020), where training, validation, and test data come from differ- ent disjoint topics. This ensures that the methods not overfitting to the particular characteristics of a topic and can generalize well to unseen data from different topics.
For ASR, we use the trained ASM model to ob- tain list of papers ranked by their probabilities of being the attributed source estimated by the model. The candidate entity sequence Oi is constructed by fetching entities associated with each potentially attributed paper in ranked order using the Paper- RelatesTo- En-tnt relations in PwC. Only entities of the same cell type identified in CTC are retained. Note that including entities associated with lower- ranked papers mitigates the errors propagated from the ASM model and the problem of imperfect entity and relation coverage that is common in real-world KBs.
5.1 Evaluating Sub-tasks
5.1.1 Cell Type Classification
We compare our method against AxCell's cell type classification component (Kardas et al., 2020), which uses ULMFiT architecture (Howard and Ruder, 2018) with LSTM layers pre-trained on arXiv papers. It takes input the contents of table cells with a set of hand-crafted features to pro- vide the context of cells in the paper. We use their publicly available implementation' with slight modification to the output layer to suit our 5-class classification.
We finally interleave O and O until reach a pre-defined entity set size K.
Entity Disambiguation with outKB Identifica- tion- - Given a table cell and its entity candidates, we fine-tune a cross-encoder architecture (Reimers and Gurevych, 2019) with a SciBERT that takes as input the fused cell representation and entity representation, followed by linear output layer. We optimize for BCE loss using the same negative examples used in CER training. The trained model is used estimate the probability that table cell matches an entity. the top-ranked entity for cell has a matching likelihood lower than 0.5, then the cell is considered to be outKB.
Table 2 shows that method outperforms Ax- Cell somewhat in terms of Fl scores. Although we do not claim our method on this particular sub-task substantially better, we provide baseline results using state-of-the-ar art transformer models.
5.1.2 Candidate Entity Retrieval
Since the goal of CER is to generate a small list of potential entities for table cell, we evaluate the performance of the CER method using recall @ K. Figure shows the results of evaluating dense retrieval (DR), attributed source retrieval (ASR), and a combination of both methods, with different candidate size limits K. We observe that seeding the candidate set with entities associated with at- tributed papers significantly outperforms DR, while interleaving candidates from ASR and DR pro- duces the most promising results. These results
5 Evaluations
As existing baselines exist for the end-to-end table EL task with outKB mention identification, we compare methods against appropriate recent
"We chose to use BM25F instead of BM25 because it can take into account entity data with several fields, including name, full name, and description.
'https: haatps::/gtthuue http://thhhu..cc//aaoooaoaa wwwppppppswwtteeeeeeecooopppeppe thcode/ axcell

---

Validation
Test
Class
AxCell
Ours
AxCell
Ours
P
R
Fi
P
R
Fi
P
R
F1
P
R
Fi
other
97.4
98.6
98.0
98.4
98.4
98.4
97.3
98.4
97.8
98.0
98.4
98.1
dataset
83.1
81.7
82.2
83.0
88.7
85.6
90.0
84.0
86.2
90.1
82.3
85.4
method
96.7
96.4
96.6
97.2
96.8
97.0
95.3
95.5
95.4
93.4
96.9
95.1
metric
71.3
72.8
71.9
88.6
79.7
83.7
86.6
86.1
85.0
88.0
87.4
86.8
dalaset&metrty
97.1
41.9
58.0
88.8
77.6
82.1
82.6
63.4
68.1
75.3
64.8
61.9
Micro F1
95.8
96.8
96.0
96.2
Table 2: Results of cell type classification on our method and AxCell, with image classification papers fixed as the validation set and papers from each remaining category as the test set in turn.
or because the authors cited the wrong paper. We additionally investigated the error cases from DR and found that considerable fraction was caused by the use of generic words refer to a specific entity. For instance, the validation of a specific dataset entity was referred to "val" the table, the method proposed the paper was referred to as "ours", and a subset of a dataset that represents data belonging to one of the classifica- tion categories was referred to as "window". Re- solving the ambiguity of such references requires the model to have understanding of the unique meaning of those words in the context.
90
800
ASR (BM25F)
DR (SciBert)
ASR (SeiBert)
ASR (SeiBert)
65
100
Candidate
When using the combined candidate sets, miss- ing gold entities were only observed when both DR and ASR failed, leading to superior performance compared to using either method alone.
Figure Evaluation of different candidate entity re- trieval methods. The method in the parenthesis indicates whether a fine-tuned SciBERT or BM25F is used.
5.1.3 Entity Disambiguation with inKB Mentions
demonstrate the effectiveness of utilizing informa- tion on attributed sources to generate high-quality candidates. worth noting that when K suf- ficiently large, ASR considers all sources as at- tributed sources for a given cell, thus returning entities that are associated with any source. How- ever, if the gold entity is not related to any cited source in the paper, it will still be missing from the candidate Increasing K further will not recover this missing entity, as indicated by the saturation observed in Figure 2.
The state-of-the-art method closest to our table EL task is TURL. (Deng et al., 2020), designed for general-domain tables with inKB cells. is a structure- aware Transformer encoder pre-trained on the general-purpose WikiTables corpus (Bhaga- vatula et al., 2015), which produces contextualized embeddings for table cells, rows, and columns that are suitable for range of downstream applications, including table EL. We used TURL's public code and fine-tuned it on the inKB cells of our dataset and compared with our method using the same entity candidate set of size 50.
Error Analysis - We examined the outputs of ASR and identified two main challenges. First, we observed that in 22.8% of the етror cases when K 100, authors did not cite papers for re- ferred concepts. These cases typically involve well-known entities such LSTM (Hochreiter and Schmidhuber, 1997). In the remaining error cases, the authors did cite papers; however, the gold entity was not retrieved due to incomplete Paper-RelatesTor RelatesTo Entity relations in the target KB
Table 3 shows that our model achieves substan- tial improvement in accuracy of ten paper folds. The examples in Table 6 (ap- pendix) demonstrate that our model is more effec- tive at recognizing the referent entity when the cell mention is ambiguous and looks similar to other en-
TURL nine
Phttps://gtt ccm comm/sunlabb http:hh.ccmm.aalaaaaa////ottt

---

titics, the method needs to determine when cell mentions refer to entities that do not exist in the Test fold
Support
TURL
Ours
Question ans.
381
15.0
36.1
Object det.
2040
34.1
41.0
Speech rec.
175
34.3
54.3
80
Image gen.
168
7.7
35.1
Machine trans.
234
15.4
39.3
Text class.
246
52.4
68.7
Pose estim.
108
36.1
36.1
Semantic seg.
641
44.8
50.9
NLI
328
30.8
58,8
Misc.
81
14.8
33.3
outK B
outK B recall
Micro avg
32.5
44.8
20
cark R
inKD bit @
overall ace
Table Accuracy for end-to-end entity linking for cells that refer to inKB entity with 10-fold-cross-domain evaluation using our approach and TURL. Our method is specialized for tables scientific papers and outper- forms the general-pure purppss TURL method.
0.0
0.2
0.4
0.6
0.8
1.0
threshold
Figure 3: Entity linking results with varying inKB thresholds. Note that the inKB hit rate is low (44.8%) even when all mentions are predicted with entity (i.e.. threshold is 0).
O/l ratio
OutKB Fi
InKB hit@1 1
Overall acc.
Test fold
Machine trans.
0.60
62.2
23.7
50.3
Image gen.
0.48
55.1
20.0
44.4
Misc.
2.74
85.1
19.5
74.9
Speech rec.
1.46
73.7
33.3
66.5
target KB. We report F1 scores for outKB entities as the prediction is binary (precision and recall are reported in Appendix Table 8). For inKB mentions, report the hit rate at top-1. Additionally, evaluate overall performance using accuracy." For each topic of pupers, we report the ratio of outKB mentions to inKB mentions. The top block of Ta- ble shows the end-to-end EL performance of our method. Our analysis shows a positive Pearson cor- relation (Cohen et al., 2009) of 0.87 between O/T ratio and overall accuracy, indicating our method tends to higher accuracy datasets with more outKB mentions. Figure shows the performance at various inKB thresholds.
Question ans.
2.33
84.2
14.0
69.3
NLI
1.11
80.6
47.4
68.3
Text class.
1.24
77.1
35.4
66.8
Object det.
0.23
49.6
35.2
45.7
Semantic seg.
0.42
73.4
39.7
55.0
Pose estim.
1.06
72.2
35.2
59.9
Micro avg
0.75
71.4
33.3
57.6
goold CTC
0.75
72.4
33.4
58.2
+ gold Can.
0.75
71.5
33.4
57.6
gold both
0.75
72.5
33.4
58.2
Table 4: End-to-end EL results with 10-fold-cross- domain evaluation of our method on learned + candidate of size 50 with the inKB threshold set 0.5. Although our model achieved reasonable overall accuracy, still far from perfect, leaving ample room for future improvements in the end-to-end table El task.
Error Analysis - We sampled 100 examples of correct predictions for both outKB and inKB men- tions and analyzed their causes of errors in Table 7 (Appendix E). Our analysis reveals that major- ity of incorrect inKB predictions are due to the use of generic words. For outKB mentions, the model tends to get confused when they are similar to existing entities in the target KB.
tities in the KB. This is because TURL generic table embedding method focuses on just cell con- tent and position while our approach combines cell content with the full document. Our analysis fur- ther reveals that TURL made incorrect predictions for all cells whose mentions were shorter than four characters (likely an abbreviation or pointer to reference paper). Mcanwhile, method correctly linked 39% of these cells.
5.3 Component- wise Ablation Study
To investigate how much of the error in our end-to- end three-step system was due to errors introduced in the first two stages (specifically, wrong cell type
5.2 End-to-end Evaluation
We now evaluate the end-to-end performance of our approach on the EL task with outKB identi- fication. addition to re-ranking candidate en-
"A cell is considered correct prediction if is an outKB mention and predicted as such, or if it an inK mention and predicted as inKB with the gold entity being ranked at top 1.

---

classifications from CTC or missing correct candi- dates from CER), we tried measuring system per- formance with these errors removed. Specifically, we tried replacing the CTC output with the gold cell labels, or adding the gold entity to the output CER candidate set, or both.
ent methods may improve performance on our task substantially.
Acknowledgments
This work was supported in part by NSF Grant 2033558.
The results in the bottom block of Table 4 show that there is no significant difference in perfor- mance with gold inputs. This could be because CTC and CER are easier tasks compared to ED, and if the model fails those tasks, it is likely to still struggle to identify the correct referent entity, even if that is present in the candidate set or the correct cell type is given.
References
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for web of open data. In Proceedings of the 6th International The Seman- tic Web and 2nd Asian Conference on Asian Se- mantic Web Conference, ISWC'07/ASWCC"0,, page 722-735, Berlin, Heidelberg. Springer- Verlag.
6 Conclusion
Iz. Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLL) pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics.
In this paper, we present S2abEL, high-quality human-annotated dataset for Entity Linking in ma- chine learning results tables, which is, to the best of our knowledge, the first dataset for table EL in the scientific domain. We propose a model that serves as a strong baseline for the end-to-end table EL task with identification of outKB mentions, as well as for each of the sub-tasks in the dataset. We show that extracting context from paper text gives a significant improvement compared to methods that use only tables. Identifying attributed source papers for concept achieves higher recall @ k com- pared with dense retrieval for candidate entity gen- eration. However, the best baselines still fall far below human performance, showing potential for future improvement.
Chandra Sekhar Bhagavatula, Thanapon Noraset, and Doug Downey. 2015. Tabel: Entity linking web tables. In The Semantic Web-ISWC 2015: 14th In- ternational Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings. Part pages 425-441. Springer.
Roi Blanco, Giuseppe Ottaviano, and Edgar 2015. Fast and space-efficiees entity linking for queries. In Proceedings of the Eighth ACM International Confer- ence on Web Search and Data Mining, WSDM 15, page 179-188, New York, NY, USA. Association for Computing Machinery.
Limitations
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for ci- tation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3586-3596, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
In this section, we discuss some limitations of our work. First, our dataset only includes tables from English-language papers in the machine learning domain, linked to the Papers with Code KB, which limits its generalizability to other domains, lan- guages, and KBs. Second, we acknowledge that the creation of S2abEL required significant manual effort from domain experts, making it a resource- intensive process that may not be easily scalable. Third, our approach of using attributed papers to aid in identifying referent entities relies on the tar- get KB containing relations that associate relevant papers and entities together. Fourth, do not compare against large GPT-series models and leave this as future work. Finally, while our experiments set one initial bascline for model performance on our task, substantially more exploration of differ-
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Ben- esty, Jacob Benesty, Jingdong Chen, Yiteng Huang. and Israel Cohen. 2009. Pearson correlation coeffi- cient. Noise reduction in speech processing, pages 1-4.
Silviu Cucerzan. 2007. Large-scale named entity disam- biguation based Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNIL) pages 708-716 Prague, Czech Republic, Association for Computational Linguistics.

---

Nicola De Cao, Gautier Izacard. Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 328-339, Melbourne, Australia. Association for Computational Linguisties.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. Turl: table understanding through representation learning. Proceedings of the VLDB Endowment, 1433)307-319
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446-3456, Online. Association for Computational Linguistics.
Mohnish Dubey, Debayan Banerjee. Debanjan Chaud- huri, and Jens Lehmann. 2018. Earl: joint entity and relation linking for question answering over knowl- edge graphs. In The Semantic Web-ISWC 2018: 17th International Semantic Web Conference, Monterey. CA. USA, October 12, 2018, Proceedings, Part I pages (08-126.66) Springer.
Sarthak Jain, Madeleine van Zuylen, Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. SciREX: challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7506- 7516, Online. Association for Computational Lin- guistics.
Sunyang Fu, David Chen, Huan He, Sijia Liu, Sun- grim Moon, Kevin Peterson, Feichen Shen, Liwei Wang, Yanshan Wang, Andrew Wen, Yiqing Zhao, Sunghwan Sohn, and Hongfang Liu. 2020. Clinical concept extraction: A methodology review. Journal of Biomedical Informatcss, 109:103526.
David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc- Farland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through eitation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Yingjic Gu, Xiaoye Qu, Zhefeng Wang. Baoxing Huai, Nicholas Jing Yuan, and Xiaolin Gui. 2021. Read, retrospect, select: An mre framework to short text entity linking. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14): 129200-222222999
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebas- tian Ruder, Sehastian Riedel, Ross Taylor, and Robert Stojnic. 2020. AxCell: Automatic extraction of re- sults from machine learning papers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8580- 8594, Online. Association for Computational Lin- guistics.
Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel Weld, and Marti Hearst. 2021. Augmenting scientific papers with just- in-time, Desition-sensitivy definitions of terms and symbols. In Proceedings of the 202/ CHI Conference on Human Factors in Computing Systems, pages 1- 18.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781 6681,, Online. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 978):1735- 1780.
Johannes Hoffart, Mohamed Amir Yosef, llriia Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Em- pirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguistics.
Rodney Michael Kinney. Chloe Anastasiades, Rus- sell Authur, Beltagy, Jonathan Bragg, Alexan- dra Buraczynski, Isabel Cachola, Stefan Candra, Yo- ganand Chandrasekhar, Arman Coban, Miles Craw- ford, Doug Downey, Jason Dunkelberger, Oren Et- zioni, Robert Evans, Sergey Feldman, Joseph Gor- ney, David W. Graham, F.Q. Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Christo- pher Newell, Smita Rao. Shaurya Rohatgi, Paul L Sayre, Zejiang Shen, Amanpreet Singh, Luca Sol- daini, Shivashankar Subramanian, A. Tanaka, Alex D Wade, Linda M. Wagner, Lucy Lu Wang. Christo- pher Wilhelm, Caroline Wu, Jiangjiang Yang, An- gele Zamarron, Madeleine van Zuylen, and Daniel S. Weld. 2023. The semantic scholar open data platform. ArXiv, abs/2301. 10140..
Tom Hope, Doug Downey, Oren Etzioni, Daniel Weld, and Eric Horvitz. 2022. A computational inflection for scientific discovery. arXiv preprint arXiv:225.000007.
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5203- 5213, Florence, Italy, Association for Computational Linguistics.

---

Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one- one-pass pass end-to-end entity linking for questions. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), , pages 6433-6441, Online. Association for Computational Linguisties.
Workshop on Biomedical Natural Language Process- ing, pages 206-215, Berlin, Germany. Association for Computational Linguistics.
Kurt Shuster. Mojtaba Komeili, Leonard Adolphs. Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion.
Lajanugen Logeswaran, Ming-Woi Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
Hongyin Tang, Xingwu Sun, Beihong Jin, and Fuzheng Zhang. 2021a. A bidirectional multi- paragraph read- ing model for zero-shot entity linking. In Proceed- ings of the AAAI Conference on Artificial Intelligence. volume 33, pages
Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Sam Madden, and Mourad Ouz- zani. 2020. Rpt: relational pre- -rairrd transformer is almost all you need towards democratizing data preparation. arXiv preprint
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Mary I. McHugh, 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3))276-28) 282.
Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Sam Madden, and Mourad Ouz- zani. 2021b. Rpt: Relational pre-trained transformer is almost all you need towards democratizing data preparation. Proc. VLDB Endow., 14(8):
Varish Mulwad, Tim Finin. Vijay S Kumar, Jenny Weisenberg Williams, Sharad Dixit, Anupam Joshi, et al. 2023. A practical entity linking system for tables scientific literature. In 3rd Workshop on Scientific Document Understanding at AAAI-2023.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledgebase. Commun. ACM, 57(10)778-8) 85.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNT.P: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP)) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2019, Global entity disambiguation with pretrained contextualized embeddings of words and entities. arXiv preprint arxiv:1909.000 1909.0046.
Stephen Robertson and Hugo Zaragoza. 2009, The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr. ((4):333-389
Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2022. Global entity disambiguation with BERT. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3264-322 3271, Seattle, United States. Association for Computational Linguistics.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Eric Michael Smith, Y-Lan Boureau, and Jason We- open-domain
ston. 2021. Recipes for building chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association.fo Compu- lational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Wenhao Yu, Zongze Li, Qingkai Zeng, and Meng Jiang. 2019. Tablepedia: Automating pdf table reading in an experimental evidence exploration and analytic system. In The World Wide Web Conference, WWW '19, page 3615-3619, New York, NY, USA. Associa- tion for Computing Machinery.
Pedro Ruas and Francisco M. Couto. 2022. Nilinker: Attention- approach nil entity linking. Jour- nal of Biomedical Informatics, 104117.
Wenhao Yu, Wei Peng, Yu Shu, Qingkai Zeng, and Meng Jiang. 2020. Experimental evidence extraction system in data science with hybrid table features and ensemble learning. In Proceedings of The Web Con- ference 2020, WWW '20, page 951-961, New York, NY, USA. Association for Computing Machinery.
Sunil Sahu, Ashish Anand, Krishnadev Oruganty, and Mahanandeeshwar Gattu, 2016. Relation extraction from clinical texts using domain invariant convolu- tional neural network. In Proceedings of the 15th

---

CTC
ASM
EL
Fold
# paper
# table
# cell
paper
# table
# cell
# paper
# table
cell
Question ans.
58
139
6,422
57
128
1,320
57
127
1,282
Object det.
54
159
17,020
52
141
2,686
51
135
2,527
Image class.
27
94
2,681
27
82
608
27
81
597
Speech rec.
22
88
3,516
21
76
649
21
74
612
Image gen.
25
1,184
34
290
23
34
288
Machine trans.
28
48
2,199
42
412
25
41
378
Text class.
21
75
4,085
20
55
688
19
51
600
NLI
32
83
3,385
30
68
787
30
66
697
Pose estim.
13
47
4,447
11
31
550
7
18
222
Semantic seg,
32
82
5,733
30
75
927
30
75
919
Misc.
15
34
1,585
13
30
308
13
30
307
Table 5: Detailed statisties of S2abEL.
Cell Content Column header
TURL result
Our result
Gold entity
InKB
Cityscapes, PoseTrack,
"[33]"
PointNet, GAN, CRF
PointNet
LAMBADA
Global Conv. Net.,
Graph Conv. Net.,
"Text GCN"
"Model"
Graph Conv. Net.
End-to-End Mem. Net.
Global Conv. Net.
Table 6: Incorrect examples for end to-end EL from TURL. The table includes the cell content and the column header in the first two columns, the top-3 ranked results from TURL. and our approach in the third and fourth columns, respectively, and the gold entity in the last column.
Shuo Zhang, Edgar Meij, Krisztian Balog. and Ridho Reinanda. 2020. Novel entity discovery from web tables. Proceedings The Web Conference 2020, WWW 20, page 1298-1308, New York, NY, USA. Association for Computing Machinery.
B Training Details
We trained all our models for two epochs with batch size of 32, using the AdamW optimizer (Loshchilov and Hutter, 2019) with linear decay warm-up. The initial learning rate was 2e-5 and the warm-up ratio was 10%. All models were trained using a single 48Gb NVIDIA A6000 GPU. For the triplet loss function in DR, we used Euclidean the distance function with margin of 1. For the Candidate Entity Retrieval and Entity Disambigua- tion tasks, we used negative examples of size 50 at training time. Additionally for the ED task, we set candidate set size limitation as 50 when making predictions.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint ar5.0:2205.01068.
A Detailed Dataset Statisties
S2abEL consists of 11 folds, each corresponding to topic. Table provides detailed statistics on the number of papers, tables, and cells for each sub-task and topic. The class distribution for CTC is as follows: other (74%), dataset (8%), method (14%), metric (3%), and dataset&metric (0.4%). For ASM, 1,532 (16.6%) cells have missing at- tributed paper, 1,095 (11.9%) cells attribute the paper itself, 6.598 (71.5%) cells attribute an en- try in the reference section of the paper. For EL, 3,610 (42.8%) cells refer to outKB entities and (57.2%) cells refer to inKB entities.
C Annotation Interface and Guidelines
Our annotation interface with annotation guide- lines is at httpts///githhbb.oom homm/mlleeaaiil s2abel /2abel//boo//mmai//comooo pdf.. Note that there might be cells that contain a subentity mentions consisting of an entity mention and a non-entity mention string, e.g., "Bert-large". "Bert
a

---

Cause
Percent
Example cell
Our result
Gold entity
InKB
PASCAL VOC,
"bottle" "
PASCAL VOC 2007
Variants
21%
PASCAL VOC 2007
"BigGAN"
BigGan-deep, BigGan
BigGan
Top below threshold
16%
"Car"
outKB, KITTI
KITTI
Abbreviations
19%
"FR-EN"
Arcade Learning Environment
WMT 2014
Generic words
39%
"Ours"
Neural Turing Machine
OHEM
OutKB
Similar names
56%
"DPN""
Dual Path Network
Deep Parsing Network
Generic words
22%
"12"
HUB5 English
bAbi
Table 7: Representative examples of erroneous end-to-end EL cases. The table includes the cause and the percentage
for that cause in the first two columns, an example of cell content for that cause and our incorrect prediction in the third and fourth columns, and the gold entity in the last column.
Test fold
Precision Recall
Feature
Description
cell content
cell's raw text
Machine trans.
46.4
94.4
region
reference lop-left numeric cell the table, top-left, top right. bottom-left, and bottom-right
Image gen.
38.7
95.7
Misc.
77.0
95.1
context sentences
top ranked sentences in full document (including table captions, section henders, etc.) regarding the cell content based on BM25 concatenated cell's row separated by speciaal tokens concatenated
Speech rec.
62.8
89.3
Question ans.
76.9
93.1
rOW content
column context
separated by special distance
NLI
74.9
87.2
position
position from the left corner cell's 2D position in the table in terms of distance bottom right corner.
Text class.
66.2
92.2
reverse position
Object det.
34.0
91.1
Semantic seg.
61.4
91.2
has reference
whether cell has reference
Pose estim.
63.8
83.3
Table 9: Features for cell representation.
Micro avg
58.6
91.4
D Annotation Error Analysis
+ gold CTC
59.5
92.7
+ gold Can.
58.7
91.4
Our investigation showed that the main cause of disagreement the EL phase was that there were cells whose matching entities were confusing to the annotators, due to their insufficient background in the specific academic area and/or the paper not clearly indicating which entity it was and/or whether should be considered variant of an exist- ing entity different entity entirely.
+ goold both
59.5
92.7
Table Additional end-to-end Entity linking results for outKB cells.
with 6 layers frozen". For these cells, we asked the annotators to focus on the primary entity and our current model considers these mentions as mentions of the main entity. Thus those two men- tions are labeled as method, and linked to https: paper swi thcode. com/ me method/bert. We also specifically asked the annotators to mark cells that contain mentions of more than one primary entity or are confusing to understand, which are excluded from the dataset. We leave the tasks of linking subentities explicitly and cells to multiple entities for future work.
E Error Case Study
Table presents examples where TURL made in- correct EL predictions while our approach made correct predictions. Table summarizes the main causes of incorrect predictions made by our ap- proach for both inKB and outKB mentions.


Reviews:
[{"review_id": "Oq5jMIJRqa", "reviewer": "Reviewer_NRhP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting this limitation in our paper. \n\nWe also mentioned the same point in our Limitations section (Lines 621-625). We totally agree with the importance of diversifying the dataset to capture broader patterns in other domains, and this is one of the future steps. \n\nHowever, we would argue that our dataset nonetheless forms a significant contribution to the community as it is the only existing scientific table entity linking dataset with out-of-KB mentions, and it covers a broad range of the large ML field (spanning eleven different sub-disciplines).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.A new dataset for EL in scientific tables is presented. The new dataset is large-scale and can be used for training and testing models on table EL.\n\n2.A baseline model for table EL in scientific papers is proposed, which achieves good results in the experiments.\n\n3.The paper is well structured and written.\n\n", "reasons_to_reject": "As discussed in Limitations, only papers in the machine learning domain are used to built the dataset. Papers in different domains may have different patterns in tables. Papers from more domains need to be included in the dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new dataset for entity linking in scientific tables. The entity linking task studied in the paper is to link entities in tables from scientific papers to entities in the Papers with Code taxonomy. To build the dataset, a special-purpose annotation interface and pipeline is developed. The paper also proposes a baseline model for the EL task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QsjyjSJqEs", "reviewer": "Reviewer_zr1c", "summary": "", "questions": "1. What is the \"Paper-RelatesTo-Entity relation in PwC\" (lines 303-304)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and pointing out the lack of clarity regarding the work of Kardas et al., (2020). We acknowledge that our description of Kardas et al., (2020) and the comparison of our work and Kardas et al., in Section 2.2 was not sufficiently detailed. We do draw upon their work significantly in ours.  \n\nHere's a clearer comparison between our work and that of Kardas et al., which we will incorporate in our revisions:\n1. **Task relation with Kardas et al., (2020)**: Given a set of papers, Kardas and team begin by manually extracting a taxonomy of tasks, datasets, and metric names from those papers. This is the most important distinction from our work: whereas our data set maps each entity to an existing canonical external KB of models and datasets (PwC), they target a taxonomy manually built from the particular papers and surface forms they extract from. We would expect that the lexical similarity between the paper surface forms and the taxonomy entity names is thus higher than it would be for an external KB.  The AxCell tool subsequently maps corresponding numeric cells to the extracted taxonomy entries under a closed-world assumption. Notably, this taxonomy emphasizes lexical representations, with entities such as \"AP\" and \"Average Precision\" treated as distinct entities in the taxonomy despite being identical metrics in reality, due to different surface forms appearing in the papers. This is evident from their released taxonomy ( https://github.com/paperswithcode/axcell/releases/download/v1.0/models.tar.xz). \nContrasting this with our approach:\n     1. Surface Form Extraction: The entity linker in Kardas et al. is based on lexical features, making it adept for their specific taxonomy. However, when aiming for Entity Linking to a canonical Knowledge Base (KB), such a method struggles due to variations in entity mentions, like \"USA\", \"U.S.\", and \"America\" all denoting the same entity.  Our dataset, by contrast, resolves different ambiguous surface forms from papers to their referent entities in te PWC knowledge base.\n     2. Entity Deduplication and Clarity: The taxonomy in Kardas et al. lacks robust deduplication or clustering of identical entities across papers and does not provide additional information other than names, resulting in potential ambiguities.\n2. **Using the linker in Karadas et al., (2020) in our task**: The entity linker in Kardas et al., (2020) extracts lexical features of a mention in a cell, including Bag-of-Words and abbreviation extension and calculates the probability of matching to a (task, dataset, metric) tiple similar to TF-IDF, with 18 manually-tuned hyperparameters in the formula (as seen in Table 9 of their paper). Our initial research suggested that KB Entity Linking demands deeper linguistic and semantic features, which a predominantly lexical linker does not effectively capture. Moreover, the choice of the 18 parameters in Kardas et al., remains opaque, making replication or adaptation to new tasks challenging.\n\nRegarding your question on the \"Paper-RelatesTo-Entity relation in PwC (lines 303-304): The Papers with Code KB provides methods and datasets that are relevant to a paper (e.g., Methods and Datasets sections on this page: https://paperswithcode.com/paper/code-llama-open-foundation-models-for-code). However, there is no guarantee of coverage or correctness of this feature. In fact, we found a significant number of papers that have incomplete or even completely blank Methods and Dataset sections in PwC. Due to this limitation, it became necessary to incorporate candidate entities from both direct dense retrieval and attributed source matching, as discussed in Section 4, line 410.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow. The authors clearly describe the dataset construction process and the model details.\n2. The dataset is useful for the community and set a good example for future dataset construction on similar tasks but different domains.\n3. The authors provide a reasonable baseline for the proposed dataset and conduct a thorough analysis of experimented methods.", "reasons_to_reject": "The biggest weakness of this paper for me is that it does not clearly discuss its relation to the previous work of Kardas et al., (2020). In Section 2.2 (lines 144-148), the authors describe Kardas et al., (2020) as \"However, they only identify entities as raw strings extracted from the set of papers they examine, without ... actual linking to an external\", which is not the case. In fact, Kardas et al., (2020) also present a similar linking framework as the proposed one in this paper (linking cells to leaderboards/KB) while focusing on the result tuples (task, dataset, metric, score) instead of the entities. To me, this paper is more like an extension of Kardas et al., (2020) to the entity level with more fine-grained sub-task definitions, e.g., the inclusion of Attributed Source Matching and Candidate Entity Retrieval, and showing more recent Transformer models' performance. The proposed dataset is constructed upon the SegmentedTables dataset proposed by Kardas et al., (2020). Moreover, the AxCell model proposed by Karas et al., (2020) can be potentially used as an end-to-end baseline for this paper, where outKB mentions are entities that have low linking scores (below a threshold).\n\nReference:\n\nKardas et al., (2020), AXCELL: Automatic Extraction of Results from Machine Learning Papers", "questions_to_authors": "1. What is the \"Paper-RelatesTo-Entity relation in PwC\" (lines 303-304)?", "paper_topic_and_main_contributions": "This paper presents a human-labeled dataset for entity linking in machine-learning result tables and a corresponding pipeline neural baseline. The unique feature of this dataset is that it explicitly addresses the problem of \"outKB mentions\", which refers to entities not yet in the reference KB.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hnFThGFKrw", "reviewer": "Reviewer_CtHC", "summary": "", "questions": "A.\tAre the internal cells - usually the numerical values of the metrics for a given method and dataset - extracted and linked?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very detailed feedback. We acknowledge the concerns raised and aim to address each one.\n\n1. **Internal cells**: We consider all cells of a table in our task, as sourced from the SegmentedTable dataset from Kardas et al., (2020) (Lines 257-258).  This dataset extracts tables into two-dimensional arrays, and each cell is annotated according to whether it is a dataset, metric, and so on (Lines 258-260). This dataset also classifies the topic of a paper into one of eleven categories (e.g, NLI and Image Generation). In our Cell Type Classification sub-task, cells that do not refer to datasets, metrics, or methods are identified (e.g., the numeric value cells). These cells, thus, are not considered in subsequent Entity Linking steps (Lines 175-179).\n2. **Knowledge Base (KB) description**:  We concur with your observation about the need for a clearer presentation of the KB. We'll enhance this description in the revised manuscript. We believe this is a beneficial presentation improvement, but not a technical issue. There was also some confusion around the “topics”. We used the topic classification as it is from the SegmentedTable dataset.\n3. **Discussion of table structure**: We used tables extracted in the SegmentedTable dataset from Kardas et al., (2020) (Lines 257-258). We acknowledge the presence of cells with diverse grid structures and have used them as parsed by the SegmentedTable, including their approach to multicolumn cells. We will add further discussion of this processing in our revisions.\n4. **IAA score**: We'd like to highlight that a Cohen Kappa score in the 0.61–0.80 range is deemed as \"substantial agreement\" as per McHugh (2012). We stopped fine-tuning the guidelines once this agreement level was achieved. The location of the guideline is given on Line 954.  Our investigation showed that the main cause of disagreement in the EL phase was that there were cells whose matching entities were confusing to the annotators, due to their insufficient background in the specific academic area and/or the paper not clearly indicating which entity it was and/or whether should be considered a variant of an existing entity or a different entity entirely. We will add a discussion of this in the paper or its appendix.\n5. **STabEL Method & Out-of-KB Entities**: We appreciate your insight into the distinction between closed-world assumptions and KB structure. We'll ensure the manuscript accurately represents our non-hierarchical \"flat\" KB and provides a more fair and balanced discussion on different types of Entity Linking.\n6. **Reproducibility**: We have provided as supplementary material our dataset along with the code to reproduce our main results. If there are any other specific reproducibility concerns leading to the current score (1), we would be very happy to address them.\n\nLastly, we genuinely appreciate the suggestions in the “Typos Grammar Style And Presentation Improvements” section. These insights will guide us in refining the manuscript.\n\nThank you for your time and consideration.\n\nReferences:\nKardas et al., (2020), AXCELL: Automatic Extraction of Results from Machine Learning Papers\nMcHugh (2012), Interrater reliability: the kappa statistic", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The aim of the work, i.e. to extract information from experimental result tables, is relevant because comparing methods on reference datasets has become a very important part of machine learning research. Linking the methods and datasets cited in the tables to the primary source of information is an important step towards reproducibility and fair comparison. Papers with code website is becoming a popular source of information for this comparison. The table mentions are highly ambiguous, mostly acronyms or abbreviations that require external text analysis to disambiguate. The dataset and the method proposed to solve this problem are relevant. The experimental protocol is detailed.", "reasons_to_reject": "The main issue is the lack of clarity on the precise objective of the task. The first point is related to the cells of the table that are effectively considered. All the examples are about the names of columns and rows, and the entity types are method, dataset, metric, and dataset&metric. It is not clear at all if the internal cells, usually the numerical values of the metrics for a given method and dataset, are also extracted and linked. Column and row names should absolutely not be called cell content if internal cells are not processed. If they are, the way the right column and row are detected from the table structure, and the way the values are obtained should be discussed (taken from the primary source cited, or reproduced by the authors). The entities of the metrics type are not normalized with the KB (line 331), this should be mentioned in the task definition section. \nThe second major problem relates to the reference KB the description of which is seriously insufficient for the reader to understand the aim of the work, its challenges, and limitations. The KB components, data types, relationships, and content must be formally described and related to the table content. The generic URL of the paperwithcode website is not enough. The canonicalized ontology (line 158) and the taxonomy (line 162) must be defined. The “topics” (section 5, line 474 and following) also play an important role in the method. The paper must explain how they are extracted from the KB, and whether they are a subset of the KB topics. Their full names should be given. It is not sufficient to list their abbreviation in Table 5. The relationships of the KB are mentioned as helpful to the method, but it is unclear what the relationships are, and how they are used by the method.\nThe lack of discussion on the structure of the table is a third shortcoming. The reader understands that the column and row names are extracted and linked to the KB independently of each other. Besides internal cell values, which are obviously dependent in 2-D table of the row and column meaning, row and column meaning is also dependent on their context within the table when tables are not just in the form of a simple grid. There are three levels of column names in Table 2 as an example. The way complex structures are taken into account by the EL method should be discussed.\nLess seriously, the low rate of inter-annotator agreement for EL (66%) is below the usual standard for this type of corpus. It is presented as satisfactory. The paper should discuss and analyze the reason for this, which may be due to the guidelines, or the lack of expertise of the annotators. Since the reader assumes that the paper content is not ambiguous, could the annotation difficulty be due to the KB, which could contain several possible references for the same entity?\nThe authors emphasize the ability of the STabEL method to detect out-of-the-knowledge-base entities compared to other methods that would not be able to do it. The authors seem to confuse about what is due to the closed-world assumption (KB incompleteness) and what is due to the KB structure. Usually, in information extraction, the entity is recognized together with its type. Depending on the tasks, the reference KB for a given type can be a flat non-hierarchical nomenclature where entity-linking consists of assigning a nomenclature entry to the entity whether there exists one. The reader understands that the scientific table entity linking task here belongs to this family. This is not more realistic or less realistic in scientific domains (Lines 54-60, lines 116-129). Many reference KB for a given type are hierarchical (diseases, anatomy parts, chemical compounds), and the entity-linking tasks consist of identifying the most relevant node, possibly general, which is much more relevant than considering the entity as out-of-knowledge base. Without judging the value of others, the authors should simply describe their own objective, EL with “flat” KB.\n", "questions_to_authors": "A.\tAre the internal cells - usually the numerical values of the metrics for a given method and dataset - extracted and linked?", "paper_topic_and_main_contributions": "This paper presents a new manually annotated dataset for the entity linking task in tables and a new method for automatically detecting and linking table cells. The task is dedicated to machine learning experimental results reported in tables. The entity to be linked are those that are defined in Paper with code website, other relevant entities are considered as out-of-knowledge-base. The dataset is annotated by two annotators in a single way with double-blind on a limited subset. The entity linking method is divided into four successive steps, namely extraction and typing of the mentions, identification of the reference paper for the mentions if relevant, and filtering a subset of mentions that are good candidates for the last entity linking step. The end-to-end method achieves better results than the general method TURL.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The goal of Section 2.2 Scientific IE could be revised to focus on the paper subject, i.e. scientific table, or ML method or dataset name extraction, entity-linking to KB, and remove more general references that are either not relevant or incomplete.\nLine 287, the URL of the guidelines must be given.\nLine 320, the number of articles must be given.\nLine 364, what is Appendix 9?\nLines 397-398, please explain how oversampling is achieved by shuffling the sentences.\nLine 453, the 0.5 threshold should be experimentally confirmed.\nThe second sentence of the Table 2 legend is redundant to the text and should be removed.\nLine 518 Have all errors been examined?\nLine 624 The expression “may limit its generalizability” is an understatement.\n"}}]

Author rebuttal:

Meta-review: The paper presents a new dataset for entity linking in scientific tables - a dataset of tables are annotated with links to the papers with code taxonomy using a custom annotation interface.  It appears this is the first work to construct such an entity linking dataset for tables in scientific papers.

As the authors discuss in the limitations section, the dataset only covers one domain (results tables in machine learning papers).  It would be nice to see more domains, but this still seems like an important contribution.
Decision: Accept-Main