Benchmarking Diverse-Modal Entity Linking with Generative Models
Year: 2023

Abstract:
Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding, or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image, and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenges of DMEL, facilitating future research on this task.

heading: 1 Introduction
text: Linking ambiguous mentions to unambiguous referent in a knowledge base (KB) such as Wikipedia, known as Entity linking (EL) (Shen et al., 2015), is an essential component for applications like question answering (Ferrucci, 2012; Chen et al., 2017; Lewis et al., 2020) and recommendation systems (Yang et al., 2018). Diverse-Modal Entity Linking (DMEL) extends the scope of interest from textual entity linking to heterogeneous input formats, such as linking visual and textual expressions to KB (Adjali et al., 2020b,a; Moon et al., 2018; Gan et al., 2021a; Zheng et al., 2022; Wang et al., 2022d; Gan et al., 2021a; Cui et al., 2021) and linking mentions in natural language to tables or database (DB)
∗Work conducted during an internship at Amazon. †Corresponding author.
schema (Liu et al., 2021; Katsakioris et al., 2022; Shi et al., 2020; Lei et al., 2020; Chen et al., 2020; Wang et al., 2022a). Figure 1 demonstrate three examples of DMEL, including (a) classical textual entity linking, (b) textual-visual entity linking in which the question or mentions are paired with image(s), and (c) tabular schema linking in which the mentions are linked to column names or cell values.
Retrieval-based contrastive learning or ranking mechanism is the mainstream for early visual entity linking by leveraging a matching score between the mention and the KB entities (Cui et al., 2021; Wang et al., 2022d; Zheng et al., 2022). However, these methods require storage of dense representations of all KB entities, and when the size of entities increases (e.g. Wikipedia has 6M articles), it raises concerns for space complexity and also inference-time latency. Meanwhile, linking mentions to tables or DB schemes, known as schema linking, remains an important but under-explored task. For example, in text-to-SQL generations, incorrect schema linking usually counts for a large portion of the errors (Zhong et al., 2017; Yu et al., 2018; Shi et al., 2020; Lei et al., 2020; Taniguchi et al., 2021). Previous string matching heuristic
7841
(Chen et al., 2020) or embedding matching methods (Chen et al., 2020; Wang et al., 2022a; Guo et al., 2019; Wang et al., 2020) lack semantic and schema understanding, and can hardly generalize well to new domains. Last but not the least, previous endeavors of entity linking are limited to individual tasks including textual EL, textual-visual EL, or schema linking, and lack a general view for the DMEL problem.
To this end, we propose a unified DMEL task that includes existing EL datasets on all three modalities – text, image, and table. The unified DMEL task is challenging because the model needs to handle a wide spectrum of modality configurations together. On the modeling side, because storing all entity information (e.g. all the images in the entire KB) is expensive at inference time, we propose to use a unified generative model that can take diverse-modal input and generate entity names in an autoregressive fashion. Additionally, the mention diversity and ambiguity issue in schema linking can be addressed by pre-training the generative model.
In this work, we build a generic diversemodal architecture for end-to-end DMEL. The DMEL dataset is constructed from five existing datasets, including GERBIL benchmark, WikiDiverse, MELBench-Wikipedia, Squall, and SLSQL, covering diverse EL tasks. The proposed generative diverse-modal model (GDMM) is first pretrained on large-scale text corpus BLINK and images corpus from Wikipedia KB, offering profound prior knowledge. Extensive experiments are then conducted on the DMEL benchmark to compare our proposed generative model to previous stateof-the-art methods. Experimental results show that GDMM achieves strong performance on the DMEL dataset and outperforms state-of-the-art task-specific EL models. Our contributions include:
• We define a novel diverse-modal Entity Linking task, which links an entity mention within heterogeneous information sources to a knowledge base. A unified dataset is constructed for rigorous DMEL examination.
• A generative diverse-modal model GDMM is proposed following a multimodal-encoderdecoder structure. The multimodal encoder allows collective representation between each modality. The autoregressive structure enables us to directly predict the entity name
without storing the entire KB. The pre-training experimental results confirm that a candidate trie created from entity names is sufficient for inference.
• The experimental results show that the proposed model obtains state-of-the-art performance on (almost) each individual EL task.

heading: 2 Problem Formulation
text: We assume to have a KB (e.g., Wikipedia or a DB schema) where each entity is a unique entry in the KB. We formulate the following DMEL problem: given a multimodal input {xi,vi,ui} of textual (L), visual (V), and tabular (U) modality respectively, an entity mention mi within the input, and a candidate set Ci = {c1i , · · · , cKi }, the task is to link the mention mi to one entity in Ci. We assume the entity span is given. Sometimes the candidate set can be the entire entity collection E . Particular instances of DMEL problem include but are not limited to: Textual Entity Disambiguation where a given mention mi in xi will be linked to one entity in Ci; Textual-Visual Entity Disambiguation where the mi and a given image vi will be linked to one entity in Ci; Schema linking where a mi in a SQL query will be linked to table schema, i.e., a column name within given tables ui. If the mention is not a valid entity or not in Ci, the target label is “nil”.

heading: 3 DMEL Benchmark
text: We build the DMEL benchmark from five existing datasets, including GERBIL benchmark (Verborgh et al., 2018), WikiDiverse (Wang et al., 2022d), MELBench-Wikidata (as MELBench in the rest of the paper) (Gan et al., 2021a), Squall (Shi et al., 2020), and SLSQL (Lei et al., 2020). We evaluate textual-visual entity disambiguation capability on WikiDiverse and MELBench, and evaluate tabular
schema linking on Squall and SLSQL. All datasets are in English, and we summarize the data statistics in Table 1. We will release this benchmark.
We assume that the mention span is given across all datasets, thus (1) on GERBIL benchmark, we investigate textual entity disambiguation using the same candidate sets as in Le and Titov (2018) and De Cao et al. (2021). (2) On WikiDiverse, we investigate entity disambiguation performance with retrieved Top-10 candidates by Wang et al. (2022d) (3) On MELBench-Wikidata, we follow the original setting in which the given entity mention will be linked to its referent in the knowledge base. (4) On Squall, the given entity mention in the natural question will be linked to column names in the target table. (5) On SLSQL, the entity mention in the natural question will be linked to the column name within multiple tables.
Statistics for each individual EL task are shown in Table 2. Note that for the GERBIL benchmark, the training split refers to AIDA-train, the validation split refers to AIDA-dev, and the test split includes all test split as in (De Cao et al., 2021).

heading: 4 GDMM Model
text: We build a generative entity linking model that enables diverse-modal vision, language and table understanding and inference. We show how this can be achieved with a generative encoder-decoder structure.

heading: 4.1 Input Processor
text: As shown in Figure 2, our model can process inputs of three modalities, including texts, images, and tables. Formally, given a multimodal input {xi,vi,ui}, the input processor serves to encode the data and group the modalities as follows. (1) Text Given an input text xi, we first tokenize and embed it into a list of word vectors xi following Devlin et al. (2019). (2) Image Given an input image vi, we first resize it to a fixed size and split it into patches, following Kim et al. (2021). (3) Table We follow the table representation proposed in TAPEX (Liu et al., 2022). The table is flattened and represented as u∗i = [head], col1, · · · , colM , [ROW], 1, cell11, · · · , cell1M , [ROW] · · · , where [head] and [ROW] are special tokens denoting the beginning of table headers and rows, and the number after [ROW] is used to denote the row index. cellij represents a cell in ith row and jth column. Then the table representation u∗i will be tokenized and embedded into ui. Finally, given the multimodal input {xi,vi,ui}, our input processor outputs {xi ⊕ ui, v i}, where ⊕ is a concatenation operator.

heading: 4.2 GDMM Model Architecture
text: Multimodal Encoder The multimodal encoder consists of an image encoder, a text encoder, and a fusion encoder, following previous work (Singh
et al., 2022; Yang et al., 2022). The text encoder and image encoder use the same ViT architecture (Dosovitskiy et al., 2021) with different parameters. The text input and visual input {xi ⊕ ui, v i} are passed into the text encoder and vision encoder individually. The text hidden state vectors {hLUi } and the image embeddings {hVi } are then projected and concatenated into a single list. A fusion encoder is applied to the concatenated list, which allows cross-attention between the projected unimodal representations and fuses the two. The output is a list of hidden states {hMi }. The multimodal encoder parameters are initialized with pre-trained FLAVA (Singh et al., 2022) parameters.
Decoder We exploit the transformer architecture (Vaswani et al., 2017) for the decoder. Previous study (Rothe et al., 2020) points out that combining models with same vocabulary has stronger overall performance, thus we initialize the decoder with the BERT (Devlin et al., 2019) pre-trained parameters.
Training and Inference The GDMM is trained with standard autoregressive objective, i.e., maximizing the output sequence likelihood pθ(y i|xi, v i,ui) with respect to the model’s parameters θ. We rank each candidate cki ∈ Ci by computing a score with an autoregressive formulation: score(cki |xi, v i,ui) = pθ(yki |xi, v i,ui) =∏N
j=1 pθ(y k j |yk<j ,xi, v i,ui), where N is the number of tokens of cki . If the score is lower than a threshold θ, the prediction becomes “nil”. The threshold will be decided by the development set.
Constrained decoding When the candidate set Ci is very large (e.g., the entire entity space E), naturally, it is intractable to compute a score for every element. Thus we exploit Constrained Beam Search (Sutskever et al., 2014; De Cao et al., 2021), a tractable decoding strategy to efficiently search the valid entity space. It is tractable as the average time cost depends on beam size and the average length of entity representations (e.g. 6 BPE tokens on average for entities in Wikipedia KB), instead of the size of Ci. An entity trie Ti for Ci will be created so that the output is limited to the target space. The constraint is defined as, for each node t ∈ T , its children indicate all allowed continuations from the prefix traversing from root to t. For example, as shown in Figure 2, given four candidates Manchester United F.C., Manchester City F.C., Manchester City W.F.C, and City College Manchester, a candidate trie will be cre-
ated as shown in the figure. The decoding will strictly follow the top-down order in the trie with a certain beam size.

heading: 4.3 Pre-training GDMM
text: Pre-training is critical to our architecture though the encoder and decoder are initialized with pretrained weights because the mapping between the encoder and decoder are randomly initialized and they have not been pre-trained simultaneously with the encoders and the decoder.
Pre-training data A pre-training corpus is constructed from BLINK (Wu et al., 2020) and images in Wikipedia KB. BLINK is a commonly used corpus for textual entity linking pre-training, including 9M unique annotations of document-mentionentity triples from Wikipedia. Meanwhile, the images in Wikipedia KB are naturally linked to their respective entity names. The two together are well-suited for pre-training DMEL models. Aside from text-only BLINK, we construct LVpaired pre-training data by linking BLINK and Wiki-images. An image pool (Wiki-images) is collected from Wikipedia KB if the entity can be linked to mentions in BLINK. The image pool contains 797,436 downloaded images of 495,149 entities in Wikipedia KB. We then randomly attach an image of the target entity, if exists, to each mention in BLINK. In total, the LV-paired pre-training data includes 5,445,264 mentions and 678,385 distinct images in the training set, and 5,816 mentions and 5,414 images in the development set.
Pre-training details We pre-train GDMM on text-only BLINK and LV-paired pre-training data in two stages. Note that not all the BLINK entities appear in Wiki-images. There are over 2.5M BLINK mentions not covered by LV-paired pretraining data. To fully leverage the BLINK annotations, we first pre-train on text-only BLINK and then pre-train on the LV-paired data. With text-only BLINK, we freeze parameters in the image encoder and fusion layers and only update parameters in the text encoder and decoder. In the second stage, all the parameters are updated.

heading: 4.4 Unified Learning
text: Upon the pre-trained model, one straightforward strategy for downstream tasks is single-task finetuning. We take one step further and investigate unified learning. Specifically, we’ll investigate (1) single-task finetuning (ST-F), which refers to
finetuning on individual tasks; (2) multi-task finetuning (MT-F) which combines the mixed training data of all datasets (Raffel et al., 2022); and (3) multitask fine-tuning with prefixes (MT-FP), where we prepend task-specific prefixes like “entity linking” and “schema linking” to the input context.

heading: 5 Experiments
text: Model Variants We primarily report results on two model variants: GDMM-base where the decoder is initialized with BERT-base parameters, and GDMM-large where the decoder is initialized with BERT-large parameters. To investigate which modality provides dominant information for visual-text entity linking, three configurations are explored: L+V where both visual and textual information are given, L where only textual input are given, and V where only image are given. We report experimental results with a single generic model for the three modality settings. It is achieved by randomly masking out one modality during training. Implementation details are in Appendix B.

heading: 5.1 Results
text: Pre-training The pre-training of GDMM consists of two stages, text-only pre-training and LVpaired pre-training. The pre-training performance is investigated with two methods: zero-shot or fine-tuned on WikiDiverse. Zero-shot refers to directly evaluating on WikiDiverse without training, while fine-tuned refers to further fine-tuning on the WikiDiverse. The first-stage pre-trained checkpoint is directly evaluated on WikiDiverse with only text information (note that WikiDiverse is a dataset with both image and text input), achieving
75.43 zero-shot F1 score. It greatly outperform the baseline model in (Wang et al., 2022d) by 4.36 F1 score, even though only text information are leveraged. The evaluation demonstrates that pretraining on BLINK builds a strong foundation for the proposed model. After that, we investigate the effect of paired pre-training data size in the second stage and visualize it in Figure 3. It shows that the pre-training data size has a positive effect on inference with both image and text modality (L+V) and with only image modality (V). Text-only (L) performance is not affected even though the visual modality is introduced in the second pre-training stage.
Experimental results on DMEL benchmark The experimental results of the proposed GDMM on DMEL are shown in Table 3. We compare visual-language entity disambiguation result on WikiDiverse with LXMERT (Wang et al., 2022d), and visual-language entity linking performance on MELBench with Gan et al. (2021a). GDMM achieves better performance on both datasets, especially on MELBench. GDMM strikingly improves the F1 score by over 31%, demonstrating the effectiveness of the proposed architecture.
The schema linking performance is evaluated on Squall and SLSQL. For schema linking, we compare our model with the baseline model GENRE (De Cao et al., 2021), as it has competitive performance in entity disambiguation. For a fair comparison, we fine-tune GENRE with their pretrained checkpoint on BLINK and investigate two options, one without a flattened table (GENRE) and another with the flattened table (GENRE+) where the table content is leveraged identically as in GDMM. Only experimental results for GENRE+ are reported in Table 3 since it has better performance than GENRE. The fact that the flattened table has better performance demonstrates the effectiveness of the table representation. Detailed results can be found in Appendix C.
Unified learning As mentioned in Section 4.4, we report unified learning results for ST-F, MT-F, and MT-FP in Table 4. To confirm whether the pre-trained checkpoints build a competitive foundation for visual-language entity linking, zero-shot (ZS) performance is also reported in the same table. The pre-trained checkpoint is competitive because the zero-shot performance (i.e. ZS column) outperforms previous state-of-the-art fine-tuned re-
sults for WikiDiverse in Table 8 and MELBench in Table 9. On average, ST-F and MT-FP achieve the best and the second-best performance, with a small gap between the two. It is expected that ST-F achieves the best performance as each fine-tuned model is able to fit the target dataset distribution. Considering ST-F trains five models while MT-FP trains a single model, the competitive MT-FP performance suggests that model efficiency can be achieved at the cost of a minor performance drop, that is, 0.30 average F1 score drop for GDMMbase. Additionally, the fact that MT-FP constantly outperforms MT-F aligns with previous findings that task-specific prefixes is effective in informing the model of the target tasks (Dong et al., 2019; Raffel et al., 2022).

heading: 5.2 Error Analysis
text: Figure 4 shows the error breakdown on WikiDiverse. The errors are divided into four categories: retrieval error where the target entity is not in the candidates; misidentification where the prediction does not match the ground truth entity; under predict where the model predicts “nil” and the ground truth entity is not “nil”; over prediction where the ground truth entity is “nil”.
Representative error examples are presented in Table 5 for (a) retrieval error, (b) misidentification, (c) over prediction, and (d) under prediction. Error type (a) contributes to over half of the errors, emphasizing the need for a good retriever. It cannot be addressed by our model, because the ground truth entity is not in the set. Example (b) is due to candidate confusion, as Cape Canaveral Air Force Station is a previously used name for Cape Canaveral Space Force Station from 1974 to 1994 and from 2000 to 2020. Such errors indicate the necessity for a coreference system at inference time. The over-prediction example as shown in (c) calls for a better discrimination strategy for plausible candidates. Example (d) is a challenging example, which asks future models to possess more profound prior knowledge.
Table 6 further shows four types of errors for schema linking, name ambiguity, inference difficulty, prime key confusion, and unknown strings. (a) Name ambiguity is a common challenge for schema linking especially when the column names have overlapped tokens. (b) Sometimes the model fails to make inferences on subtle entity expressions. (c) Another common error type for schema linking is the confusion in prime keys, as the prime key “pet id” is shared by multiple tables in the example. (d) Another challenge is unknown strings or composite tokens since it is usually intractable to recover the original expression from those mentions.

heading: 5.3 Ablation Study
text: We further discuss the indispensability of information within each modality through an ablation study on WikiDiverse. Three evaluations are conducted under settings: (1) L+V where both text and image are leveraged; (2) L where only texts are used for prediction; (3) V where only images are used for prediction. The experimental results are reported in Table 8. It shows each model achieves the best performance with both text and image modalities (L+V). While textual content provides the most inference clues, visual content provides complementary information. Additionally, the proposed model GDMM outperforms LXMERT with a single generic model trained for various con-
figurations, while three modality-specific models for each configuration (L+V, L, V) are trained in LXMERT. This observation demonstrates the effectiveness of the proposed model for various modality configurations.
Table 7 shows several misinformation examples where the model fails without visual information. In (a), the image of a soccer stadium provides extra semantics when the model misses the semantic indicator from “the 2006 World Cup.” The image in example (b) is in Wiki-KB and has been used for pre-training. Its visibility during the pretraining makes the image a strong indicator of the target entity. The optical character “NEW SCOTLAND YARD” in example (c) is indispensable for the mention to be correctly identified. Without the
optical features, inferring the mention of “headquarters” is challenging given the ambiguous text. Examples (d) and (e) emphasize the dependence on facial features for entity disambiguation. In the absence of the image in (d), it is impossible to disambiguate between “Cristiano Ronaldo” and “Ronaldo ( Brazilian Footballer )” as both players served Real Madrid.

heading: 6 Related Works
text: Textual Entity Linking Early entity linking researches (Hoffart et al., 2011; Daiber et al., 2013) reply on probabilistic approaches, based on textual similarity and corpus occurrence. A more recent line of research is neural networks based retrievalreranking approaches (El Vaigh et al., 2020; Zhang et al., 2022; Mrini et al., 2022), which first retrieve top candidates given the input text, and then score each candidate with semantic similarity or correlation. End-to-end entity linking models (Broscheit, 2019; Martins et al., 2019; El Vaigh et al., 2020) approach this problem by directly detecting the entity mentions and linking them to their corresponding entities in the KB. For example, autoregressive entity linking models (De Cao et al., 2021; De Cao et al., 2021; Petroni et al., 2021; Mrini et al., 2022) formulate entity linking as a language generation problem using an encoder-decoder model.
Textual-Visual Entity Linking The growing trend towards multimodality significantly advanced research in multimodal entity linking. Due to the difficulty in collecting and cleaning multimodal entity linking data, previous researchers limit their attention to a specific domain such as social media data (Adjali et al., 2020b,a; Moon et al., 2018; Gan et al., 2021a) and news domain(Zheng et al., 2022; Wang et al., 2022d), or a limited scope like person and organization recognition (Gan et al., 2021a; Cui et al., 2021). Previous work (Wang et al., 2022d) represents each entity with one image, which limits the visual expression of entities. We
overcome this limitation by pre-training GDMM with multiple images per entity to obtain diverse visual representations.
Tabular Schema Linking Schema linking (Guo et al., 2019; Wang et al., 2020) is an instance of entity linking in the context of linking to the relational database schema. Previous research shows that good schema linking (Liu et al., 2021; Katsakioris et al., 2022; Shi et al., 2020; Lei et al., 2020; Chen et al., 2020) can substantially improve downstream tasks such as Text-to-SQL parsing. However, entity mentions in existing benchmarks such as Spider (Yu et al., 2018) can almost exactly match the corresponding schema entities (Chen et al., 2020). Therefore, current Text-to-SQL semantic parsers normally address this problem with string-matching heuristics (Chen et al., 2020) or embedding matching modules (Chen et al., 2020; Wang et al., 2022a; Guo et al., 2019; Wang et al., 2020). However, due to the diversity and ambiguity in natural language mentions, such heuristics are hard to generalize to new domains (Chen et al., 2020; Wang et al., 2022a).
Multimodal Models Multimodal models have attracted increasing attention in computer vision and natural language processing communities. Recent transformer-based approaches (Kim et al., 2021; Radford et al., 2021; Singh et al., 2022) that leverage the attention between the visual and textual embeddings manifest the effectiveness of the attention mechanism. However, the proposed learning objectives are usually limited to predefined scopes, such as text-image matching or alignment (Xu et al., 2021; Radford et al., 2021; Biten et al., 2022; Ho et al., 2022; Li et al., 2022; Yang et al., 2022; Huang et al., 2023), semantic segmentation, object detection, classification (Xu et al., 2022; Guo et al., 2022; Assran et al., 2022), and masked language modeling (Li et al., 2020; Ni et al., 2022; Tong et al., 2022; Appalaraju et al., 2021). Instead, we proposed a generic generative model that is open to diverse downstream tasks. Additionally, GDMM differs from previous generative multimodal models(Li et al., 2023; Wang et al., 2021) in that GDMM can process and comprehend information from heterogeneous instead of a single source; GDMM differs from VL-T5 (Cho et al., 2021) and others (Wang et al., 2022b; Bao et al., 2022; Wang et al., 2022c) in that GDMM enables thorough encoding for each modality, instead of
discretizing visual content.

heading: 7 Conclusion
text: In this paper, a novel DMEL problem is formulated, which links the entity mention within heterogeneous information to a defined KB. A generic DMEL dataset is built covering diverse EL tasks. We propose a unified generative model for DMEL, GDMM. Comprehensive experiments are conducted over the DMEL dataset. Experimental results show that the proposed GDMM outperforms state-of-the-art models on almost each individual EL task.
Broader Impact In contrast to previous work (Pan et al., 2022; OpenAI, 2022) that only leverage textual content, the proposed model has the potential to deal with misinformation (text only as a data source might be prone to misinformation or fake context/information). This research will lead to a clearer understanding of misinformation issues and encourage better leverage of multimodal information.
Limitations
GDMM establishes a compelling starting point for DMEL research. In spite of this, the proposed approach has several shortcomings. First, GDMM currently generates entity name within the entity candidate set, however, we saw how retrieval errors limit entity linking performance. Thus, how to work collectively with the retrieval system to diminish errors takes appropriate action. Second, how to handle large tables still remains under-explored. It is infeasible to represent a huge database with the table flattening technique. In practice, it is possible to filter out less likely candidates to compress the search space, but a more promising approach is to represent the table more efficiently.
GDMM also enables studies on more diversemodal tasks. New tasks can be easily framed based on the proposed architecture, such as visual question answering, grounded generation, and diversemodal commonsense reasoning. We believe that with more follow-up work on diverse tasks, this approach will turn out to be a more comprehensive generative diverse-modal framework.

heading: A DMEL Selection
text: The DMEL benchmark’s datasets were carefully chosen after conducting extensive research on publicly available datasets. The DMEL benchmark includes five datasets: GERBIL, WikiDiverse, MELBench, Squall, and SLSQL. Each of these datasets is necessary and represents the best option for a comprehensive evaluation for Diverse-Modal Entity Linking. The selected datasets best align with the DMEL problem, among more than 20 datasets we looked into. The reasons why other datasets are excluded in the benchmark are: i) most dataset collected from social media cannot be reproduced because some of the data are no longer accessible. This category includes Twitter-MEL (Adjali et al., 2020b), SnapCaptionsKB(Moon et al., 2018); ii) Dataset is not publicly available (Zheng et al., 2022); iii) Annotated dataset for schema linking is limited. Existing work that investigates entities in tables are Text-to-SQL datasets. However, annotations for schema linking are not available, such as Spider (Yu et al., 2018), Spider-Syn (Gan et al., 2021b), Spider-DK (Gan et al., 2021c), WikiSQL(Zhong et al., 2017), ATIS (Price, 1990), Freebase917 (Cai and Yates, 2013), and WikiTableQuestions (Pasupat and Liang, 2015). Furthremore, there is no overlap between the datasets. For tabular datasets, Squall is built upon WikiTableQuestions, while SLSQL is based on the Spider text-to-SQL dataset. Lastly, GENRE is widely recognized as the standard dataset for the task of textual entity linking.
B Implementation Details
For every dataset in DMEL dataset, the fine-tuning procedure runs for 5 epochs with a batch size of 16. For both pre-training and fine-tuning, the learning rate is 3 × 10−5, with a linear scheduler with 0.1 warmup ratio. Fine-tuning takes 5 hours for WikiDiverse, 2 for MELBench-Wiki, 1 for Squall, and 2 for SLSQL.
We run each setting 5 times and report the mean and variance unless stated otherwise (except the zero-shot setting when evaluated with the pretrained checkpoint, since there is no randomness with the pre-trained checkpoint ). One MELBench, since the dataset split is not given along with the released MELBench-Wikidata data, we randomly split the dataset according to their split statistics and repeat the experiment 5 times to get average evaluation metrics. On Squall, we reported a 5-fold
cross-validation result following the released split. For Squall and SLSQL, all the hyperparameters are tuned on the training set since there is no test set.
Additionally, Wikipedia images are collected through hyperlinks shared by Wang et al. (2022d) at https://github.com/wangxw5/wikidiverse. Annotations on SLSQL are adapted from Spider, excluding train_others.json that are from Restaurants, GeoQuery, Scholar, Academic, IMDB, and Yelp prepared by (Finegan-Dollak et al., 2018). For GERBIL benchmark results, we report average F1 scores on six test sets, including Aidatest, MSNBC-test AQUAINT-test, ACE2004-test, WNED-CWEB-test, and WNED-WIKI-test following De Cao et al. (2021).

heading: C Detailed Experimental Results
text: Experimental results for each individual dataset are shown in this section. Specifically, Table 9 shows results for MELBench, Table 10 shows the experimental result for Squall, and Table 11 shows the experimental result for SLSQL.

heading: D Domain Adaption Results
text: We investigate zero-shot performance in unseen domains on the WikiDiverse dataset. Specifically, we choose the five domains as seen domains, including politic, crime, sports, entertainment, and technology, and the rest five domains as the unseen domains, including disaster, health, economy, weather, and education. Training data includes instances from the seen domain, and the instances from the unseen domains are randomly split into validation and test set. Note that the data used for this experiment is from the WikiDiverse training and validation set, the data in the test set are excluded.
The domain adaption result on WikiDiverse is shown in Table 12. These experiments facilitate studying knowledge transfer between seen and unseen domains. The experiment results show that (a) pre-training is indispensable for new domains as it provides profound prior knowledge for the MEL task in general; (b) knowledge learned from seen domains can indeed transfer to the unseen domain as the average F1 score improves by 3.61 percentage points.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 5 and Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 3, 4, 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 3, 4, 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix B

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix B
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 and Appendix B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Appendix B
C 3 Did you run computational experiments? Section 5 and Appendix B

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5 and Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Example (d) is a challenging example, which asks future models to possess more profound prior knowledge. Table 6 further shows four types of errors for schema linking, name ambiguity, inference difficulty, prime key confusion, and unknown strings. (a) Name ambiguity is a common challenge for schema linking especially when the column names have overlapped tokens. (b) Sometimes the model fails to make inferences on subtle entity expressions. (c) Another common error type for schema linking is the confusion in prime keys, as the prime key “pet id” is shared by multiple tables in the example. (d) Another challenge is unknown strings or composite tokens since it is usually intractable to recover the original expression from those mentions