LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models (2024)

Abstract: Many computational analyses require linking information across noisy text datasets. While large language models (LLMs) offer significant promise, approximate string matching in popular statistical softwares such as R and Stata remain predominant in academic applications. These packages have simple interfaces and can be easily extended to a diversity of languages and settings, and for academic applications, ease-of-use and extensibility are essential. In contrast, packages for record linkage with LLMs require significant familiarity with deep learning frameworks and often focus on applications of commercial value in English. The open-source package LinkTransformer aims to bridge this gap by providing an endto-end software for performing record linkage and other data cleaning tasks with transformer LLMs, treating linkage as a text retrieval problem. At its core is an off-the-shelf toolkit for applying transformer models to record linkage. LinkTransformer contains a rich repository of pre-trained models for multiple languages and supports easy integration of any transformer language model from Hugging Face or OpenAI, providing the extensibility required for many scholarly applications. Its APIs also perform common data processing tasks, e.g., aggregation, noisy de-duplication, and translation-free cross-lingual linkage. LinkTransformer contains comprehensive tools for efficient model tuning, allowing for highly customized applications, and users can easily contribute their custom-trained models to its model hub to ensure reproducibility. Using a novel benchmark dataset geared towards academic applications, we show that LinkTransformerwith both custom models and Hugging Face or OpenAI models off-the-shelf outperforms string matching by a wide margin. By combining transformer LMs with intuitive APIs, LinkTransformer aims to democratize these performance gains for those who lack familiarity with deep learning frameworks.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 221–231 August 11-16, 2024 ©2024 Association for Computational Linguistics
Many computational analyses require linking information across noisy text datasets. While large language models (LLMs) offer significant promise, approximate string matching in popular statistical softwares such as R and Stata remain predominant in academic applications. These packages have simple interfaces and can be easily extended to a diversity of languages and settings, and for academic applications, ease-of-use and extensibility are essential. In contrast, packages for record linkage with LLMs require significant familiarity with deep learning frameworks and often focus on applications of commercial value in English. The open-source package LinkTransformer aims to bridge this gap by providing an endto-end software for performing record linkage and other data cleaning tasks with transformer LLMs, treating linkage as a text retrieval problem. At its core is an off-the-shelf toolkit for applying transformer models to record linkage. LinkTransformer contains a rich repository of pre-trained models for multiple languages and supports easy integration of any transformer language model from Hugging Face or OpenAI, providing the extensibility required for many scholarly applications. Its APIs also perform common data processing tasks, e.g., aggregation, noisy de-duplication, and translation-free cross-lingual linkage. LinkTransformer contains comprehensive tools for efficient model tuning, allowing for highly customized applications, and users can easily contribute their custom-trained models to its model hub to ensure reproducibility. Using a novel benchmark dataset geared towards academic applications, we show that LinkTransformer- with both custom models and Hugging Face or OpenAI models off-the-shelf - outperforms string matching by a wide margin. By combining transformer LMs with intuitive APIs, LinkTransformer aims to democratize these performance gains for those who lack familiarity with deep learning frameworks.

heading: 1 Introduction
text: Linking information across sources is fundamental to a variety of analyses in social science, business, and government. A recent literature, focused on matching across e-commerce datasets, shows the promise of transformer large language models (LLMs) for improving record linkage (alternatively termed entity resolution or approximate dictionary matching). Yet these methods have not yet made widespread inroads in social science applications, with rule-based methods continuing to overwhelmingly predominate (e.g., see reviews by Binette and Steorts (2022); Abramitzky et al. (2021)). In particular, researchers commonly employ stringbased matching tools available in statistical software packages such as R or Stata.
In academic applications, extensibility to a diversity of human societies (historic and present) and ease of use for those not familiar with deep learning are essential. String matching algorithms in widely used statistical packages meet these requirements because they require little coding expertise and can easily be applied across different languages and settings. In contrast, existing tools for large language model matching require considerable technical expertise to implement. This makes sense in the context for which these models were developed - classifying and linking products for e-commerce firms, which employ data scientists - but it is a significant impediment for scholarly use.
To bridge the gap between the ease-of-use of widely employed string matching packages and the power of modern LLMs, we developed LinkTransformer, a general purpose, user friendly package for record linkage with transformer LLMs. LinkTransformer treats record linkage as a text retrieval problem (See Figure 1). The API can be thought of as a drop-in replacement to popular dataframe manipulation frameworks like pandas or tools like R and Stata, catering to those
221
who lack extensive coding experience. LinkTransformer integrates:
1. An off-the-shelf toolkit for applying transformer models to record linkage
2. A rich repository of pre-trained models, supporting multiple applications and languages and evaluated on novel social science-oriented benchmarks
3. Easy integration of any Hugging Face or OpenAI transformer LLM, for extensibility
4. APIs to support common data processing tasks: aggregation, de-duplication, classification, and translation-free cross-lingual linkage
5. Comprehensive tools for model tuning
6. Easy sharing to the LinkTransformer model hub, as reproducibility is essential for academic applications
While transfer learning can facilitate strong off-the-shelf performance, heterogeneity in how out-of-domain applications are from LLM training corpora - combined with settings that demand extremely high accuracy - create many scenarios where custom training may be needed. LinkTransformer makes it straightforward for users to tune their own customized models. LinkTransformer performs well on challenging record linkage applications. It is equally applicable to tasks with a single field - e.g., linking 1940s Mexican tariff product classes across time - and applications that require concatenating an array of noisily measured fields - e.g., linking 1950s Japanese firms across different largescale, noisy databases using the firm name, location, products, shareholders, and banks. This
type of linkage problem would be highly convoluted with traditional string matching, as there are many noisily measured fields (e.g., products can be described in different ways, different subsets of managers and shareholders are listed, etc). Using LinkTransformer to automatically concatenate the information and feed it to a LLM handles these challenges with ease. LinkTransformer has a GNU General Public License and is being actively maintained. A demo is available at https://youtu.be/hFrh8k1pukI. More resources are available on our package website https://linktransformer.github.io/.
This study is organized as follows: Section 2 provides an overview of related work. The core LinkTransformer library is described in Section 3. Section 4 evaluates LinkTransformer performance on various use cases, and Section 5 considers ethics.

heading: 2 Relation to the Existing Literature
text: The record linkage literature is sprawling - with large literatures in quantitative social science (particularly economics), statistics, computer science, and industry applications. These literatures are highly disjoint, taking very different approaches and even using different terms (record linkage, entity resolution/matching, approximate dictionary matching, etc.) to refer to the same task. A 2022 interdisciplinary Science Advances review, “(Almost) All of Entity Resolution” (Binette and Steorts, 2022), concludes that deep neural models are unlikely to be applicable to record linkage using structured data. It argues that training datasets are small and there is not much to be gained from LLMs since text fields are often short. Yet there is an active literature on e-commerce applications that underscores the utility of LLMs for linking structured datasets, even when text fields are short. Bench-
marks in this literature (e.g., Köpcke et al. (2010); Das et al. (2015); Primpeli et al. (2019)) focus on high resource commercial applications in English, such as matching electronics and software products between Amazon-Google and Walmart-Amazon listings, matching iTunes and Amazon music listings, and matching restaurants between Fodors and Zagat. Recent studies have used masked language models (Li et al., 2020; Joshi et al., 2021; Brunner and Stockinger, 2020; Zhou et al., 2022), GPT (Peeters and Bizer, 2023; Tang et al., 2022), or both, significantly outperforming static word embedding and other older linkage methods.
The siloed nature of the literature is reflected in softwares. The main existing package for record linkage with LLMs is Ditto (Li et al., 2020), which implements Li et al. (2023). It requires significant programming expertise to deploy. While this is appropriate for an e-commerce target audience - where data scientists predominate - the technical expertise required and the lack of pre-trained models targeted to multilingual social science applications has likely hindered further takeup. Moreover, most of the literature examining record linkage with LLMs poses record linkage as a classification task (Barlaug and Gulla, 2021), which is appropriate for the e-commerce benchmarks. However, this significantly limits extensibility, as in many social science and government applications the number of entities to be linked numbers in the millions, making it computationally infeasible to compute a softmax over all possible classes (entities). In the social sciences, string matching with statistical packages predominates. LinkTransformer frames record linkage as a knn-retrieval task, in which the nearest neighbor for each entity in a query embedding dataset is retrieved from a key embedding dataset, using cosine similarity implemented with an FAISS backend (Johnson et al., 2019). LinkTransformer includes functionality to tune a no-match threshold - since not all entities in the query need to have a match in the key - and allows for retrieving multiple neighbors, to accommodate many-tomany matches between the query and the key. The LinkTransformer architecture was inspired by biencoder applications with unstructured texts, e.g., passage retrieval (Karpukhin et al., 2020), entity disambiguation (Wu et al., 2019), and entity coreference resolution (Hsu and Horwood, 2022). The knn retrieval structure of LinkTransformer also supports noisy de-duplication, a closely re-
lated task that finds noisily duplicated observations within a dataset, following the methods developed in Silcock et al. (2023). LinkTransformer departs from much of the literature in utilizing LLMs trained for semantic similarity, combined with a supervised contrastive loss (Khosla et al., 2020). Off-the-shelf LLMs such as BERT have anisotropic geometries (Ethayarajh, 2019), which makes them unsuitable off-the-shelf for metric learning problems like LinkTransformer nearest neighbor retrieval. Contrastive training for semantic similarity reduces anisotropy, improving alignment between semantically similar pairs to be linked and improving sentence embeddings (Wang and Isola, 2020; Reimers and Gurevych, 2019). LinkTransformer builds closely upon the contrastively trained Sentence BERT (Reimers and Gurevych, 2019), whose semantic similarity library inspired many of the features in LinkTransformer.
3 The LinkTransformer Library

heading: 3.1 Off-the-shelf Toolkit
text: At the core of LinkTransformer is an off-the-shelf toolkit that streamlines record linkage with transformer language models. The record linkage models enable using pre-trained or self-trained transformer models with minimal coding required. Any Hugging Face or OpenAI model can be used by configuring the model and openai_key arguments. This future-proofs the package, allowing it to take advantage of the open-source revolution that Hugging Face has pioneered. Here is an example of the core merge functionality, based on embeddings sourced from an external language model.
1 #pip install linktransformer 2 import linktransformer as lt 3 df1 = pd.read_csv("df1.csv") 4 df1 = pd.read_csv("df2.csv") 5 df_matched = lt.merge(df2 , df1 ,
merge_type='1:m', on=["Varname"], model="sentence -transformers/all - MiniLM -L6 -v2", openai_key=None)
We recommend that users new to LLMs deploy the package using a cloud service optimized for deep learning to avoid the need to resolve dependencies, and our tutorials use Colab.
In addition to supporting Hugging Face and OpenAI models, LinkTransformer provides pretrained models, currently encompassing six languages (English, Chinese, French, German, Japanese, and Spanish) plus a multilingual model.
These models are trained and evaluated using novel datasets that reflect common record linkage tasks in quantitative social science:
1. Firm aliases: these are drawn from Wikidata for 6 languages. Firm alias models learn to recognize the different ways that firm names are written and abbreviated.
2. Homogenized industry and product names: These are drawn from the United Nations economic classification schedules (International Standard Industrial Classification, Standard International Trade Classification, and Central Product Classification), that map different country classifications to an international standard. We include models trained on these for 3 official UN languages.
3. Historical datasets: linked product-level Mexican tariff schedules from 1947 and 1948, and a dataset linking 1950s Japanese firms across noisy text databases. Historical data are central to better understanding economic and social processes; for example, these datasets could be used to elucidate the political determinants of tariff policy or the role of supply chain linkages in Japan’s spectacular 20th century growth performance.
We also provide models for the standard industry benchmarks.
We name these models with a semantic syntax: {org_name}/lt-{data}-{task}-{lang}. Each model has a detailed model card, with the appropriate tags for model discovery. Additionally, we provide a high-level interface to download the right model by task through a wrapper that retrieves the best model for a task chosen by the user. LinkTransformer makes no compromise in
scalability. All functions are vectorized wherever possible and the vector similarity search underlying knn retrieval is accelerated by an FAISS (Johnson et al., 2019) backend that can easily be extended to perform retrieval on GPUs with billion-scale datasets. We also allow “blocking” - running knnsearch only within “blocks” that can be defined by the blocking_vars argument.
Record linkage frequently requires matching databases on multiple noisily measured keys. LinkTransformer allows a list of as many variables as needed in the "on" argument. The merge keys specified by the on variable are serialized by
concatenating them with a < SEP > token, which is based on the underlying tokenizer of the selected base language model. Since we have designed the API around dataframes - due to their familiarity amongst users of R, Stata or Excel - all import/export formats are supported.
The LinkTransformer API supports a plethora of other features that are frequently integrated into data analysis pipelines. These include:
Aggregation: Data processing often requires the aggregation of fine descriptions into coarser categories, that are consistent across datasets and time or facilitate interpretation. This problem can be thought of as a merge between finer categories and coarser ones, where LinkTransformer classifies the finer categories by means of finding their nearest coarser neighbor(s). lt.aggregate_rows performs this task, with a similar syntax to the main record linkage API.
Deduplication: Text datasets can contain noisy duplicates. Popular libraries like dedupe (Gregg and Eder, 2022) only support deduplication using metrics that most closely resemble edit distance. LinkTransformer allows for semantic deduplication with a single, intuitive function call.
1 df_dedup=lt.dedup_rows(df,on=" CompanyName",model="sentence - transformers/all -MiniLM -L6-v2", 2 cluster_params= {'threshold ': 0.7})
LinkTransformer de-duplication clusters embeddings under the hood, with embeddings in the same cluster classified as duplicates. LinkTransformer supports SLINK, DBSCAN, HDBSCAN, and agglomerative clustering.
Cross-lingual linkage: Analyses spanning multiple countries often require cross-lingual linkage. Machine translation followed by string matching methods tend to perform very poorly, necessitating costly hand linking. LinkTransformer users can bypass translation by using multilingual transformer models.
Text Classification: While Hugging Face provides an accessible API, text classification can still be challenging for users who haven’t been exposed to NLP libraries. Our API requires only one line of code to use a classification model on Hugging Face or the ChatGPT (3.5 and 4) API to classify texts.
Notebooks and tutorials outline the use of these functionalities on toy datasets.1 We also have a tutorial to help those who are less familiar with language models to select ones that fit their use
1https://linktransformer.github.io/
case. More detailed information and additional features can be found in the online documentation.2

heading: 3.2 Customized Model Training
text: Record Linkage Record linkage tasks are highly diverse and may demand very high accuracy; hence, fine-tuning on target datasets may be necessary. LinkTransformer supports easy model training, which can be initialized using any Hugging Face transformer model.
Training data are expected in a pandas data frame, removing entry barriers for the typical social science user. A data frame can include only positive labeled examples (linked observations) as inputs, in which case the model is evaluated using an information retrieval evaluator that measures top-1 retrieval accuracy. Alternately, it can take a list of both positive and negative pairs, in which case the model is evaluated using a binary classification objective.
Only the most important arguments are exposed and the rest have reasonable defaults which can be tweaked by more advanced users. Additionally, LinkTransformer supports logging of a training run on Weights and Biases (Biewald, 2020).
1 best_model_path=lt.train_model( 2 model_path="hf-path -model", 3 data="df1.csv", 4 left_col_names =["left_var"], 5 right_col_names =['right_var '], 6 label_col_name=None , 7 log_wandb=False , 8 training_args ={"num_epochs": 1})
Default training expects positive pairs. A simple argument that specifies label_col_name switches the dataset format and model evaluation to adapt to positive and negative labels. To make this extensible to most record linkage use-cases, the model can also be trained on a dataset of cluster ids and texts by simply specifying clus_id_col_name and clus_text_col_names. LinkTransformer is sufficiently sample efficient that most models in the model zoo were trained with a student Google Colab account, an integral feature since the vast majority of potential users have constrained compute budgets.
Classification We added classification at the request of LinkTransformer users. Users can train custom models with a single line of code, using training
2https://github.com/dell-research-harvard/ linktransformer
data in the form of a data frame. They simply specify the on columns containing the text and a column for annotations, label_col_name. We have helpful guides on our website to allow users to effectively tune hyperparameters.
Since this function wraps around the Trainer class from Hugging Face, it can make use of multiple GPUs. training_args allow an advanced user to fully customize the Trainer by providing arguments with the same format as Hugging Face’s TrainingArguments class.

heading: 3.3 User Contributions
text: LinkTransformer aims to promote reusability and reproducibility, which are central to academic applications. End-users can upload their self-trained models to the LinkTransformer Hugging Face hub with a simple model.save_to_hub command. Whenever a model is saved, a model card is automatically generated that follows best practices outlined in Hugging Face’s Model Card Guidebook.

heading: 4 Applications
text: The LLMs in the LinkTransformer model zoo excel at a variety of tasks. Table 1 evaluates performance linking Wikidata firm aliases (panel A), linking product descriptions from different countries’ classification schemes (panel B), linking products to their industries (panel C), and aggregating fine product descriptions to coarser descriptions (panel D). It compares the accuracy of Levenshtein edit distance matching (Levenshtein et al., 1966), popular off-the-shelf semantic similarity models from Hugging Face (see Appendix Table A-1 for a listing of models used), OpenAI embeddings (the better of text-embedding-3-small and text-embedding-3-large, which outperformed Ada embeddings), and LinkTransformer tuned models. The supplementary materials describe the models and training datasets in detail.
As expected, custom-tuned models typically achieve the best performance, with off-the-shelf models still outperforming edit distance matching, typically by a wide margin. The custom-trained models are often plausibly achieving human-level accuracy, as cases that they get wrong are often impossible to resolve from the information provided, e.g., in cases where a firm is referred to by two completely disparate acronyms.
Second, we examine historical applications, which are central to understanding long-run phe-
nomena like economic growth or social mobility and typically lack unique identifiers for linkage. First, we link two tariff schedules published by the Mexican government in the 1940s (Secretaria de Economía de Mexico, 1948). Tariffs were applied at an extremely disaggregated product level and each of the many thousands of products in the tariff schedule is identified only by a text description, which can change each time the tariff schedule is updated. Around 2,000 products map to different descriptions across the schedules in a rare crosswalk published by the government (typically, homogenized crosswalks do not exist). We link the tariff schedules using an off-the-shelf semantic similarity model, as well as a model tuned on the in-domain historical data and Open AI embeddings. All transformer models widely outperform edit distance. While there are considerable debates on the role that trade policies have played in long-run development, empirical evidence is limited largely due to the considerable challenges of homogenizing tariff schedules across time. Language model
linking offers the opportunitiy to bring novel quantitative evidence to this important question.
We also link firms across two different 1950s publications created by different Japanese credit bureaus (Jinji Koshinjo, 1954; Teikoku Koshinjo, 1957). One has around 7,000 firms and the other has around 70,000, including many small firms. Firm names can be written differently across publications and there are many duplicated or similar firm names. We concatenate information on the firm’s name, prefecture, major products, shareholders, and banks. These variables contain OCR noise and the information included varies, e.g. in terms of how a firm’s products are described, which shareholders are included, etc. This makes rule-based methods quite brittle, whereas the custom-tuned model links 91% of firms correctly.
In the supplemental materials, we examine the various e-commerce and industry benchmarks that prevail in this literature. We use the same training procedure for each benchmark, to avoid overfitting, which is often not the case in the literature. We have generally comparable performance, sometimes outperformed by other models (that could be used with LinkTransformer if on Hugging Face) and sometimes outperforming other models.
When OCR errors are severe, too much information may have been destroyed to achieve the desired accuracy with the garbled texts. A multimodal matching framework (Arora et al., 2023) that uses aligned language and vision transformers to incorporate the original image crops or a matching framework that incorporates character visual similarity (Yang et al., 2023) - as OCR confuses visually similar characters - may be required. Vision and multimodal linking support will be incorporated into future releases of LinkTransformer.

heading: 5 Ethics Statement
text: LinkTransformer is ethically sound. It is built using public domain training data. Because it is built
upon transformer language models, it will not be suitable for lower resource languages that lack pretrained LLMs. However, it can utilize any Hugging Face or OpenAI embedding model and hence will be extensible as the low-resource transformer literature expands to lower resource settings, as long as relevant embedding models are posted on Hugging Face or made available commercially by OpenAI.

heading: A Supplementary Materials
text: A.1 Training and other details LinkTransformer models use AdamW as the optimizer with a linear schedule with a 100% warm-up with 2e-6 as the max learning rate. We use a batch size of 64 for models trained with Wikidata (companies) and UN data (products). For industry benchmarks, we used a batch size of 128. We trained the models for 150 epochs for industrial benchmarks and 100 epochs for UN/Wikidata/Historic applications. We used Supervised Contrastive loss (Khosla et al., 2020) and Online Contrastive loss with default hyperparameters as the training objective depending upon the structure of the training dataset (as specified in Table A-4). The implementation for the losses was based on the implementation shared on the sentence-transformers repository (Reimers and Gurevych, 2019). LinkTransformer uses IndexFlatIP from FAISS (Johnson et al., 2019) as the index of choice, allowing an exhaustive search to get k nearest neighbours. We use the inner-product as the metric. All embeddings from the encoders are L2normalized such that the distances (inner-products) given by the FAISS indices are equivalent to cosine similarity.
Code to replicate the below tables and train the models is available on our repository, which also contains links to our training data.
A.2 Datasets and Results Table A-1 lists the base sentence transformer models that we used to initialize the custom LinkTransformer models. Table A-2 describes the datasets used for training the LinkTransformer model zoo. They are drawn from multilingual UN product and industry concordances, Wikidata company aliases, a 1948 Mexican government concordance between tariff schedules (Secretaria de Economía de Mexico, 1948), and a hand-linked dataset between two 1950s Japanese firm-level datasets collected by credit bureaus, one containing around 7,000 firms and the other around 70,000 (Teikoku Koshinjo, 1957; Jinji Koshinjo, 1954). Table A-3 describes the train-val-test splits for each of these datasets. Table A-5 reports results on standard industry and e-commerce benchmarks for record linkage.

heading: Future Work
text: Vision and multimodal linking support will be incorporated into future releases of LinkTransformer