Quantifying Character Similarity with Vision Transformers
Authors: Xinmei Yang
Abhishek Arora
Shao-Yu Jheng
Melissa Dell
Venue: EMNLP 2023 Main (2023)
Keywords: record linkage
homoglyphs
character similarity

Abstract: Record linkage is a bedrock of quantitative social science, as analyses often require linking data from multiple, noisy sources. Off-the-shelf string matching methods are widely used, as they are straightforward and cheap to implement and scale. Not all character substitutions are equally probable, and for some settings there are widely used handcrafted lists denoting which string substitutions are more likely, that improve the accuracy of string matching. However, such lists do not exist for many settings, skewing research with linked datasets towards a few high-resource contexts that are not representative of the diversity of human societies. This study develops an extensible way to measure character substitution costs for OCR'ed documents, by employing large-scale self-supervised training of vision transformers (ViT) with augmented digital fonts. For each language written with the CJK script, we contrastively learn a metric space where different augmentations of the same character are represented nearby. In this space, homoglyphic characters - those with similar appearance such as "O" and "0" - have similar vector representations. Using the cosine distance between characters' representations as the substitution cost in an edit distance matching algorithm significantly improves record linkage compared to other widely used string matching methods, as OCR errors tend to be homoglyphic in nature. Homoglyphs can plausibly capture character visual similarity across any script, including low-resource settings. We illustrate this by creating homoglyph sets for 3,000 year old ancient Chinese characters, which are highly pictorial. Fascinatingly, a ViT is able to capture relationships in how different abstract concepts were conceptualized by ancient societies, that have been noted in the archaeological literature.

Quantifying Character Similarity with Vision Transformers
Xinmei Yang', Abhishek Arora', Shao-Yu Jheng', Melissa Dell 1.2
HHrvard University; Cambridge, MA, USA.
National Bureau of Economic Research; Cambridge, MA, USA.
"Coresponning author:
melissadell fas.harvard.edu
Abstract
require linking information from multiple datasets. For example, researchers and governments link his- torical censuses, match hand-written records from vaccination campaigns to administrative data, and de-duplicate voter rolls. The sources to be linked often contain noise, particularly when they were created with optical character recognition (OCR), String matching methods are widely used to link entities across datasets, they are straightfor- ward to implement off-the-shell and can be scaled to massive datasets (Binette and Steorts, 2022: Abramitzky et al., 2021).
Record linkage is bedrock of quantitative cial science, as analyses often require linking data from multiple, noisy sources. Off-the- shelf string matching methods are widely used, as they are straightforward and cheap to imple- ment and scale. Not all character substitutions are equally probable, and for some settings there are widely used handcrafted lists denot- ing which string substitutions that improve the accuracy of string matching. However, such lists not exist for many set- tings, skewing research with linked datasets towards few high-resource contexts that are not representative of the diversity human societies. This study develops an extensible way to measure character substitution costs for OCR'et R'ded documents, by employing large-scale self-supervised training of vision transformers (ViT) with augmented digital fonts. For each language written with the CJK script, con- trastively leam a metric space where different augmentations the same character are rep- resented nearby. In this space, homoglyphic characters those with similar appearance such "O" and "0" have similar vector representa- tions. Using the cosine distance between char- acters' representations as the substitution cost in edit distance matching algorithm signif- icantly improves record linkage compared to other widely used string matching methods, as OCR errors tend be homoglyphic in nature. Homoglyphs can plausibly capture character visual similarity across any script, including low-resource settings. We illustrate this by cre- ating homoglyph sets for 3,000 year old ancient Chinese characters, which are highly pictorial. Fascinatingly, VIT is able to capture relation- ships how different abstract concepts were conceptualized by ancient societies, that have been noted in the archaeological literature.
more likely,
Most simply. approximate string matching meth- ods count the number of edits (insertions, deletions, and substitutions) to transform one string into an- other (Levenshtein et al., 1966). Another common approach computes the similarity between n--ram representations of strings, where n-grams are all substrings of length (Okazaki and Tsujii, 2010).
In practice, not all string substitutions are equally probable, and efforts to construct lists that vary their costs date back over a century. For example, in 1918 Russell and Odell patented Soundex (Russell, 1918; Archives and Administration, 2023), sound standardization toolkit that accounts for the fact that census enumerators often misspelled names ac- cording to their sound. Together with the updated New York State Identification and Intelligence Sys- tem (Silbert, 1970). it remains a bedrock for linking U.S. historical censuses (Abramitzky et al., 2021). Similarly, Novosad (2018 8) adjusts Levenshtein dis- tance to impose smaller penalties for common al- ternative spellings in Hindi, and the FuzzyChinese package (znwang25, 2020) uses strokes as the unit for n-grams substring representations, where the strokes for a given character are drawn from an external database (kfed, 2015) covering a subset of the CJK script. Characters sharing strokes are more likely to be matched.
1 Introduction
Many quantitative analyses in the social seiences as well as government and business applications
Such methods can perform well in the contexts for which they are tailored but are labor-intensive

---

extend to new settings, due to the use of hand- crafted features. Low extensibility skews research with linked data necessary to examine intergener- ational mobility, the evolution of firm productivity, the persistence of poverty, and many other topics towards a few higher resource settings that are not representative of the diversity of human societies.
ters as the substitution cost within a Levenshtein edit distance framework (Levenshtein al., 1966) significantly improves record linkage, relative to other string matching methods. We first examine linking real world data on supply chains, which are central to variety of cconomic questions. We use three very different open-source OCR archi- tectures to digitize supply chain and firm informa- tion from two 1950s Japanese publications (Jinji Koshinjo, 1954; Teikoku Koshinjo, 1957). We then use HOMOGL YPH to link them, significantly improv- ing linkage over other string matching methods. This exercise illustrates that OCR errors tend to be homoglyphic regardless of the OCR architecture used.
This study aims to preserve the advantages of string matching methods. simple off-the-shelf im- plementation and high scalability while devel- oping an extensible, self-supervised method for determining the relative costs of character substi- tutions in databases created with OCR. OCR often confuses characters with their homoglyphs, which have similar visual appearance (e.g. "0" and "O"). Incorporating character visual similarity into string matching can thus plausibly improve record link- age. Homoglyphs can be constructed by hand for small script sets such as Latin, as in psychology literature on literacy acquisition (Simpson al., 2013). For a script such as CJK, containing over 38,000 characters, this is infcasible.
We provide evaluations for additional languages using synthetically generated data, as creating eval- uation data is very costly. We augment image ren- ders of place and firm names written with different fonts, for the Simplified and Traditional Chinese, Japanese, and Korean character sets. We then OCR two different views of each entity with different OCR engines. The different augmentations and OCR engines lead with high frequency to differ- ent text string views of the same entity. We then link these using string matching methods. Homo- glyphic matching outperforms other widely used string matching techniques for all four languages.
Following a literature on sulf-suuervision through simple data augmentation for image en- coders (Grill et al., 2020: Chen et al., 2021; Chen and He, 2021), this study uses augmented digital fonts to contrastively learn metric space where different augmentations of character (e.g., the character rendered with different fonts) have simi- lar vector representations. The resulting space can be used, with a reference font, to measure the vi- sual similarity of different characters. This purely self-supervised approach can be extended to any character set. Due to space constraints, this study focuses on languages written with CJK: Simplified and Traditional Chinese, Japanese, and Korean.
While end-to-end deep neural methods could plausibly outperform string matching, the data required for them are not always available and technical requirements for implementation are higher, explaining why string matching methods predominate in social science applications. Homo- glyphic matching is a cheap and extensible way to improve string matching. Our python package HomoglyphsCJK cnttps://pypii Homoglyphsche yphsCJK/) provides simple, off-the-shelf implementation.
We train the HOMOGLYPH model on augmenta- tions of the same character rather than paired data across characters- because self-supervised approach is more extensible. Paired character simi- larity data are limited. Unicode maintains a set of confusables constructed with rule-based methods but for CJK the only confusables are structurally identical characters with different Unicode code- points. Despite a large post-OCR error correction literature (Lyu al., 2021; Nguyen et al., 2021; van Strien. et al., 2020), there is also limited ground truth data about the types of errors that OCR makes across architectures, languages, scripts, layouts, and document contexts.
Homoglyphs can be extended to any script. To explore this, we contrastively train a HOMOGL YPH model for ancient Chinese characters, using a database that provides views of the same charac- ter from different archaeological sites and time periods (Academia Sinica et al., 2023). Ancient characters are much more pictorial than their more abstract, modern equivalents. Fascinatingly, homo- glyphs constructed with a VIT for the Shang Dy- nasty (1600 BC-1045 BC) capture ways which ancient Chinese society related abstract concepts that have been noted in the archaeological literature
Using the cosine distance between two charac-

---

(e.g. Wang (2003)).
To compute characters' similarity, we embed their image crops, created with a reference font (Google Noto) chosen for its comprehensiveness, and compute cosine similarity with a Facebook Artificial Intelligence Similarly Search backend (Johnson et al., 2019).
The rest of this study is organized as follows: Section develops methods for learning character similarity and incorporating it into string match- ing, and Section 3 describes the evaluation datasets. Section 4 compares the performance of homo- glyphic edit distance to other string matching meth- ods for record linkage, and Section 5 introduces the HomoglyphsCJK package. Section 6 examines ex- tensibility by constructing homoglyphs for ancient Chinese, and Section 7 discusses the limitations of homoglyphs.
Figure 3 shows representative examples of char- acters and their five nearest homoglyphs. Charac- ters with similar vector representations have quali- tatively similar appearances.
HOMOGL YPH shares common elements with Ef- ficientOCR (Carlson al., 2023), an OCR archi- tecture that learns to recognize characters by con- trastively training on character crops rendered with augmented digital fonts. Different augmentations of a character provide positive examples. At in- ference time, localized characters are OCR's ed by retrieving their nearest neighbor from an offline index of exemplar character embeddings. Efficien- tOCR aims retrieve the same character in an offline index, whereas HOMOGL YPH measures simi- larity across characters. While HOMOGL YPH shares the architecture of the EfficientOCR character rec- ognizer, it does not use the same model weights or training data.
2 Methods
2.1
The HOMOGLYPH model
The HOMOGLYPI YPH model contrastively learns a map- ping between character crops and dense vector rep- resentations, such that crops of augmentations of the same character are nearby, as illustrated in Fig- ure 1. HOMOGLYPH is trained purely on digital fonts. Figure 2 shows variations of the same characters rendered with different fonts, which form positive examples for training. Variations across fonts are non-trivial, forcing the model learn character similarities at varying levels of abstraction.
We use a DINO Soff-Distillation, No Labels) pre-trained ViT as the encoder (Caron et al., 2021). DINO ViT embeddings perform well as nearest neighbor classifier. making them well-suited for ho- moglyphic matching. The model is trained using Supervised Contrastive loss function (Khosla et al., 2020), a generalization of the InfoNCE loss (Oord et al., 2018) that allows for multiple positive and negative pairs for a given anchor:
2.2 String Matching Methods
Dunn (1946) in one of the first treatments of record linkage wrote: "Each person in the world creates Book of Life. This Book starts with birth and ends with death. Its pages are made up of the records of the principal events in life. Record link- age is the name given to the process of assembling the pages of this Book into a volume."
Edit distance metrics are widely used for this task e.g. Levenshtein et al. (1966); Jaro (1989); Winkler (1990). Another common approach com- putes the cosine similarity between n-gram repre- sentations of strings (Okazaki and Tsujii, 2010).
(x, xp/T)
EPPO FEr
log
(1)
P(]
Decat exp (#: #c/T)
PEP
where: T is a temperature parameter (equal to 0.1), i indexes a sample in a "multiviewed" batch (in this case multiple fonts/uugmentations of char- acters with the same identity), P(i) is the set of indices of all positives in the multiviewed batch that are distinct from i, A(i) is the set of all indices excluding i, and an embedding of a sample in the batch. Training details are describe in the supplementary materials.
There are variety of ways that character-level visual similarity could be incorporated into record linkage. We follow the literature modifying Lev- enshtein distance, e.g. Novosad (2018), by using cosine distance in the HOMOGL YPH space as the sub- stitution cost. Insertion and deletion costs are set to one. It is straightforward to scale the insertion and deletion costs using parameters estimated on a val- idation set, but we focus on performance without any tuned parameters to maintain purely off-the- shelf, sepe-super implementation.
WWe use 62 open-source fonts for Korean, 27 for Simplified Chinese, 17 for Traditional Chinese, and 13 for Japanese, Our training budget required us to base the number of fonts for each script on the total number of characters in that script, using more fonts for characters with a smaller Unicode range.
We compare matching with homoglyphic edit

---

Vision Transforme
High Dimensional Embedding Space
Shared Weights
Contrastive Loss
X
Vision Transflrmm
Figure I: Training Architecture. Contrastive training architecture for the HOMOGLYPH model.
Divarsity of Characteee hcreen Torts
Chanactor Irner Froduct Sime ciliiittty Fline
laggaago
Diancter
Chanscter
3
X
H
kM
Bilf
+ii
M
B
4
OFFF
O0
OF
10
50
Om
OE
5
777
194
#
Teditool
Traekiooal
1
Sarphiiee Careew
Sarphtted
+
#
x
Figure 2: Character variation across fonts. This fig- ure illustrates examples of the same character rendered with different fonts. Augmentations of these comprise positives in the HOMOGLYPH training data.
Figure Homoglyphs. This figure illustrates the five nearest neighbors in the HOMOGL YPH embedding space for representative characters.
(kfed, 2015) covering subset of the CJK script.
distance variety of other methods. The first comparison is to classic Levenshtein distance (in- sertions, deletions, and substitutions are equally costly), to isolate the effect of varying the substitu- tion cost. We also compare to the popular Simstring package, which uses variety of similarity metrics (Jaccard, cosine, and Dice similarity), computed with 2-gram substrings (Okazaki and Tsujii, 2010). The third comparison is to FuzzyChinese, widely used package that uses strokes characters as the fundamental unit for n-gram substring represen- tations (we use the default 3-grams). These are compared using the TF-IDF vectors. The strokes in each character are drawn from an external database
Evaluation Datasets
To our knowledge, there are widely used bench- marks for evaluating record linkage for the CJK script, and thus we develop datasets for evaluation. Table contains information about their sizes.
First, we link dataset on the customers and sup- pliers of major Japanese firms, drawn from 1956 Japanese firm publication (Jinji Koshinjo, 1954). firm index of around 7,000 firms. The index is from the same publication but written a different font. Firm names are localized with LayoutParser (Shen et al., 2021), using a custom layout analysis

---

Task
Query
Target
fied Chinese (synth-zhs), Japanese (synth-ja), and Korean (synth-ko), we draw placenames from the Geonames database (Geonames, 2023). Because Traditional Chinese placenames in Geonames are rare, we instead draw from a list of Taiwanese firms (Taiwan Ministry of Economic Affairs, 2023),
Hist. Jap. Companies Across Ohistoric-
238
68,352
ja-across)
Hist. Jap. Companies Within (historic-
1,092
6,725
ja-within)
Synthetic Japanese (synth-jaa
86,470
86,470
Synthetic Korean (synth-ko)
48,809
48,809
Synthetic Trud. Chinese (symthhzhr)
66,943
66,943
as Taiwan still uses Traditional Chinese (forming the dataset synth-zht). We augment the images of each entity name, randomly select two image crops for each entity, and OCR them using EasyOCR and PaddleOCR. Anywhere from 40% (Simplified Chinese) to 88% (Traditional Chinesc) of OCR' ed string pairs differ between the two OCR engines, and we use homoglyphic edit distance to match these strings.
Synthetic Simp. Chinese (synth-zhs)
20.162
20,162
Table Query (items match) and target (reference dataset) for each record linkage task.
model that detects individual customers and sup- pliers in lists. Custom layout analysis is necessary, some OCR engines fail to detect the character separating firms in the customer-supplier list. We then hand-link a randomly selected sample of 1092 customers and suppliers to the index, keeping only one instance of each firm in the customer-supplier lists to create a diverse evaluation set. We call this dataset wiiiinn
4 Results
Homoglyphic edit distance outperforms the other string matching methods in all three evaluation exercises across different OCR engines and lan- guages typically by an appreciable margin. This illustrates that homoglyphic errors in OCR are com- mon and can be captured with self-supervised vi- sion transformers.
Firm names are OCR'ed ed twice, to shed light on whether errors tend to be homoglyphic for both vision-only and vision-language sequence- to-sequence OCR architectures. We employ two widely used, open-source engines: EasyOCR and PaddleOCR. EasyOCR uses convolutional recur- rent neural network (CRNN) (Shi et al., 2016), with learned embeddings from vision model serving as inputs to learned language model. PaddleOCR abandons language modeling, dividing text images into small patches, using mixing blocks to perceive inter- and intra-character patterns, and recognizing text by linear prediction (Du et al., 2022). Neither engine localizes individual characters.
We show that homoglyphs are useful for real- world research tasks, starting with supply chains. Supply chain widely studied, they are fun- damental to the transmission of economic shocks (Acemoglu al., 2016, 2012), agglomeration (EI- lison et al., 2010), and economic development (Hirschman, 1958; Myrdal and Sitohang, 1957; Rasmussen, 1956; Bartelme and Gorodnichenko. 2015; Lane, 2022). Yet their role in long-run eco- nomic development has been difficult to study due to the challenges of accurately linking large-scale historical records
In a second exercise, we use a hand-curated dataset that links 238 firms to a much larger firm directory containing over 70,000 firms (Teikoku Koshinjo, 1957). This latter source is written ver- tically, which PaddleOCR and EasyOCR do not support. Instead, we digitize them with Efficien- 1OCR (Bryan et al., 2023), custom-trained on both target publications. We would expect Efficien- iOCR's character retrieval framework to make ho- moglyphic errors. The large firm directory con- tains some firms with duplicate names; these are removed for the purposes of this analysis, as space constraints do not permit a discussion of varied blocking methods (Binette and Steorts, 2022).
Our first evaluation exercise. with linked Japanese supply chain data (historie-ja-wtthin) aims to elucidate whether homoglyphic matching is as helpful for linking datasets created with vision- language OCR (EasyOCR) as for linking datasets created with vision-only OCR (PaddleOCR), and whether it can similarly be useful for linking datasets created with different OCR architectures. We hence separately consider results linking Pad- dleOCR ed customers and suppliers to the Easy- OCR' ed firm index, and vice-versa, as well as link- ing when both are OCR' ed by either PaddleOCR
Because creating ground truth data for record linkage is costly, we use synthetically generated data for a third set of evaluations. For Simpli-
'The sample size is 20,162 for Simplified Chinese, 66,943 for Traditional Chinese, 86,470 for Japanese, and 48, 48,809 for Korcan.

---

OCR Engines
Method
Accuracy
Method
Paddle
Lasy
Paddle to Paddle
Dasy to Easy
Homoglyphic distance
0.824
to Easy
to Paddle
Levenshtein distance
0.731
Homoglyphic
0.808
0.753
0.844
0.728
Simstring (cosine)
0.748
distance
Simstring (dice)
0.752
Levenshtein
0.766
0.697
0.807
0.693
Simstring (jaccard)
0.752
distance
Simstring
0.762
0.662
0.787
0.673
FuzzyChinese (stroke)
0.735
(cosine)
FuzzyChinese (character) 0.618
Simstring
0.763
0.663
0.788
0.673
(dice)
Table Linking Japanese Firms to a Large Direc- tory: historic-ja-acren This table links Japanese firms from supply chain records (Jinji Koshinjo, 1954) to an extensive firm directory (Teikoku Koshinjo, 1957), com- paring various string matching methods.
Simstring
0.763
0.663
0.788
0.673
(jaccard)
FuzzyClinn Chinese (stroke)
0.690
0.567
0.717
0.554
FuzzyChinese
0.533
0.445
0.559
0.464
(character)
synth-ja synth-ko
synth shih--zhs
synth-zht
Table 2: String Matching Across OCR Engines for wistoric-ja-withinn This table reports accuracy using a variety different methods for linking Japanese firms from supply chain records (Tinji Koshinjo, 1954) to firm index created from the same publication. The four columns report results when (I) PaddleOCR is used to OCR the firm list and EasyOCR the directory, (2) EasyOCR is used to OCR the firm list and PaddleOCR the directory, (3) Pad is used to OCR both lists,
Homoglyphic distunce
0.456
0.292
0,476
0.465
Levenshtein
0.396
0.188
0.375
0.407
distance
Simstring
0.376
0.247
0.425
0.383
(cosine)
Simstring (dice)
0.380
0.248
0.426
0.385
Simstring
0.380
0.248
0.426
0.385
(jaccard)
(4) EasyOCR is used to OCR both lists.
FuzzyChinese (stroke)
0.168
0.000
0.473
0.372
FuzzyChinese (character)
0.230
0.110
0.137
0.197
EasyOCR. Homoglyphic edit distance outper- forms other string matching methods and does so by a similar margin (around percentage points higher accuracy) regardless of the OCR architec- ture used. FuzzyChinese has the weakest perfor- mance, expected since many Japanese characters are not covered in their stroke dictionary.
Table 4: Matching Results: Synthetic Data. This table reports accuracy linking synthetic paired data generated by OCR 'ing location and firm names rendered with augmented digital fonts with different OCR gines, four the four languages in out exercise.
Our second ovaluation, historic-jer ja-rriis----coosss exam- ines matching to a different, vertically written book, with around ten times more firms, digitized with EfficientOCR because it supports vertical Japanese. Homoglyphic distance outperforms all other string matching methods, with matching accuracy of 82%. This matching exercise also examined in Arora al. (2023), using customized, end-to- end multimodal deep neural model to link firms across the books. This achieves an accuracy of 87.8 with Vision-only contrastive training and a peak accuracy of 94.5 using Language-Ime Image model (CLIP) backbone with multimodal pooling. The end-to-end deep neural methods outperform string matching, would be expected, as they are leveraging more information (the image crops as well as the texts) and language understanding. Yet string matching methods remain prevalent in the literature, due to their straightforward off-the-
shelf implementation, interpretability, familiarity to users, minimal input requirements (only OCR' ed strings), and the fact that language understanding while relevant for linking firms may be of less relevance for linking individuals, placenames, etc.
Finally, Table reports results with the syntheti- cally generated record linkage datasets, elucidate the performance of homoglyphic matching across languages that use the CJK script. Homoglyphs out- perform other string-matching methods. The only case where the performance of another method is similar is Simplified Chinese, with the FuzzyChi- package using stroke level n-grams. The stroke dictionary that underlies FuzzyChinese crafted for Simplified Chinese, yet homoglyphs can perform similarly with self-supervised meth- ods. On Traditional Chinese, which proliferates historical documents, homoglyphic edit distance

---

(1)
(2)
(3)
(4) Homoglyphic Match
(5) Levenshtein Match
(6) Simstring Cosine Match
Ground Truth String
PaddleOCR Recognition
EasyOCR Recognition
FuzzyChinese Stroke Match
(Gorrect Match)
Panel A: Homoglyphic Matching Error Cases
TJ
BA
E
7
g0l
1
XT
p
E
Panel Homoglyphic Matching Correct Cases
/
g
O
SVE
#115
g0
-
T
x
T
M
Figure Error analysis. Panel A shows representative errors from homoglyphic matching. Panel B shows representative cases that homoglyphic matching gets correct. The ground truth string is shown in column (I). PaddleOCR is used to OCR the query images (column (2)) and EasyOCR is used OCR their corresponding keys (column (3)). Columns (4) through (7) give the selected match the query using different string matching methods, with the correct match shown in column (3). Bold characters differ from the query.
offers a nine percentage point accuracy advantage over FuzzyChinese, illustrating the extensibility ad- vantages of self-supervised methods. The accuracy rates are low, but this must be interpreted in the context of the dataset, which only includes paired records where the OCR differs.
barrier to using it not only for Python users but also those experienced in using R or Stata for data analysis. All the user needs are the two datasets to match (with the key match on) and one line of code. The package comes pre-loaded with the lookup tables that contain pairwise distances be- tween different characters for a script. The files are also available on HuggingFace"".
Figure 4 provides an error analysis for the syn- thetic record linkage exercise. The ground truth string, taken from the original image, is shown in the first column. PaddleOCR is used to OCR the query (column 2). EasyOCR is used to OCR the key, with the resulting string shown in column (3).
We provide two main functions in the pack- age one to calculate the homoglyphic distance fig_distance) between pair of strings and the other for matching two data frames [hg_merge) Here is example usage.
The matches selected from the key by different string-matching methods are shown in columns (4)
import pandas as pd from HomoglyphsCJK import hg_distance, hg_merge
through (7). Bold characters differ from the query OCR. Panel A shows cases where homoglyphic edit distance selects an incorrect match. This typi- cally occurs when the OCR' ed key has a similar vi- sual appearance to an incorrect match in the query, showing the limits of homoglyphs to fully alleviate the OCR information bottleneck. Panel B shows cases where homoglyphic edit distance selects correct match.
dfi=pd. read_csv("dfl df2=pd. read_csv ("df2.csv') df_merged = hg_merge (hzss" df1, df2,'
query' "key" homo_ 1ambda=1, insertion=1, deletion=1)
tiadisstancee "" ""A
zhss " howohoooolaooo 1ambda11,, insertion=1, deletion=1)
5 The HomoglyphsCJK package
The distance calculations are fully customizable, with the weights of insertion and deletion tunable. "homo lambda" allows the user to scale the homo- glyphic substitution cost, thereby adding another dimension to tune the match quality. (In the results
We distribute the homoglyphic matching approach for CJK languages as a Python package which has an API designed around the standard merge operation in pandas- effectively reducing the entry
"htrp://nuggingfacc.ooooaa htt.:///aaaaaccooooooooaae/ee/ee harvaa/HoonggyyhaCKKKaannnngg

---

above, the default of setting these all to 1.) The package repo contains more documentation and a colab notebook to allow quick exploration the package's functionality." For users who are interested in computing their own character visual similarity indices, we provide our training code in a separate repo.
Dansk
Cianeerr Inver Pidduct Simlyty
Thilvigg: Abundance
Wotting
Law
Leaming
Noming
6 Extending Homoglyphs
+
-
H
+
While this study focuses on the modern CJK script, HOMOGL YPH can be extended to any character set. As proof of concept, we explore its extensibility to ancient Chinese characters. Like other early forms of human writing, ancient Chinese scripts are highly pictorial relative to modern characters.
Middle or otficial
Nesee lagi-loot
miing
Jelsing Acsiating
logchai semestng
Since we do not have multiple fonts for ancient Chinese characters, we instead use an existing database of grouped ancient characters- from dif- ferent archaeological sites and periods that cor- respond to the same concept (Academia Sinica al., 2023). These ancient characters also linked their descendant modern (Traditional) Chinese character, identified by the database. We contrastively learn a metric space where the repre- sentations of ancient characters denoting the same concept are nearby. We train on 25,984 ancient characters, as well as the corresponding augmented Traditional Chinese characters. The dataset in- cludes characters from the Shang Dynasty (1600 BC-1045 BC), the Western Zhou (1045 BC-771 BC), the Spring and Autumn Warring States Era (770 BBC -221 BC), and the Qin-Han Dynasties (221BC circa third century).? To illustrate ho- moglyphs, we create a reference set for the Shang Dynasty, randomly choosing one character for each concept.
Ilstory! Eistectin
Tylng up
Conquest
Searchin
Figure Ancient Homoglyphs. This figure shows ho- moglyph sets constructed for ancient Chinese, with the descendant modern Chinese character and a description of the character's ancient meaning.
chaeological literature. The first line shows that the concepts of writing, law, learning, and morning ("recording the sun") are homoglyphs, and the sec- ond line shows that characters for different types of officials are homoglyphs, as are characters denot- ing "joining." The final line shows that history and government official are homoglyphs underscoring the central role of the government in writing history as characters denoting conquest, tying up, and city center (denoted by prisoner to be exccuted by the government, which occurred in the city center).
Not all concepts within each many of the connections above have been noted in an archaeological literature examining how an- cient peoples conceptualized the world (e.g. Wang (2003)). That these meanings can be captured using vision transformers a fascinating illustration of the relationship between images, written language. and meaning in ancient societies.
related, but
Figure shows representative examples of these homoglyphs, consisting of character and its five nearest neighbors. The modern character descen- dant as well as a short description of the ancient concept (which may differ significantly in meaning from the modern descendent) are provided. The description draws upon Li (2012).
Another test of extensibility is to compute ho- moglyphs for all (ancient and modern) characters in Unicode, clustering the embeddings to create homoglyphic sets (termed "Confusables" (Consor- tium, 2023) by Unicode). Training a homoglyphic space for all of Unicode generally gives sensible results and also has the advantage of being able measure similarity within a set, something standard Unicode confusables cannot do. An example set, spanning different scripts, is displayed in Figure 6.
Fascinatingly, the homoglyph sets are able to capture related abstract concepts noted in the ar-
harvard/HommglyppssC.
com//hll--esserrchh harvard//omoolyphsCC Trriiionnnngyisiiiiiiiiinnngg
"W exclude images from the Shuowen Jiezi a book on ancient characters limiting to the most reliable character
renders, which were drawn from archacological sites.

---

to a specific Japanese use case. To the extent that labels are required develop specialized models, the data can be costly to create at a sufficient scale for training end-to-end models. Finally, most so- cial science researchers lack familiarity with deep learning methods but are comfortable processing strings.
h
18A19 BAMUM LETTER P
LETTER
Figure 6: A confusable across writing systems de- tected by our model that was trained on all of Unicode
For these reasons, off-the-sheif string match- ing algorithms often preferred by practition- ers and can be the most suitable tool given the constraints. Homoglyphic edit distance integrates information about character similarity from purely self supervisd vision transformers. It can be im- plemented using our publicly available, simple, off- the-shelf string matching package and is highly extensible, as illustrated by our exercise with char- acters from ancient Chinese societies. We hope that highly extensible string-matching methods will make the contexts that are feasible for quantitative social scientists to study more representative of the diversity of human societies.
7 Limitations
Using homoglyphs for string matching inherits the well-known limitations of string matching. In some cases, OCR destroys too much information for record linkage to be feasible with the resulting strings. Even with clean OCR, sometimes language understanding is necessary to determine the correct match, as in the case of firm names that can be written differently. Homoglyphs do not address other types string substitutions, like those that result from enumerator misspellings, although in principle similar contrastive approach could also be developed to quantify other types of string sub- stitutions.
More sophisticated methods have been devel- oped as alternatives to string matching. For exam- ple, Ventura et al. (2015) use random forest classi- fier trained on labeled data to disambiguate authors of U.S. patents, applying clustering to the resulting dissimilarity scores enforce transitivity. Arora et al. (2023) develop multimodal record linkage methods that combine the image crops entities and their OCR. They also develop a vision-only linkage method, which avoids the OCR informa- tion bottleneck. Bayesian methods have also been used, e.g. Sadinle (2014, 2017). They offer the advantage of uncertainty quantification another well-known limitation of string matching- but do not scale well.
While these methods offer various advantages, they not always applicable. Researchers may lack access to the original document images, or may lack the compute or technical resources to process images, limiting the use of OCR-free or multimodal approaches. Language models are less likely to be useful linking individual or place names, common applications.
Moreover, general purpose, off-the-shelf, end-to- end deep neural methods for record linkage do not exist, e.g. the model in Arora al. (2023) is tuned

---

References
Yongkun Du, Zhineng Chen, Caiyan Jía, Xiaoting Yin. Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. 2022. Svtr: Scene text recognition with sin- gle visual model. arXiv preprint vrXiv:2255.01555
Ran Abramitzky, Leah Boustan, Katherine Eriksson, James Feigenbaum, and Santiago Pérez. 2021. Auto- mated linking historical data. Journal of Economic Literature, 59(3):865-995 -98
Halbert I. Dunn. 1946. Record linkage. American
Journal of Public Health and the Nations Health, 36(12):442-1446
Institute of History Academia Sinica, Philology, and Institute of Information Science Academia Sinica. 2023. Xiaoxuetang yanbian database.
Glenn Ellison, Edward I. Glaeser, and William Kerr. 2010. What causes industry agglomeration? idence from coagglomeration patterns. American Economic Review, 100(3): 195-1213.
Daron Acemoglu, Ufuk Akcigit, and William Kerr. 2016. Networks and the macroeconomy: An em- pirical exploration. Nber macroeconomics annual, 30( 30(1):2733335.
Geonames. 2023. Geonames.
Daron Acemoglu, Vasco M Carvalho, Asuman Ozdaglar, and Alireza Tahbaz-Salehi. 2012. The network origins of aggregate fluctuations. Econometrica, 80(5):1977-2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaoban Daniel Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent: A new ap- proach self-supervised learning. arXiv preprint arXiv:2006.0773:
U.S. National Archives and Records Administration. 2023. The soundex indexing system. Accessed: 11/10/2023.
Albert O Hirschman. 1958. The strategy of economic development. Yale Univ. Press, New Haven, Conn.
Abhishek Arora, Xinmei Yang, Shao Yu Jheng, and Melissa Dell. 2023. Linking representations with multimodal contrastive learning. arXiv preprint arXiv:2304.03466
Matthew A Jaro. 1989. Advances record- -inkage methodology applied to matching the 1985 census of tampa, florida. Journal the American Statistical Association, 84(406):411
Dominick Bartelme and Yuriy Gorodnichenko, 2015. Linkages and economic development. Technical re- port, National Bureau of Economic Research.
Jinji Koshinjo. 1954. Nihon shokuinrokj. Koshinjo.
Jinji
Olivier Binette and Rebecca C Steorts. 2022. (al- most) all entity resolution. Science Advances, 8( 812))eabi8021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-5477 547.
Tom Bryan, Jacob Carlson, Abhishek Arora, and Melissa Dell. 2023. Efficientocr: An extensible, open-source package for efficiently digitizing world knowledge" Empirical Methods Natural Lan- guage Processing (Systems Demonstrations Track).
kfcd. 2015. chaizi. https://gtts http://ggtuuu.oooo... com/kfed/ chaizi.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron
Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive lcarning. arXiv preprint DrXiv:2004.11366
Jacob Carlson, Tom Bryan, and Melissa Dell. 2023. Fifficient ocr for building a diverse digital history. arXiv preprint ar4.0:2304.02737
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. 2021. Emerging properties in self- supervised vision transformers. arXiv preprint arXiv:2104. 14294.
Nathan Lane. 2022.
Manufacturing revolutions: In- dustrial policy and industrialization in south korea. Available at SSRN 3890311.
Vladimir I Levenshtein al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Sovict Union
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -7750-15758.
Xueqin Li, editor. 2012. 7 Yun (The Origin of Char- acters). Tianjin Ancient Works Publishing House, Tianjin.
Xinlei Chen, Saining Xie, and Kaiming He. 2021. empirical study of training self-supervised vision transformers. arXiv preprint 1rXiv:2104.02055:
Lijun Lyu, Maria Koutraki, Martin Krickl, and Besnik Fetahu. 2021. Neural ocr post-hoc correction of his- torical corpora. Transactions of Association for Computational Linguistics, 9:479-482
Unicode Consortium. 2023. Unicode security mecha- nisms. Accessed: 10/11/2023.

---

Gunnar Myrdal and Paul Sitohang. 1957. Economic the- ory and under-developed regions. Regional Studies.
Daniel van Strien., Kaspar Beelen.. Mariona Coll Ar- danuy,, Kasra Hosseini., Barbara McGillivray. and Giovanni Colavizza. 2020. Assessing the impact of oct quality on downstream nlp tasks. In Proceedings of the 12th International Conference on Agents and Artificial Intellgence- Volume 1; ARTIDIGH,, pages 484 496. INSTICC, SciTePress.
Thi Tuyet Nguyen, Adam Jatowt, Mickael Cous- taty, and Antoine Doucet. 2021. Survey of post-ocr processing approaches. ACM Compul. Surv., 54(6).
Paul Novosad. 2018. Masala merge. https://gi thub. pttppauooo/mssalaaaaaaaaaaaaaaaa merge.
Samuel L Ventura, Rebecca Nugent, and Erica RH Fuchs. 2015. Seeing the (some)) sources of bias in past disambiguation approaches and public tool leveraging labeled records. Research Pol- icy, 4)): 1672-1701
Naoaki Okazaki and Jun' ichi Tsujii. 2010. Simple and efficient algorithm for approximate dictionary matching. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 851-859.
Guowei Wang. 2003. Shishi (deciphering "history") ). In Guantan Gjilinn Hebei Education Publishing House, Shijiazhuang.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03748.
William Winkler. 1990. String comparator metrics and enhanced decision rules in the Fellegi-Sune Sunter model of record linkage. ERIC.
Poul Nørregaard Rasmussen. 1956. Studies in inter- sectoral relationn, volume 15. E. Harck.
znwang25, 2020. fuzzychinese. Comm/zzwenng25/uzzyyhnneess
https://githhn
Robert C. Russell. 1918. U.s. patent no. usi 167a. U.S. Patent and Trademark Office. httpss//ppatents.gogge
(11nn A67A/cn.
Mauricio Sadinle. 2014. Detecting duplicates in homi- cide registry using bayesian partitioning approach. The Annals of Applied Statisties, 8(4):2404 (44:2440-22434
Mauricio Sadinle. 2017. Bayesian estimation of bipar- tite matchings for record linkage. Journal of American Statistical Association, 1 11(6518)600-612
Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lec, Jacob Carlson, and Weining Li. 2021. Layoutparser: unified toolkit for deep leamn ing based document image analysis. International Conference Document Analysis and Recognition, 12821.
Baoguang Shi, Xiang Bai, and Cong Yao. 2016. end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, (0(11):2298-2)
Jeffrey M Silbert. 1970. The world's first computer- ized criminal-justic information-sharte system- thh new york state identification and intelligence system (nysiss). Criminology, 8: 107.
Ian Simpson, Petroula Mousikou, Juan Manuel Mon- toya, and Sylvia Detior. 2013. A letter visual- similarity matrix for hatin- alphabets. Behavior research methods, 45:431-439.
Taiwan Ministry of Economic Affairs. 2023. Registered factory directory.
Teikoku Koshinjo. 1957. Teikoku Ginko Kaisha Yoroku. Teikoku Koshinjo.

---

Supplementary Materials
Script
Training
Inference
S-1
HOMOGLYPH Model Details
Japanese
17,963
17,963
Simplified Chinese
6,621
7,806
S-1.1 Encoder
Traditional Chinese
8,415
8,628
For both of applications, trained (Caron et al., 2021) vision transformer (VTT) as the encoder. Our implementation of the VIT comes from the Pytorch Image Models library (timm) (Wightman, 2019). Specifically, we use the vis_base lot.baseeepphchl..cc (achhh1_2224ddee 224.dino model that corre- sponds to the official PINO-pretainer VīT-base model with patch size of 16 and with input reso- lution of 224 The pretrained checkpoint does not have classification head.
DINO pre-
Korean
3,686
3,729
Total
36,685
38,126
Table S-1: Training and Inference Sizes. This table shows the training and inference sizes for different lan- guages.
its digital renders belong. Characters do not need to be seen during training to be considered at infer- ence time, advantage if users wish to expand the homoglpyph sets (e.g. because an OCR engine uses different character set). We illustrate this empiri- cally by expanding the character set to characters covered by the three OCR engines we explore that were not included in our character ranges used ini- tially for training.
S-1.2 Loss function
We use Supervised Contrastive loss (Khosla et al., 2020) training objective, implemented in the PyTorch Metric Learning library (Musgrave al., 2020). where the temperature parameter is set to 0.1.
S-2.1.2 Batching
S-1.3 Data Augmentation
We deploy several image augmentations, using transformations provided in the Torchvision library (Torch 2016). These include Affine transfor- mation (only slight translation and scaling allowed), Random Color Jitter, Random Autocontrast, Ran- dom Gaussian Blurring, and Random Grayscale. Additionally, we pad the character to make the im- age square while preserving the aspect ratio of the character render. We do not use common augmen- tations like Random Cropping or Center Cropping, avoid destroying too much information.
Without hard-negative mining
Let B = 128 denote the batch size. A batch consists of m = views of B m without replacement. When all the views for a character are utilized, all images are replaced and the sampling process without replacement starts again. "Views" of character are augmented digital renders using the fonts and transformations described above. One training epoch is defined seeing the views of all characters exactly once.
characters sampled
For augmenting the skeleton the rendered character itself, we use a variety of digital fonts to render the images. We use 27 fonts for Simpli- fied Chinese, 17 fonts for Traditional Chinese (for both string matching and ancient Chinese), 62 fonts for Korean, and 14 fonts for Japanese.
With hard-negative mining
We find the k = 8 nearest neighbors of cach character on a checkpoint trained without hard neg- atives. We do this by rendering all characters in language with reference font, Noto Serif CJK font (Tc/Sc/Ip/Ko), chosen for its broad coverage of characters. Each hard negative set includes 4 views of the anchor character, as well as views each of its k nearest neighbor characters. We ran- domly intersperse hard negative sets in the batches. One training epoch is now defined as seeing all hard negative sets once. Table S-2 contains the number of epochs we trained each model for.
S-2
Application-speerffc details
S-2.1
Record Linkage
S-2.1.1
Data
For each language, the datasct consists of images of characters from the corresponding language ren- dered with different fonts and augmented during training. The number of characters for each lan- guage seen during training is given Table S-1. Each character can considered a "class" which
S-2.1.3 Model Validation
80% of characters are used for training, 10% are used to select hyperparameters, and 10% are used

---

Model
Ir
weight decay
T_D
T_malt
Ejochs
S-2.2
Homoglyph Sets
Japanese- (No HIN)
2e-5
5o-3
100
distance
We allow for the expansion of the character set beyond what is seen in training because different OCR engines use different character dictionaries (a list of characters supported by the engine). We take the union of characters from the character dic- tionaries of PaddleOCR, EasyOCR, and Efficien- tOCR. For each language, we render all its char- acters using the reference font and embed them using the language-specific HOMOGL YPH encoder. For cach character, we then find 800-ncarest neigh- bours (measured by Cosine Similarity between the embeddings) among the set of all renders in the reference set. We store these look-up dictio- nary that contains, for each character a language, its 800 neighbors and its Cosine Similarity with all of them. This look-up dictionary is used in our modified Levenshtein distance implementation to modify the substitution cost. The dictionaries are available in our GitHub repository.
Jaunese- (HHN)
2e-5
5o-3
1
distance
Simplittd Chinese (No HIN)
2
5--3
1
2
30
distance
Simplifed Chinese- (HHN)
5o-3
30
distunce
Traditional Chinese- - (No HN)
2e-5
3e-3
distance
Traditional Chinese - (IIN)
2e-5
5e-3
distance
2e-5
5e-3
distance
(HIN)
20-5
5e-3
distance
Ancienl Chineso- (No
2e-5
5o-3
300
distance
Ancient Chinese- (TNN)
20-5
5o-3
300
distance
Table S-2: Training Hyperparameters. This table reports the training hyperparameters used for the models. The stands for learning rate, weight decay represents the weight decay factor. T_O 0 is the number of steps until the first restart of the learning rate scheduler, T_ mult denotes the factor by which T_0 is multiplied at each restart, and Epochs indicates the total number of training epochs. Parameters not mentioned here use Py Torch defaults. HN denotes offline hard-negative mining.
Table S-1 contains the number of characters that were used to prepare these sets for each language.
S-2.3 Implementing Homoglyphic Edit Distance
to select the best checkpoint. We embed the vali- dation images and find the nearest neighbor among the embeddings of digital renders of the universe of characters in the language, rendered with the reference font described above. The top-1 retrieval accuracy is used as the validation metric for the selection of the best checkpoint. We see a peak validation accuracy of 90% for Japanese, 98% for Korean, 91% % for Traditional Chinese, and 91% for Simplified Chinese
We use standard algorithm to calculate Leven- shtein distance that uses dynamic programming (Wagner and Fischer, 1974). The space and time complexity of the algorithm is (mn) where m and n are the lengths of the two strings that are being compared.
We modify this algorithm by switching the stan- dard substitution cost À between two characters a and b with A*(1 CosineSimilarity (u(()) u(b)). Here u(a) and ub)) are the embeddings of the HOMOGL YPH encoder for the language to which a and b belong. À is a tunable hyperparameter but for simplicity, we fix it as 1 for the results shown in the paper. We also fixed the addition and deletion cost as I but in the implementation provided in our package and our GitHub repository, the costs are tunable hyperparameters.
S-2.1.4 Other training details
CJK glyphs are somewhat similar across languages. To converge faster, we initialize the weights of the encoders for Traditional and Simplified Chinese and Korean with the checkpoint used for Japanese. which has the largest number of characters. We use AdamW (Loshchilov and Hutter, 2019) as the opti- mizer and Cosine Annealing with Warm Restarts (Loshchilov and Hutter, 2016) as the learning rate schedule. We use the standard Pytorch implemen- tation for both. The relevant hyperparameters are listed in Table S-2. We stop training the mod- els once the validation accuracy stagnates and the checkpoint with the best validation accuracy is cho- sen.
S-2.4
Ancient Chinese Homoglyphs
S-2.4.1 Data
The source database (Academia Sinica et al., 2023) from which we collect the ancient Chinese char- acter crops contains 5,024 concepts, comprised of 25, 25,984 character renderings. Each of these con- cepts is mapped to a modern character. This en- ables us to insert digital renders of these modern

---

characters using the same fonts above (for tradi- tional Chinese) create more variation. A "class" in this case comprises character cluster- with both ancient crops and modern digital renders forming the positive samples for a class.
digital renders from particular font the "refer- ence set", we look at the five nearest neighbors of ancient characters within period. We illustrate ho- moglyphs using The Shang Dynasty period (1600 BC-1045 BC), the most ancient.
We slightly modify the data augmentation scheme for this application account for the wide variation in writing styles across centuries. We allow for slight (-10 to +10 degree) rotation and also add more transformations tailored this use case - Random Equalize, Random Posterize, Random Solarize, Random Inversion and Random Erasc (randomly crase 0-5% of the image). We ap- ply all augmentations to the digital renders but only apply Random Affine transformation and Random Inversion the ancient crops.
S-2.4.2 Batching
We use the same sampling and batching process (this time with a larger batch size 256) as we did for the modern homoglyph models. The only difference is in how the hard-negative sets are defined. Instead of one nearest neighbor per concept, for each ancient crop within a concept cluster, we find k 8 nearest neighbors. This gives many nearest neighbor sets (hard-negative sets) ancient crops in our dataset. This allows us account for the fact that the homoglyphs of a character may differ across different historical periods, spanning millennia.
S-2.4.3 Model Validation
We use 1op-1 accuracy as our validation metric, de- fined as the proportion of correct retrievals of the corresponding modern render (using the reference font Noto Serif CJK Tc) for each ancient image in the validation set. During training, the model reached a peak validation accuracy of 50% demon- strating the difficult nature of this task. We use this metric for selecting the best checkpoint.
S-2.4.4 Other training details
We again use the AdamW optimizer and Cosine Annealing with Warm Restarts as the learning rate schedule. Relevant Hyperparameters are listed in Table S-2. We stop training when validation accu- racy stagnates.
S-2.4.5 Creation of Ancient Chinese Homoglyphs
The creation of homolgyph sets analogous to the case of modern characters. Instead using

---

References
Institute of History Academia Sinica, Philology, and Institute of Information Science Academia Sinica. 2023. Xiaoxuetang yanbian database.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. 2021. Emerging properties in self- supervised vision transformers. arXiv preprint arXiv:2104. 14294.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. arXiv preprint arXiv:2004.11362.
Ilya Loshchilov and Frank Hutter. 2016. stochastic gradient descent with restarts. CoRR,
SGDR:
abs/1608.03988
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. 2020. Pytorch metric learning.
Viscon. 2016. Torchvision: Pytorch's computer vision library. https://gitthu: thwb. wttp:ttthh.....ooeeeeeeeeeeeette com/pytorch/ vision.
Robert A. Wagner and Michael Fischer. 1974. The string-to-string correction problem. J. ACM, 11(1):168-173.
Ross Wightman. 2019.
Pytorch image mod- https: / /gi thub. wtt./wwi/twwwweeee rwightman/ pytorch-image-ne image-models.
els.


Reviews:
[{"review_id": "E10w7klUfi", "reviewer": "Reviewer_Qkwc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. Below, we provide responses to their questions and concerns. \n\n*One limitation not discussed in the paper is that these experiments focus on a single script, and it’s not demonstrated how the approach would generalize to other alphabets, but the authors picked a good example of a high cardinality alphabet where the limitations of an unmodified edit distance in record linking would be most pronnounced.* \n\nResponse: The reviewer highlights exactly our reasoning for focusing on the CJK script, and we now explicitly acknowledge this limitation. We provide well-documented and easy-to-implement code to facilitate the extension of our approach to other scripts. We do not provide evaluations for other high cardinality scripts primarily because our research team does not have the expertise or budget to create gold quality record linkage data for these scripts, and to our knowledge no benchmarks currently exist. We do hope this work will encourage other researchers to develop such evaluations. \n\nWe have tried an exercise computing homoglyphs for all of Unicode; e.g., embedding all of Unicode. This yielded generally sensible results - e.g., Unicode characters 78128, 42884,123196,92697 (unfortunately, these Egyptian Hieroglyph, Latin, Nyiakeng, and Bamum characters will not display properly in OpenReview, and EMNLP prohibits us from posting links to an image). We didn’t pursue this further because there was no ground truth data to use for evaluation. However, - along with the Ancient Chinese exercise in the paper - it suggests that the approach produces sensible results across a wide variety of (ancient and modern) scripts. We can include some of these universal unicode confusable sets in the supplemental materials.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is novel, well-argued, and effective, and addresses several inherent difficulties of the problem\n- Training data are reasonably augmented to focus on the composition & structure of characters rather than style of writing / font\n- The authors overcome a paucity of evaluation data by synthesizing their own, but they do provide some evaluation on some real data that are available — this may be a limitation of the evaluation, however the methods for data synthesis proposed seem sound and it seems likely results on a synthetic evaluation will generalize to real data\n- Limitations are clearly discussed and accurate, and the authors acknowledge that there are more sophisticated end-to-end deep neural methods that may solve the record linking problem more directly and decisively, and incorporate language understanding into the process, but these methods may not always be practical.", "reasons_to_reject": "This is a well-written paper which clearly and effectively argues for a novel and interesting approach, providing an adequate evaluation of the approach and a clear discussion of its limitations — this reviewer sees no clear or overwhelming reason to reject.\n\nOne limitation not discussed in the paper (edit: the authors have, in their rebuttal, agreed to revise the paper to explicitly address this) is that these experiments focus on a single script, and it’s not demonstrated how the approach would generalize to other alphabets, but the authors picked a good example of a high cardinality alphabet where the limitations of an unmodified edit distance in record linking would be most pronnounced.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel approach to record linking based on string similarity as measured by edit distance which builds on existing methods which contemplate that not all edits or substitutions are equally probable.  The focus is on visual similarity between characters, and to this end the authors develop a self-supervised embedding of images of characters based on vision transformers, such that edit distance can be weighted by visual similarity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2HFs6DntKM", "reviewer": "Reviewer_BJgQ", "summary": "", "questions": "How many digital fonts were used in training the embedding model?  \n\nWere the ancient Chinese characters written on something like paper or ceramics, or were they carved?   (Just curious.)\n\nP. 5 column 2 right near the bottom:   What was the actual accuracy of the end-to-end neural network methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful comments. We respond to the reviewer's questions and concerns below:\n\n*“The idea isn't rocket science (but I do think it's nice).”*\n\nResponse: We agree that the main contribution is to take existing tools (DINO ViT embeddings, supervised contrastive loss) and apply them to solve a novel problem, versus innovating on the architectures themselves. We wanted this technology to be accessible to those with limited coding experience, who form the bulk of potential users, and a simple architecture greatly facilitates this. Our homoglyphic edit distance package links data frames with a single line of accessible code.\n \n*”The paper is somewhat repetitive.”*\n \nRespnose: We thank the reviewer for the helpful suggestions on how to make the writing clearer and more effective. We have followed the reviewer’s advice, editing for concision and clarity. \n \n*”How many digital fonts were used in training the embedding model?”*\n\nResponse: We use 62 open-source fonts for Korean, 27 for Simplified Chinese, 17 for Traditional Chinese,  and 13 for Japanese. Our training budget required us to base the number of fonts for each script on the total number of characters in that script, using more fonts for characters with a smaller Unicode range.  \n\n*”Were the ancient Chinese characters written on something like paper or ceramics, or were they carved?”*\n\nResponse: According to Tsuen-hsuin Tsien’s seminal work on ancient Chinese writing, *Written on Bamboo and Silk: The Beginnings of Chinese Books and Inscriptions* (2004), characters written during the Shang period tended to be discovered archaeologically from inscriptions on bone tortoise shells, ox bones, or bronze wares. Several archaeological findings suggest that such inscriptions would first be written with brush pens with inks and later carved with knives or other tools. \n \n*”P. 5 column 2 right near the bottom: What was the actual accuracy of the end-to-end neural network methods?”*\n\nResponse: As we now clarify, Arora et al. (2023) achieve an accuracy of 87.8 with Vision-only contrastive training with a Vision Transformer backbone and a peak accuracy of 94.5 using a Language-Image model (CLIP) backbone with multimodal pooling, trained specifically on the train split of this dataset and hence not extensible zero shot to other applications. \n \n*Soundex reference*\n\nResponse: While we have likewise never seen an original paper describing Soundex, we now cite the patent: Russell, Robert C. U.S. Patent #US1261167A and point readers seeking a readily accessible description to U.S. Census Bureau (2007), “Soundex System: The Soundex Indexing System.” https://www.archives.gov/research/census/soundex\n\n*”Make a table of the key facts for all datasets, e.g. language, how many strings to be matched, which OCR engines used. And give each dataset a short name.”*\n\nResponse: We have made a table with key facts for all the datasets and also given them a short name. Here is a markdown-friendly version: \n| Task                            | Query Sample Size | Target Sample Size | Dataset Name   |\n|---------------------------------|-------------------|--------------------|----------------|\n| Synthetic Japanese              | 86,470            | 86,470             | synth-ja       |\n| Synthetic Korean                | 48,809            | 48,809             | synth-ko       |\n| Synthetic Traditional Chinese   | 66,943            | 66,943             | synth-zht      |\n| Synthetic Simplified Chinese    | 20,162            | 20,162             | synth-zhs      |\n| Historic Japanese Companies     | 238               | 68,352             | historic-japan |\n \n*”Perhaps an architecture diagram for the main algorithm in section 2.1?”*\n\nResponse: We agree that this will be helpful. EMNLP does not provide us with a mechanism to share this now but we will add it to the paper. \n \n*Various suggestions to streamline and clarify the writing*\n\nResponse: We greatly appreciate these helpful suggestions and have incorporated them into the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is simple but looks very useful and straightforward to reproduce.   A quick reference (p. 3 column 1 second paragraph) suggests they have a package for distribution.   \n\nThe new method outperforms previous string matching algorithms on this task.   They make the believable claim that previous methods (e.g. end-to-end neural networks) may have somewhat higher accuracy but are costly to (re-)tune for each use case.   \n\nThe paper is mostly clear and well written, with some exceptions noted below.    ", "reasons_to_reject": "The idea isn't rocket science (but I do think it's nice).  \n\nThe paper is somewhat repetitive.   The content seems a bit short for a long paper.   Then again, I think it would be quite difficult to squeeze it into the length of a short paper.   I guess it just needs to be a short long paper.\n\nThe abstract and the limitations/conclusions section have appropriate content but need editing (see below).", "questions_to_authors": "How many digital fonts were used in training the embedding model?  \n\nWere the ancient Chinese characters written on something like paper or ceramics, or were they carved?   (Just curious.)\n\nP. 5 column 2 right near the bottom:   What was the actual accuracy of the end-to-end neural network methods?", "paper_topic_and_main_contributions": "This paper presents a novel algorithm for matching strings in OCR text for languages using CJK script.   The intended application is linking instances of the same entity between different historical documents.   So OCR is necessary and character recognition errors are a significant problem.    \n\nThe algorithm uses renderings of each character in several digital fonts as positive examples for contrastive training of an algorithm to embed renderings of characters in a vector space.   (This is somewhat similar to EfficientOCR.)   Cosine distance in this space (between two renderings in a fixed standard font) is used as the substitution penalty in the string matcher's Levenshtein distance computation.   In other words, the substitution penalty is less if the characters look similar.   This is a simple combination of existing methods but it seems novel and well placed.\n\nThe algorithm is tested on two small hand-curated datasets of Japanese firm names:   1092 pairs of customer-supplier entries in one case (two different OCR engines) and then 238 firm names to be located in a very large list of firm names (a third OCR engine).  Simulated ground truth data was created for four large datasets of place names by running each through two different OCR engines and seeing how well the matcher could correctly pair the output images.   This is a nice job of combining simulated and manually curated data in a situation where it could be expensive to get large amounts of gold standard data.\n\nThey also run their embedding algorithm on a dataset of images of ancient Chinese characters.  There is no formal evaluation of performance, but the resulting nearest neighbor sets give a sense of the algorithm's ability to find similar-looking characters in a less well controlled dataset.   Also this is fun.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Give some reference for Soundex.   (Yes, I realize that it might be hard to find a primary reference.   I couldn't do that quickly.   Perhaps a secondary source?)\n\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Make a table of the key facts for all datasets, e.g. language, how many strings to be matched, which OCR engines used.   And give each dataset a short name.   That will allow you to refer to them easily and not end up with so much duplicative description of the datasets.\n\nTry to reduce the repetition between different sections of the paper, e.g. the introduction and section 2.2 and section 6.   \n\nNotice that the limitations section does not count towards the 8-page limit.   So you have space to add examples (e.g. some of the linked text from the first two experiments) and/or make the algorithm easier to follow.   \n\nPerhaps an architecture diagram for the main algorithm in section 2.1?    When I got to the third paragraph in the second column of page 3 (\"To compute character's similarity\") I had to re-read a couple times to see how this part fit together with what you had just done in the first two paragraphs.   2.1 is the critical main part of your algorithm, so use the available page space to make it very very clear and easy to follow.\n\nIn the early parts of section 3, remind us of the language for the first two datasets.\n\nThe abstract needs to be rewritten.   Shorten it and divide into 2-3 paragraphs.   Some key definitions (e.g. ViT) should be repeated in the main text.   \n\nThe paper doesn't have a conclusions section.   The limitations section repeats too much from earlier sections and some of its material feels like it belongs in the missing conclusions.   Also, mention the fact that the test datasets are small.   Everyone will understand why but it is definitely a limitation and perhaps some future author will be inspired to make more gold standard data for this task.\n\n\"Image render\" ---> \"image rendering\"\n\nTable 2:  boldface the best result number"}}, {"review_id": "VHKRTEI5SG", "reviewer": "Reviewer_3GU8", "summary": "", "questions": "A) More clear description about the technical contribution may help understand the paper.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and suggestions, and detail our response below. \n\n*”The way the method uses contrastive learning looks straightforward, so the technical contribution is limited.”*\n\nWe view the technical contributions of the paper as: \n1. **Application of SOTA methods to a novel problem**: To the best of our knowledge, we are the first to use Dino-pretrained ViT’s excellent features (Caron et al 2021) to contrastively learn a metric space for character visual similarity. We do so using a state-of-the-art training objective (Khosla et al 2021) and a highly scaleable FAISS inference architecture. While we have not innovated a new backbone or loss function, we have shown how to apply existing methods to a novel setting in a technically rigorous way. \n2. **Novel data augmentation strategy:** Creating paired data is very costly, and one of the main innovations of the paper is to show that digital fonts can replace costly paired data for training a vision model to encode character visual similarity. \n3. **Novel edit distance metric**: We modified edit distance - the most widely used metric for record linkage - to incorporate character visual similarity, as measured by a learned metric space. The method can be used off-the-shelf by those who lack familiarity with deep learning frameworks (or coding more generally) in one line of code. It can be cheaply extended to any language, unlike the predominant practice of modifying the edit distance substitution cost with hand-crafted features.\n\nWe agree that the contribution is not to improve deep learning in a general sense, but rather to apply state-of-the-art deep learning in such a way that the output can be broadly used, democratizing the benefits of vision transformers for the majority of social scientists who have no familiarity with deep learning frameworks. As mentioned in the paper, end-to-end deep learning approaches fail to generate widespread usage - despite being technically innovative - due to the requirement of in-domain training of models, which requires proficiency with deep learning frameworks. In contrast, with our package users can link two data frames with one line of code, with an API that will be familiar to those users who have experience with statistical packages like R and Stata (which predominate in social science). We feel that the nature of the technical contribution is appropriate for the “Computational Social Science and Cultural Analytics” track. \n\n*”Details of string matching over the proposed similarity metric are not provided. This may significantly reduce the reproducibility of the method if it does not publish the code.”*\n\nWe will distribute a package on PyPi for easy application of homoglyphic distance - with no model training required. Unfortunately, we couldn’t share it as a PyPi package to preserve anonymity, but we have shared the code as part of the supplementary materials. We will also provide a public Github Repo for our training and inference code, useful for replicating our indices and extending our results to other scripts and languages. Besides training and inference code, the repo contains implementations of all string-matching approaches mentioned in the paper that will make it simple to re-generate Table 2 for replication. We will also make all of our data publicly available using Hugging Face Hub once the anonymity period is lifted. Unfortunately, we could not share the image data in the submission packet due to file size limitations. These efforts ensure that we fully adhere to EMNLP’s reproducibility checklist. \n\n*”The paper is redundant, repeating similar sentences. This makes the paper hard to read.”*\n\nBased on your and reviewer 3GU8’s comments, we have edited the paper for concision.\n\n*”The paper claims that the method to train the similarity metric is self-supervised, but for me, this is not self-supervised because it has labels to represent the same characters.”*\n\nWe understand that the reviewer defines label-supervision as something that disqualifies an approach from the paradigm of self-supervised training, but we would also like to highlight that there are precedents for our usage of self-supervised. Conceptually, rendering characters with different fonts is a form of data augmentation, with augmentation on the skeleton of the characters instead of on the entire image. Contrastive training with a standard augmentation scheme that involves operations like affine transformations, color inversion etc. is widely called self-supervised learning (eg, the seminal contribution SimCLR by Chen at al (2020)), and our font-based augmentations are analogous. We simply use instructions for how to render fonts, rather than other image augmentation libraries, to create these augmentations. Just like in standard image augmentations, we generated the labels for free, with no need for human annotations. Semantics aside, the main fact of practical importance is that the method can be extended to virtually *any* Unicode character without the need to create labels, making it highly extensible. We agree that defining self-supervised learning is a tricky task in the context of contrastive learning and thank the reviewer for providing us the space to discuss this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the topic is interesting.", "reasons_to_reject": "The main technical component of the paper is an application of contrastive learning with multiple fonts to learn the similarity between two characters. The way the method uses contrastive learning looks straightforward, so the technical contribution is limited. \n\nDetails of string matching over the proposed similarity metric are not provided. This may significantly reduce the reproducibility of the method if it does not publish the code. \n\nSome minor points:\n\nThe paper is redundant, repeating similar sentences. This makes the paper hard to read. \n\nThe paper claims that the method to train the similarity metric is self-supervised, but for me, this is not self-supervised because it has labels to represent the same characters.", "questions_to_authors": "A) More clear description about the technical contribution may help understand the paper.", "paper_topic_and_main_contributions": "This paper presents a method for matching (hand-written) strings. I think the paper claims that the main contribution of the paper is to use contrastive learning to train a model to compute the similarity between two strings with various fonts. The experimental results show that the method outperforms other similarity metrics for multiple datasets.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new technique for record linkage that uses the visual features of characters, rather than simple edit distance, arguing that this addresses a common problem in OCRed historical text. It uses a clever and novel combination of existing tools (DINO ViT, contrastive loss) to solve a real problem that applied researchers have. While the paper does not make a novel contribution in loss function or model architecture, the combination of tools and the task framing are novel and the paper will be useful. 

One reviewer raised concerns about replicability or whether the process will be easy to use, but another reviewer and the authors point out that the pipeline will be available as a Python package, which will be useful to applied researchers.

The reviewers find that the paper is somewhat repetitive at times and an architecture figure would be helpful, but the authors have agreed to revise the paper to address these issues.

Reviewers raise a number of clarification questions, comments about terminology, and requests for minor extensions/additional citations. The authors thoughtfully respond to each of these and propose changes to the paper that will address them.
Decision: Accept-Main