Cheap Character Noise for OCR-Robust Multilingual Embeddings
Authors: Andrianos Michail, Juri Opitz, Yining Wang, Robin Meister, Rico Sennrich, Simon Clematide
Accepted at: acl25

Cheap Character Noise for OCR-Robust Multilingual Embeddings

Andrianos Michail    Juri Opitz    Yining Wang    Robin Meister
Rico Sennrich    Simon Clematide
University of Zurich
<firstname>.<lastname>@uzh.ch

Abstract

The large amount of text collections digitized by imperfect OCR systems requires semantic search models that perform robustly on noisy input. Such collections are highly heterogeneous, with varying degrees of OCR quality, spelling conventions and other inconsistencies—all phenomena that are underrepresented in the training data of standard embedding models, with ramifications for their generalization. In our paper, we show that this problem can be alleviated with a simple and inexpensive method that does not require supervision or in-domain training. Specifically, we fine-tune existing multilingual models using noisy texts and a contrastive loss. We show that these models show considerable improvements across different noise conditions. Control experiments indicate minimal, and occasionally positive, impact on standard similarity tasks. These findings suggest that embedding models can be inexpensively adapted for cross-lingual semantic search in heterogeneous, digitized corpora. We publicly release our code, datasets, and models at https://github.com/impresso/ocr-robust-multilingual-embeddings.

1 Introduction

Optical Character Recognition (OCR) technology plays a central role in the digitization of historical documents, rendering large volumes of textual data accessible to Natural Language Processing (NLP) tools. To enable effective retrieval from such collections, semantic search based on text embeddings presents an appealing approach (Reimers and Gurevych, 2019; Gao et al., 2021). However, OCR output is inherently imperfect and often introduces errors due to factors such as poor image quality, complex layouts and fonts, low contrast and degradation of the source material (Dhingra et al., 2008). Consequentially, the noisy OCR output can severely degrade the performance of NLP systems by disrupting the syntactic and semantic structure of the text, possibly further exacerbated in multilingual contexts, where models show different performance degradations for different input languages (Lopresti, 2008; Todorov and Colavizza, 2022). A particular problem is that this noise disrupts tokenization processes, resulting in fragmented or incorrect sub-word units (van Strien et al., 2020; Todorov and Colavizza, 2022). Figure 1 showcases an example of ramifications for text embedding models: A multi-lingual embedding model assigns vastly different similarities to two parallel texts, just depending on their font before undergoing the digitization process.

In this work, we empirically analyze how OCR-induced noise affects multilingual embedding models and propose a method based on contrastive fine-tuning with randomly noised text. Our main contributions are:

1. We introduce a simple, cheap, and unsupervised adaptation strategy that improves the robustness of off-the-shelf multilingual embedding models to OCR noise in digitized text collections.

![Same meaning, but different similarity scores. The cosine similarity assigned by the multilingual-gte-base model differs by 28 points when comparing clean French–German sentence pairs to a version in which the German sentence has been OCRed from a Black-letter font. The distorted output contains errors such as “Jahren”, “Niickerstattungsanträge”, and “Grensen gebalten”.](page_1047_355_367_181.png)

Figure 1: Same meaning, but different similarity scores. The cosine similarity assigned by the multilingual-gte-base model differs by 28 points when comparing clean French–German sentence pairs to a version in which the German sentence has been OCRed from a Black-letter font. The distorted output contains errors such as “Jahren”, “Niickerstattungsanträge”, and “Grensen gebalten”.
2. We evaluate our method on challenging text similarity tasks in German and French, using digitized data with diverse types of OCR noise—including realistic, historically difficult conditions (e.g., Blackletter fonts) and minimal-noise, modern digitization. Despite its simplicity, our approach improves performance across all conditions, including on a language unseen during training. Moreover, it does not degrade performance on clean text similarity benchmarks.

3. We conduct detailed analyses of training strategies and investigate subtokenization mismatches as a potential source of performance degradation in the off-the-shelf models (an issue that our method then helps to mitigate).

2 Background and Related Work

Semantic search refers to a class of tasks aimed at retrieving texts that are semantically similar, or relevant to a given query. Today’s dominant paradigm to address this tasks is embedding texts as vectors with pre-trained encoders fine-tuned on contrastive tasks (Reimers and Gurevych, 2019). In many tasks, this technique has been proven efficient and effective. However, these tasks typically involve contemporary and more normative text. Hence, it is unclear whether the models maintain their performance in noisy and highly heterogeneous text collections, as we would encounter in large digitized collections of text, e.g., within institutions such as libraries.

Typo Robustness. A related line of research investigates the robustness of dense retrieval and embedding models to typos and character-level perturbations. While OCR errors and human typos differ in origin, both introduce noise that disrupts tokenization and embedding similarity. Tasawong et al. (2023) propose a robust dense retrieval training method that aligns representations of pristine and misspelled queries while enforcing contrast to distinguish unrelated inputs. Their approach combines query augmentation with a dual self-teaching loss, significantly improving performance on both synthetic and real-world misspelled queries without sacrificing accuracy on clean inputs. Similarly, Sidiropoulos and Kanoulas (2022) examine the vulnerability of dual encoder architectures to misspellings, finding that even minor character-level changes can substantially degrade retrieval performance. They propose contrastive learning-based strategies to mitigate this sensitivity.

Training models on character-level noise has been shown to be effective for machine translation (Belinkov and Bisk, 2018; Vaibhav et al., 2019; Sperber et al., 2017) and cross-lingual transfer between closely related languages (Aeppli and Sen- nrich, 2022). We demonstrate that multilingual embedding models have worse semantic search capabilities within digitized collections due to the presence of OCR errors. We then evaluate versions of further training approaches that aim to bring closer the representations of randomly noised text with its original version and show increased capabilities of semantic search in digitized collections even for languages pairs not included in our noised fine-tuning.

3 Notation and Preliminaries

Throughout this work, we make use of multilingual parallel datasets that may contain noise of different degree. In particular, the term parallel here does not necessarily only mean that there are pairs with texts from two different languages, but it can also mean, e.g., that the dataset is monolingual but contains pairs of texts where one part contains OCR noise. To denote such a dataset we use \( \text{DataSets}^{ns}_{ls\rightarrownt} \). In this notation, \( ns \) (\( nt \)) are variables that denote the type of observed noise, whereas \( ls \) (\( lt \)) refer to the source (target) language within the dataset (if they differ). We formalize forms of OCR noise within the following categories:

• Random Noise (RN): A fully unsupervised stochastic noise. This will only be used for training in our proposed method.

• Minimal Noise (MN): Simplest of noise produced by OCR engines when the fonts, font size and resolution of the image is clear.

• Salt and Pepper Noise (SnP): A noising condition which emulates tear and wear on paper by adding small (very subtle) sprinkles of (“Salt”) and (“Pepper”) throughout the scan, hence the name SnP. See also the example in Figure 1, top right.

• BlackLetter (BL): Blackletter fonts, also known as Gothic script or Fraktur (Figure 1, bottom right), were prevalent in German-speaking regions until the early 20th century. These typefaces are characterized by their
dense, angular letterforms. Since Blackletter wasn’t used in French texts, for French texts we assume an equivalent (in terms of OCR parsing difficulty) of Scanned Distorted Noise (\( SD \)). This is characterized by irregular distortions and artifacts introduced during the scanning or image processing stages, which make the text difficult to recognize accurately by OCR systems.

The languages that we mention in our paper are German (de), French (fr), English (en), Luxembourgish (lb), Turkish (tr) and Arabic (ar). For example, in a parallel DATASET, we consider a case where French source sentences in standard clean text are paired with German text OCRed from Blackletter font. We denote this dataset as DATASET\(_{fr\rightarrow de}^{OCRed}\). A full description of the generation process for our multilingual noised training data and a separate description of generating the realistic OCR evaluation data follows in §5.

4 Method Overview

The core idea of our method is straightforward: We generate a training set by inducing random character-level noise into a given dataset. This enables us to fine-tune a pre-trained embedding model to better align clean and character-noised versions of the same text using a contrastive loss objective. Our study focuses on lightweight adaptations of multilingual models, requiring only 20,000 positive pairs and modest computational resources (approximately 20 minutes on a single Tesla T4 16GB GPU). Because the method is fully unsupervised and language-agnostic, it can be easily applied to adapt multilingual embedding models for improved robustness to OCR-induced noise.

Figure 2: Lengths per sentence in tokens for German and French in the TED and X-News corpora.

5 Experimental Setup

5.1 Creating Noised Training Data

To train our proposed models, we use two bilingual corpora. The first is the NeuLabs (TED) corpus (Qi et al., 2018), which contains French and German translations of TED Talk transcripts. The second is the ELRC-CORDIS News corpus (X-News), a parallel dataset of French and German sentences from the news domain—our primary application focus. For fine-tuning, we sample 20,000 positive pairs from each corpus. Descriptive statistics for both datasets are shown in Figure 2.

We augment the texts in these datasets by generating noisy variants through stochastic, unsupervised character-level modifications that simulate typical OCR errors. Specifically, we apply character-level perturbations to 5% of the characters in the original text. These modifications are randomly drawn from three types of noise: (1) substitution, where a character is replaced with a randomly selected alternative to mimic misrecognition; (2) insertion, where a random character is inserted at an arbitrary position; and (3) deletion, where a randomly chosen character is removed. This process produces a noisy version of each sentence, which we then pair with its original clean counterpart to form positive training examples.

We refer to the resulting datasets as *noised TED* and *noised X-News*. In each dataset, one side contains the original (clean) text, while the other contains its stochastically noised counterpart. In our dataset notation, *noised TED* is defined as TED\(_{de\rightarrow en}^{OCRed}\) \( \cup \) TED\(_{fr\rightarrow de}^{OCRed}\), comprising monolingual sentence pairs. In contrast, *noised X-news* is defined as X-News\(_{de\rightarrow fr}^{OCRed}\) \( \cup \) X-News\(_{fr\rightarrow de}^{OCRed}\), containing cross-lingual sentence pairs. Both datasets consist exclusively of positive pairs and are used to fine-tune embedding models for greater robustness to OCR noise.

5.2 Main Evaluation Datasets

We evaluate the effectiveness of our proposed adaptation method using cross-lingual datasets that include realistic and naturally occurring OCR noise. This evaluation is organized in two parts: *Evaluation task* and *Inducing realistic OCR noise*.

5.2.1 Evaluation task.
We adopt the Cross-Lingual Semantic Discrimination (CLSD) German-French evaluation benchmark introduced by Michail et al. (2025a). This
task assesses whether multilingual embedding models can accurately identify the correct cross-lingual semantic match in the presence of challenging distractors. It is based on parallel sentence pairs from the WMT19 and WMT21 DE–FR news test sets (Barrault et al., 2019; Akhbardeh et al., 2021). Specifically, each datum consists of a source sentence, its correct translation in the target language, and a set of four semantically similar distractor sentences in the target language. The model must produce an embedding for the source sentence that is more similar to the target sentence than to any distractors. Performance is evaluated using accuracy, reported as Precision@1.

5.2.2 Inducing realistic OCR noise

We then construct our primary evaluation datasets by generating three realistic OCR noise variants of the CLSD dataset: Minimal Noise (MN), Blackletter (BL), and Salt and Pepper (SnP).

To produce the Minimal Noise (MN) realistic OCR noise, we print the text in font Times New Roman at font size 10 and save it as an image at 300 pixels per inch (PPI). We then apply OCR using Tesseract 3\(^1\), a widely used open-source OCR engine. This setup introduces light OCR errors, resulting in average character error rates (CER) of 0.4% for German and 0.6% for French.

For the Blackletter (BL) condition, we follow the same procedure but render the German text using Canterbury, a blackletter-style font commonly found in historical print. We pair this with a Scanned Distorted (SD) variant on the French side, which simulates scanning artifacts by introducing horizontal offsets and random spacing distortions within characters. This setting yields CERs of 2.8% for German and 2.4% for French.

For the Salt and Pepper (SnP) condition, we again follow the base procedure but add synthetic background noise by randomly scattering black and white pixels at a density of 0.45%. This visual noise mimics degradation due to paper aging or scanning artifacts, producing average CERs of 5.4% for German and 5.1% for French.

In evaluation, we annotate the source and target language noise types independently. For instance, evaluating CLSD WMT19 where both the French source and German target contain SnP noise is denoted as WMT19\(_{de\rightarrow fr}^{SnP\rightarrow SnP}\).

\footnotetext{1https://github.com/tesseract-ocr/tesseract}

5.3 Control Task and Dataset

To ensure that our adaptation method does not negatively affect performance on standard semantic search tasks involving clean text, we conduct a set of control experiments. Specifically, we assess whether improvements on OCR-noisy inputs come at the cost of degrading the model’s ability to handle more normative, noise-free text.

Thus, as a baseline for our evaluations, we use the clean (OCR-free) variant of the CLSD dataset, along with the multilingual Semantic Textual Similarity (STS) benchmark introduced by Cer et al. (2017). For STS, we focus on language pairs that are not present in our adaptation training data: STS\(_{en\leftrightarrow tr}\) (English–Turkish), STS\(_{en\leftrightarrow es}\) (English–Spanish), and STS\(_{en\leftrightarrow ar}\) (English–Arabic).

5.4 Historic Luxembourgish Bitext Mining

We further evaluate our adapted models using the Historical Bitext Mining task introduced by Michail et al. (2025b), referred to here as HISTLUX. This dataset consists of digitized historical Luxembourgish newspaper articles (published between 1840 and 1950), which have been segmented into sentences and manually translated into clean, modern German and French. As it contains naturally occurring digitization errors, HISTLUX provides a valuable testbed for assessing model robustness beyond simulated OCR noise.

5.5 Embedding Models

We evaluate our adaptation strategies using two state-of-the-art multilingual embedding models:

• M-E5B: Proposed by Wang et al. (2024a,b), this model is trained via weakly supervised contrastive pre-training on multilingual data, followed by supervised fine-tuning on information retrieval tasks. We use the multilingual-E5-base version, available on Hugging Face.

• M-GTE: Introduced by Zhang et al. (2024), this model is based on a multilingual long-context encoder and trained using a similar contrastive objective to M-E5B, but with a greater emphasis on difficult negatives. Our experiments use the gte-multilingual-base variant.

Our primary baseline to compare against is the unmodified base model (aka off-the-shelf model),
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Approach</th>
    <th colspan="4">clean → OCR</th>
    <th colspan="4">OCR → OCR</th>
    <th colspan="2">Averages</th>
  </tr>
  <tr>
    <th>WMT19<sup>de→fr</sup></th>
    <th>WMT21<sup>de→fr</sup></th>
    <th>WMT19<sup>fr→de</sup></th>
    <th>WMT21<sup>fr→de</sup></th>
    <th>WMT19<sup>de→fr</sup></th>
    <th>WMT21<sup>de→fr</sup></th>
    <th>WMT19<sup>fr→de</sup></th>
    <th>WMT21<sup>fr→de</sup></th>
    <th>clean → OCR</th>
    <th>OCR → OCR</th>
  </tr>
  <tr>
    <th colspan="13">Blackletter / Scanned Distorted Noise</th>
  </tr>
  <tr>
    <td>M-GTE</td>
    <td>base</td>
    <td>80.9</td>
    <td>80.0</td>
    <td>77.5</td>
    <td>78.1</td>
    <td>78.2</td>
    <td>76.5</td>
    <td>76.4</td>
    <td>77.7</td>
    <td>79.1</td>
    <td>77.2</td>
  </tr>
  <tr>
    <td></td>
    <td><i>clean X-News</i></td>
    <td>80.6</td>
    <td>79.6</td>
    <td>80.6</td>
    <td>77.8</td>
    <td>77.7</td>
    <td>75.5</td>
    <td>75.9</td>
    <td>76.4</td>
    <td><b>79.6<sub>+0.5</sub></b></td>
    <td><b>76.4<sub>-0.8</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised TED</i></td>
    <td>85.3</td>
    <td>84.3</td>
    <td>82.0</td>
    <td>81.9</td>
    <td>82.3</td>
    <td>81.2</td>
    <td>80.3</td>
    <td>80.7</td>
    <td><b>83.4<sub>+4.3</sub></b></td>
    <td><b>81.1<sub>+3.9</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised X-News</i></td>
    <td>82.3</td>
    <td>80.2</td>
    <td>79.1</td>
    <td>79.1</td>
    <td>79.4</td>
    <td>75.9</td>
    <td>77.6</td>
    <td>77.9</td>
    <td><b>80.2<sub>+1.1</sub></b></td>
    <td><b>77.7<sub>+0.5</sub></b></td>
  </tr>
  <tr>
    <td>M-E5B</td>
    <td>base</td>
    <td>79.6</td>
    <td>75.3</td>
    <td>72.4</td>
    <td>69.2</td>
    <td>76.3</td>
    <td>71.8</td>
    <td>72.3</td>
    <td>68.1</td>
    <td>74.1</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td></td>
    <td><i>clean X-News</i></td>
    <td>78.3</td>
    <td>77.3</td>
    <td>75.4</td>
    <td>74.3</td>
    <td>75.0</td>
    <td>74.2</td>
    <td>74.7</td>
    <td>73.0</td>
    <td><b>76.3<sub>+2.2</sub></b></td>
    <td><b>74.2<sub>+2.1</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised TED</i></td>
    <td>81.0</td>
    <td>78.7</td>
    <td>78.7</td>
    <td>76.3</td>
    <td>77.1</td>
    <td>76.3</td>
    <td>77.5</td>
    <td>75.8</td>
    <td><b>78.7<sub>+4.6</sub></b></td>
    <td><b>76.6<sub>+4.5</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised X-News</i></td>
    <td>80.2</td>
    <td>78.0</td>
    <td>78.5</td>
    <td>77.7</td>
    <td>77.2</td>
    <td>75.2</td>
    <td>76.9</td>
    <td>76.6</td>
    <td><b>78.6<sub>+4.5</sub></b></td>
    <td><b>76.4<sub>+4.3</sub></b></td>
  </tr>
  <tr>
    <th colspan="13">Salt and Pepper Noise</th>
  </tr>
  <tr>
    <td>M-GTE</td>
    <td>base</td>
    <td>82.2</td>
    <td>81.8</td>
    <td>81.3</td>
    <td>80.1</td>
    <td>82.1</td>
    <td>82.9</td>
    <td>81.9</td>
    <td>80.9</td>
    <td>81.3</td>
    <td>81.9</td>
  </tr>
  <tr>
    <td></td>
    <td><i>clean X-News</i></td>
    <td>81.8</td>
    <td>82.1</td>
    <td>80.2</td>
    <td>80.9</td>
    <td>80.8</td>
    <td>82.1</td>
    <td>81.8</td>
    <td>80.6</td>
    <td><b>81.2<sub>-0.1</sub></b></td>
    <td><b>81.4<sub>-0.5</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised TED</i></td>
    <td>85.0</td>
    <td>84.5</td>
    <td>82.9</td>
    <td>81.4</td>
    <td>85.4</td>
    <td>85.0</td>
    <td>84.5</td>
    <td>83.1</td>
    <td><b>83.5<sub>+2.2</sub></b></td>
    <td><b>84.5<sub>+2.6</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised X-News</i></td>
    <td>82.3</td>
    <td>82.2</td>
    <td>81.0</td>
    <td>80.1</td>
    <td>81.5</td>
    <td>83.2</td>
    <td>82.2</td>
    <td>81.0</td>
    <td><b>81.4<sub>+0.1</sub></b></td>
    <td><b>82.0<sub>+0.1</sub></b></td>
  </tr>
  <tr>
    <td>M-E5B</td>
    <td>base</td>
    <td>81.8</td>
    <td>74.9</td>
    <td>76.9</td>
    <td>71.8</td>
    <td>77.1</td>
    <td>70.2</td>
    <td>78.5</td>
    <td>75.4</td>
    <td>76.4</td>
    <td>75.3</td>
  </tr>
  <tr>
    <td></td>
    <td><i>clean X-News</i></td>
    <td>77.9</td>
    <td>78.4</td>
    <td>79.6</td>
    <td>77.2</td>
    <td>77.9</td>
    <td>78.6</td>
    <td>79.6</td>
    <td>77.4</td>
    <td><b>78.3<sub>+1.9</sub></b></td>
    <td><b>78.4<sub>+3.1</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised TED</i></td>
    <td>80.3</td>
    <td>79.1</td>
    <td>82.1</td>
    <td>78.0</td>
    <td>81.1</td>
    <td>78.9</td>
    <td>82.5</td>
    <td>79.9</td>
    <td><b>79.9<sub>+3.5</sub></b></td>
    <td><b>80.6<sub>+5.3</sub></b></td>
  </tr>
  <tr>
    <td></td>
    <td><i>noised X-News</i></td>
    <td>78.5</td>
    <td>79.0</td>
    <td>79.5</td>
    <td>77.4</td>
    <td>78.9</td>
    <td>79.7</td>
    <td>79.9</td>
    <td>78.4</td>
    <td><b>78.6<sub>+2.2</sub></b></td>
    <td><b>79.2<sub>+3.9</sub></b></td>
  </tr>
</table>

Table 1: Main German–French results. All values are averaged over five fine-tuning seeds. <i>Italicized</i> entries indicate results from our proposed adaptation approach.

which allows us to assess how well these embedding models perform on digitized text without any additional training—reflecting their typical use in real-world applications.

To isolate the effect of our noising strategy, we additionally fine-tune the model on the X-News corpus without any injected OCR noise. We refer to this condition as <i>clean X-News</i>.

All model training follows a consistent setup: we use the standard <i>MultipleNegativesRankingLoss</i> (Henderson et al., 2017), a batch size of 8, and train for one epoch using 20,000 positive pairs.

6 Experimental Results

We evaluate our approach in two scenarios: (1) clean → OCR, representing a typical user-query setting where clean input is used to search over noisy, digitized content; and (2) OCR → OCR, representing semantic search conducted entirely within OCR-processed corpora.

6.1 Main DE–FR Results

Our primary German–French results are presented in Table 1. Across both embedding models, our simple adaptation method yields substantial performance gains. Notably, these improvements persist even when training uses out-of-domain data. When
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Approach</th>
    <th colspan="4">X-Semantic Search (CLSD)</th>
    <th colspan="3">Similarity</th>
    <th colspan="2">Averages</th>
  </tr>
  <tr>
    <th>WMT19<sup>de→fr</sup></th>
    <th>WMT19<sup>en→de</sup></th>
    <th>WMT19<sup>en→fr</sup></th>
    <th>WMT19<sup>en→en</sup></th>
    <th>STS<sub>en→es</sub></th>
    <th>STS<sub>en→ar</sub></th>
    <th>STS<sub>en→ar</sub></th>
    <th>CLSD</th>
    <th>STS</th>
  </tr>
  <tr>
    <td rowspan="4">M-GTE</td>
    <td>base</td>
    <td>90.2</td>
    <td>90.5</td>
    <td>89.7</td>
    <td>91.6</td>
    <td>83.9</td>
    <td>76.4</td>
    <td>76.4</td>
    <td>90.5</td>
    <td>78.9</td>
  </tr>
  <tr>
    <td><i>clean X-News</i></td>
    <td>91.1</td>
    <td>91.0</td>
    <td>91.1</td>
    <td>92.5</td>
    <td>82.6</td>
    <td>77.0</td>
    <td>76.9</td>
    <td>91.4<sup>+0.9</sup></td>
    <td>78.9<sup>+0.0</sup></td>
  </tr>
  <tr>
    <td><i>noised TED</i></td>
    <td>93.3</td>
    <td>93.4</td>
    <td>92.5</td>
    <td>93.3</td>
    <td>80.8</td>
    <td>75.4</td>
    <td>76.4</td>
    <td>93.1<sup>+2.6</sup></td>
    <td>77.5<sup>-1.4</sup></td>
  </tr>
  <tr>
    <td><i>noised X-News</i></td>
    <td>91.7</td>
    <td>91.4</td>
    <td>90.6</td>
    <td>92.7</td>
    <td>82.3</td>
    <td>76.1</td>
    <td>76.7</td>
    <td>91.6<sup>+1.1</sup></td>
    <td>78.4<sup>-0.5</sup></td>
  </tr>
  <tr>
    <td rowspan="4">M-E5B</td>
    <td>base</td>
    <td>91.5</td>
    <td>86.3</td>
    <td>88.5</td>
    <td>82.0</td>
    <td>76.6</td>
    <td>63.3</td>
    <td>71.3</td>
    <td>87.1</td>
    <td>70.4</td>
  </tr>
  <tr>
    <td><i>clean X-News</i></td>
    <td>90.9</td>
    <td>91.5</td>
    <td>90.5</td>
    <td>89.7</td>
    <td>77.4</td>
    <td>68.7</td>
    <td>74.9</td>
    <td>90.7<sup>+3.6</sup></td>
    <td>73.7<sup>+3.3</sup></td>
  </tr>
  <tr>
    <td><i>noised TED</i></td>
    <td>90.1</td>
    <td>89.9</td>
    <td>91.2</td>
    <td>87.6</td>
    <td>75.2</td>
    <td>66.7</td>
    <td>73.9</td>
    <td>89.7<sup>+2.6</sup></td>
    <td>71.9<sup>+1.5</sup></td>
  </tr>
  <tr>
    <td><i>noised X-News</i></td>
    <td>91.3</td>
    <td>92.0</td>
    <td>91.0</td>
    <td>90.0</td>
    <td>76.3</td>
    <td>69.1</td>
    <td>74.1</td>
    <td>91.1<sup>+4.0</sup></td>
    <td>73.2<sup>+2.8</sup></td>
  </tr>
</table>

Table 2: Results of control experiments on clean test data, i.e., without OCR noise. STS evaluation is based on Spearman correlation with human Likert-scale scores. All values are averaged over five fine-tuning seeds. <i>Italicized</i> entries indicate results from our proposed adaptation approach.

<table>
  <tr>
    <th>Approach</th>
    <th>lb→fr</th>
    <th>lb→en</th>
    <th>lb→de</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>base</td>
    <td>83.7</td>
    <td>80.1</td>
    <td>87.6</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td><i>clean X-News</i></td>
    <td>83.6</td>
    <td>81.6</td>
    <td>86.0</td>
    <td>83.7<sub>-0.1</sub></td>
  </tr>
  <tr>
    <td><i>noised TED</i></td>
    <td>87.1</td>
    <td>86.2</td>
    <td>90.8</td>
    <td>88.0<sup>+4.2</sup></td>
  </tr>
  <tr>
    <td><i>noised X-News</i></td>
    <td>84.7</td>
    <td>83.0</td>
    <td>87.1</td>
    <td>84.9<sup>+1.1</sup></td>
  </tr>
</table>

Table 3: Accuracy of our adapted models on the HistLUX Bitext Mining evaluation set using M-GTE.

fine-tuning on the <i>noised TED</i> dataset and evaluating on Blackletter (BL) OCR text, we observe average improvements of +4.3 and +4.6 points (for M-E5B and M-GTE, respectively) when clean queries are matched against noisy candidates, and gains of +4.0 and +4.6 points when both query and candidate contain BL noise.

Similarly, for Salt and Pepper (SnP) noise, fine-tuning on <i>noised TED</i> results in average gains of +2.2 and +3.6 points when only the candidate text is noisy, and +2.6 and +5.3 points when noise is present in both query and candidate.

Fine-tuning on domain-aligned data (<i>noised X-News</i>) yields improvements in a comparable range, indicating the robustness and generalizability of our adaptation strategy.

Looking more closely at individual evaluations, we observe consistent improvements across all dataset variations and both directions of the language pair. The largest gain is observed in WMT19<sup>SnP→SnP</sup><sub>de→fr</sub>, with an improvement of +8.7 points. Notably, some improvements also occur when fine-tuning on the <i>clean X-News</i> baseline, particularly for the less robust M-E5B model.

Finally, we perform statistical significance testing. Our null hypothesis is that the proposed random noise strategy yields correct predictions with equal likelihood as any of the baseline strategies (i.e., using the base model or training on clean data). Using Fisher’s exact test, we reject the null hypothesis with high confidence (p ≪ 0.001). This result holds consistently across both embedding models (M-E5B and M-GTE).

6.2 Performance on Historic Luxembourgish

In this experiment, we evaluate our models on HistLUX, a dataset consisting of historical Luxembourgish text that has been OCR-processed by real digitization facilities. Notably, Luxembourgish is not included in any of our adaptation training data, making this a challenging generalization task for our method.2

The results are presented in Table 3. We observe consistent improvements with both of our noise-based adaptation strategies, particularly when fine-tuning on <i>noised TED</i>, which yields an average improvement of +4.2 points. In contrast, fine-

2Since the M-E5B model does not support Luxembourgish, we restrict this evaluation to the M-GTE model, which was exposed to 48,000 Luxembourgish sentence pairs during its pre-release contrastive pre-training (c.f. Table 10 in Zhang et al., 2024).
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Approach</th>
    <th colspan="4">clean → OCR</th>
    <th colspan="4">OCR → OCR</th>
    <th colspan="2">Averages</th>
  </tr>
  <tr>
    <th>WMT19<sup>de→en</sup></th>
    <th>WMT21<sup>de→en</sup></th>
    <th>WMT19<sup>fr→de</sup></th>
    <th>WMT21<sup>fr→de</sup></th>
    <th>WMT19<sup>de→en</sup></th>
    <th>WMT21<sup>de→en</sup></th>
    <th>WMT19<sup>fr→de</sup></th>
    <th>WMT21<sup>fr→de</sup></th>
    <th>clean → OCR</th>
    <th>OCR → OCR</th>
  </tr>
  <tr>
    <th colspan="13">Minimal Noise | Morgen | Bonjour</th>
  </tr>
  <tr>
    <th rowspan="4">M-GTE</th>
    <td>base</td>
    <td>87.3</td>
    <td>88.7</td>
    <td>89.5</td>
    <td>90.6</td>
    <td>87.2</td>
    <td>88.7</td>
    <td>88.8</td>
    <td>90.4</td>
    <td>89.0</td>
    <td>88.8</td>
  </tr>
  <tr>
    <td><i>clean X-News</i></td>
    <td>88.4</td>
    <td>89.3</td>
    <td>89.8</td>
    <td>91.2</td>
    <td>88.2</td>
    <td>88.9</td>
    <td>89.6</td>
    <td>90.7</td>
    <td>89.7<sup>+0.7</sup></td>
    <td>89.3<sup>+0.5</sup></td>
  </tr>
  <tr>
    <td><i>noised TED</i></td>
    <td>91.2</td>
    <td>92.4</td>
    <td>91.9</td>
    <td>92.7</td>
    <td>91.1</td>
    <td>92.5</td>
    <td>91.4</td>
    <td>92.5</td>
    <td>92.0<sup>+3.0</sup></td>
    <td>91.9<sup>+3.1</sup></td>
  </tr>
  <tr>
    <td><i>noised X-News</i></td>
    <td>89.3</td>
    <td>89.9</td>
    <td>90.4</td>
    <td>91.8</td>
    <td>89.2</td>
    <td>89.6</td>
    <td>90.0</td>
    <td>91.4</td>
    <td>90.3<sup>+1.3</sup></td>
    <td>90.0<sup>+1.2</sup></td>
  </tr>
  <tr>
    <th rowspan="4">M-E5B</th>
    <td>base</td>
    <td>88.9</td>
    <td>84.9</td>
    <td>86.7</td>
    <td>80.4</td>
    <td>88.7</td>
    <td>84.5</td>
    <td>86.2</td>
    <td>80.7</td>
    <td>85.2</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td><i>clean X-News</i></td>
    <td>87.8</td>
    <td>90.4</td>
    <td>89.9</td>
    <td>88.0</td>
    <td>87.7</td>
    <td>90.1</td>
    <td>89.4</td>
    <td>88.1</td>
    <td>89.0<sup>+3.8</sup></td>
    <td>88.8<sup>+3.8</sup></td>
  </tr>
  <tr>
    <td><i>noised TED</i></td>
    <td>88.3</td>
    <td>88.4</td>
    <td>90.6</td>
    <td>86.9</td>
    <td>88.1</td>
    <td>88.4</td>
    <td>90.3</td>
    <td>86.5</td>
    <td>88.5<sup>+3.3</sup></td>
    <td>88.3<sup>+3.3</sup></td>
  </tr>
  <tr>
    <td><i>noised X-News</i></td>
    <td>89.0</td>
    <td>90.8</td>
    <td>90.5</td>
    <td>89.1</td>
    <td>88.9</td>
    <td>90.9</td>
    <td>90.0</td>
    <td>89.0</td>
    <td>89.8<sup>+4.6</sup></td>
    <td>89.7<sup>+4.7</sup></td>
  </tr>
</table>

Table 4: Results under minimal noise conditions. All values are averaged over five fine-tuning seeds. <i>Italicized</i> entries indicate results from our proposed adaptation approach.

tuning on clean, domain-aligned data (<i>clean X-News</i>) does not lead to any measurable gains. The most substantial improvement is observed for the Luxembourgish–English direction, with a gain of +6.1 points following adaptation on <i>noised TED</i>.

6.3 Control Experiments

Having observed substantial improvements on OCR-noisy texts, a key question arises: does this adaptation come at the cost of performance on standard, clean data? To evaluate this, we conduct control experiments on <i>contemporary, clean, cross-lingual, and out-of-language</i> semantic textual similarity tasks. Specifically, we use the English–Turkish, English–Spanish, and English–Arabic test sets from the multilingual STS benchmark: STS<sub>en+tr</sub>, STS<sub>en+es</sub>, and STS<sub>en+ar</sub>.

The results of these experiments are shown in Table 2. On the clean CLSD evaluation set, we observe no performance degradation. In fact, fine-tuning on <i>noised X-News</i> yields average gains of up to +4.0 points. For the STS benchmarks in previously unseen languages, results vary depending on the model: M-E5B shows modest improvements (up to +3.3 points), while M-GTE experiences small declines (up to −1.4 points).

Overall, we find that adaptation to OCR noise does not negatively impact performance on clean, cross-lingual tasks and remains stable across unrelated language pairs.

6.4 Minimal Noise Conditions

To assess whether our approach remains beneficial when applied to high-quality OCR text with minimal degradation, we evaluate the Minimal Noise (MN) variant of the CLSD dataset. This variant, which simulates modern digitized text with clean layout and high-resolution input, has a Character Error Rate (CER) below 0.6%.

We evaluate both the baseline and adapted models on this dataset, as shown in Table 4. As expected, the base models suffer significantly less performance degradation under MN conditions compared to the more challenging Blackletter (BL) and Salt-and-Pepper (SnP) scenarios. Nonetheless, our adaptation strategies continue to yield gains. Specifically, the M-GTE model improves by up to 3.0 points when fine-tuned on <i>noised TED</i>, and the M-E5B model improves by up to 4.7 points when trained on <i>noised X-News</i>.

These results indicate that even in cleaner OCR scenarios, our models benefit from the proposed adaptation method and maintain strong performance in modern digitization settings.
Figure 3: Precision, Recall and F1 scores of the sets of subtokens in the noisy text against their clean equivalent.

7 Discussion

Generalization issues in off-the-shelf models: Is it the tokenizer’s fault? Our adaptation strategies reduce the performance drop that standard multilingual embedding models exhibit when processing OCR-noisy text. A central question, however, is what causes these models to generalize poorly to such noisy input.

We hypothesize that the tokenizer plays a crucial role in this degradation. Modern embedding models rely on subword tokenization, and OCR-induced character perturbations often result in token boundary mismatches. These mismatches propagate through the network layers, distorting semantic representations.

To investigate this hypothesis, we examine the XLM-RoBERTa tokenizer (Conneau et al., 2020) used by both M-ESB and M-GTE. This tokenizer is built on the SentencePiece unigram algorithm (Kudo, 2018; Kudo and Richardson, 2018). We start by analyzing words and short phrases from original noisy text pairs in the CLSD evaluation set (see Appendix Table 6 for examples). Our qualitative analysis reveals that even minor recognition errors can significantly disrupt the tokenization process. For instance, misreading the character “ö” as “ö” in the German word “können” (English: “can”) causes what is normally a single meaningful subtoken to be split into three smaller, less interpretable fragments. Similarly, inserting an extra “i” in the French word “comme” (English: “like”, used as a conjunction) results in the unintended subtokens "com" and "mie".

To quantify the effect of the subtokenization changes, we compare the tokenizations of clean and OCRed text by measuring their vocabulary overlap. Specifically, we compute precision, recall and F1 scores based on the sets of produced subtokens. The results are presented in Figure 3. We observe that the more noisy the conditions, the less of the original subtokens are preserved by the subtokenizer, a finding that seems to align well with the observed performance degradation of the baseline models in our main experiments. It is also interesting that while texts with BL noise have about half the character error rate of SnP, they preserve about the same amount of subtokens from the original texts. This suggests that misrecognitions, which occur commonly in difficult-to-parse fonts such as BL, alter the subtokenization more than insertion errors, which are more prevalent under SnP conditions.

These findings suggest that our adaptation strategy may help correct some of the harmful effects by misaligned tokenization, thereby reducing the propagation of errors through the model. A more detailed investigation of this mechanism is beyond the scope of the present study and is left for future work.

<table>
  <tr>
    <th>Method</th>
    <th>BL/SD</th>
    <th>SnP</th>
    <th>HistLUX</th>
  </tr>
  <tr>
    <td>Base</td>
    <td>78.1</td>
    <td>81.6</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td>Random Noise</td>
    <td>82.2</td>
    <td>84.0</td>
    <td><b>87.9</b></td>
  </tr>
  <tr>
    <td>BL/SD Noise</td>
    <td><b>83.2</b></td>
    <td>86.0</td>
    <td>84.3</td>
  </tr>
  <tr>
    <td>SnP Noise</td>
    <td>82.8</td>
    <td><b>86.1</b></td>
    <td>82.8</td>
  </tr>
</table>

Table 5: Performance comparison of models trained with different noise types across three evaluation settings: CLSD with Blackletter/Scanned (BL/SD) noise, CLSD with Salt-and-Pepper (SnP) noise, and the historical HistLUX benchmark. While training on realistic noise (BL/SD or SnP) improves performance on the corresponding noise type, models trained with random noise show more consistent generalization, including on naturally noisy historical data.
Figure 4: Ablation study comparing the effects of batch composition and training domain on model performance. “TED Noised (Mono)” refers to monolingual batching (one language per batch), while “TED Noised (Mixed)” uses mixed-language batches. “X-News Noised (Mixed)” applies the same mixed batching strategy on a different domain. Results are shown for three evaluation settings: CLSD with Blackletter/Scanned (BL/SD) noise, CLSD with Salt-and-Pepper (SnP) noise, and the historical HistLUX benchmark. In this experiment, X-News is used as a monolingual training set.

Training regime: Which strategy performs best?
We examine how the batch mixing strategy affects performance in our best adaptation setup, noised TED. To do so, we replicate the results with monolingual batches3, allowing us to isolate the effect of language mixing during training. In addition, to assess the impact of domain similarity, we conduct parallel experiment using the noised X-News corpus, also with monolingual batching, while keeping the training procedure identical to noised TED.

We show the results of this experiment in Figure 4. We observe negligible differences between mixed and monolingual batching strategies. However, training on the X-News dataset results in slightly lower performance than TED, suggesting that while domain may have some influence, the choice of training corpus plays only a minor role in the overall effectiveness of the noise adaptation.

Noise type: Does training on realistic OCR noise help? We investigate whether training models on more realistic OCR noise improves performance and generalization across different noise types. To this end, we train two variants of our models using the noised TED dataset, with noise generated via our realistic Blackletter (B)L and Salt-n-Pepper (SnP) pipelines.

3“Mixed” refers to batches with samples from multiple languages; “Mono” to batches with samples from a single language.

As shown in Table 5, training on realistic noise improves robustness within the respective noise condition to a similar extent as training on random noise. However, it yields smaller gains on other types of noise. Notably, performance drops most clearly on the HistLUX benchmark, which contains naturally occurring OCR noise. In this setting, both realistic-noise models perform worse than the base model (by -0.4 and -2.0 points) and substantially worse than the model trained on random noise (by -3.8 and -5.4 points).

These results indicate that while training on (more costly) realistic noise can enhance robustness to that specific noise type, it reduces generalization to other types of noise and real-world OCR artifacts. In contrast, our random noise approach offers broader transferability at a lower cost.

8 Conclusions

We propose an inexpensive and effective strategy for adapting multilingual embedding models to be more robust to heterogeneous digitized text. Our adaptation approach improves performance across different noise conditions and even over historical digitized text in related languages. Through control experiments, we show that our method has minimal impact on the overall embedding quality. We find that OCR noise increases token fragmentation, which may explain the observed performance degradation. We believe that our work is an important step towards building and evaluating reliable semantic search systems for large and diverse digitized text collections.

Acknowledgments

We thank the anonymous reviewers for their suggestions. The authors received funding through the project Impresso – Media Monitoring of the Past II Beyond Borders: Connecting Historical Newspapers and Radio. Impresso is a research project funded by the Swiss National Science Foundation (SNSF 213585) and the Luxembourg National Research Fund (17498891).

Limitations

Our study demonstrates that multilingual embedding models can be made more robust to noise commonly found in digitized texts. However, such noise may not only consist of actual errors (e.g., from OCR systems), but it can also include other phenomena, such as typos in newspapers, regional
or historical spelling variants. It is possible that the methodology proposed here improves the representation of such texts, but we are unable to assess this in our study. Next, we plan to scale our adaptation procedure to a massively multilingual version, with careful ablations about the effect of scaling the data, increasing the training noise, and mixing different noise adaptations. This could lead to a generalized multilingual model that is able to robustly represent highly heterogeneous texts across languages.

References

Noëmi Aeppli and Rico Sennrich. 2022. Improving zero-shot cross-lingual transfer between closely related languages by injecting character-level noise. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4074–4083, Dublin, Ireland. Association for Computational Linguistics.

Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussà, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kopec, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaki Nagata, Ajay Nagesh, Yoshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1–88, Online. Association for Computational Linguistics.

Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics.

Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

K. D. Dhingra, S. Sanyal, and P. K. Sharma. 2008. A robust ocr for degraded documents. In X. Huang, YS. Chen, and SI. Ao, editors, Advances in Communication Systems and Electrical Engineering, volume 4 of Lecture Notes in Electrical Engineering. Springer, Boston, MA.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. Preprint. arXiv:1705.00652.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Daniel Lopresti. 2008. Optical character recognition errors and their effects on natural language processing. In Proceedings of the Second Workshop on Analytics for Noisy Unstructured Text Data, AND ’08, page 9–16, New York, NY, USA. Association for Computing Machinery.

Andrianos Michail, Simon Clematide, and Rico Sennrich. 2025a. Examining multilingual embedding models cross-lingually through llm-generated adversarial examples. Preprint. arXiv:2502.08638.

Andrianos Michail, Corina Julia Raclé, Juri Opitz, and Simon Clematide. 2025b. Adapting multilingual embedding models to historical luxembourgish. Preprint. arXiv:2502.07938.

Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and
why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 529–535, New Orleans, Louisiana. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Georgios Sidiroopoulos and Evangelos Kanoulas. 2022. Analysing the robustness of dual encoders for dense retrieval against misspellings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2132–2136, New York, NY, USA. Association for Computing Machinery.

Matthias Sperber, Jan Niehtes, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 90–96, Tokyo, Japan. International Workshop on Spoken Language Translation.

Panuthep Tasawong, Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Ekapol Chuangsuvanich, and Sarana Nutanong. 2023. Typo-robust representation learning for dense retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1106–1115, Toronto, Canada. Association for Computational Linguistics.

Konstantin Todorov and Giovanni Colavizza. 2022. An assessment of the impact of ocr noise on language models. In International Conference on Agents and Artificial Intelligence.

Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1916–1920, Minneapolis, Minnesota. Association for Computational Linguistics.

Daniel Alexander van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020. Assessing the impact of ocr quality on downstream nlp tasks. In International Conference on Agents and Artificial Intelligence.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024a. Text embeddings by weakly-supervised contrastive pre-training. Preprint, arXiv:2212.03533.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. Multilingual e5 text embeddings: A technical report. Preprint, arXiv:2402.05672.

Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1393–1412, Miami, Florida, US. Association for Computational Linguistics.

A Appendix
<table>
  <tr>
    <th>Noise Type</th>
    <th>Original and Noisy Text</th>
    <th>Subtokenization Changes</th>
  </tr>
  <tr>
    <th colspan="3">German Text from CLSD WMT19/21</th>
  </tr>
  <tr>
    <td rowspan="2">Simple Noise</td>
    <td><b>Original:</b> Staatssozialismus<br><b>Noisy:</b> Staatssozialiismus</td>
    <td><b>Original Tokens:</b><br>[Staats][sozial][ismus]<br><b>Noisy Sub-Tokens:</b><br>[Staats][sozial][i][s][mus]</td>
  </tr>
  <tr>
    <td><b>Original:</b> mit<br><b>Noisy:</b> mit</td>
    <td><b>Original Sub-Tokens:</b> [mit]<br><b>Noisy Sub-Tokens:</b> [m][t]</td>
  </tr>
  <tr>
    <td rowspan="2">Blackletter</td>
    <td><b>Original:</b> historischen<br><b>Noisy:</b> bistorisehen</td>
    <td><b>OriginalSub-Tokens:</b><br>[historische][n]<br><b>Noisy Sub-Tokens:</b><br>[bis][tori][sehen]</td>
  </tr>
  <tr>
    <td><b>Original:</b> Europa<br><b>Noisy:</b> Curopa</td>
    <td><b>Original Sub-Tokens:</b> [Europa]<br><b>Noisy Sub-Tokens:</b> [Cu][ropa]</td>
  </tr>
  <tr>
    <td rowspan="2">Salt and Pepper</td>
    <td><b>Original:</b> können<br><b>Noisy:</b> können ; ;</td>
    <td><b>Original Sub-Tokens:</b> [können]<br><b>Noisy Sub-Tokens:</b> [k][ôn][nen][;][;]</td>
  </tr>
  <tr>
    <td><b>Original:</b> andere<br><b>Noisy:</b> .ander.e</td>
    <td><b>Original Sub-Tokens:</b> [andere]<br><b>Noisy Sub-Tokens:</b> [.][ander][.][e]</td>
  </tr>
  <tr>
    <th colspan="3">French Text from CLSD WMT19/21</th>
  </tr>
  <tr>
    <td rowspan="2">Simple Noise</td>
    <td><b>Original:</b> présidente<br><b>Noisy:</b> preresidente</td>
    <td><b>Original Sub-Tokens:</b> [président][e]<br><b>Noisy Sub-Tokens:</b> [presidente]</td>
  </tr>
  <tr>
    <td><b>Original:</b> leçon<br><b>Noisy:</b> lecon</td>
    <td><b>Original Sub-Tokens:</b> [le][çon]<br><b>Noisy Sub-Tokens:</b> [le][con]</td>
  </tr>
  <tr>
    <td rowspan="2">Scanned Noise</td>
    <td><b>Original:</b> telle qu’elle<br><b>Noisy:</b> ’teHÉ’ : qu’elle</td>
    <td><b>Original Sub-Tokens:</b><br>[telle][qu][’][elle]<br><b>Noisy Sub-Tokens:</b><br>[’][te][H][É][’][:][qu][’][elle]</td>
  </tr>
  <tr>
    <td><b>Original:</b> comme<br><b>Noisy:</b> commie</td>
    <td><b>Original Sub-Tokens:</b> [comme]<br><b>Noisy Sub-Tokens:</b> [com][mie]</td>
  </tr>
  <tr>
    <td rowspan="2">Salt and Pepper</td>
    <td><b>Original:</b> autre Europe<br><b>Noisy:</b> autré ... Europe</td>
    <td><b>Original Sub-Tokens:</b> [autre][Europe]<br><b>Noisy Sub-Tokens:</b><br>[au][tré][...][Europe]</td>
  </tr>
  <tr>
    <td><b>Original:</b> de gauche<br><b>Noisy:</b> de.gauge</td>
    <td><b>Original Sub-Tokens:</b> [de][gauche]<br><b>Noisy Sub-Tokens:</b><br>[de][.][gau][che]</td>
  </tr>
</table>

Table 6: Examples of subtokenization changes in different noise conditions

Reviewer paper summaries: The paper addresses the problem of OCR errors in digitized text collections and their impact on the performance of multilingual embedding models used for semantic search. The authors propose a simple method to improve the robustness of these embedding models by fine-tuning them on texts with artificially induced character-level noise: insertions, deletions, and substitutions. They evaluate their approach on German-French cross-lingual semantic search tasks under various OCR noise conditions (minimal noise, salt-and-pepper noise, and blackletter/scanned distorted noise). They also test on historical Luxembourgish texts to assess generalization to unseen languages with real OCR errors. The authors show that the method generalizes out-of-domain and compare with a control condition to isolate the effects of robustness training.
Summary of strengths: - The paper is written super carefully and is very easy to follow.
- The method is simple and inexpensive, requiring only 20 minutes of training on a single GPU. It also doesn't require extra data compared to normal semantic search training.
- There are significant improvements shown in various settings (OCR->OCR, Clean->OCR semantic search).
- Out-of-domain generalization is shown.
Summary of weaknesses: - The paper doesn't discuss a potential pipeline where the text is first denoised through some kind of specialized denoising model, and only then fed into an embedding model that is not finetuned. How would this compare to the presented method in terms of compute and latency? One substantial advantage of such a pipeline would be to avoid potential performance deterioration on clean data (the authors conduct a good analysis of this on their data, e.g. Clean X-News, but in real-world practical applications you can never be sure that this is the case on your data which can be very different).
- The performance of the method is not always consistent; sometimes it performs worse than training on clean data (e.g. the case of M-GTE / noised X-News / Salt and Pepper Noise in Table 1)
- The evaluation is limited to 2 embedding models; running the same experiments through a larger suite of models (including larger versions of these embedding models) would enhance the claims and demonstrate consistent improvement in noise conditions.
Comments, suggestions and typos: - Related work could be a bit more extensive, incorporating a comparison between the type of noise injection done in this paper and those done in other papers, as well as citing more related papers (e.g. possibly Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining by Stickland et al.)
- For the tokenization impact analysis, it would be very interesting to see the correlations between the relative performance of your method and the noise rate in the text, and between the relative performance of your method and the mistokenization rate in the text. If it correlates more with mistokenization than with character-level noise across the board, that can serve as extra evidence that mistokenization is an aggravating factor.
- It would be interesting to see an analysis (or a reference to an analysis) of the distribution over types of OCR errors across different types of digitized data and how this distribution aligns with the distribution of character-level transformations you use for synthetic noising.
- Figure 1 was a bit unclear during the first pass, in particular, the relationships between text pieces were initially unclear, as well as the abbreviations (BL, SnP).
- To show it's reliable for real-world applications, it would help to see a much more extensive analysis confirming that the performance on clean data doesn't drop across more domains and possibly even more tasks.
Overall assessment scores: 3.5