PHD: Pixel-Based Language Modeling of Historical Documents
Authors: Nadav Borenstein
Phillip Rust
Desmond Elliott
Isabelle Augenstein
Venue: EMNLP 2023 Main (2023)
Keywords: Visual language modelling
Historical documents
Multimodal models
TL;DR: We use pixel-based language models to train a general purpose model for historical documents that is OCR-free

Abstract: The digitisation of historical documents has provided historians with unprecedented research opportunities. Yet, the conventional approach to analysing historical documents involves converting them from images to text using OCR, a process that overlooks the potential benefits of treating them as images and introduces high levels of noise. To bridge this gap, we take advantage of recent advancements in pixel-based language models trained to reconstruct masked patches of pixels instead of predicting token distributions. Due to the scarcity of real historical scans, we propose a novel method for generating synthetic scans to resemble real historical documents. We then pre-train our model, PHD, on a combination of synthetic scans and real historical newspapers from the 1700-1900 period. Through our experiments, we demonstrate that PHD exhibits high proficiency in reconstructing masked image patches and provide evidence of our model's noteworthy language understanding capabilities. Notably, we successfully apply our model to a historical QA task, highlighting its usefulness in this domain.

PHD: Pixel-Based Language Modeling of Historical Documents
Phillip Rust Desmond Elliott Isabelle Augenstein { Department of Computer Science, University of Copenhagen p.rust de, augenstein @di. ku. dk
Nadav Borenstein
Abstract
strips away any valuable contextual meaning em- bedded within non-textual elements, such as page layout, fonts, and figures.' Moreover, historical documents present numerous challenges to OCR systems. This can range from deteriorated pages, archaic fonts and language, the presence of non- textual elements, and occasional deficiencies in quality (e.g., blurriness), all of which tribute to the introduction of additional noise. Con- sequently, the extracted text often riddled with errors at the character level (Robertson and Gold- water, 2018; Bollmann, 2019), which most large language models (LLMs) are not tuned to process. Token- LLMs are especially sensitive to this, as the discrete structure of their input space cannot handle well the abundance of out-of-vocabulary words that characterise OCRed historical docu- ments (Rust al., 2023). Therefore, while LLMs have proven remarkably successful in modern do- mains, their performance is considerably weaker when applied to historical texts (Manjavacas and Fonteyn, 2022; Baptiste et al., 2021, inter alia). Fi- nally, for many languages, OCR systems either do not exist perform particularly poorly. As train- ing OCR models is laborious and expensive (Li et al., 2021a), the application of NLP tools historical documents in these languages is limited.
The digitisation of historical documents has provided historians with unprecedented re- search opportunities. Yet, the conventional ap- proach to analysing historical documents in- volves converting them from images to text us- ing OCR, a process that overlooks the potential benefits of treating them as images and intro- duces high levels of noise. To bridge this gap, we take advantage of recent advancements in pixel-based language models trained to recon- struct masked patches of pixels instead of pre- dicting token distributions. Due to the scarcity of real historical scans, we propose novel method for generating synthetic scans to resem- ble real historical documents. We then pre-train model, PHD, combination of synthetic scans and real historical newspapers from the 1700- 1900 period. Through our experiments, demonstrate that PHD exhibits high profi- ciency in reconstructing masked image patches and provide evidence of model's notewor- thy language understanding capabilities. No- tably, we successfully apply our model to a historical QA task, highlighting its utility in this domain.
1 Introduction
Recent years have seen boom in efforts to digitise historical documents in numerous languages and sources (Chadwyck, 1998; Groesen, 2015; Moss, 2009), leading to a transformation the way histo- rians work. Researchers are now able to expedite the analysis process of vast historical corpora using NLP tools, thereby enabling them to focus on in- terpretation instead of the arduous task of evidence collection (Laite, 2020; Gerritsen, 2012).
This work addresses these limitations by taking advantage of recent advancements in pixel-based language modelling, with the goal of construct- ing general- purppss,, image-based and OCR-frec language encoder of historical documents. Specif- ically, we adapt PIXEL (Rust et al., 2023), lan- guage model that renders text as images and is trained to reconstruct masked patches instead predicting a distribution over tokens. PIXEL's train- ing methodology is highly suitable for the histori- cal domain, as (unlike other pixel-based language models) it does not rely on pretraining dataset
The primary step in most NLP tools tailored for historical analysis involves Optical Character Recognition (OCR). However, this approach poses several challenges and drawbacks. First, OCR
CConsider, for example, the visual data that is lost by pro- cessing the newspaper page in Fig 18 in App C as text.
*Tis paper shows datasct samples that are racist in nature

---

Laws
Laws Goll the hyymemtt of Proramm TocLowat u.. 2tc Streod (neer Teup Bar))
sild the FatabBllrroom Tartewar 1k4, Shownd (cnuc Trmple Har.) Len fow rike by all seayoreeble Dreguiiie séwilens to ided cinen tiremt olellann mucid as There
Prommaro
the k Hohment Promarre
RosLowa, atc. od (near Tea Ban)
Landow: ; alao by
apole and dellladd
Londow: cleo by al moseebbee ragge and dhilliand
ydoaler tn Madeinss
rdsaber Mad
There
conseeralle mning
There coneeeaa
inking largee slone.
eaking
taking 4lo
W. B.. B--Deeellin reetionn for the guilence prstents in viey dacrder are whteee Fos.
B
culdenoe petieel
s..
guldenon of pisirela
evary
to ssth
THE FRIEND OF A. L HOLLOWAY' PILLS
IEED
L
THE LLIEDD
LLO
PILES
HOLLO A PILLS
Nericus DisordeTs. the (owrtfl pierous ao smal deggio saees diarssiing
dera.
Na Disorders.
(earfall
Te ancliable
the grevent
grevent in
(a) Input example.
(b) Masking the input.
(c) Model predictions.
Figure 1: Our proposed model, PHD. The model is trained to reconstruct the original image (a) from the masked image (b), resulting in (c). The grid represents the 16 x 16 pixels patches that the inputs are broken into.
composed of instances where the image and text are aligned. Fig visualises our proposed training approach.
mann, 2019; Lyu et al., 2021). This has been done with the aim of aligning historical texts with their modern counterparts. However, these methods are not without flaws (Robertson and Goldwater, 2018; Bollmann, 2019), and any errors introduced during these preprocessing stages can propagate to down- stream tasks (Robertson and Goldwater, 2018; Hill and Hengchen, 2019). As result, historical texts remain persistently challenging domain for NLP research (Lai et al., 2021; De Toni et al., 2022; Borenstein al., 2023b). Here, we propose novel approach to overcome the challenges associated with OCR in historical material, by employing an basge-based language model capable of directly processing historical document scans and effec- tively bypassing the OCR stage.
Given the paucity of large, high-quality datasets comprising historical scans, we pretrain our model using a combination of 1) syntheric scans designed to resemble historical documents faithfully, pro- duced using novel method we propose for syn- thetic scan generation; and 2) real historical En- glish newspapers published in the Caribbeans in the 18th and 19th centuries. The resulting pixel- based language encoder, PHD (Pixel-based model for Historical Documents), is subsequently eval- uated based on its comprehension of natural lan- guage and its effectiveness in performing Question Answering from historical documents.
We discover that PHD displays impressive recon- struction capabilities, being able to correctly pre- dict both the form and content of masked patches of historical newspapers ($4.4). We also note the chal- lenges concerning quantitatively evaluating these predictions. We provide evidence of model's s noteworthy language understanding capabilities while exhibiting an impressive resilience to noise. Finally, we demonstrate the usefulness of the model when applied to the historical QA task ($5.4).
2.2 Pixel-based Models for NLU
Extensive research has been conducted on models for processing text embedded in images. Most existing approaches incorporate OCR systems as an integral part of their inference pipeline (Appalaraju et al., 2021; Li et al., 2021 Delteil et al., 2022). These approaches employ multimodal architectures where the input consists of both the image and the output generated by OCR system.
To facilitate future rescarch, we provide the dataset, models, and code at Ω htttp:://iiii ub. wwt..///aaaaarbbbeeeeeeeeeeieeee
Recent years have also witnessed the emergence of OCR-free approaches for pixel-based language understanding. Kim et al. (2022) introduce Donut, an imgg--nncoder--xx--eeeoooe model for docu- ment comprehension. Donut is pretrained with the objective of extracting text from scans, task they refer to "pseudo-OCR". Subsequently, it is fine- tuned on various text generation tasks, reminiscent of T5 (Roberts et al., 2020). While architecturally similar to Donut, Dessurt (Davis al., 2023) and Pix2Struct (Lee et al., 2022) were pretrained by masking image regions and predicting the text in
2 Background
2.1 NLP for Historical Texts
Considerable efforts have been invested in improv- ing both OCR accuracy (Li et al., 2021a; Smith, 2023) and text normalisation techniques for his- torical documents (Drobac et al., 2017; Robertson and Goldwater, 2018; Bollmann et al., 2018; Boll-

---

both masked and unmasked image regions. Unlike our method, all above-mentioned models predict in the text space rather than the pixel space. This pre- supposes access to a pretraining dataset comprised instances where the image and text are aligned. However, this assumption cannot hold for histori- cal NLP since OCR-independent ground truth text for historical scans is, in many times, unprocurable and cannot be used for training purposes.
#Tain Scans
#Test Scans
Source
Issues
Caribbean Project
7 4887
675 172
87 721
Danish Royal Library
566
300 780
15 159
Total
13 148
975 952
102 880
Table 1: Statistics of the newspapers dataset.
Text-free models that operate at the pixel level for language understanding are relatively uncom- mon. One notable exception is Li et al. (2022), which utilises Masked Image Modeling for pre- training on document patches. Nevertheless, their focus lies primarily on tasks that do not necessitate robust language understanding, such as table detec- tion, document classification, and layout analysis. PIXEL (Rust et al., 2023), conversely, is a text-free pixel-based language model that exhibits strong language understanding capabilities, making the ideal choice for our research. The subsequent sec- tion will delve into a more detailed discussion of PIXEL and how we adapt it to our task.
training and data processing procedures to make it compatible with our use case ($4 and $5). alter the dimensions of the Most crucially, model's input: The text renderer of PIXEL renders strings as a long and narrow mmge with resolution of 16 x 8464 pixels (corresponding to 529 patches), such that the resulting image resembles ribbon with text. Each input character is set to be not taller than 16 pixels and occupies roughly one patch. However, real document scans cannot be represented this way, as they have natural two- dimensional structure and irregular fonts, Fig la demonstrates (and compare to Fig 17a in App C). Therefore, we set the input size of PHD to be 368 x 368 pixels (or 23 x 23 patches).
3 Model
PIXEL We base PHD on PIXEL, a pretrained pixel-bacc pixel-based encoder of language. PIXEL has three main components: A text renderer that draws texts as images, a pixel-based encoder, and a pixel-based decoder. The training of PIXEL is analogous to BERT (Devlin et al., 2019). During pretraining, input strings are rendered images, and the en- coder and the decoder are trained jointly to recon- struct randomly masked image regions from the unmasked context. During finetuning, the decoder is replaced with a suitable classification head, and no masking is performed. The encoder and decoder are based on the ViT-MAE architecture (He et al., 2022) and work at the patch level. That is, the en- coder breaks the input image into patches of 16 x 16 pixels and outputs an embedding for each patch. The decoder then decodes these patch embeddings back into pixels. Therefore, random masking performed at the patch level as well.
Training Pixel-Based Historical LM
We design PHD to serve as pureres-purpose, pixel-based language encoder of historical docu- ments. Ideally, PHD should be pretrained on a large dataset of scanned documents from various histori- cal periods and different locations. However, large, qugh-ty datasets of historical scans are not easily obtainable. Therefore, we propose a novel method for generating historical-looking artificial data from modern corpora (see subsection 4.1). We adapt model to the historical domain by contin- uously pretraining it on medium-sized corpus of real historical documents. Below, we describe the datasets and the pretraining process of the model.
4.1 Artificially Generated Pretraining Data
Our pretraining dataset consists of artificially gen- erated scans of texts from the same sources that BERT used, namely the BookCorpus (Zhu et al., 2015) and the English Wikipedia.? We generate the scans as follows.
We follow the same approach as PIXEL's pretraining and finetuning schemes. However, PIXEL's intended use is to process texts, not natu- ral images. That is, the expected input PIXEL is a string, not an image file. In contrast, we aim to use the model to encode real document seans. Therefore, we make several adaptations to PIXEL's
PHD
We generate dataset samples on-the-fly, adopt- ing similar approach as Davis et al. (2023). First,
We use the version "20220501.cr enn" hosted on hugg! nggt ace. co datasetss /wi wikk htw.kiiiiiiee pedia..

---

Sefwel In Pon Vellewspllkkonn to HIZ lt then rousled for ds nandeeis frret Bertin and Switooekndd hlse his vondeeshr he oetled in ss easty los the Entrenctty of Heshel Hee aloo lonel studett carps ihere.
Senodl in Pln, Sthienatg celee in MI. He frem trame for eie mondbe in lives Brisdo and Sorterrandd fife sts sonderjabe hh urtled to in enltyy her the iniweeyy of Helleeherg He also jeined a studen caye these.
Scked in Hon, Schlowiig beech tn 1922 de then pusiled for ets inresshe in fired lifiin and Switereindd Afber his wonderthhe for wrteel in to sauh les the Univeriiy Hetfelberg lic dee joned stelens corps theve
Wol wan Etleee Babsoor was comrrter und se lnstraer [lecteront] in Apeil MBI and spalled for planks voining the Inforreterrtt (farman str servies] Akby with kis troher Inrend he trained on Replaceenne Division 5 in Hanovee Coroorgh IS September 395.
Waber voi Below Nowwlhhh camp wes maneeneeee os a leuuuer (eereeaae) io Aget E0S and appted for plade trining in the inf Blernuan aat service] Allng with his looteer Gornad le Iesined in Replamerent Diviten 5.5r Haroowe Ghorghh 15 Seplember 1915.
1928, L2
bas lean
(a) Embedding one paragraph.
(b) Adding more paragraphs.
(c) Adding noise.
Figure 2: Process of generating single artificial scan. Refer $4.1 for detailed explanations.
we split the text corpora into paragraphs, using the new-line character as a delimiter. From a para- graph chosen at random, we pick a random spot and keep the text spanning from that spot to the paragraph's 'ss end. We also sample a random font and font size from a pre-definets list of fonts (from Davis et al. (2023)). The text span and the font then embedded within an HTML template using the Python package Jinja, set to generate a Web page with dimensions that match the input dimension of the model. Finally, we use the Python package WeasyPrint* to render the HTML file as a PNG image. Fig 2a visualises this process' outcome. In some cases, if the text span is short or the selected font is small, the resulting image contains large empty space (as in Fig 2a). When the empty space within an image exceeds 10%, new image is generated to replace the vacant area. We create the new image by randomly choosing one of two options. In 80% of the cases, we retain the font of the original image and seleet the next paragraph. In 20% of the cases, new paragraph and font are sampled. This pertains to the common case where a historical scan depicts a transition of context or font (c.g.. Fig la). This process can repeat multiple times, resulting images akin to Fig 2b.
scans of real historical newspapers. Specifically, we collect newspapers written in English from the "Caribbean Newspapers, 1718-1876" database, the largest collection of Caribbean newspapers from the 18th-19th century available online. We extend this dataset with English-Danish Danish newspa- pers published between 1 1770-1850 in the Danish Caribbean colony of Santa Cruz (now Saint Croix) downloaded from the Danish Royal Library's web- site. See Tab I for details of dataset sizes. While confined in its geographical and temporal context, this dataset offers a rich diversity in terms of con- tent and format, rendering it an effective test bed for evaluating PHD.
Newspaper pages are converted into a 368 x 368 pixels crops using a sliding window approach over the page's columns. This process is described in more detail in App A.2. We reserve 5% of newspa- per issues for validation, using the rest for training. See Fig 10 in App C for dataset examples.
4.3 Pretraining Procedure
Like PIXEL, the pretraining objective of PHD is to reconstruct the pixels in masked image patches. We randomly occlude 28% of the input patches with 2D rectangular masks. We uniformly sample their width and height from [2, 6] and [2, 4) patches, respectively, and then place them in random image locations (See Fig lb for an example). Training hyperparameters can be found in App A.I.
Finally, to simulate the effects of scanning age- ing historical documents, we degrade the image by adding various types of noise, such as blurring. rotations, aalt-and-pepper noise and bleed-through effect (see Fig 2c and Fig 9 in App C for examples). App A.2 enumerates the full list of the degradations and augmentations we use.
4.4 Pretraining Results
Qualitative Evaluation. We begin by conducting a qualitative examination of the predictions made by our model. Fig 3 presents a visual representa-
4.2 Real Historical Scans
We adapt PHD to the historical domain by contin- uously pretraining it on a medium-size corpus of
$ fteadex.com/prodeeee http://////.rtttttt/ebbbeeeeeeee ers- 1re-series-1-117 11t-1--1111------ aaarr---aaeiinaaaaan antiqua rian- society "statsbiblioteker hagattaiibiitteeeee...//eeeettte
"jinja. srinja.paaiee pprooest wwtpetts.commeeeeeeeeeeeeeeeeeee com/en/2 3.1.x 'weasyprint. org

---

neturntaal Society quite
Secretar
Listed
EYETT inf
Belggo and teo fadls sbore and
require,, with the
the
mation given
M. CAVAN CO..
the anrsendnng kilk
a
Pox
Che Sockety, enable
dimatng ovnny, only
any Secabbreeeheeeee mi te to thoronghly
stamine th
Tike rer Dom
PHENIX FIRE INSURANCE C OTPPANY LONDON
iginona of lederry though
degatbbed
TO THE ADITOR OF THE peroaren
1l obelesa.a-quod quad
oa
Ge Pascooss dedlid
Todion toetherteeeeeee dalivery of letters bod Intto
th dveey they taseadio
st our now Post-odice It. appaars that the
diretion 31 ahhote
would
folivery made
four windows,
mos produt by
Rdtettr Cook
sxme spstets the may be
coming and
edoptel fuilltate the
pites MiC alfurd
EST ABLSSHED
mayy ond oaee that Medile would
more Lasily ention app
Figure 3: Examples of some image completions made by PHD. Masked regions marked by dark outlines.
B
Pue in the cuartima
palt wii the On- by which
we exe undertake authonitatieely repa- dleate, on Mrr Ghast's bohalf, aey naporal- sility the alleged igucetace of the Bector, -iin foot appreherd that auch mulacioss baranded with vjowr
aboo le Blowed pay perpetiig aenny beffir borpod
deemcee the 24 Frov
the
othar samsally, with inttreat the pate eegt 95 desrged Boun the woll.. Mirr.
fillsey has only
to drace from Mr GRase sous public
MARSHAL'S OFFICE
dleation of himself for the landable pur-
GENERATITE and MUSCOLAR POWER REGAINE::
120 THE CAn oF
Jumicial Dale,
Lucia 55th December 1884
to the istanes solace gratifiattinn all
dengroenn into position of anfoguaists
peen, prolusbyy
bringing two brither-
"ERRY'S-CORIAA' B BALM OP
me-tes Jomos, Laoce, Quites, snd others whose clement auarily it saticipote the disgpsittrmmn
SYRIACUM
Mr. and Máe. ANDRE LIRADRD,
Rtiallished f Cornighens hont mond as rax GUSVE
Casmeronp Dureissr, elininprrater to the J
with the lst Jona Myres
howorer, thess elaritable intentions, such are estertiined since Mr. Guart's
malorbboss Jole manly
Res Goas of this Tiberd and mendde
piraowee the
comemiiinn
(a) world
(b) 1893
(c) every
Figure 4: Single word completions made by our model. Figure captions depict the missing word. Fig (a) depicts successful reconstruction, whereas Fig (b) and (c) represent fail-cases.
tion of the model's predictions on three randomly selected scans from the test set of the Caribbean newspapers dataset (for additional results on other datasets, refer to Fig 12 App C). From a visual inspection, it becomes evident that the model ac- curately reconstructs the fonts and structure of the masked regions. However, the situation is less clear when it comes to predicting textual content. Sim- ilar to Rust et al. (2023), unsurprisingly, predic- tion quality is high and the results are sharp for smaller masks and when words are only partially obscured. However, as the completions become longer, the text quality deteriorates, resulting in blurry text. It is important to note that evaluat- ing these blurry completions presents a significant challenge. Unlike token-based models, where the presence of multiple words with high, similar like- lihood can easily be detected by examining the discrete distribution, this becomes impossible with pixel-based models. In pixel-based completions, high-likelihood words may overlay and produce a blurry completion. Clear completions are only observed when single word has a significantly higher probability compared to others. This limita- tion is an area that we leave for future work.
scans and OCRed them using Tesseract. 7 Next, we randomly select single word from the OCRed text and use Tesseract's s word-to-iman tomae location fune- tionality to (heuristically) mask the word from the image. Results are presented in Fig 4. Similar to our earlier findings, the reconstruction quality of single-word word completion varies. Some completions are sharp and precise, while others appear blurry. In some few cases, the model produces sharp reconstruction of an incorrect word (Fig 4c). Un- fortunately, due to the blurry nature of many of the results (regardless of their correctness). a quantita- tive analysis of these results (c.g.. by OCRing the reconstructed patch and comparing it to the OCR output of the original patch) is unattainable.
Semantic Search. A possible useful application of PHD is semantic search. That is, searching in a corpus for historical documents that are seman- tically similar to a concept of interest. We now analyse PHD's 's ability to assign similar historical scans with similar embeddings. We start by tak- ing random sample of 1000 images from our test set and embed them by averaging the patch em- beddings of the final layer of the model. We then reduce the dimensionality of the embeddings with
We now move to analyse PHD's ability to fill in single masked words. We randomly sample test

---

Port of Chrittanited.
Bock Rore
Fram Mi Kotts Lesk Lenkiegick
lng d Pallned de Aenndi of
HANEE.
RiEa, SE No. Seraliant,,
day Oenated df araernee
1ith
Port of Chriltanfiad
Trykt hos G. Harninge
Dell han ynn
fuddle is GENEGEELLAA
Port of Chrittianied.
Jecnin Ditts
Artived. Junn 55. Stip Coldired Cillle,- W.. Danlen, from (Guddsooges Brig Jeni Joha, Marayjen, Irimn
Chrilfianitzd 5- Feb, 1800, loing
may 7o0 Aal
16 Im Veren Jah Crekey to Lie
Wallagges
goils Lanber.
57. Sch's Carhariee,_Jokk Stves, dottiem
Sailed,
Cuodolouga
Wed Ead Piaerry sheck frws
DIED Widweler Aborscen Isf Cyp.
Rith
Juns Beig Renighsionn.-.ol. Moreufin, Co-
jprubagen.
Ship Siz Rubderm,-Miiiee Mirbad (oologgaand
Selt Widko's Son
Toby Del
Brig Blicobeth- Saplins Markroning Fro- ddenichioed.
Biseen Cig
Skop Da anhin Botty
sile Toe Derien, Tralien, Nerill
"ngren sourl PCC ane cillineeeen
park Red of
Seop Tines
La Perle Estate, Choisou
bleege Adree Foado
(a) Semantic search target.
(b) Retrieved scans.
Figure 5: Semantic search using our model. (a) is the target of the search, and (b) are scans retrieved from the newspaper corpus.
E-SN (van der Maaten and Hinton, 2008). Upon visual inspection (Fig 13 in App C), we see that scans are clustered based on visual similarity and page structure.
Gerbence & Now doed out submerin deptr. without the inflow water
Seowre a Wir mlontw ioplica
senin donk, les morer cusial!
Sentence 2 Why does submartne implede of carhan depih, some
doyes
oceen morie (iAe can live witoue floor, in the deepeat drushed?
Fig 13, however, does not provide insights re- garding the semantic properties of the clusters. Therefore, we also directly use the model in se- mantic search settings. Specifically, we search our newspapers corpus for scans that are semanti- cally similar to instances of the Runaways Slaves in Britain dataset, well as scans containing ship- ping ads (See Fig 16 in App C for examples). To do so, we embed IM random scans from the corpus. We then calculate the cosine similarity between these embeddings and the embedding of samples from the Runaways Slaves in Britain and embed- dings of shipping ads. Finally, we manually exam- ine the ten most similar scans to each sample.
Figure 6: Samples from the clean and noisy visual GLUE datasets.
How much reward is offered?
BAS AS Molatto Bey (his Name Dosch;
young Man lately Doi Eift Iudich, abrente himdet en Mon- the soth fest Se I on when went awaa a Thickeer Frock and idrest, whil red Crlar.
Breecher,
Perfon that will oppreheud the aovementenne Bay, dire tay Inueligence where may taken,
and
thail reesive a Reward of fhoet black Hiirr not
about Kind,
high,
N. B. If taken, to be brought to Sign of the Gesage, Queen-Ann Street, Cavencifh-Squue Square.
Figure 7: Example from the Runaways Slaves in Britain dataset, rendered visual question answering task. The gray overlay marks the patches containing the answer.
Our results (Fig and Fig 14 in App C) are en- couraging, indicating that the embeddings capture not only structural and visual information, but also the semantic content of the scans. However, the results are still far from perfect, and many retrieved scans not semantically similar to the search's 's target. It is highly plausible that additional spe- cialised finetuning (e.g.. BER'' (Reimers and Gurevych, 2019) training scheme) is necessary to produce more semantically meaningful embed- dings.
5 Training for Downstream NLU Tasks
After obtaining pretrained pixel-based lan- guage model adapted to the historical domain ($4), we now move evaluate its understanding of natural language and its usefulness in addressing historically--rienneed NLP tasks. Below, we describe the datasets we use for this and the experimental settings.

---

MNLI 393k
QQP
QNLI
SST-2
COLA
STS-B 5.8k
MRPC 3.7k
RTE 2.5k
WNLI
Noise
Images
Model
AVG
364k
105k
67k
8.6k
635
BERT
84.1
87.6
91.0
92.6
60.3
88.8
90.2
69.5
51.8
80.0
x
PIXEL
78.5
84.5
87.8
89.6
38.4
81.1
88.2
60.5
53.8
74.1
x
CLIP
50.2
64.7
67.4
79.8
4.2
56.4
74.1
51.5
25.6
52.7
Donut
61.0
77.8
69.7
82.1
13.9
14.4
81.7
54.0
57.7
57.2
Ours
70.1
82.7
82.3
82.5
15.9
80.2
83.4
59.9
54.1
67.9
OCR+BERT
71.7
77.5
82.7
85.5
39.7
68.4
86.9
58.8
51.3
69.2
OCR+PIXEL
70.6
78.5
81.5
83.6
68 8
59.7
58.6
68.5
CLIP
45.3
67.4
64.4
79.2
3.5
57 .9
78.8
47.3
32.7
52.9
Donut
61.6
74.1
75.1
75.5
10.2
20.6
81.9
56.7
60.0
57.3
Ours
68.0
80.4
81.8
83.9
15.1
80.4
58.5
57.8
67.2
Table 2: Results for PHD finetuned on GLUE. The metrics are Fj score for QQP and MRPC, Matthew's correlation for COLA, Spearman's p for STS-B, and accuracy for the remaining datasets. Bold values indicate the best model in category (noisy/clcan), while underscored values indicate the best pixel-based model.
5.1 Language Understanding
patch of I is labelled with if it contains part of a or otherwise. This generates binary label mask M for 1, which our model tries to predict (Fig 1lb). If any degradations or augmentations are later applied to I, we ensure that M is affected accordingly. Finally, similarly to Lee et al. (2022), we concatenate to rendering of and crop the resulting image to the appropriate input size (Fig lc).
We adapt the commonly used GLUE benchmark (Wang al., 2018) to gauge model's under- standing of language. We convert GLUE instances into images similar to the process described in $4.1. Given a GLUE instance with sentences 81, 82 (52 can be empty), we embed 81 and 82 into an HTML template, introducing line break between the sen- tences. We then render the HTML files as images.
We generate two versions of this visual GLUE dataset clean and noisy. The former rendered using a single pre-defined font without applying degradations or augmentations, whereas the latter is generated with random fonts and degradations. Fig 6 presents sample of each of the two dataset versions. While the first version allows to mea- sure PHD's understanding of language in "sterile" settings, use the second version estimate the robustness of the model noise common to historical scans.
Generating the binary mask M is not straightfor- ward, as we not know where a is located inside the generated image J. For this purpose, we first use Tesseract to OCR and generate è. Next, we use fuzzy string matching to search for within Tesseract
č. If a match à E è is found, find the pixel coordinates of within I. We then map the pixel coordinates patch coordinates and label all the patches containing à with In about 15% the cases, Tesseract fails to OCR / properly. and cannot be found é, resulting a higher proportion of SQuAD samples without an answer compared to the text-based version.
5.2 Historical Question Answering
QA applied to historical datasets can be immensely valuable and useful for historians (Borenstein et al., 2023a). Therefore, assisting historians with this important NLP task. We finetune the model on two novel datasets. The first an adaptation of the classical SQuAD-v2 dataset (Rajpurkar et al., 2016), while the second is a genuine historical QA dataset.
As with GLUE, generate two versions of visual SQuAD, which we use evaluate PHD's performance in both sterile and historical settings.
assess PHD's potential for
Historical QA Dataset Finally, we finetune PHD for a real historical QA task. For this, we use the English dataset scraped from the website of the Runaways Slaves in Britain project, a searchable database of over 800 newspaper adverts printed between 1700 and 1780 placed by enslavers who wanted capture enslaved people who had self- liberated (Newman al., 2019). Each ad was man- ually transcribed and annotated with more than 50 different attributes, such as the described gender
SQuAD Dataset We formulate SQuAD-v2 as a patch classification task, as illustrated in Fig 11 in App C. Given SQuAD instance with question g. context and answer a that span in c, we render c as an image, I (Fig la). Then, each

---

and age, what clothes the enslaved person wore, and their physical description.
Binary
Patch acc
One
Task
Model
Noise / Image
Overlap
BERT
x
72.3
47.3
53.9
Following Borenstein et al. (2023a), we convert this dataset to match the SQuAD format: given an ad and an annotated attribute, we define the transcribed ad as the context c, the attribute as the answer a, and manually compose appropriate question q. We process the resulting dataset sim- ilarly how SQuAD is processed, with one key difference: instead of rendering the transcribed ad c as an image, we use the original ad scan. Therefore, we also do not introduce any noise to the images. See Figure 7 for an example instance. We reserve 20% of the dataset for testing.
S
Ours
Ki /
60.3
16.4
42.2
Ours
ViV
61.7
14.4
41.2
BERT
-/x
78.3
52.0
55.8
Ours
-1V
74.7
20.0
48.8
Table 3: Results for PHD finetuned on our visual
SQuAD (S) and the Runaways Slaves (R) datasets.
ear probe setting, which was shown to be effecttive in range of settings that require a joint understand- ing of image and text text-including rendered SST-2 (Radford et al., 2021). While we only train a linear model on the extracted CLIP features, compared to full finetuning in PHD, CLIP is about 5x the size with ~427M parameters and has been trained longer on more data. Second, we finetune Donut ($2.2), which has ~200 parameters and is the closest and strongest OCR-free alternative to PHD. Moreover, finetune BERT and PIXEL on the OCR output of Tesseract. Both BERT and PIXEL are comparable in size and compute budget to PHD. Although BERT has been shown to be overall more effective on standard GLUE than PIXEL., PIXEL is more robust to orthographic noise (Rust et al., 2023). Finally, to obtain an empirical upper limit to model, we finetune BERT and PIXEL on standard, not-OCRed version of GLUE Like- wise, for the QA tasks, we compare PHD to BERT trained on non-OCRed version of the datasets (the Runaways dataset was manually transcribed). We describe all baseline setups in App B.
5.3 Training Procedure
Similar to BERT, PHD is finetuned for downstream tasks by replacing the decoder with suitable head. Tab 4 in App A.I details the hyperparameters used to train PHD on the different GLUE tasks. We use the standard GLUE metries to evaluate our model. Since GLUE is designed for models of modern English, we use this benchmark to evaluate a checkpoint of our model obtained after training on the artificial modern scans, but before training on the real historical scans. The same checkpoint is also used to evaluate PHD on SQuAD. Conversely, we use the final model checkpoint (after introduc- ing the historical data) to finetune on the historical QA dataset: First, we train the model on the noisy SQuAD and subsequently finetune it on the Run- aways dataset (see App A.I for training details).
To evaluate our model's performance on the QA datasets, we employ various metrics. The primary metrics include binary accuracy, which indicates whether the model agrees with the ground truth regarding the presence of an answer in the con- text. Additionally, we utilise patch-based accuracy, which measures the ratio of overlapping answer patches between the ground truth mask M and the predicted mask M, averaged over all the dataset instances for which an answer exists. Finally, we measure the number of times predicted answer and the ground truth overlap by at least single patch. We balance the test sets to contain an equal number of examples with and without an answer.
GLUE Tab summarises the performance of PHD on GLUE. Our model demonstrates notewor- thy results, achieving scores of above 80 for five out of the nine GLUE tasks. These results serve as evi- dence of our model's language understanding capa- bilities. Although our model falls short when com- pared text-based BERT by 13 absolute points on average, it achieves competitive results com- pared to the OCR-then-finetune baselines. More- over, PHD outperforms other pixel-based models by more than 10 absolute points on average, high- lighting the efficacy of our methodology.
Question Answering According to Tab 3, our model achieves above guess-level accuracies on these highly challenging tasks, further strengthen- ing the indications that PHD was able to obtain im- pressive language comprehension skills. Although the binary accuracy on SQuAD is low, hovering
5.4 Results
Baselines We compare PHD's performance on GLUE a variety of strong baselines, covering both OCR-free and OCR-baser methods. First, we use CLIP with a ViT-L/14 image encoder in the lin-

---

What does the contact of the ad do for living?
historical documents at the pixel level. We present novel pretraining method involving a combina- tion of synthetic seans that closely resemble histor- ical documents, as well as real historical newspa- pers published the Caribbeans during the 18th and 19th centuries. Through our experiments, we observe that PHD exhibits high proficiency in re- constructing masked image patches, and provide evidence of our model's noteworthy language derstanding capabilities. Notably, we successfully apply our model to a historical QA task, achieving a binary accuracy score of nearly 75%, highlight- ing its usefulness in this domain. Finally, we note that better evaluation methods are needed to further drive progress in this domain.
R UN awey from Colonel M-Dowarl of Caffee- Sempill, upez the 3oth of Jamary, a Negro Man, named CdTO, alies JOHN; be is middl- aged, protey sall,
Legs, with feuat broad Any Perfin ruba apprebendi Information
gōves any Mr Deavitt or to Mr. Alxxander Houfon Merbard r Glaftev foall bovee Peffeien
(a)
How much reward is offered?
W ENI away from his Maftrr, Whitfon- Thurfda lift, an Indian Black, about
20 Years Age, Well ler, Speaks Good Eng- lift, goes by the Name Anthony Stanle, had on Calico Walt Coar, Strip'd Flat hee'dd
Black, Dark Fuftian Breechees,
Shoes :
Whoever brings him to Robert Powel Seee in Linse- fhall have
Cuinea Reward, and Charges.
(h)
Figure 8: Saliency maps PHD fine-tuned on Run- aways Slaves in Britain dataset. Ground truth label in grey box. The figures were cropped in post-processing.
Acknowledgements
This research was partially funded by a DFF Sapere Aude research leader grant under grant agreement No 017 0171-00034B, the Danish-Israeli Study Foun- dation in Memory of Josef and Regine Nachem- sohn, the Novo Nordisk Foundation (grant NNF 20SA0066568), as well by rescarch grant (VIL53122) from VILLUM FONDEN. The search was also supported by the Pioneer Centre for AI, DNRF grant number PI.
around 60% compared to the 72% of BERT, the rel- atively high "At least one overlap" score of above 40 indicates that PHD has gained the ability to lo- cate the answer within the scan correctly, Further- more, PHD displays impressive robustness to noise, with only a marginal decline in performance ob- served between the clean and noisy versions of the SQuAD dataset, indicating its potential in handling the highly noisy historical domain. The model's performance on the Runaways Slaves dataset is par- ticularly noteworthy, reaching a binary accuracy score of nearly 75% compared to BERT's 78%, demonstrating the usefulness of the model in appli- cation historically-oriene
NLP tasks. We be- lieve that the higher metrics reported for this dataset compared to the standard SQuAD might stem from the fact that Runaways Slaves in Britain contains repeated questions (with different contexts), which might render the task more trackable for our model. Saliency Maps Our
QA approach can also produce visual saliency maps, allowing for more fine-grained interpretation of model pre- dictions and capabilities (Das al., 2017). Fig 8 presents two such saliency maps produced by applying the model to test samples from the Run- aways Slaves in Britain dataset, including failure case (Fig 8a) and successful prediction (Fig 8b). More examples can be found in Fig 15 in App C.
6 Conclusion
this study, we introduce PHD, an OCR-free lan- guage encoder specifically designed for analysing

---

Limitations
End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 993-1003.
We see several limitations regarding work. First, focus on the English language only, high-resource language with strong OCR systems developed for it. By doing so, we neglect low- resource languages for which our model can poten- tially be more impactful.
Blouin Baptiste, Benoit Favre, Jeremy Auguste, and Christian Henriot. 2021. Transferring modern named entity recognition to the historical domain: How to take the step? In Workshop Natural Language Processing for Digital Humanities (NI.PDII)..
On the same note, we opted to pretrain our model on single (albeit diverse) historical corpus of newspapers, and its robustness in handling other historical sources is yet to be proven. To address this limitation, we plan to extend our historical cor- pora in future research endcavours. Expanding the range of the historical training data would not only alleviate this concern but also tackle another limi- tation; while our model was designed for historical document analysis, most of its pretraining corpora consist of modern texts due to the insufficient avail- ability of large historical datasets.
Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3885-389 Minneapolis, Min- nesota. Association for Computational Linguistics.
Marcel Bollmann, Anders Søgaard, and Joachim Bin- gel. 2018. Multi-task learning for historical text normalization: Size matters. In Proceedings of the Workshop on Deep Learning Approaches for Low- Resource NLP, pages 19-24.
We also see limitations in the evaluation of PHD. As mentioned in Section 4.4, it is unclear how to empirically quantify the quality of the model's re- construction of masked image regions, thus neces- sitating reliance on qualitative evaluation. This qualitative approach may result in suboptimal model for downstream tasks. Furthermore, the eval- uation tasks used to assess our model's language understanding capabilities are limited in their scope. Considering our emphasis on historical language modelling, it is worth noting that the evaluation datasets predominantly cater to models trained on modern language. We rely a single historical dataset to evaluate our model's performance.
Nadav Borenstein, Natalia da Silva Perez, and Isabelle Augenstein. 2023a. Multilingual event extraction from historical newspaper adverts. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguisties, Toronto, Canada. Association for Computational Linguistics.
Nadav Borenstein, Karolina Staáczak, Thea Rolskov, Natacha Klein Käfer, Natalia Silva Perez, and Is- abelle Augenstein. 2023b. Mcasuring intersectional biases in historical documents. Association for Com- putcaiional Linguistics.
Chadwyck. 1998. Early english books online Eebo.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. 2017. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, Language in Vision.
Lastly, due to limited computational resources, we were constrained to training a relatively small- scale model for a limited amount of steps, poten- tially impeding its ability to develop the capabilities needed to address this challenging task. Insufficient computational capacity also hindered us from con- ducting comprehensive hyperparameter searches for the downstream tasks, restricting our ability to optimize the model's performance to its full poten- tial. This, perhaps, could enhance our performance metries and allow PHD to achieve more competi- tive results on GLUE and higher absolute numbers on SQuAD.
Brian Davis, Bryan Morse, Brian Price. Chris Tens- meyer, Curtis Wigington, and Vlad Morariu. 2023. End-to-end document recognition and understand- ing with dessurt. In Computer Vision ECCV 2022 Workshops: Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part IV, page 280-296, Berlin, Heidel- berg. Springer-
Francesco De Toni, Christopher Akiki, Javier
De La Rosa, Clémentine Fourrier, Enrique Manjava- cas, Stefan Schweter, and Daniel Van Strien. 2022. Entities, dates, and languages: Zero-shot on histor- ical texts with t0. Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspec- rives in Creating Large Language Models, pages 75- 83, virlual Association for Computational Linguistics.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer:

---

Thomas Delteil, Edouard Belval, Lei Chen, Luis Goncalves, and Vijay Mahadevan. 2022. MATrXX Modality-Aware, Transformer for Information eXtrac- tion. arXiv e-prints, page arXiv:2205.0800 000944.
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu. Fangyu Liu, Julian Eisenschlos, Urvashi Khandel- wal, Peter Shaw, Ming-Wei Seig-Wei Chang, and Kristina Toutanova. 2022. Pix2struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprint arXiv: 221.02221100037.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Junlong Li, Yiheng Xu, Tengehao Lv, Lei Cui, Cha Zhang, and Furu Wei. 2022. Dit: Self-supervised pre-training for document image transformer. In Pro- ceedings of the 30th ACM International Conference on Multimedia, MM '22, page 3530-3539, New York. NY. USA. Association for Computing Machinery.
Minghao Li. Tengchao Lv, Jingye Chen, Lei Cui. Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 202 Trocr: Transformer-basee optical character recogmition with pre-trained models. arXiv preprimt arXiv:2 10282.
Senka Drobac, Pekka Kauppinen, and Krister Lindén. 2017. Oer and post-correction of historical finnish texts. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 70-76.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021b. Selfdoc: Self-supervised doc- ument representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652 56660.
Anne Gerritsen. 2012. Scales of a local: the place of locality in a globalizing world. A Companion to World History, pages 213-226.
Michiel van Groesen. 2015. Digital gatekeeper of the past: Delpher and the emergence of the press in the dutch golden age. Tijdschrift voor Tijdschiffstuddies 38:9-19.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 17X
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16000-16009.
Lijun Lyu, Maria Koutraki, Martin Krickl, and Besnik Fetahu. 2021. Neural ocr post-hoc correction of his- torical corpora. Transactions of the Association for Computational Linguistics, 9:479-493.
Enrique Manjavacas and Lauren Fonteyn. 2022. Adapt- ing vs. Pre-training Language Models for Historical Languages. Journal of Data Mining & Digital Hu- munities, NLP4DH.
Mark J Hill and Simon Hengchen. 2019. Quantifying the impact of dirty ocr on historical text analysis: Fighteenth century collections online as case study. Digital Scholarship in the Humanities, 34(4):825- 843.
Janalyn Moss. 2009. Guides: News and newspapers:
Historical newspaper collections. Iowa's 's University Libraries.
Geewook Kim, Teakgyu Hong. Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- seok Hwang, Sangdoo Yun. Dongyoon Han, and Se- unghyun Park. 2022. Ocr-free document understand- ing transformer. In Computer Vision -ECC 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings. Part XXVIII, page 597, Berlin, Heidelberg. Springer-Verlag.
Simon P. Newman, Stephen Mullen, Nelson Mundell. and Roslyn Chapman. 2019. Runaway Slaves in Britain: bondage, freedom and race in the eighteenth century. https: / www. runaways glaa.. ac. uk. Accessed: 2022-12-10.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 1CML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1oXiv:1412.6988
Viet Lai, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021. Event extraction from historical texts: new dataset for black rebellions. In Findings of the Association for Computational Linguistics: ACL-ICNLLP 2021, pages
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1- 1400667.
Julia Laite. 2020. The emmet's 's inch: Small history in a digital age. Journal of Social History, 53(4):963- 989

---

Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page MrXiv:1606.05220
A
Reproducibility
A.1 Training
Pretraining We pretrain PHD for IM steps on with the artificial dataset using a batch size of 176 (the maximal batch size that fits our system) us- ing AdamW optimizer (Kingma and Ba, 2014; Loshchilov and Hutter, 2017) with linear warm- up over the first 50k steps to a peak learning rate of 1.5e-1 and a cosine decay to a minimum learning rate of 1e-5. We then train PHD for additional 100k steps with the real historical scans using the same hyperparameters but without warm-up. Pre- training took 10 days on 2 x 80GB Nvidia A100 GPUs.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages
Online. Association for Computational Linguistics.
Alexander Robertson and Sharon Goldwater. 2018. Evaluating historical text normalization systems:
How well do they generalize? In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pa- pers), pages 720-725, New Orleans, Louisiana. As- sociation for Computational Linguistics.
GLUE Table 4 contains the hyperparameters used to finetune PHD on the GLUE benchmark. We did not run a comprehensive hyperparameter search due to compute limitations; these settings were manually selected based on a small number of preliminary runs.
Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023, Language modelling with pixels. Inter- national Conference Learning Representations.
SQuAD To finetune PHD on SQuAD, we used learning rate of 6.75e-6, batch size of 128, dropout probability of 0.0 and weight decay of 1e 5. We train the model for 50 000 steps.
Phillip Rust, Jonas Pfeiffer. Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 3115 3183313 Online, Association for Computational Linguistics.
Runaways Slaves in Britain To finetune PHD on the Runaways Slaves in Britain dataset, first trained the model on SQuAD using the hyperpa- rameters mentioned above. Then, we finetuned the resulting model for an additional 1000 steps the Runaways Slaves in Britain. The only hyperpa- rameter we changed between the two runs is the dropout probability, which we increased to 0.2.
Ray Smith. 2023, tesseract: Open source ocr engine. https:/ //gp::///iie htt///gttttttt tttpbttttuuugog//g////ggg wwtp/////eeacct.ooooo.o/ o esseract.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of Machine Learning Research, 9866)2557-26655
A.2 Dataset Generation
List of dataset augmentations To generate the synthetic dataset described in Section 1, we ap- plied the following transformations to the rendered images: text bleed-through effect; addition of ran- dom horizontal and lines; salt and pepper noise; Gaussian blurring; water stains effect; "holes-in- image" effect; colour jitters on image background; and random rotations.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like like visual explanations by watching movies and reading books. In The IEE. International Con- ference on Computer Vision (ICCV).
Converting the Caribbean Newspapers dataset into 368 x 368 scans We convert full newspaper pages into a collection of 368 x 368 pixels using the following process. First, we extract the layout of the page using the Python package Eynollah. 8
'nttps:/ //g thub. wtt. hatn//prattrrspk/eymme lah

---

Parameter
MNLI
QQP
QNLI
SST-2 COLA
STS-B
MRPC
RTE
WNLI
Classifiatioon-hcdd-ooooo
Mcan
Optimizer
AdamW
Adam B
(0.9, 0.999)
Adam
1e-8
Weight decay
1e-5
Learning rate
5e-2
Learning rate warmup steps
100
Learning rate schedule
Cosine annealing
Batch size
172
172
128
128
128
128
172
172
172
Max steps
10 /
Early stopping
Eval interval (steps/epoch)
500
500
500
500
100
100
100
250
100
Dropout probability
0.0
Table 4: hyperparameters used to train PHD on GLUE tasks.
This package provides the location of every para- graph the page, well their reading order. As newspapers tend to be multi-columned, we "lin- earise" the page into single-column document. We crop each paragraph and resize such that its width equals 368 pixels. We then concatenate all the resized paragraphs with respect to their reading order to generate a long, single-column document with width of 368 pixels. Finally, we use slid- ing window approach to split the linear page into 368 x 368 crops, applying stride of 128 pixcls. We reserve 5% of newspaper issues for validation, using the rest for training. See Fig 10 in App C for dataset examples.
We run linear probing CLIP using an CLIP
adaptation of OpenAl's official codebase." 10 We first extract image features from the VIT-L/14 CLIP model and then train a logistic regression model with L-BFGS solver for all classification tasks and an ordinary least squares linear regression model for the regression tasks (only STS-B).
Donut We finetune Donut-base using an adapta- tion of ClovaAl's official codebase." !I We frame each of the GLUE tasks image-to-text tasks: the model receives the (noisy) input image and is trained to produce output text sequence such as </ //s_class>
>> In this example, taken from SST-2, the < x > tags are new vocabulary items added to Donut and the label is an added vocab- ulary item for the positive sentiment class. All classification tasks in GLUE can be represented in this way. For STS-B, where the label is a floating point value denoting the similarity score between two sentences, we follow Raffel al. (2020) round and convert the floats into strings. 12 We fine- tune with batch size 32 and learning rate between le-5 and 3e-5 for a maximum of 30 epochs or 15 000 steps on images resized to a resolution 320 320 pixels.
B Historical GLUE Baselines
For all baselines below, we compute and average scores over random initializations.
OCR + BERT/PIXEL For each GLUE task, we first generate 5 epochs of noisy training data and run Tesseract on it to obtain noisy text datasets. Similarly, however without oversampling, tain noisy versions of our fixed validation sets. We then finetune BERT-base and PIXEL-base in the same way as Rust et al. (2023), with one main dif- ference: the noisy OCR output prevents us from separating the first and second sentence in sentence- level tasks. Therefore we treat each sentence pair single sequence and leave it for the models to identify sentence boundarics itself, similar to how PHD has to identify sentence boundaries in the im- ages. We use the codebase and training setup from Rust et al. (2023).9
ob-
OCR-free BERT/PIXEL For GLUE, we take results reported in (Rust et al., 2021). For SQuAD, take BERT model finetuned on SQuAD-v2,"
r- prppoo--ealllatttee
"ntrps://github.
loCdde example hwtp://c///.....///oo//ooo/
L816-L855
frrmm https: http:///ggggtteccc///eee/eeeeee
"https://gt thub. com/ www.xpw..ccccccpoootooooooo.w... i pixelel
t- ba-ese--e cased- 2quad

---

and evaluate it the validation set of SQuAD-v2. after being balanced for the existence of an an- swer. For the Runaways Slaves in Britain dataset, we finetune a BERT-base-cased model on a man- ually transcribed version of the dataset. We use the default SQuAD-v2 hyperparameters reported in the official Huggingface repository for training on SQuAD-v2. 15 We then evaluate the model balanced test set, containing 20% of the ads.
Additional Material
Figure 9 additional examples from our artificially generated dataset.
Figure 10 Sample scans from the real historical dataset, described in Section 4.2.
Figure 11 The process of generating the Visual SQuAD dataset. We first render the context as an image (a), generate patch-level label mask high- lighting the answer (b), add noise and concatenate the question (c).
Figure 12 Additional examples of PHD's comple- tions over test set samples.
Figure 13 Dimensionality reduction of embedding calculated our model historical scans. We that scans are clustered based visual similarity and page structure. However, further investigation is required to determine whether scans are also clustered based on semantic similarity.
Figure 14 Using PHD for semantic search. Fig- ure 14a and is the target of the search (the concept we are looking for), while Figure 14b and are the retrieved scans.
Figure 15 Additional examples of PHD's saliency maps for samples from the of the Runaways Slaves in Britain dataset.
Figure 16 Examples of shipping ads Newspapers. Newspapers in the Caribbean region routinely re- ported on passenger and cargo ships porting and departing the islands. These ads are usually well- structured and contain information such as relevant dates, the ship's captain, route, and cargo.
Figure 17 Input samples for PIXEL. The images are rolled, i.e., the actual input resolution is 16 8461 pixels. The grid represents the 16 16 patches that the inputs are broken into. Figure 18 An example of a full newspaper page downloaded from the "Caribbean project".
https::t..... / http:/:::///aaaaaacceeeeeeeeeeee http://ga.c........tttteeee cased 16 "nttps://colab. http://:///.aaabbeearr.....o/// gowgle...ggoooe...ooo.ooooo www.g....ooog......ooooooooooo
thub/ thub/hugg htt.://ottaa////ooooooo/es/ ob/master/a
xamples/ ittps//stsssttooneeseeeeeeee 1pynb

---

Ii of Ne Aoe Bst mistapoup of
Sagint
Tyy rok op hot detrnnieed by To make ahee oryy ghetorre cocosalinn whih swed of cognnts hat capssies heo o skay brow or orarge op and not lound amang tect
oppoction feney
plrtgrath to
Uoimssy Sonaonnto obeer be baggl
poolic Caffinns Si
Belly prosoed back tydle
wat 2005 Rphidbuce show offushoo
whyh wes
poblo conopieet work Aatand
odoping the person of a "opoos Apis
end fimmuni turakle latles
Bomrle lemeter 1801 lonk
lapu Thlee
book on periot that
the thatress dofuts the trtmee the
foom Jopontion
the rere hall teue Mersorn to- prancc merted
Takes milend the socrrppey faens by
dayy the
and food under troat voavel voos porttueeyy bechrrttion mach sphagrm nor foord
He is the igonn for shet inginn a
fwom
produced dhasings emsporns filre
which deconved by uelnn
way and Rowe Naviusa io has
thes. rn adoutts in the auns of leve ougi upi
pape wich puled
Tcort 5er tomppe Wew zhardd wolcor Jopan and Diin
actiorad groupe
foun larrh
ate hitt caale plloo sdype boo Bitis viiee o
secally Bagn Arcade humantia publobed Stetted Univerity from 2012 2005 he boatd webcan by Alseea Hears Clacktzer
Tansir for scit 1s, Asdes
Troe T
schieved qualifying
patogedd
following evenisy top
the
wose Couatiee indapendens
notting titto 2nd tha not
mysterious peness worting to denoiliie gerpllited balower
Olympicc Qualityig Time the
each
1' things I'd people canton plating
gigh foom druus
cavernating canoderces Antred and adteey together mas thy
has ostered athiete into Time (OST]):
potintially
Olympic
19t yeant was mis advice, wurnings
tensis
two decades of operatig Bussland, It was
Pugta unin and haw
the souhh babrn "Gouar befow is too
Olympic spot the
Gamess Grending Arvelo
the
the
Story podiaat.
"oo gox was ourt of the cytlang as Strapes oven comatee rookig most. the clstrotoo munds
by virtue top aix fisish the
singles
Selkman's is uer eicec took Takys
creat and the Thu puamered than how the nurorr
2016 Latin American Qualfication
served os one the "Mor antet Mor"
oscapation
Toumament Sartigp, Chile.
aovernnt
al. ("1nE" [1936]); "Cortriiuçto in contectrentó des ofidics de Beosil. VII. Novos géneros e espécies de Co Celsbrideus... opisthogl yphus Iestituto
were craeddoreed the balion. Froet Officndee Hhe had
Wool P Tonler Provacon appdined 09 Prevtncon saoes for.
adrrsicaatve diteed
County morthen Nownn with
coups conenndder His
Not of Decia regioval cipts
in nech-
had usted him saliabe higher communtd
Carputiion. bim after they served
saibbee ho on if
Butanton
Wenoiios
many onthenla
Alys
Jurtomet by
Eniondae dee
(Calansdon poucidens, species, p. 204)..
milkemmns Drowors netholasd Wes crevibi Costih Teaeeiin Symphaic Orthestes
In carty 1917 be remorned to the Bheern Corps belf bete Kleek soveeded Berutytsuty
election Hte pedenee 20 ao0
Portuguese). eallage in Bystrico Region of
malk
balet chavogrpred by hane Tourss centertes National Thatrs as Zagen
be bad beon pronottd to Greerabbure
controt Slovekip. Baing
Coutton
feseues the Bhom sabort Bhe mose Ap encop pine the
riss shod
bog
gare Face 170 Joon scouttng nows
harry
other organisations for improved pay and working conditions, and achieved
in the 1970s equalisation welfare and social security with induerial just
lhe for
Wed airppip
comete Bilt fan bouie South
the (laas
coom daace
laid workers
ai wee baik part jave dinch Spont
th
Henny mamred Lacy kone does the Eorahs
aad from Nis father in
Employnent sectur gradually 348,661
1987, it only merged with the Italian Federation of Food Industry and Tobacco Workers, form the Itaiin Federation of Agroindustiial
Chings
the
Artpert ate hes inpood dhr foum
tig.
bhssttund tle ds the Ean les
on
Figure 9: Samples of our artificially generated dataset, and compare to Figure 10.

---

MENT in our sald Bermuda Somers' Iehand, sakick Writt are to bear Teate on Wedsesdoy PifA Day of March, 1858, and retarnalie ToEsnAY, the 15th day of spril, 1868.
Fmmmedigtely after, siill he sold by the
ur
same requese, and at the absremention. ed Time and Place 1. the Estate Mount W Washingtons-
In testimony whersof We have caeeed the Great Seal of our sid Ielanda to be hereto sfased
Witness our Trusty and well Beloved
TRACT of Land called Parera, Kasted East Eidt Quarterr Let, A.un- <ler No. 18, nise belonging to Mr. Con- NaLits HOsT's babkeupt Deeling
FREEMAN MURRAY, Colonel in
Our Army, and Governor, Com-
mander-in-Chae Chief and Vice-Ad-
miral in and over Our said
W. MOURITZEN V Vendue Master
Bermuda Somers' Islands,
at Our Town of Hamilton, in our said Islanda this Twenty- Third day of February, 1858 and the Nineteenth Year of
Christinnated May 3, 1815.
Efttr Frederilotadd 8oifooe Porlmn.
geade Mieer,, dea Mi Mowned.
onynng home consumption, and mashed Potato.
enang ievertment
dresse Contoire Efferretainger tousiske Markur, eg ligele med 3 Man- meders Farull at regue foa dea Tid, Pro- elama bleens last and Ouer eg Under- vetterue pea Fare eestiudikke Bilande samt 3 G Gayg indfirt des danke destbudikke Regierings Auis indkalde alle bemeldte Juvisrradt GIBLLERU'S og Sorken afdide Hatros falles Boer pas forasforte Bilande beelle elter opholdende Credi- tueer aod deres Pardriger at konne
den Al.
wne sugerr (ragrancs to the breath, Price WHFTE GUTTA PERCHA EN- .AMEL, for stopping decayed Teeth renders Toolh sound for masticttion no matter how far decaved.Price-1/66.
W. KEJJ
Potatoes for the Million,
RS.
useful
IN barrels 8.1. by the single pound Themo Putstoes really good
LS
and Houry, and can recommennddd as of the finest trooe lately imported.
OSTEO-ENAMEES STOPPING, warranied remain white and firm as the Tooth iteelf. This Sesutifal preparation restores frent teath, and easily used, Nuficientt to stop six Teath. Price 5/-
W. KELL
33.
Octr. 3. TO TOBACCONISTS
117
og reedest deres Aavende Kew inden fore- streuse Tela Forlib for St. Thona Skifle-
ODONTALGIQUU ELIXIR. This celebrated Mouth W Wash is most re- freshing, It strengthene the gams tarbare all injarious se- cretions, sweetens the for eleansing artificial teeth
T THE undersigned have steck Hoge-
hesile superior Light TOBAC CO, suitable for Cigar making.
ret avaelde og besitaliggiire oy shal
D&COSTA C0.
beaeldte stiftefreattee On elstelige Kan-
jeke GOrPREd SCHMIDFEN
18.
2178
and
Colonnade October 1874.
aliatia ril lads dane Vae
Figure 10: Sample scans from the real historical dataset.
Beyonod Gindle KnowlesCarter Vhi jonneel/ bee. boN.Yo (born September 1981) is songweiter, record producer and aetress. Boen and mised in Houston, Texus. she performed in various singing and daneing competitions child, and the late 1990s lead singer of R&B gir-group Destiny's Child. Managed by her father. Mathew Knowlex the group became one of the would's bestaelling girl became one of the world's bestsslling gid droute all time. Their histus saw the
Beyonce Giselle KaowkesCurter (hi: jünsel/ ber-YONaay) (born September American singer.
When did Beyonce start becoming popular?
Gisdle KnowlesCCsterr Vbi : 'jOnset/ bea. bo..YONary) (born September
American singer.
4 1981) is songwriter, record producer and setrem. Born and raised in Houston, Temas she performed various singing and dancing fame
Beyoned
4. 1981) is Ameriian singer. songwriter. record producer and actrem. Born and raised Howston. Texas she perf performed ormed various singing and dancing compatitions
fame
competitions in the late 1990s lead singer of R&B girl-group Destiny's Child. Managed by her father. Mathew Knowles the group
child, and
child. and to fame in the late 1990s lend singer of R&B group Destiny's Child. Managed her father. Mathow Knowles the grout 13.. hosteellng gir (c) Adding 9 and degradations.
drouts of all time. Their histus saw the (a) Rendcring context
image
(h) Generating a label mask
Figure 11: Process of generating the Visual SQuAD dataset. We first render the context as an image (a), generate a patch-level label mask highlighting the answer (b), add noise and concatenate the question (c).

---

BROKSTANS
Infldmaation
Jaundice :: L
FOR NEW YORK
Liver Complaint
THELERIGA HAIEERIIATTION NTINE
Lumb baga
F.Ecelssio "x
Piles
Bilious Com plaints
Blotch on the skin
Sale 63 a hove
Bop
Power Complasnts
On Genenc day
boality
seal bakes Baaen
Colics
TaYSAGE.
agglark
DNTHUASDAY TEXT
Pupier d Turissoe
city
tam
M Maa
pele gulged
the
BREAD BEST mi
Ba Amandes App Appree
PORK, Aoa D60 Bar
Poriet
dach
MUSSEE ARBELL
Auction.
of FRED P
Wataondie and Lakes
The 10th Instaut T... 19 LLD.
aurty end
Sturecty Shl I
bests gonnn coeen Dae
PURNE PLATE de be.
Condaena of me
votae telabberio wheck a muly
"o sot Pracic
oditer doas
2HR Reihe de
maaas Pienh
Jcbah Toeeh con
and Hoems the can
anticted
oakey
rack
Sulphate Ammonia.
Doal Rowes
Far
taken
dle beminis
PUN
Pettagal
MANURE
Taiter
and
Planters' Book Club
the
hnd
BA 1 A
B
beel
An
hangs
lng by toch to raflrr reed
Figure 12: Additional examples of PHD's completions.

---

Figure 13: Dimensionality reduction of embedding calculated by our model on historical scans.
Chippenham, Manchester, July 1892.
colsonal BolEETTni
Taraly-Chasn St. Plliabei''s
June 16, 1884.
E. ELOPEDD The 9t FEESUURY, 176t, from Joiss STONE, EGG. CHIPPERAAM,
HOUHLON BBW
Silver- Hill, George's
Sept. 1. 1829
NEGRO SERVANT,
Named GLOUCESTER: marked with lang
Twentr-one
Agt, shout fioe Fost fox
Inches high, flender
Star down sie Middle of his For Forchesd, and foesks
D. G. Bratteler
LEATGRAPT & CO.
Tinns Caiman Jahole, Vel. F
Mo.
Englihh tolumbly mull, Work, when want off,
light coloocoloee Clock Livery Cast, and
ARTICLES.
Coachmaan & Poftillion
compleat negro
Walhooat,
with Red
Callary red
Holes, Lining, and Slsh Sleaves. Had lakeiid a beack Veivet Capy with a Silver Band, in dii Silver laced Hat, and an old Pair of Leather Brnscher.
DOUND weat age
brown Bay
W Wheerer fecures the fidd Negro and given
Natser of to Joan SToRt, Eig. aforetiad, to
that may be brought back agsin, But Peddin rewerded
Purt Dosal Mouraaty
THE 2009
Trouble-
harbouring the fadd Black, will proficuted agreabie Lan.
REUSEIAA MALTE A88ES, &c
JANE DUNEBETT
Dha
16
Auth hall
qusnilty of Carpailanns Tools
Turning Lathe
Right Huad Cattle
Elizabeth's Workhouse
(a) Semantic search target.
(b) Retrieved scans.
Figure 14: Semantic search using our model. (a) is the target of the search, and (b) are seans retrieved from the
newspaper corpus.

---

Poue BuzZANCIA, Cup, bost
R
RIGNTED Madee's
T Negro
LaMils ner to
ded sms interver
Robate
Thone (la) Rogin Right
EET
Toolly badia Hob, doat
B.L.
Biok Dutt fabin Bemtoon sert
hrop faben
to Co pus Thome baith
UN avay fiom Srilluss
the bad
1o gani Tipt
popirtt
A12313 comared peEarT
W VIL
W WHEREA
CEBROO a C
camer can
ther
Cimeyat infinm
Figure 15: Additional examples of PHD's saliency maps for samples from the of the Runaways Slaves in Britain dataset.

---

Fost
pá Book fommm E.EWW
Ariired
wrate de wit pangs
fris desghninmmdde Menginn
sp. adt Cattanite-blle (Don Gradecape
FORL HONTON
bi All Alt Nov Co-
Zip for Siw.titmmm----ee.. (ochal chagamm
de Caplia
parip OPS
Figure 16: Shipping ads samples. Newspapers in the Caribbean region routinely reported on passenger and cargo ships porting and departing the islands. These ads are usually well- structured and contain information such as relevant dates, ship's 's captain, route, and cargo.
the ukulele based sev eral kmall, quitar-like instruments of Portuquese ori gin. the machete. cayaguinho, timple, and raião, intr oduced the Hawaiian Islands by Portuquese immi arants from Madeira, Azores and Cape Verde. TI ree immigrants in particular, Madeiran sabinet mak Manuel Nunes,, José do Espirito Santo, isto Dias, are aenerallv. credited the first ukulele nakers,, Two weeks after thev dise mbarkeeddd from the ss Ravenscraq in late Auqust 1879, the Hawaiian azette reported that "Madeira Islonders recently ived here. have been deliahtina the people with nia htlv street concerts." One of the most important fa tors establi shina culture was the ar ardent support and promotion of the instrument by King Kalákaua. A patron of the arl incorporated it performances verinas, in the Hawaiian lanquade the word ukulele oughly translates as "jumping flee perhaps becaus the movement of the plaver's fincers. Legend ttributes the nickname of Enalishman Edward villiam Purvis, one of Kina Kalákaua's s officers, beca ise of his small size, fidgety manner, and plaving
Developed
Auax
ukulele Hawaiian music
royal oath
PIXEL's input.
Develcped in the 188ds, eral kmall, qu
is bast
ts of P
ain, machete, tavaguin mble, and rajão, intr
od bt awalial nds by Port bs
ants from Made
and Cape Verde. Ti
miiaral
particular, Madeira
ers Manuel Nunee
Espirito Santo, an
Dias, are denerallv credited
first uku
they disembarked from
: 55 Ravenscrad in late Auqust
the Hawaiian G
azette ro orted that Madeira Islanders recentlv
here, have delia
people with
htlv
s.
the most important
tors in establishing the
e in Hawaiiari music ar
culture the
oil
the ent by Kind Kalák
is, he incorporated it into perform
aat
Jerings
Hawaiian lan
vord ukulele
oughly translates as "jumping flea". pertaps beca
of the plaver's fincers. Le
utes it the nickname of Enalish
ard
Villim
Kalákaua's
ise
hnnner. and plavina exi
(b) PIXEL's masking.
Figure 17: Input samples for PIXEL. The images
rolled, i.e., the actual input resolution 16 x 8464 pixels. The grid represents the 16 x patches that the inputs are broken into.

---

Tit
Koyal
Casette.
BERMUDA COMMERCIAL AND GENERAL ADVERTISE AND RECORDER.
V4. XXXIX.
<TATE SUTAL VIAS ANTIQUAL
2de. Aua
Hauritonn Bermuda, Tuesloy, June 10, 1566.
Commissariat, Bermuda,
Merhanine Indnerisl 20 SeEEE H2 23
TersuR Zasnson the Jon, 15a.
GEORGE
Coestmee Gpeass, Sear S. 3 R D of Y,
by
Ur-
odhi der de ady Rom In Gamphotine the Asooiatien's
of for
&
HERMIRO, ROWEN DOLEXUSS
The 23rd June.
W flans ae-
BIS EXCERLENCE encoavaiton
Me
Pootbay DARSY u7,
Poncce vtite seppt with Crasttioo
pitt
EOPS.
anoud Bown Aueaion
Welnedar Sith Thunday 2843, and
Geerr Ord,
Bonc Deree to Andd
the fertir the
isme "iichmoond Grounds,"
thes
Widk far insouatio ou deraed
Than
WIEEEAA Pretionaalas
lhhes ir. Sr. So.
T PErrIte
KADIL LAG Tur co
T. W. GOLDIR
New Vonki-) bo
it
The Ban stors kost [eia]
a be wreckey
Acticlas Adaated the wanta of ail
Deal the Sermayyss relle
dentory
via the
Cless of Sccinty,
OSt OLVARORTAISED ALOUCOO PAZES
For Salo, BY-ELIZA BARSS" AND STORE.
and frmm lacl we baro
upplication St. Georges, General Store.
an
le af Pork
B. BaanTLaN here BeEA2 PLoti Uiiie MRAL
THE UNDERSIC CENENE
Oe moro ih andl dhreppo dfty the
HRCRIVING bilnele lIs the Bast Pairy.
The fdiwaggt-
bes THtA polk
you
thrse my fiodt Creet Intadariar
I. "boundonn 1001-4
la mas any nas
Th The 22111.22 Oue the PO., 10re00
de
WALKER.
C. W. GAUNTLETTE
MILES GEBALD KBON
DENSCOOE
The Sure Gezen
Ursuuii tb, to ba
Offere Sals,
I IC E.
Ex barque Eliza Harss, FRUM NEW YORK.
THE SUTRSCHIBRRS di fow kaniiig tHEm TEDAL SUFPET OF
MILLS (ed Dela Pubs
10 Barrels Tá R. How Som usc.
HORERES POLINUS
MARRY exnoes
boneatad MAllikAA
Curtes ntaril
HAS RECEIVED,
THE
IE.F3 IL BL George ded,
Eue Fires
Gode D0213
Hi usaal suppy of SCHLER GOODS.
THEMS sil in ande lonve aplledin
On the lot Jane
Chly. and
Soal.
Dutlte them wits
23 POO Le 12:5 D
Call
GOSLING BROS.
Cussk Glase wides TEAMLES
strean Gus
VAGTORIA BOTEL
Mo.
Gue Lare saLTs
FOSTER BONNELL
Frome Hamsiton
LOUTE on 2u Faes the fulk Gete and
SVAR: Bay lene da, o8k
Chill shine 10 poll
N.B. Repairad
eastly Madi
TEAS and COFFEE.
padd som cacaeni Fiom Headees Apd this 1901.
Geimenea Lousr by with
OXHORROW & HUGHRS
Chat 721, lud the the timm curres. Jare attio.
Guapp, Jun,
For Sale,
thesssunn ioreenen thame, Tass,
thanl fed
Per Bocer Impottaiion,
dud par ELIZA B-ISss,
Bortin.
ane trken the te-di
Tas gi
Bus Tua Maen PUCK
Bdtiill snall makes
B holesalc or Retail
GOSLING BROTHERS, Baalla and design
SODA WATER.
glto
Bottled Soda Water,
Hoie Contannt VILk
CHEESE,
Of Superior quality be
by
da.
Be.. de.
H. A. Frithh HIOTOGRAPHER ST. GEORGES.
pliel any queatit frou the Merical
Green GINGER,
Goocaes
each paa Riqunt's
HIGENBOTHOM. 2. Bore Niy to. 1h-
For Kioussni da.
H. B. DICKINSON
Rr. Ao
Late
Stedage
"ENEEEEED' 2 EZZZ, FOR RENE
By MILES GBRALD KBON,
Gelenn denitay.
Cartes de Visito, Pigweetts thase
Gos
Just Arrived,
This Commsodias Mousion
Bermuda.
Call RARIETIT Paal STANDS
PLATED
bo Lodkete labrosin
legs Macly
there
TTE Edowwke APP with Jorc 1. 1806 lartet
Calonial Becwetary
Prose dloert thi pow
Buel
wil lake dh Lut Higily
ALah
gro. Ihe Weats Sealeryy
Mourning Brooches,
Ear
NOT NOTICE.
dog
Eings and silver Thimbis
An Apprentice W Wanted Appl
The Salverdor offecs for Rost
GHILD & GAULTS.
Warehouuee an Queen Maset.
MILES GEILLLD KEON
Ziailion
1850
Ball Eaae,
Belllown, Mhandoo
KERWISK
Boae Unit
Haaciio,, 18i.
WM. I. COX.
(E-te, Ar fartur
Chinaal Shuiiays
Figure 18: An example of a full newspaper page downloaded from the "Caribbean project". Section 4.2 details the way of processing full newspaper pages so that they can be inputted to our model.


Reviews:
[{"review_id": "0e20iCHSdA", "reviewer": "Reviewer_nvph", "summary": "", "questions": "See above. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and kind words. Your recognition of our work’s merits and novelty is greatly appreciated.\n\n >*“It would be helpful to include examples … to show what happens if the answer patch lies at the boundary of the image patch and is half cropped”.*\n\nA: This is a good suggestion. We will supplement the appendix with such examples, as well as other edge cases. We note, however, that the accuracy of the page-layout analysis tool that we used (Eynollah) is very high, so such edge cases are uncommon. \n\n\n> *“... Include a discussion of the efficiency comparison between PHD and BERT… it is unclear whether PHD can encode the same number of actual text compared to BERT.”*\n\nA: Another good idea, which we will add to the camera-ready. Specifically, we will include an analysis of the performance of PHD as a function of font size and text length.\n\n\n> *“The choice of the 16-pixel height for each patch seems to be chosen specifically for the documents you have…”*\n\nA: We want to clarify that the Vision-transformer architecture we used (like most popular ViT models), is capable of processing only 16x16 pixel patches. Therefore, we didn't select this particular hyperparameter to align with the characteristics of our documents; rather, we had to adhere to it. Additionally, we adjust the resolution of documents, either upscaling or downscaling them, to conform to this 16x16 patch size (refer to line 897 in Appendix A.2). This ensures that PHD can effectively handle documents with varying DPIs. We will better elucidate this point by integrating Appendix A.2 into the main body of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a pretty novel work and has the potential to inspire many follow up work for understanding scanned (historical) documents. It has the following strengths: \n- Simplicity: the image-based modeling approach is a generalization of the PIXEL model in the previous work. No special technique is involved in pre-processing the images – simply dividing any input image to patches and PHD can treat each individual patch as a token and model them like a “language”. \n- PHD has a clever design that uses the same representation space in both the input and the output. Compared to previous work like TrOCR that tries to convert input images to text output, in PHD, the input and output are both image patches. It eliminates some difficulties like aligning image and text (which often involves much post-host processing and needs special loss like CTC to handle this). \n- Another interesting aspect in PHD is how it can be used for “text-only” classification or QA tasks like GLUE or SQUAD. The authors firstly render the text in images, and the PHD model generates a prediction based on the image. \n", "reasons_to_reject": "I don’t see any reasons why this paper should be rejected, while the paper can benefit with a discussion of the following: \n- For almost all the examples shown in the paper, the cropping seems to be perfect – that is, no cross text line cropping. It would be helpful to include examples (e.g., in fig 13) to show what happens if the answer patch lies at the boundary of the image patch and is half cropped. \n- It will be helpful to include a discussion of the efficiency comparison between PHD and BERT. For example, given the same 512 token width, it is unclear whether PHD can encode the same number of actual text compared to BERT. \n- The choice of the 16 pixel height for each patch seems to be chosen specifically tuned for the given documents you have. However I can imagine there could be  images with different scanning resolution (e.g., 200 dpi), sticking to 16 as the patch height might not lead to optimal results. It would be better to show some results testing the generalizability of the trained models. \n", "questions_to_authors": "See above. ", "paper_topic_and_main_contributions": "This paper proposes a new modeling approach called PHD with pixel-based language modeling for scanned (historical) documents. Instead of modeling with text tokens (typically generated by OCR engines), the method takes in patches of document images (pixels) and trains a language model on top of it. A similar masked language modeling objective is used to pre-train such models on a combination of synthetic and real data; the objective is to “black out” specific image patches and the goal is to recover the original text image. Empirical results on both general NLU tasks like GLUE and historical document QA tasks show that PHD can achieve good and sometimes even matching performances compared to text-based language models like BERT. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GTc9SFJ8ep", "reviewer": "Reviewer_Uoyc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our manuscript and for your useful suggestions. We are glad that you enjoyed our work and believe that our method is well-argued and our evaluation robust.\n\n\n> *“The cluster analysis may be superfluous — it’s not clear what the authors hoped to understand by performing that study…”*\n\nA: This is a valid criticism. In this experiment, we hoped to gain initial insights into the usefulness of PHD’s encoding in search and retrieve tasks. However, we acknowledge the experiment’s shortcomings. Therefore, for the camera-ready version, we will augment this section by conducting comprehensive experiments involving semantic search. Specifically, we will use PHD to encode a large set of scans **S** and a target scan **t** containing a specific topic of interest. We will then use cosine similarity to find scans in **S** that share the same topic as **t**.\n\n\n> *“...the synthetic training corpus consists of contemporary English that’s rendered in the same font and style as a historical document... [this] could limit the applicability of the model’s language understanding…”*\n\nA: While we note this concern in the manuscript (e.g., lines 221), we agree that the paper can benefit from further developing this discussion. We will do so for the camera-ready version. We will better clarify the following key points:\n1. Historical corpora are either too small to train a strong LLM successfully or cannot be easily obtained. Therefore, using modern texts is a compromise we must make.\n2. We claim that the model can learn general, transferable language understanding skills from contemporary English texts, while their adaptation to the historical domain can be learned from the smaller historical corpus. This pretraining approach is commonly (and successfully) used in many analogue problems. For example, to train a model on a low-resource language, it is beneficial to also train the model on another, high-resource language.  \n\n> *“also, much of the language understanding evaluation is based on contemporary English as well;...”*\n\nA: This is an important limitation of our work, which we also acknowledged in the limitation section of the paper (line 605). While we leave the development of a historical variant of GLUE to future work, we will include in the camera-ready version a benchmark of semantic search over historical texts (as mentioned above). We hope this additional benchmark will give more insights into PHD’s ability to understand historical language. \n\n\n> *“The contribution of the authors’ can be characterized as adapting an existing method to a novel domain, which may not be compelling to some…”*\n\nA: While we acknowledge and are pleased that you do not share this view, we want to stress that our work goes beyond simply applying an existing model to a new domain. Rather, several methodological innovations had to be developed in creating PHD, such as the pretraining method, dataset generation and processing, and evaluation schemes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method proposed is well-argued and the limitations are clearly discussed.  The evaluation is robust, evaluating both the model’s ability to resolve corruption of both synthetic and actual historical documents, as well as measuring the model’s language understanding against standard benchmarks.", "reasons_to_reject": "This reviewer sees no core or overwhelming reason to reject this study; the contribution of the authors’ can be characterized as adapting an existing method to a novel domain, which may not be compelling to some, but it is an interesting and thorough study of the applicability of those ideas to said domain.   \n\nThe cluster analysis may be superfluous — it’s not clear what the authors hoped to understand by performing that study if not the effectiveness of the encoder at providing a deep semantic representation of historical documents, however the authors noted that they only evaluated the visual similarity of similarly encoded documents.\n\nOne limitation perhaps not adequately discussed is that much of the synthetic training corpus consists of contemporary English that’s rendered in the same font and style as a historical document, and also much of the language understanding evaluation is based on contemporary English as well; depending on the time period in which the documents of interest are written, shifts in style, the meaning of words, etc., could limit the applicability of the model’s language understanding — however, evaluation on actual historical documents is performed, so this limitation is to a degree quantified.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method for historical document reconstruction based on PIXEL, a language model that’s unique in that it deals in visual, rather than token-based, representations of language.  The main contribution of this paper is a language model that follows the general design of PIXEL but is trained on a synthetically generated corpus of historical documents.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xg3MAC1hDT", "reviewer": "Reviewer_NET7", "summary": "", "questions": "The previous studies utilized datasets such as IIT-CDIP and DocVQA for pre-training and evaluation. Can you discuss why you did not consider them?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and detailed feedback. We are pleased that you believe that our work is interesting and novel and that the paper is well-written. We hope that we can address some of your concerns:\n\n> *“Most of the pretraining data is modern text, not historical…”*\n\nA: Reviewer **Uoyc** raised this concern as well, which we also note in the manuscript (e.g., lines 221). We will better clarify the following key points in the camera-ready version:\n1. Historical corpora are either too small to train a strong LLM successfully or cannot be easily obtained. Therefore, using modern texts is a compromise we must make.\n2. We claim that the model can learn general, transferable language understanding skills from contemporary English texts, while their adaptation to the historical domain can be learned from the smaller historical corpus. This pretraining approach is commonly (and successfully) used in many analogue problems. For example, to train a model on a low-resource language, it is beneficial to also train the model on another, high-resource language. \n\n\n> *“...previous work, such as Donut, DiT, Dessurt, and LayoutLM, pre-trained their models on IIT-CDIP…”*\n\nA: We considered adding IIT-CDIP to our pretraining corpora. However, we dismissed this idea as the dataset's domain is narrow and dissimilar to historical texts (it consists of documents from a state's lawsuit against the tobacco industry in the 1990s). We will add this consideration to a footnote in the camera ready.\n\n\n> *“The evaluation tasks, apart from one historical QA dataset, predominantly involve modern text. More historical evaluation data could better assess performance.”*\n\nA: This is an important limitation of our work, which we also acknowledged in the limitation section of the paper (line 605). Following your comment and Reviewer **Uoyc**'s request, we will augment the end of section 4.4 (Clustering) by conducting experiments involving semantic search over historical texts. Specifically, we will use PHD to encode a large set of historical scans **S** and a target scan **t** containing a specific topic of interest. We will then use cosine similarity to find scans in **S** that share the same topic as **t**. We hope this additional benchmark will give more insights into PHD’s ability to understand historical language.\n\n\n\n> *“There are also some document understanding benchmarks, such as DocVQA…, which can be used to evaluate the question-answering performance of models.”*\n\nA: This is an interesting suggestion. While these are modern datasets and, as such, not the focus of our paper, they can still provide some insights into the capabilities of PHD. We will add an appendix where we analyse the performance of PHD on these datasets.\n\n\n> *“OCR techniques continue to improve over time. At some point, OCR quality may be sufficient to apply standard NLP pipelines to historical texts without needing to bypass OCR.”*\n\nA: This is an interesting and valid perspective. Nevertheless, we anticipate that it will require many years before OCR models reach a level of quality that guarantees freedom from errors. Furthermore, as Nguyen et al. [Survey of Post-OCR Processing Approaches. ACM Computing Surveys, 2021] point out, OCR engines will face difficulties with historical texts even if they perform well on modern texts, because *“they still lack adequate training data composed of past documents”*. In the meantime, OCR models could potentially propagate errors into downstream tasks and pipelines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explores an interesting new direction and provides a thorough evaluation of the proposed techniques. Releasing the datasets and models could catalyze more work in this area.\n1. Well-written paper that clearly explains the motivation, proposed approach, experiments, results, and limitations.\n2. Novel application of pixel-based language models to historical document analysis, bypassing the need for OCR. This is an interesting new direction for processing historical texts.\n3. Releases new datasets, models, and code to facilitate research in this area. The historical QA dataset created from real newspaper ads could be valuable for the community.", "reasons_to_reject": "1. Most of the pretraining data is modern text, not historical. More diverse historical data could help the model better adapt to that domain. Previous work, such as Donut, DiT, Dessurt, and LayoutLM (v1, v2, v3), pre-trained their models on IIT-CDIP. IIT-CDIP is a large-scale scanned document corpus used for pre-training language models.\n\n2. The evaluation tasks, apart from one historical QA dataset, predominantly involve modern text. More historical evaluation data could better assess performance. \n\n3. There are also some document understanding benchmarks, such as DocVQA (also used in Donut, DiT, Dessurt, and LayoutLM v1, v2, v3), which can be used to evaluate the question-answering performance of models.\n\n4. As the paper mentions, evaluating the pixel-based completions is challenging. More robust quantitative evaluation methods are needed.\n\n5. OCR techniques continue to improve over time. At some point, OCR quality may be sufficient to apply standard NLP pipelines to historical texts without needing to bypass OCR.", "questions_to_authors": "The previous studies utilized datasets such as IIT-CDIP and DocVQA for pre-training and evaluation. Can you discuss why you did not consider them?", "paper_topic_and_main_contributions": "This paper introduces PHD, a pixel-based language model for analyzing historical documents without using OCR. The main contributions are:\n\n1. Proposes a novel method to generate synthetic scans that resemble historical documents for pretraining. This helps address the scarcity of large historical scan datasets.\n\n2. Pretrains PHD on a combination of synthetic scans and real historical newspapers from the 18th-19th centuries.\n\n3. Evaluates PHD on image reconstruction, clustering, language understanding tasks like GLUE, and question answering on both SQuAD and a real historical QA dataset.\n\n4. Provides evidence that PHD can effectively understand language and has potential for assisting with NLP tasks involving historical documents.\n\n5. Releases the datasets, models, and code to facilitate future research.\n\nOverall, this paper explores using recent advances in pixel-based language modeling to process historical scans directly at the pixel level. This allows bypassing the OCR stage which can introduce noise when applied to historical documents. The proposed pretraining methodology and evaluations demonstrate the promise of this approach for historical document analysis.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Lewis, David D. et al. “Building a test collection for complex document information processing.” Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval (2006): n. pag.\n\n- Mathew, Minesh et al. “DocVQA: A Dataset for VQA on Document Images.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 2199-2208."}}]

Author rebuttal:

Meta-review: The reviewers all agree that the paper proposed a novel and useful approach to analyzing scanned historical documents, building on previous work in the area. The problem of analyzing historical text is of great interest to social scientists and researchers in the humanities, and the proposed method could greatly improve their ability to work with historical text. The use of synthetic training data is clever, and the evaluations are well done. One concern is that the training data is taken from modern, as opposed to historical text. The authors, however, provide a reasonable data availability justification for this point. The potential reasons to reject (modern text, 16 pixel size, confusing cluster analysis, missing comparisons to corpora and BERT) are well responded to by the authors in the rebuttal period.
Decision: Accept-Main