RANLP-ALP 2023

Proceedings of theAncient Language Processing Workshop
associated with
The 14th International Conference on
Recent Advances in Natural Language Processing
RANLP 2023

8 September, 2023

ANCIENT LANGUAGE PROCESSING WORKSHOPASSOCIATED WITH THE 14TH INTERNATIONAL CONFERENCE
RECENT ADVANCES IN
NATURAL LANGUAGE PROCESSING (RANLP 2023)

PROCEEDINGS
8 September 2023
ISBN 978-954-452-087-8
Designed by INCOMA Ltd.
Shoumen, BULGARIAii

Preface
The First Workshop on Ancient Language Processing (ALP-2023) is co-located with the fourteenth
edition of Recent Advance in Natural Language Processing (RANLP) in Varna, Bulgaria on 8 September,
2023.
Ancient languages function as stores of the historical and cultural legacy of humanity. In recent years,
significant advancements have been made through the utilization of language technologies in the analysis
and interpretation of archaic languages. The purpose of the workshop is to establish a reputable platform
for both scholars and practitioners, facilitating the exchange of their most recent research findings andfostering meaningful discussions.
The workshop has received thirty-six submissions covering a wide variety of ancient languages, including
Ancient Chinese, Ancient Tibetan, Ancient Greek, Latin, Etruscan, Akkadian, Sumerian, Ancient Syriac,
Ancient Hebrew, Basquenglish, Classical Arabic, Meroitic, Middle High German, Pali, and Sanskrit.
Among these, Latin, Greek, ancient Chinese, and Sumerian are prominently featured. Within the realm of
natural language processing, this collection of papers employs techniques that encompass a diverse rangeof methodologies, spanning from qualitative analyses and corpus construction strategies to sophisticatedmachine learning algorithms.
We have accepted sixteen papers for oral presentations and nine for poster presentations. The topics ofthe accepted submissions include: morphological analysis, POS-tagging and lemmatization, parsing,evaluation of LLMs, text annotation, corpus construction, distributional semantic models, emotion
recognition, machine translation, corrupted text correction, hand-written text recognition, intertextual
identification, stylistic analysis, named entity recognition, input method, and NLP pipeline systems. The
quality resonating through these submissions is genuinely commendable, highlighting the excellence thatcharacterizes the content of this workshop.
The field of ancient language processing is witnessing an expanding research community, driven by theincreasing availability of ancient language resources and the growing interest of scholars with machinelearning expertise in this domain. The strong turnout in the inaugural year of the ALP workshop serves
as a testament to this trend. Hence, we are confident that this event will continue to thrive, fostering
an environment for dynamic discussions and interdisciplinary collaborations in both the research andapplications of this domain.
We extend our gratitude to the members of the Program Committee for their exhaustive reviews,
including the acceptance of additional reviews. We are also appreciative of the RANLP conference
chairs, whose tremendous and timely assistance proved invaluable. Last but not the least, we would liketo extend our thanks to the student voluteers: Bolin Chang, Zhixiao Zhao, Yutong Zhang, Yixuan Zhang,
Kaixin Yin, Feng Xie, and Zhixing Xu.
The ALP-2023 Organizers
Adam Anderson, UC Berkeley, USA
Shai Gordin, Ariel University, Israel
Stav Klein, Tel Aviv University, Israel
Bin Li, Nanjing Normal University, China
Yudong Liu, Western Washington University, USA
Marco C. Passarotti, Università Cattolica del Sacro Cuore, Italyiii

Organizers:
Adam Anderson (UC Berkeley, USA)
Shai Gordin (Ariel University, Israel)
Stav Klein (Tel Aviv University, Israel)
Bin Li (Nanjing Normal University, China)
Yudong Liu (Western Washington University, USA)
Marco C. Passarotti (Università Cattolica del Sacro Cuore, Italy)

Programme Committee:
Alaa Mamdouh Akef (Beking University, China)
Masayuki Asahara (National Institute for Japanese Language and Linguistics, Japan)
Jonathan Berant (Tel Aviv University, Israel)
Monica Berti (Leipzig University, Germany)
Gregory Crane (Tufts University, USA)
Sanhong Deng (Nanjing University, China)
Minxuan Feng (Nanjing Normal University, China)
Toon Van Hal (University of Leuven, Belgium)
Renfen Hu (Beijing Normal University, China)
Heidi Jauhiainen (University of Helsinki, Finland)
Kyle P. Johnson (Berlin-Brandenburg Academy of Sciences, Germany)
Orly Lewis (Hebrew University of Jerusalem, Israel)
Johann-Mattis List (Max Planck Institute for Evolutionary Anthropology, Germany)
Chao-Lin Liu (National Chengchi University, Taiwan)
Liu Liu (Nanjing Agricultural University, China)
Congjun Long (Chinese Academy of Social Sciences, China))
Longlong Ma (University of Chinese Academy of Sciences, China)
Francesco Mambrini (Università Cattolica del Sacro Cuore, Milan, Italy)
Martijn Naaijer (University of Copenhagen, Denmark)Christian M. Prager (University of Bonn, Germany)
Luis Sáenz (Ariel University/Heidelberg University, Israel/Germany)
Si Shen (Nanjing University of Science and Technology, China)
Thea Sommerschield (Ca’ Foscari University of Venice, Italy)
Rachele Sprugnoli (Università degli Studi di Parma, Italy)
Qi Su (Peking University, China)
Xurui Tang (Huazhong University of Science and Technology China)
Niek Veldhuis (University of California, Berkeley, USA)
Dongbo Wang (Nanjing Agricultural University, China)

Invited Speakers:
Dr. Chu-ren Huang (The HongKong Polytechnic University, China)
Dr. Thea Sommerschield (Ca’ Foscari University of Venice, Italy)
Dr. Gabriel Stanovsky (Hebrew University of Jerusalem, Israel)vTable of Contents
Training and Evaluation of Named Entity Recognition Models for Classical Latin
Marijke Beersmans, Evelien de Graaf, Tim Van de Cruys and Margherita Fantoli . . . . . . . . . . . . . . . 1
Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation
Kevin Krahn, Derrick Tate and Andrew C. Lamicela . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
A Transformer-based parser for Syriac morphology
Martijn Naaijer, Constantijn Sikkel, Mathias Coeckelbergs, Jisk Attema and Willem Th. Van
Peursen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23
Graecia capta ferum victorem cepit. Detecting Latin Allusions to Ancient Greek Literature
Frederick Riemenschneider and Anette Frank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Larth: Dataset and Machine Translation for Etruscan
Gianluca Vico and Gerasimos Spanakis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map
for Future WorkSilvia Stopponi, Nilo Pedrazzini, Saskia Peels, Barbara McGillivray and Malvina Nissim . . . . . . 49
Latin Morphology through the Centuries: Ensuring Consistency for Better Language Processing
Federica Gamba and Daniel Zeman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Cross-Lingual Constituency Parsing for Middle High German: A Delexicalized Approach
Ercong Nie, Helmut Schmid and Hinrich Schütze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Can Large Langauge Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE
Yixuan Zhang and Haonan Li . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Unveiling Emotional Landscapes in Plautus and Terentius Comedies: A Computational Approach for
Qualitative Analysis
Davide Picca and Caroline Richard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Morphological and Semantic Evaluation of Ancient Chinese Machine Translation
Kai Jin, Dan Zhao and Wuying Liu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
A tailored Handwritten-Text-Recognition System for Medieval Latin
Philipp Koch, Gilary Vera Nuñez, Esteban Garces Arias, Christian Heumann, Matthias Schöffel,
Alexander Häberlin and Matthias Assenmacher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Evaluating Existing Lemmatisers on Unedited Byzantine Greek Poetry
Colin Swaelens, Ilse De Vos and Els Lefever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Vector Based Stylistic Analysis on Ancient Chinese Books: Take the Three Commentaries on the Springand Autumn Annals as an Example
Yue Qi, Liu Liu, Bin Li and Dongbo Wang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
A Joint Model of Automatic Word Segmentation and Part-Of-Speech Tagging for Ancient Classical Texts
Based on Radicals
Bolin Chang, Yiguo Yuan, Bin Li, Zhixing Xu, Minxuan Feng and Dongbo Wang . . . . . . . . . . . . 122

vii

Introducing an Open Source Library for Sumerian Text Analysis
Hansel Guzman-Soto and Yudong Liu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Coding Design of Oracle Bone Inscriptions Input Method Based on "ZhongHuaZiKu" Database
Dongxin Hu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Word Sense Disambiguation for Ancient Greek: Sourcing a training corpus through translation alignment
Alek Keersmaekers, Wouter Mercelis and Toon Van Hal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Enhancing State-of-the-Art NLP Models for Classical Arabic
Tariq Yousef, Lisa Mischer, Hamid Reza Hakimi and Maxim Romanov . . . . . . . . . . . . . . . . . . . . . 160
Logion: Machine-Learning Based Detection and Correction of Textual Errors in Greek Philology
Charlie Cowen-Breen, Creston Brooks, Barbara Graziosi and Johannes Haubold . . . . . . . . . . . . . 170
Classical Philology in the Time of AI: Exploring the Potential of Parallel Corpora in Ancient Languages
Tariq Yousef, Chiara Palladino and Farnoosh Shamsian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Using Word Embeddings for Identifying Emotions Relating to the Body in a Neo-Assyrian Corpus
Ellie Bennett and Aleksi Sahala . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
A Neural Pipeline for POS-tagging and Lemmatizing Cuneiform Languages
Aleksi Sahala and Krister Lindén . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Tibetan Dependency Parsing with Graph Convolutional Neural Networks
Bo An . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
On the Development of Interlinearized Ancient Literature of Ethnic Minorities: A Case Study of theInterlinearization of Ancient Written Tibetan Literature
Congjun Long and Bo An . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

viii

Program
Friday, Sept 8, 2023
9:00–9:10

Opening Remarks

9:10–9:40

Invited Talk by Chu-Ren Huang: Processing of a ‘living’ ancient language: Issues
and Insights from Chinese

9:40–10:55

Oral Session 1:

9:40–9:55

Training and Evaluation of Named Entity Recognition Models for Classical Latin
Marijke Beersmans, Evelien de Graaf, Tim Van de Cruys and Margherita Fantoli

9:55–10:10

Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation
Kevin Krahn, Derrick Tate and Andrew C. Lamicela

10:10–10:25

A Transformer-based parser for Syriac morphology
Martijn Naaijer, Constantijn Sikkel, Mathias Coeckelbergs, Jisk Attema and Willem
Th. Van Peursen

10:25–10:40

Graecia capta ferum victorem cepit. Detecting Latin Allusions to Ancient Greek
Literature
Frederick Riemenschneider and Anette Frank

10:40–10:55

Larth: Dataset and Machine Translation for Etruscan
Gianluca Vico and Gerasimos SpanakisixFriday, Sept 8, 2023 (continued)
10:55–11:15

Coffee break

11:15–12:00

Oral Session 2:

11:15–11:30

Evaluation of Distributional Semantic Models of Ancient Greek: Preliminary Results and a Road Map for Future WorkSilvia Stopponi, Nilo Pedrazzini, Saskia Peels, Barbara McGillivray and Malvina
Nissim

11:30–11:45

Latin Morphology through the Centuries: Ensuring Consistency for Better Language Processing
Federica Gamba and Daniel Zeman

11:45–12:00

Cross-Lingual Constituency Parsing for Middle High German: A Delexicalized Approach
Ercong Nie, Helmut Schmid and Hinrich Schütze

12:00–13:00

Poster Session
Can Large Langauge Model Comprehend Ancient Chinese? A Preliminary Test on
ACLUE
Yixuan Zhang and Haonan Li
Unveiling Emotional Landscapes in Plautus and Terentius Comedies: A Computational Approach for Qualitative Analysis
Davide Picca and Caroline Richard
Morphological and Semantic Evaluation of Ancient Chinese Machine Translation
Kai Jin, Dan Zhao and Wuying Liu
A tailored Handwritten-Text-Recognition System for Medieval Latin
Philipp Koch, Gilary Vera Nuñez, Esteban Garces Arias, Christian Heumann,
Matthias Schöffel, Alexander Häberlin and Matthias Assenmacher
Evaluating Existing Lemmatisers on Unedited Byzantine Greek Poetry
Colin Swaelens, Ilse De Vos and Els Lefever
Vector Based Stylistic Analysis on Ancient Chinese Books: Take the Three Commentaries on the Spring and Autumn Annals as an Example
Yue Qi, Liu Liu, Bin Li and Dongbo Wangx

Friday, Sept 8, 2023 (continued)
A Joint Model of Automatic Word Segmentation and Part-Of-Speech Tagging forAncient Classical Texts Based on Radicals
Bolin Chang, Yiguo Yuan, Bin Li, Zhixing Xu, Minxuan Feng and Dongbo Wang
Introducing an Open Source Library for Sumerian Text Analysis
Hansel Guzman-Soto and Yudong Liu
Coding Design of Oracle Bone Inscriptions Input Method Based on“ZhongHuaZiKu” Database
Dongxin Hu
13:00–14:30

Lunch break

14:30–15:00

Invited Talk by Thea Sommerschield: When the past meets the future at Odessus

15:00–16:00

Oral Session 3:

15:00–15:15

Word Sense Disambiguation for Ancient Greek: Sourcing a training corpus throughtranslation alignment
Alek Keersmaekers, Wouter Mercelis and Toon Van Hal

15:15–15:30

Enhancing State-of-the-Art NLP Models for Classical Arabic
Tariq Yousef, Lisa Mischer, Hamid Reza Hakimi and Maxim Romanov

15:30–15:45

Logion: Machine-Learning Based Detection and Correction of Textual Errors in
Greek Philology
Charlie Cowen-Breen, Creston Brooks, Barbara Graziosi and Johannes Haubold

15:45–16:00

Classical Philology in the Time of AI: Exploring the Potential of Parallel Corpora
in Ancient Languages
Tariq Yousef, Chiara Palladino and Farnoosh Shamsianxi

Friday, Sept 8, 2023 (continued)
16:00–16:20

Coffee break

16:20–17:20

Oral Session 4:

16:20–16:35

Using Word Embeddings for Identifying Emotions Relating to the Body in a NeoAssyrian Corpus
Ellie Bennett and Aleksi Sahala

16:35–16:50

A Neural Pipeline for POS-tagging and Lemmatizing Cuneiform Languages
Aleksi Sahala and Krister Lindén

16:50–17:05

Tibetan Dependency Parsing with Graph Convolutional Neural Networks
Bo An

17:05–17:20

On the Development of Interlinearized Ancient Literature of Ethnic Minorities: A
Case Study of the Interlinearization of Ancient Written Tibetan Literature
Congjun Long and Bo An

17:20–17:50

Invited Talk by Gabriel Stanovsky: Harnessing Multilingual Models for Ancient
Language Processing

17:50–18:00

Open Discussion and Closingxii

Training and Evaluation of Named Entity Recognition Models
for Classical Latin
Marijke Beersmans and Evelien de Graaf and Tim Van de Cruys and Margherita Fantoli
KU Leuven, Faculty of Arts
Blijde Inkomststraat 21, 3000 Leuven, Belgium
marijke.beersmans@kuleuven.beevelien.degraaf@kuleuven.be
tim.vandecruys@kuleuven.bemargherita.fantoli@kuleuven.be
Abstract

annotation by the best-performing modelof a large corpus of literary classical Latin
texts and documenting the strengths and weaknesses of the resulting annotation.

We evaluate the performance of various models on the task of named entity recognition(NER) for classical Latin. Using an existingdataset, we train two transformer-based LatinBERT models and one shallow conditional random field (CRF) model. The performance isassessed using both standard metrics and a detailed manual error analysis, and comparedto the results obtained by different already released Latin NER tools. Both analyses demonstrate that the BERT models achieve a better
f1-score than the other models. Furthermore,
we annotate new, unseen data for further evaluation of the models, and we discuss the impactof annotation choices on the results.

1

The paper contributes to the application of NLP
to Latin on a methodological level, since we propose a thorough analysis of the results of NER on
Latin and identify the most critical points. In addition, the paper is associated with the publicationof NER models and datasets, and documents the
choices that have been implemented. The paper
is structured as follows: after introducing existing
work and datasets related to NER for Classical Languages (Section 2), we describe the data used, andthe training of the models and their performance on
in-domain and out-of-domain test sets (Section 3).
Section 4 provides a qualitative error analysis of
the best performing model based on F1 metrics. In
section 5, we introduce the annotation of new datafrom the LASLA corpus, and analyse the resultsof the automatic annotation by the best-performing
model. The data and code related to this paper are
made available on a Github repository.1

Introduction

Commonly an important precursor to information
extraction, text summarisation and the creationof knowledge bases, Named Entity Recognition(NER) has become a ubiquitous task in Natural
Language Processing (NLP). For modern highresource languages, generic NER off-the-shelf solutions, focusing mainly on identifying locations,organizations and people, can produce highly accurate annotations. For historical languages, evenprolific ones like Latin, the task remains a challenge, in part due to a lack of annotated corpora
and tools (Ehrmann et al., 2021).
We pursue three main objectives with this paper:

2

Related work

Previous work has highlighted the challenges
linked to NER for Latin. Ehrmann et al. (2021)
identified among others the following relevant challenges concerning NER on historical documents:
variable and sparse feature space (generalizing overdifferent genres and domains, cf. Erdmann et al.
(2016)), dynamics of language such as spellingvariations and change in naming conventions, general lack of resources (e.g. typologies from modern languages not fitting for historical documents).
In addition, Burns (2023) underlined another difficulty of the already scarce resources: differences in orthographic conventions and annotation• We compare the performance of three different models for Latin NER using pre-existing,openly available data. The comparison is bothquantitative and qualitative.
• Based on the analysis of existing annotations
and the results of automatic annotation, we
publish a new set of gold data, providing documentation of the most critical choices.
• By using the newly annotated data to assess
the results of NER, we publish the automatic

1
https://github.com/NER-AncientLanguages/
Ner-Latin-RANLP.

1
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 1–12,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_001

schemes. Lastly, both Chastang et al. (2021) and
Torres Aguilar (2022) consider the frequency ofoverlapped and nested entities in Latin as a challenge.

text
BGall.
NHEp.
Ars am.
BCiv.

When it comes to existing models, Chastang
et al. (2021) trained a CRF-based model on Latin
medieval charters from Burgundy. Later Torres Aguilar (2022) tested two approaches for creating a multilingual pipeline for medieval charters(French, Spanish and Latin): the first uses contextual and static embeddings coupled to a BiLSTMCRF (Bidirectional Long-Short Term Memory)classifier, and the second employs a fine-tuning
method using the pre-trained multilingual BERT
and RoBERTa models. For both of these efforts,
custom charter corpora were annotated. In the context of the Herodotos project — which aims tocatalogue ancient ethno-political groups and theirinteractions — Erdmann et al. (2016, 2019) createda neural, BiLSTM-CRF based entity recognizer(Lample et al., 2016) trained on classical Latin
texts. In addition, NER is included in text analysis
pipelines for Latin, such as the Classical Language
Toolkit (CLTK; Johnson et al., 2021) and LatinCy(Burns, 2023).

# tokens
58,62135,672
18,571
17,102
4,819

Table 1: Number of tokens per text in the Herodotos
datasetable online.2 We decided to annotate new material
to augment the availability of data for classical
Latin.

3

Data and methods

3.1

Data

The Herodotos dataset contains two full texts, Caesar’s Bellum Gallicum (BGall.) and Ovid’s Ars
Amatoria (Ars am.), and excerpts from three othertexts: a part of the first book of Caesar’s Bellum
Civile (BCiv.); book 1, book 2 and a part of book3 of Pliny the Younger’s Epistulae (Ep.); the preface, first and a part of the second book of Pliny
the Elder’s Naturalis Historia (NH). The editions
were taken from the Latin Library (Carey, s.d.) and
the Perseus Project (Smith et al., 2000). Table 1
contains an overview of the dataset sizes.
The texts are manually annotated for location(‘LOC’), person (‘PERS’) and (socio-ethnic) group(‘GRP’) entities (Erdmann et al., 2016). The annotations are encoded in BIO-format, where eachtoken is mapped to an ‘O’ (for ‘outside’, not an entity) or an entity type with either a B- or an I-prefix.
The B-prefix, for ‘beginning’, indicates the firstor only word of an entity whereas the I-prefix, for‘inside’, specifies a continuation of a multi-word
entity. Nested entities were not considered.
On the whole dataset, minimal preprocessing
was performed to iron out formatting mistakes. Afterwards, the five works were divided into two parts:
in-domain, used for training and in-domain testing,and out-domain, used exclusively for out-domain
testing. The latter should assess the model’s generalizing capabilities to texts that are significantly
different from the data it was trained on. In this experiment, the in-domain part consisted of the prosetexts, (BGall., Bciv., Ep. and NH.) The out-domainpart consisted of the one poetry text, Ars. Am..

In recent years, transformer-based models (withthe BERT architecture as one of the prime instantiations) have become the norm for various NLP
applications (Ehrmann et al., 2022; Sprugnoli et al.,2022; Sommerschield et al., 2023). These models have been leveraged, inter alia, for Latin morphosyntactic tagging (Wróbel and Nowak, 2022;
Mercelis and Keersmaekers, 2022; Nehrdich, 2022)
and translation alignment for ancient languages(Yousef et al., 2022b), which could also be leveraged for named entity projection from modern languages given a parallel corpus (Yousef et al., 2023).
For Greek NER, a BERT-based approach equally
proved to be effective (Yousef et al., 2022a). Therealready exists a transformer-based model for Latin(LatinBERT; Bamman and Burns, 2020) but tothe best of our knowledge, it has not yet been finetuned on the task of named entity recognition.
Regarding datasets, the Herodotos dataset (atthe time of training) is the only available NER
dataset for classical Latin (Erdmann et al., 2019,2023). Additionally, the authors of the LatinCy
pipeline are planning to make their custom dataset
publicly available (Burns, 2023). Lastly, the multilingual Medieval charter dataset, which includes
non-classical Latin (Torres Aguilar, 2022), is avail-2
https://gitlab.com/magistermilitum/ner_medieval_multilingual/

2

type

frequency
Train ValidationO
B-PERS
I-PERS
B-LOC
I-LOC
B-GRP
I-GRP

82,696
2,706
618
839
31
1,271
4

The results of these models are compared to LatinCy, a SpaCy pipeline for Latin, and for the
LASLA test set (see below) to the Herodotos entity recognizer (Erdmann et al., 2016) as well. In
order to train several SpaCy pipelines (Honnibaland Montani, 2017) for Latin (viz. a small, medium
and large model), Burns (2023) leveraged the five
Latin Universal Dependencies treebanks and several large Latin corpora. LatinCy’s named entity
recognizers were trained separately from the restof their respective pipelines, on a custom-madedataset based on the UD treebanks and the datasetof the Herodotos project. For this paper, we tested
the large (‘la_core_web_lg’) pipeline, as well as
the ‘la_core_web_trf’ pipeline, which is backed
by the multilingual BERT transformer architecture(Devlin et al., 2018).
The next two subsections describe the training
setup for our models; section 3.3 discusses theresults of the models we trained, as well as a comparison to LatinCy’s performance.

13,846
473
125
169
10
207
2

Table 2: Frequency of entity types in train (left) andvalidation set (right)

The in-domain texts were then split into three
sets: a training set (75%), a validation set (12.5%)
and an in-domain test set (12.5%). As the BERTmodel processes input on the sentence level, thesentence order was randomized. The sentences
containing rare multi-word locations and groups
were identified and split separately. Each of thosesplits was later appended to one of the three sets
to ensure that each contained entities of every type.
The frequencies of the entity types can be foundin Table 2 (train and validation split) and in the‘support’ column of Table 5 (test split).
To ensure representative testing, the data wasaugmented with manually annotated test sets fromthe LASLA corpus in the second part of this paper(see Section 5), both for in-domain prose and outdomain poetry.
3.2

3.2.1

CRF

For the CRF model, we made use of an implementation based on CRFsuite (Okazaki, 2007). We specified the optimization method as l-bfgs, set the maximum number of iterations to 100 and considered
all possible transitions, The following hand-craftedfeatures are incorporated: whether the word is adigit, capitalised or fully upper-cased; whether theword is the first or last word of a sentence; the last
three letters; the last two letters; a context windowof two left words and two right words. Following
Palladino et al. (2020), the whole word itself was
not included, because this might aid generalization
to other contexts.
Hyperparameter optimization was performed using a 50-fold random search, to optimize the tworegularisation coefficients c1 (search space exponentially distributed on scale 0.5) and c2 (search
space exponentially distributed on scale 0.05). The
best hyperparameters were 0.183 and 0.086 for c1
and c2 respectively.

Model training and evaluation

We created two models on the Herodotos dataset
and compared the results of these models to
those obtained using the recently released LatinCy
toolkit. The models we trained (finetuned) ourselves are:
• A conditional random field (CRF) model. Erdmann et al. (2016) use a CRF-based baseline
in a similar context. This model is fairly simple and will serve as a starting point for comparison.

3.2.2• LatinBERT (Bamman and Burns, 2020), a
specialized BERT model for Latin, trained using the Masked Language Modeling objective
on a corpus of 642.7M words, ranging fromclassical Latin (from 200 BCE onwards) to
Neolatin from Wikipedia. We made use of
the pre-trained model, and finetuned it on the
NER dataset.

LatinBERT

Prior to the finetuning of LatinBERT, we incorporated the original subword tokenizer into our own,
custom tokenizer to ensure the model was fullycompatible with the transformers library (Wolfet al., 2020). All words were lowercased during
tokenization. We proceeded to utilize the transformers trainer API both with and without hyperpa3

Hyperpar.

InitialOptimized

Learning rate
Weight decay
Number of train epochs

2.00e-50.01
37.89e-50.10
3els being able to recognize some in-domain (Seeagain Table 10).

Table 3: initial hyperparameters (LatinBERT1) vs. optimised hyperparameters (LatinBERT2)

Error analysis

4.1

Ambiguous annotations in the training
data

Although guidelines for named entities in classicalscholarship exist (Romanello and Najem-Meyer,2022), for classical Latin texts, they are still lacking (see Section 5). This is reflected in our dataset.
We can hypothesize that this impacts the overall
performance of the models. In particular, sometokens are annotated as different entities throughout the dataset. In some cases, this is due to theinherent ambiguity of the token, as in the following
examples:rameter optimization (results reported under LatinBERT2 and LatinBERT1 respectively). During theexperiments with hyperparameter optimization, we
specified the optimization method as random. The
metric for evaluation is the validation loss, and thegoal is to minimize it based on a ten-fold search.
Table 3 provides a comparison of the hyperparameters used. In both cases the per-device train batch
size is 16 and the warmup ratio is 0.1.
3.3

4

• Homonyms: Galli (genitive singular of ‘Gallus’, name of a man) as ‘PERS’ in Ars am.
3.334 or ‘GRP’ in BGall. 1.1 (‘the Gauls’);

Results

In Table 4 we report the micro-averaged f1 (oraccuracy) based on the token labeling. The microaveraged f1 computes the proportion of correctlyclassified observations out of all observations. In
Table 5, for every entity type (‘PERS’, ‘LOC’,‘GRP’), we report the f1 score (harmonic meanof precision and recall) on the entity level, where
the full entity is only considered correct if the annotations for all its comprising tokens match thegold standard exactly, and the macro f1, where theresults for each model are averaged across the various labels without taking class size into account.
In Appendix A, more detailed counts per label areprovided (Table 10).
The overall results in Table 4 show that there
is a drop in performance going from in- to out-ofdomain, signaling a difficulty to generalize fromprose to poetry. Both LatinBERTs outperform theother models in- and out-of-domain. However, it
is important to note that optimizing the hyperparameters causes a slight increase in macro-f1 on
the in-domain dataset, but a symmetrical, decreaseon the out-of-domain dataset. Looking at the entity
level metrics in Table 5, ‘PERS’ is the class that isthe easiest to predict for every model. For the models exclusively trained on the Herodotos data (theCRF and LatinBERTs), single word groups are a
relatively well-understood category in-domain, butcause problems out-of-domain. Unfortunately, nomulti-token ‘GRP’ were correctly detected, which
can be explained by their rarity. Multi-token ‘LOC’
are also rarely detected, with only the BERT mod-• Tokens that occur both as entity and nonentity in the dataset: e.g. Liber (a divinity,
but also ‘book’), forms of Sol (divinity ‘Sun’and the sun), and Gratia (‘grace’, but also
the divinity ‘Grace’) appear both as entities(personifications, usually capitalized) and nonentities (regular use);• Patronyms such as Atrides (‘descendant of
Atreus’): sometimes forms of these refer to
one specific person, sometimes to a group.
In other cases, the differences seem to stem frominconsistent annotation choices:• Multi-token entities that contain a toponym:
e.g. the entity Amphilocho Athenaeo (‘Amphilochus of Athens’) in NH is annotatedboth as ‘B-PERS B-GRP’ and as ‘B-PERS
I-PERS’; or a building with a name aedem
Larum (‘the temple of the Lares’, NH 2.5) isannotated as ‘O B-GRP’, while aedem Feroniae (‘the temple of Feronia’, NH 2.56) isannotated as ‘B-LOC I-LOC’;• Persons referred to with only a toponym: e.g.
Cressa (‘the Cretan woman’, Ars am. 1.327)
is annotated as ‘B-GRP’, while Cynthius (‘theCynthian’, Ars am. 2.239) is annotated as‘LOC’;• Unnamed entities annotated in some cases andnot in others: e.g. some of the occurrences of4

micro f1

CRF

LB1

LB2

LatinCy lg

LatinCy trf

supportCaesar/Pliny’s (IN)

BIO-labels
BI-labels

0.98
0.79

0.99
0.90

0.99
0.92

0.96
0.60

0.95
0.58

14,686
1,048

Ars am. (OUT)

BIO-labels
BI-labels

0.97
0.39

0.98
0.65

0.98
0.60

0.96
0.39

0.95
0.31

17,102570

Table 4: micro f1 on the Herodotos selected test-set; LB stands for LatinBERT

Caesar/Pliny’s (IN)Ars Am. (OUT)

CRF

LB1

LB2

LatinCy lg

LatinCy trf

support

PERS
LOC
GRP

0.80
0.66
0.74

0.91
0.85
0.89

0.92
0.87
0.91

0.64
0.61
0.02

0.64
0.54
0.06

474
218
247

macro f1
PERS
LOC
GRP

0.74
0.44
0.30
0.25

0.88
0.76
0.43
0.45

0.90
0.72
0.38
0.40

0.43
0.47
0.28
0.00

0.44
0.36
0.18
0.05

939
375
87
107

macro f1

0.33

0.54

0.50

0.25

0.20

569

Table 5: f1-score per entity type on the Herodotos selected test-set

prouincia (‘province’) and terra (‘region’) areannotated as ‘LOC’, and some of the occurrences of equestri and praetori as ‘GRP’.

LatinBERTs predict ‘I-LOC’ for Memphitidos(‘of Memphis’, Ars am. 3.393) (‘B-GRP’ is
the gold data) without assigning ‘B-LOC’ toa previous token.

In addition, entire parts of text are not annotated in
Ars am. and NH. The scarcity of data also appears
to be a problem: out of the 180 unique tokens thatwere not correctly identified by any model, 132 do
not occur in the training data.
4.2• Entities with foreign names are often predicted as non-entity: e.g. Adadu, Calymne,
Therapnaeus, and Andromeda.
• Complete sentences with clear entities predicted as non-entities in out-of-domain data
(entities in bold): e.g. Dextra Lebinthos eratsiluisque umbrosa Calymne | Cinctaque piscosis Astypalaea uadis (Ars am. 2.81-2) non-entity predictions for all entities by LatinBERT2; LatinBERT1 only for Astypalaea.

Qualitative analysis LatinBERTIn this section, we perform a qualitative error analysis of the performance of the two best-performing
models, LatinBERT1 and 2, on both the in-domain
and out-of-domain sets, in order to better understand the origin of the errors. First, LatinBERT1
and LatinBERT2 share common issues, that are
generally not encountered by at least one of theother two models:

LatinBERT1 and LatinBERT2 differ only in the
optimization of the hyperparameters, which seems
nonetheless to have a relevant impact on the performance. In a total of 223 cases, the predictionof LatinBERT2 differs from LatinBERT1. Table 8
in Appendix A shows that LatinBERT1 slightly
outperforms LatinBERT2 on the label ‘B-PERS’.
However, in several cases, the prediction of LatinBERT2 classifies the category correctly but withwrong segmentation, predicting ‘I-PERS’ insteadof ‘B-PERS’, whereas LatinBERT1 also classifies
incorrectly. In 46 of the cases where only LatinBERT1 is correct, LatinBERT2 predicts a non-

• Boundary detection proves particularly difficult with lists of names: Lysiae Demosthenen
Aeschinen Hyperiden multosque praeterea,
Gracchis et Catoni Pollionem Caesarem
Caelium [...] (Ep. 1.20.4). Both models
correctly identify 4 separate entities in thefirst part (Lysiae .. Hyperiden) but label ‘Pollionem Caesarem Caelium’ as one entity. In
addition, we find I-labels predicted for entities
not occurring after B-label: for instance, both5

entity. 42 of these tokens did not appear in the train
or validation set and the others were either annotated both as entities and ‘O’ or appeared only oncein the training data. Besides this, many differences
can be explained by the difficulties in ‘GRP’/‘LOC’
distinction identified in Section 4.1.

domain testing of Ovid’s Ars am., on the contrary,
primarily mentions mythical persons. Good performance on these texts would indicate the models’
generalisability.

5

5.2

Annotation of the LASLA corpus

In what follows, we discuss the performance of thesame NER models on the LASLA Latin corpus.3
As the LASLA corpus includes a diverse range ofclassical Latin texts, it represents an interesting test
set to investigate the generalisability of the models.
With this procedure, we also establish criteria forthe annotation of the most problematic classes. In
addition, we augment the test set by including bothprose and poetry works (resp. in-domain and outof-domain) which do not appear in the training data
and that belong to different genres with respect tothe training data. Overall, this process allows us toreach conclusions on the urgency of guidelines, ofdata generation, and the generalisability of existingmodels across different projects.
The portion of the LASLA corpus used for thisexperiment is composed of 1,738,435 tokens, belonging to 130 Latin literary texts by 21 authors
ranging from the 2nd century BCE to the 2nd
century CE. It is linked to the LiLa Knowledge
Base, an open-ended Knowledge Base of linguistic
Linked Data (Passarotti et al., 2020). The URIs
for lemmas and tokens provided by the linking are
published to ensure interoperability and reusabilityof the data.4
5.1

Annotation process and choices

The texts were annotated by two Latin experts using the BIO-format for the entities location, person,and group (see Section 3.1). The Herodotos project
annotation was taken as a reference, and the challenging points were discussed in order to address
the shortcomings identified in Section 4.1. Cohesion between the annotations of the two experts was
guaranteed by joint annotation of 4,463 tokens of
the Saturae (Juv. 1-3). The Inter-Annotator Agreement (IAA) was calculated using Cohen’s Kappascore (Cohen, 1960). The IAA is calculated bothincluding and excluding the label ‘O’. The resulting values are 0.87 (incl. ‘O’) and 0.74 (excl. ‘O’).
The confusion matrix (excl. ‘O’) is shown in Figure 1 of the Appendix A. The biggest disagreement
concerns the label ‘B-GRP’. The difficulties withthe annotation of ‘GRP’ can be divided into two
categories: annotation of adjectives derived fromtoponyms (Tuscus - ‘Tuscan’, Aegyptius - ‘Egyptian’, Graecus - ‘Greek’) and groups of individuals that do not fit the definition of political/ethnic
groups as defined by the Herodotos project. Examples of this last category are names of families(e.g Gracchos (2.24) - ‘The Gracchii’), names used
as a generic category (e.g. Proculas et Pollittas(2.68) - ‘women like Procula and Pollitta’), gods
(Asianorum ... deorum (3.218) - ‘Asian gods’),
and other groups such as Socraticos ... cinaedos(2.10 - ‘Socratic catamites’) and Manes (2.149 ‘Shades’). For adjectives derived from toponyms,
the annotators agreed to use ‘GRP’ to align with
the Herodotos project. For the other categories,‘GRP’ is used following the definition of the subcategory ‘PER.Group’ from the Automatic Content
Extraction Guidelines (Consortium, 2008) for any
Person entity referring to more than one person.
Finally, we chose not to annotate nicknames as‘PERS’ entities (e.g. Uenusina ... lucerna (1.51)- ‘The Venusinian light’, Horace, was only annotated as ‘B-LOC ... O’). Following the first roundof joint annotation, an agreement was reached on
problematic points to enhance the consistency of
the remaining annotation.

Texts annotated

To evaluate the performance of the models on the
LASLA corpus, we annotated texts from three different authors. As in-domain data, we chose toannotate Tacitus’ Historiae (Hist.) book 1 and thefirst of Cicero’s Orationes Philippicae (Phil.) andfor out-of-domain the first three of Juvenal’s Saturae (Juv.). Tacitus and Cicero were selected as‘in-domain’ data since they belong to non-fictional
prose. Moreover, the Phil. are a different genre(oratory) than the Herodotos training data and Tacitus (Historiography and Epistolography). Juvenal’s
poetry, with its mentions of historical people, wasselected to challenge the model, since the out-of3

https://www.lasla.uliege.be
https://github.com/NER-AncientLanguages/
Ner-Latin-RANLP
4

6

micro f1

CRF

LB1

LB2

LatinCy lg

Herodotos

supportTac. and Cic. (IN)

BIO-labels
BI-labels

0.96
0.61

0.97
0.78

0.98
0.79

0.96
0.66

0.97
0.72

15,737
1,320

Juv. (OUT)

BIO-labels
BI-labels

0.96
0.45

0.96
0.48

0.96
0.50

0.96
0.51

0.96
0.48

4,399
284

Table 6: micro f1 on the LASLA corpus; LB stands for LatinBERTTac. and Cic. (IN)

Juvenal (OUT)

CRF

LB1

LB2

LatinCy lg

Herodotos

support

PERS
LOC
GRP

0.65
0.31
0.43

0.83
0.51
0.61

0.85
0.55
0.64

0.66
0.53
0.02

0.74
0.49
0.60

711
222
154

macro f1
PERS
LOC
GRP

0.46
0.48
0.32
0.47

0.65
0.53
0.46
0.40

0.68
0.64
0.36
0.52

0.40
0.64
0.44
0.00

0.61
0.59
0.27
0.23

1,087
143
83
36

macro f1

0.43

0.46

0.51

0.36

0.37

262

Table 7: f1-score per entity type & macro f1 on the LASLA corpus

5.3

Results of running the model

to ambiguous tokens that occur both as entity andnon-entity, albeit slightly more present in out-ofdomain, e.g. Pax atque Fides, Uictoria, Uirtus(‘The Goddesses Peace, Faith, Victory and Virtue’,Juv. 1.115). Also for the LASLA test-set, tokens
annotated differently across the Herodotos training data result in multiple errors. For instance,
non-capitalized forms of prouincia and urbs areannotated as ‘LOC’ in the training data only when
they refer to a precise location. Likewise, princeps
and imperator are annotated as ‘PERS’ only where
they refer to specific emperors. Lastly, words like
domus and aedes are sometimes annotated when
they indicate a specific location: for example, aede
Apollinis - ‘the temple of Apollo’ and Tiberianam
domum - ‘the palace of Tiberius’. Even thoughthe Herodotos training data are not fully consistentin these annotations, the LASLA annotation did
strictly follow these guidelines, which highlighted
the inconsistent behavior of models with respect tothese points.

Table 6 shows that when labelling single tokens
LatinBERT2 outperforms the other models on
in-domain data, whereas the models score very
close on out-of-domain data, with LatinCY scoringslightly higher than LatinBERT2.5 Table 7 showsthat LatinBERT2 predicts entire entities better thanthe other models, except for the category ‘LOC’ on
out-of-domain data, where LatinBERT1 performs
better. These results confirm LatinBERT2’s general good performance, but also its again somewhat
unexpected behavior on poetry.
5.4
5.4.1

Error Analysis
Challenging aspects of NER prediction

Similarly as to the Herodotos data, many errors
can again be related to the inherent ambiguity of
Latin and/or the choices made in annotation (cf.
Section 4.1). Both on the in- and out-of-domain
LASLA data, errors were made that are related
5
The major increase in performance of LatinCy on the
LASLA data can be explained by two reasons: first, 38% of
total errors of LatinCy concern the GRP-entities, of which
there are relatively less in the LASLA test data (23.5% of thetotal entities are ‘GRP’s in Herodotos, whereas in the LASLA
14.1%); second, many other errors are caused by the tendencyof LatinCy to predict entities for any and all capitalized words.
In the Herodotos data, all sentences start with a capital, creating many errors for LatinCy; in the LASLA, capitalization isabsent, hence such errors do not occur.

5.4.2

Qualitative analysis LatinBERT on the
LASLA dataset

In Section 4.2 we observed that LatinBERT1 andLatinBERT2 share common issues, that are generally not encountered by at least one of the othertwo models. On the LASLA corpus, similar andadditional observations can be made. Boundary
7

detection issues occur in comparable instances on
the LASLA corpus, such as predicting separate entities in lists and predicting I-labels for entities notoccurring after B-label. However, an additionalboundary complication occurs in poetry in difficult
nested cases such as the entity Cecropiam ... Cotyton (Juv. 1.7-9) separated by the entity Baptae
occurring in between (this created the annotation‘B-PERS B-GRP I-PERS’). Both LatinBERTs predict a non-entity for Baptae and Cotyton. As in
the Herodotos test set, foreign names again proved
particularly difficult, in the LASLA out-of-domain
especially those with a Greek accusative ending in‘n’ (e.g. Euphraten (Juv. 1.104). Of the 10 tokens
with this ending only Deucalion (1.81) is predicted
correctly as an entity by LatinBERT1.6 Lastly, inthe out-of-domain data we again find complete
sentences that contain multiple entities for which
non-entities are predicted.
A close analysis of the performance on tokens
where the manual annotation differed shows some
additional challenging categories. Of the 69 tokens
where the manual annotation differed, LatinBERT1
got 39 wrong (accounting for 20.5% of its total
errors), and LatinBERT2 got 41 wrong (accountingfor 22.5% of its total errors). For instance, both
LatinBERTs predict ‘O’ for most groups of individuals that did not fit the political/ethnical ‘GRP’
category, except for some family names (e.g. Catuli, Fabii). For Literary works identified by apersonal name, another category where the annotators disagreed but were eventually not annotated,
LatinBERT2 predicts an entity but LatinBERT1 ‘O’(e.g. Theseide (1.2); Heracleas | aut Diomedeas(1.52-3)). Lastly, for the category of persons referred to with only a toponym, also identified as
an issue in Section 4.1, we annotated ‘LOC’ but
the LatinBERTs predicted ‘GRP’: e.g. non Maurus erat neque Sarmata nec Thrax (‘it was not a
Moroccan nor a Sarmatian nor a Thracian’, 3.79).
The comparison between the two LatinBERTs
shows that on the in-domain LASLA data, LatinBERT2 outperforms LatinBERT1, especially on
I-labels (cf. Appendix A, Table 9). When considering I-label errors, both LatinBERTs classify thecategory correctly for more than half of these errors(40 out of 78 for LatinBERT1; 32 out of 62 for LatinBERT2), but wrongly assign the ‘B-’ label: theproblem thus lies again with the boundary detec-tion. On the out-of-domain data, LatinBERT2 outperforms LatinBERT1 in the ‘B-PERS’ category.
As on the Herodotos project test data, in the majority cases where only LatinBERT1 is correct, LatinBERT2 predicts a non-entity: for the in-domain set
22 out of 27 total cases concern words absent fromthe train/validation set, for out-of-domain 16 out of
18.
This analysis confirmed that the categories identified in Section 4.2 are difficult for NER. It alsoemphasised the differences between in- and out-ofdomain data: models only trained on prose perform
worse on poetry due to stylistic and thematic differences.

6

Conclusions and future work

The process of training two new models on existing data, comparing their results on previously andnewly annotated data, and comparing their performance to existing models allows us to draw several conclusions. First, the good performance of
LatinBERT1 and 2 demonstrates the interest of applying transformer-based models for the NERtask on Latin. Especially for the category ‘PERS’
the two models yield satisfactory results. However,
the analysis of the annotations and the errors has
shown that the development of guidelines is crucial to ensure the consistent annotation of datasetsthat can be reused as training- and test-sets acrossdifferent projects and for different models. In addition, the significantly worse performance of themodels on poetry indicates the need for training
data for this specific type of texts. Future work
should also consider improving the preprocessing
and normalization of training data (e.g. harmonizing the use of the ‘v/u’ ‘i/j’ pairs), and testing theuse of multilingual BERT models that include Latin(mBERT, XLM-Roberta) (Sprugnoli et al., 2022;
Nehrdich, 2022). Likewise, additional linguistic
information available in the LASLA corpus (e.g.
lemmatization and PoS tagging) might improve theresults of the NER. Finally, after we establish a system for Named Entity Disambiguation employing
information from existing extensive resources, wewill explore the potential of mutual reinforcement,
i.e. we will consider whether results from one system can improve the other and vice-versa as argued
by Kolitsas et al. (2018).

6

This is particularly surprising since in the Herodotos testset LatinBERT1 correctly predicted 29 out of 40 of such forms,
and LatinBERT2 22.

8

References

Alexander Erdmann, David Joseph Wrisley, Benjamin
Allen, Christopher Brown, Sophie Cohen-Bodénès,Micha Elsner, Yukun Feng, Brian Joseph, Béatrice Joyeux-Prunel, and Marie-Catherine de Marneffe. 2023.
Herodotos-project-latin-ner-taggerannotation.

David Bamman and Patrick J. Burns. 2020. Latin bert:
A contextual language model for classical philology.
(arXiv:2009.10053). ArXiv:2009.10053 [cs].
Patrick J. Burns. 2023. Latincy: Synthetic trained
pipelines for latin nlp.
(arXiv:2305.04365).
ArXiv:2305.04365 [cs].

Matthew Honnibal and Ines Montani. 2017. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental
parsing. To appear.

William L. Carey. s.d. The latin library. Accessed: July
1st, 2023.

Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd
Cook, Clément Besnier, and William J. B. Mattingly.
2021. The classical language toolkit: An nlp framework for pre-modern languages. In Proceedingsof the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing:
System Demonstrations, page 20–29, Online. Association for Computational Linguistics.

Pierre Chastang, Sergio Torres Aguilar, and Xavier Tannier. 2021. A Named Entity Recognition Model forMedieval Latin Charters. Digital Humanities Quarterly, 015(4).
Jacob Cohen. 1960. A Coefficient of Agreement for
Nominal Scales. Educational and Psychological
Measurement, 20(1):37–46.
Linguistic Data Consortium. 2008. ACE (AutomaticContent Extraction) English: Annotation Guidelines
for Entities.

Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas
Hofmann. 2018. End-to-End Neural Entity Linking.
In Computational Natural Language Learning, pages
519–529. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. BERT: pre-training ofdeep bidirectional transformers for language understanding. CoRR, abs/1810.04805.

Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.
Neural architectures for named entity recognition.
In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
pages 260–270, San Diego, California. Association
for Computational Linguistics.

Maud Ehrmann, Ahmed Hamdi, Elvys Linhares Pontes,
Matteo Romanello, and Antoine Doucet. 2021.
Named entity recognition and classification on historical documents: A survey. (arXiv:2109.11406).
ArXiv:2109.11406 [cs].
Maud Ehrmann, Matteo Romanello, Sven NajemMeyer, Antoine Doucet, and Simon Clematide. 2022.
Overview of hipe-2022: Named entity recognitionand linking in multilingual historical documents. In
Experimental IR Meets Multilinguality, Multimodality, and Interaction, Lecture Notes in Computer Science, page 423–446, Cham. Springer International
Publishing.

Wouter Mercelis and Alek Keersmaekers. 2022. An
ELECTRA model for Latin token tagging tasks. In
Proceedings of the Second Workshop on Language
Technologies for Historical and Ancient Languages,
pages 189–192, Marseille, France. European Language Resources Association.
Sebastian Nehrdich. 2022. SansTib, a Sanskrit - Tibetan parallel corpus and bilingual sentence embedding model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages
6728–6734, Marseille, France. European Language
Resources Association.

Alexander Erdmann, Christopher Brown, Brian Joseph,Mark Janse, Petra Ajaka, Micha Elsner, and MarieCatherine de Marneffe. 2016. Challenges and solutions for Latin named entity recognition. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH),
pages 85–93, Osaka, Japan. The COLING 2016 Organizing Committee.

Naoaki Okazaki. 2007. Crfsuite: a fast implementationof conditional random fields (crfs).
Chiara Palladino, Farimah Karimi, and Brigitte Mathiak.
2020. Ner on ancient greek with minimal annotation.
https://dh2020.adho.org/.

Alexander Erdmann, David Joseph Wrisley, Benjamin
Allen, Christopher Brown, Sophie Cohen-Bodénès,Micha Elsner, Yukun Feng, Brian Joseph, Béatrice
Joyeux-Prunel, and Marie-Catherine de Marneffe.
2019. Practical, efficient, and customizable active
learning for named entity recognition in the digital
humanities. In Proceedings of the 2019 Conferenceof the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page2223–2234, Minneapolis, Minnesota. Association for
Computational Linguistics.

Marco Passarotti, Francesco Mambrini, Greta Franzini,
Flavio Massimiliano Cecchini, Eleonora Litta, Giovanni Moretti, Paolo Ruffolo, and Rachele Sprugnoli.
2020. Interlinking through lemmas. the lexical collection of the lila knowledge base of linguistic resources
for latin. Studi e Saggi Linguistici, 58(1):177–212.
Matteo Romanello and Sven Najem-Meyer. 2022.
Guidelines for the annotation of named entities inthe domain of classics.

9

David A. Smith, Jeffrey A. Rydberg-Cox, and G. Crane.
2000. The perseus project: a digital library for thehumanities. Literary and Linguistic Computing.

Historical and Ancient Languages, pages 101–107,
Marseille, France. European Language Resources
Association.

Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer,
John Bodel, Jonathan Prag, Ion Androutsopoulos,
and Nando de Freitas. 2023. Machine learning for
ancient languages: A survey. Computational Linguistics, page 1–44.
Rachele Sprugnoli, Marco Passarotti, Flavio Massimiliano Cecchini, Margherita Fantoli, and Giovanni
Moretti. 2022. Overview of the EvaLatin 2022 evaluation campaign. In Proceedings of the Second
Workshop on Language Technologies for Historicaland Ancient Languages, pages 183–188, Marseille,
France. European Language Resources Association.
Sergio Torres Aguilar. 2022. Multilingual named entity
recognition for medieval charters using stacked embeddings and bert-based models. In Proceedings of
the Second Workshop on Language Technologies for
Historical and Ancient Languages, page 119–128,
Marseille, France. European Language Resources
Association.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations, pages 38–45, Online. Association
for Computational Linguistics.
Krzysztof Wróbel and Krzysztof Nowak. 2022.
Transformer-based part-of-speech tagging andlemmatization for Latin. In Proceedings of theSecond Workshop on Language Technologies for
Historical and Ancient Languages, pages 193–197,
Marseille, France. European Language Resources
Association.
Tariq Yousef, Chiara Palladino, Gerhard Heyer, andStefan Jänicke. 2023. Named entity annotation projection applied to classical languages. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social
Sciences, Humanities and Literature, page 175–182,
Dubrovnik, Croatia. Association for Computational
Linguistics.
Tariq Yousef, Chiara Palladino, and Stefan Jänicke.
2022a. Transformer-Based Named Entity Recognition for Ancient Greek.
Tariq Yousef, Chiara Palladino, David J. Wright, andMonica Berti. 2022b. Automatic translation alignment for Ancient Greek and Latin. In Proceedings of
the Second Workshop on Language Technologies for10

A

Appendix

Figure 1: IAA on Juv. Saturae 1-3, label ‘O’ excluded

Gold label

1 & 2 wrong

1 correct

2 correctO
B-PERS
I-PERS
B-LOC
I-LOC
B-GRP
I-GRP

0
13
2
14
0
18
1

11
47
1
12
0
16
0

25
24
1
14
2
12
0

Total

58

87

78

Table 8: Comparison of differences in prediction between LatinBERT1 (1) and LatinBERT2 (2) on the
Herodotos data.

Gold label

1 & 2 wrong
IN
OUT

1 correct
IN OUT

2 correct
IN OUTO
B-PERS
I-PERS
B-LOC
I-LOC
B-GRP
I-GRP

0
1
2
2
3
14
1

0
6
3
7
3
3
0

7
14
4
6
0
6
0

5
8
0
15
0
4
0

7
20
14
11
5
10
0

7
22
0
9
0
5

Total

23

22

37

32

67

43

Table 9: Comparison of differences in prediction between LatinBERT1 (1) and LatinBERT2 (2) on in andout-of-domain LASLA data.

11

Caesar/Pliny’s (IN)Ars Am. (OUT)

CRF

LB1

LB2

LatinCy lg

LatinCy trf

support

B-PERS
I-PERS

0.83
0.86

0.93
0.89

0.94
0.91

0.75
0.43

0.73
0.52

474
98

B-LOC
I-LOC

0.70
0.00

0.87
0.33

0.90
0.50

0.64
0.00

0.56
0.00

218
8

B-GRP
I-GRP
B-PERS
I-PERS

0.77
0.00
0.48
0.06

0.90
0.00
0.76
0.00

0.92
0.00
0.72
0.00

0.02
0.00
0.47
0.00

0.06
0.00
0.36
0.00

247
3
375
1

B-LOC

0.30

0.43

0.38

0.28

0.18

87

B-GRP

0.25

0.45

0.40

0.00

0.05

107

Table 10: f1-score per entity type on the Herodotos dataset; LB stands for LatinBERTTac. and Cic. (IN)Juv. (OUT)

B-PERS
I-PERS
B-LOC
I-LOC
B-GRP
I-GRP
B-PERS
I-PERS
B-LOC
I-LOC
B-GRP
I-GRP

CRF

LB1

LB2

LatinCy lg

Herodotos

support

0.71
0.78
0.33
0.00
0.43
0.00
0.55
0.23
0.35
0.00
0.47
0.00

0.88
0.81
0.58
0.00
0.61
0.00
0.55
0.00
0.50
0.00
0.40
0.00

0.89
0.85
0.60
0.18
0.62
0.00
0.65
0.00
0.39
0.00
0.52
0.00

0.84
0.24
0.57
0.00
0.03
0.00
0.66
0.00
0.44
0.00
0.00
0.00

0.81
0.79
0.52
0.13
0.60
0.00
0.64
0.19
0.27
0.00
0.23
0.00

711
188
222
42
154
3
143
7
83
14
36
1

Table 11: f1-score per entity type on the LASLA corpus

12

Sentence Embedding Models for Ancient Greek Using Multilingual
Knowledge Distillation
Kevin Krahn and Derrick Tate and Andrew C. Lamicela
Sattler College, Boston, MA{kevin.krahn24, dtate, alamicela}@sattler.edu

Abstractfrom most other ancient languages, which makes
sentence embedding models both more feasible anduseful.
Several previous works have trained language
models for Ancient Greek. Johnson et al. (2021)
introduced the Classical Language Toolkit (CLTK)
which includes several tools for Ancient Greek processing, including static word embeddings. Singhet al. (2021) fine-tuned a Modern Greek BERT
model (Koutsikakis et al., 2020) on Ancient Greek
text for PoS tagging, morphological tagging, andlemmatization tasks. Yamshchikov et al. (2022)
trained a BERT model for authorship classification of Pseudo-Plutarch texts. Cowen-Breen et al.
(2023) trained another BERT model for the purpose of identifying errors in scribal transmission.
Riemenschneider and Frank (2023) produced themost comprehensive work on Classical language
models to date, training multiple models on a large
multilingual corpus of Ancient Greek, Latin, andEnglish texts and comprehensively evaluating andcomparing their new models to previous models ona variety of tasks. None of these works, however,produce sentence embedding models for Ancient
Greek.
Although there are many digitized Ancient
Greek texts available, there is a lack of suitabletraining data for training sentence embedding models from scratch. The best approaches for highresource languages involve large human-annotateddatasets, such as the natural language inference(NLI) datasets used by Sentence-BERT (Reimers
and Gurevych, 2019). Needless to say, suchdatasets are not available for Ancient Greek.
Following Reimers and Gurevych (2020), we
use multilingual knowledge distillation to train sentence embedding models with an aligned vector
space for Ancient Greek and English. Given ateacher model M for a language s, and a datasetof translated sentences ((s1 , t1 )..(sn , tn )) where si
and ti are parallel sentences, we train a new stu-Contextual language models have been trainedon Classical languages, including Ancient
Greek and Latin, for tasks such as lemmatization, morphological tagging, part of speechtagging, authorship attribution, and detectionof scribal errors. However, high-quality sentence embedding models for these historicallanguages are significantly more difficult toachieve due to the lack of training data. In
this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek
text. The state-of-the-art sentence embedding
approaches for high-resource languages use
massive datasets, but our distillation approachallows our Ancient Greek models to inherit
the properties of these models while using a
relatively small amount of translated sentence
data. We build a parallel sentence dataset using a sentence-embedding alignment method toalign Ancient Greek documents with English
translations, and use this dataset to train ourmodels. We evaluate our models on translation search, semantic similarity, and semantic
retrieval tasks and investigate translation bias.
We make our training and evaluation datasets
freely available at this url.

1

Introduction

Sentence embedding models, which map sentencesor other sequences of text to a dense vector space,such that semantically similar sentences are close
together in the vector space, have many applications in NLP. Current state-of-the-art sentence embedding models, however, are trained on modern,high-resource languages such as English and use
massive datasets consisting of billions of sentencepairs (Ni et al., 2022). A different approach isneeded for historical languages, which have muchless data available.
In this work, we train several sentence embedding models for Ancient Greek. Many more Ancient Greek texts have survived compared to texts
13

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 13–22,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_002

MSE Loss

uses sentence embeddings for state-of-the-art alignment quality.
We create new evaluation datasets for Ancient
Greek translation search, semantic textual similarity (STS), and semantic retrieval (SR) and weevaluate our models on these datasets.
In summary, our contributions are as follows:

MSE Loss
Teacher EN vector

Student EN vector

Student GRC vector

0.1 -0.5 0.3 …

0.6 -0.2 0.2 …

0.2 -0.6 0.5 …

Student
Model

TeacherModel

1. We use a multilingual knowledge distillation
approach to train several Ancient Greek sentence embedding models.

Hello WorldΧαῖρε Κόσμε

Figure 1: Multilingual knowledge distillation for English to Ancient Greek sentence pairs.

2. We use translation alignment to produce adataset of Ancient Greek sentences and theirEnglish translations.

dent model M̂ to mimic the sentence embeddingsof the teacher M using mean squared loss, suchthat M̂ (si ) ≈ M (ti ) and M̂ (ti ) ≈ M (si ). In ourcase, the teacher model is English and the student
model learns both Greek1 and English embeddings.
This approach has numerous advantages: 1) it
requires a relatively small amount of training data,2) the student model inherits the vector space properties of a state-of-the-art English sentence embedding model, 3) the student model is multilingual,and 4) the vector spaces are aligned across languages.
The cross-lingual nature of this approach is especially useful for Ancient Greek semantic retrieval,since it is much easier to formulate search queries
in English than in Ancient Greek. Although it ispossible to operate on the English translations of
Greek texts, translations are not readily available
for all Greek texts, and the available translations areusually not aligned at the sentence level, making
it difficult to quickly find the corresponding Greek
text. Furthermore, English translations can sufferfrom various kinds of translator bias, whereas alanguage model that operates directly on the Greek
text can offer an “average” of multiple translators’interpretations of the text (See Section 4.4).
We produce a training dataset of parallel sentences using a two-step translation alignment process: an initial, smaller dataset was produced using
a sentence-length heuristic and dictionary-basedalignment technique (Halácsy et al., 2007), andthis initial dataset was used to train an intermediate multilingual sentence embedding model, which
was used to align a larger dataset using the approach introduced by Liu and Zhu (2023), which

3. We develop evaluation datasets for translation
search, semantic retrieval, and semantic textual similarity, and we evaluate our sentence
embedding models on these tasks.

2 Training
2.1

Base Models

To train a sentence embedding model, we first needa base language model trained on Ancient Greek
text. The existing Ancient Greek language models
were unsuitable for our purposes; most of them
are monolingual, but we are training a multilingual model. The models trained by Riemenschneider and Frank (2023) would be the best candidates because they include English, but one of theirgoals was to avoid contamination from modern languages, such as modern concepts and technology
like cellphones which were unknown in antiquity.
However, for us this is not a concern, since one ofour goals is to train a model to facilitate semantic
search with modern language and terminology.
Instead, we fine-tune multilingual BERT-base(mBERT) (Devlin et al., 2019) and XLMRoBERTa-base (XLM-R) (Conneau et al., 2020)
for our base models. Pires et al. (2019) showsthat low-resource languages can benefit from multilingual pre-training. We use masked language
modeling (MLM) to fine-tune mBERT, (denoted as
GRC mBERT) and XLM-R (denoted as GRC XLM-R)
with Ancient Greek text, and we use these as basemodels. See Appendix A for training details.
Both mBERT and XLM-R were trained on Modern Greek, among many other languages, but not
on Ancient Greek, and hence one disadvantage ofthese models is that their tokenizers are not optimized for Ancient Greek morphology, which could

1
When we refer to “Greek” in an unqualified way in thispaper we are referring to Ancient Greek.

14ModelSymbols/token

Words/token

mBERT
XLM-R

2.29
2.66

0.37
0.43

2. sentence-t5-large,3 a T5 model tuned
for sentence similarity tasks, trained on 2B
pairs (Ni et al., 2022) (Denoted as st5).
Both above models have a final normalization
layer which we remove prior to training to allow
student model to learn the original vector space
properties of the teacher model.

Table 1: The XLM-R tokenizer produces longer tokens
and a higher number of words per token on Ancient
Greek text compared to the mBERT tokenizer.

negatively impact performance (Park et al., 2021;
Hofmann et al., 2021).
We use a similar approach to Yamshchikov et al.
(2022) to compare the mBERT and XLM-R tokenizers. We take a random sample of 20k Ancient Greek sentences from the pre-training corpus and compute the average token length and average words per token for a rough estimation of
tokenization quality (See Table 1). The XLM-R
tokenizer scores higher on both metrics comparedto the mBERT tokenizer. However, a higher scorefor either metric does not guarantee superior performance in downstream tasks, since it does notmeasure how well the sub-word tokens capture Ancient Greek morphology.
2.2

Figure 2: Translation search accuracy over training steps
with grc XLM-R student model.

We compare GRC mBERT and GRC XLM-R as thestudent model M̂ . We add a mean pooling layer
and pair both student models with both teacher
models (4 configurations) and train all the student
parameters. With mpnet as the teacher, we train
for 15 epochs, but with st5 the student model
took twice as long to converge (See Figure 2), so
we train for 30 epochs. We use a batch size of
128, a max sequence length of 128 tokens, 2000
warmup steps, and a learning rate of 2e-5. Every
500 training steps we measure STS performance aswell as MSE loss and translation search accuracy
on 5k hold-out pairs, keeping the model with best
average performance across these tasks. Regardlessof teacher model, GRC XLM-R took many moretraining steps to converge than GRC mBERT andwas prone to catestrophic forgetting, which wasalleviated by increasing the number of warmup
steps.
We also experiment with training on parallel
Modern Greek data from Wikipedia for 3 epochs
and then on Ancient Greek data for 15 epochs if
mpnet is the teacher and 6 and 30 epochs if st5
is the teacher. Although Modern Greek differs inmany significant ways from Ancient Greek, training on this data gives the model additional exposure

Knowledge Distillation

To train multilingual sentence embedding models
on English and Ancient Greek with an aligned
vector space we use multilingual knowledge distillation (Reimers and Gurevych, 2020). This process requires a teacher model M for a source
language s, and a dataset of translated sentences((s1 , t1 )..(sn , tn )) where si and ti are parallel sentences. We train a student model M̂ to mimic thesentence embeddings of the teacher M such thatM̂ (si ) ≈ M (ti ) and M̂ (ti ) ≈ M (si ). The following mean squared loss function is minimized foreach mini-batch β:i
1 Xh((M (sj ) − M̂ (sj ))2 + (M (sj ) − M̂ (tj ))2|β| j∈βThus, the student M̂ learns to map each target
and source sentence to the same location in vector
space.
For the teacher M we compare two models:
1. all-mpnet-base-v2,2 a model tuned forsemantic search, trained on a large and diverse
training set of 1B+ pairs (Denoted as mpnet).
2https://huggingface.co/sentence-transformers/all-mpnet-base-v2

3
https://huggingface.co/sentence-transformers/sentence-t5-large

15

to aspects of Greek that have remained unchanged
since antiquity, such as historical proper nouns. Allevaluations are reported with and without training
on this additional data.
2.3

This dataset consist of Greek texts spanning a
thousand years, covering different dialects and time
periods of the language. We do not filter out any
texts based on their dialect or time period.
In addition to the Greek text, we also collect
all the English translations in the Open Greek and
Latin project to finetune our models with an additional 10 million words of historical English text.

Contrastive Learning

As a baseline against which to compare the models
trained via the distillation method, we also train
sentence embedding models using Simple Contrastive Learning of Sentence Embeddings (SimCSE), the contrastive learning method introduced
by Gao et al. (2021). Contrastive learning pulls
semantically-close neighbors together and pushes
apart non-neighbors, and has been shown to beeffective for training multilingual sentence embeddings (Gao et al., 2021; Tan et al., 2023). In addition to using dropout as noise, we use each Greek
sentence and its English translation as positive pairs
and other pairs in the same batch as negatives.
We use the CLS token representation and train
for a maximum of 10 epochs with a batch size of82, a max sequence length of 128 tokens, 2000
warmup steps, and a learning rate of 2e-5. Every
500 training steps we measure performance on the
STS evaluation and translation search accuracy on
the 5k hold-out pairs, keeping the highest performing model. As above, we also experiment withtraining on Modern Greek data for 3 epochs, andthen Ancient Greek data for 10 epochs.

3

Training Data

3.1

Pre-training

3.2

Preprocessing

Following Yamshchikov et al. (2022) and Singhet al. (2021), we lowercase all the Greek text andstrip diacritics, but keep punctuation. Althoughdiacritics contain important information for disambiguating between words that only differ by breathing marks or accent marks, the correct word canusually be inferred from context. The contextual
nature of BERT models allows them to learn to use
context to disambiguate.
3.3

Parallel Data

Human Aligned A portion of our parallel sentence dataset is taken from human aligned sources:
1. Verses of the Greek New Testament with English translations (15k pairs),
2. Verses of the Greek Septuagint with English
translations (29k pairs),
3. Verses of the Greek works of Flavius Josephus
with English translations (15k pairs),
4. Other minor sources: OPUS (Tiedemannand Nygaard, 2004), Greek Learner Texts7 ,manually aligned passages from Perseus andFirst1KGreek (total 23k pairs).

Our pre-training dataset consists of the Ancient
Greek text from the Perseus Digital Library4 andFirst1KGreek,5 which are part of the Open Greek
and Latin project.6 Different documents containing
the same Greek work were removed. These sources
contain approximately 32 million words of Ancient
Greek text. Although Riemenschneider and Frank(2023) produced a much larger corpus of Greek
text (100+ million words) using additional sources,
at the time of writing their data is not publicly
available. Our smaller dataset is sufficient for our
purposes, as Reimers and Gurevych (2020) showthat even languages with little pre-training in a
multilingual student model can be effective targets
for knowledge distillation.

Translation Alignment The bulk of the parallel
data is produced using translation alignment. We
take all the texts from our pre-training corpus thathave English translations and split them into sentences or sub-sentence segments (see Appendix B).
We then use a two-step process to align Greek sentences with their English translations. First, we use
Hunalign (Halácsy et al., 2007), a sentence-length
heuristic and dictionary-based alignment techniqueon the translated texts. This produced an initial
dataset of approximately 150k parallel sentences(including the human-aligned sources listed above).
Using this initial dataset, we trained a sentence
embedding model with an aligned vector space for4
https://github.com/PerseusDL/
canonical-greekLit
5https://github.com/OpenGreekAndLatin/
First1KGreek
6
https://opengreekandlatin.org

7

16

https://greek-learner-texts.org

ModelEnglish and Ancient Greek using SimCSE (See
Section 2.3). Next, we use this model to align all
the texts again, using a better alignment method
introduced by Liu and Zhu (2023), dubbed Bertalign, which uses multilingual sentence embeddings
to achieve state-of-the-art alignment quality. If the
Greek and English documents are already alignedby sections, we align the sentences in each section
individually. This increases alignment accuracy
and makes it possible to keep the parts of the document that have good alignments and to discard therest. Otherwise, if no section alignments exist, werun the aligner on the entire text.
We do not filter out multiple translations of thesame Greek texts, since different translations canhave different nuances and word choices, with thehope that the resulting sentence embeddings will
be more robust to translation differences.
Finally, we remove all duplicate sentence pairs
from the dataset and all pairs with very short sentences (<5 characters). We also ensure that nosentence pairs from the STS dataset (See Section
4.2) are included in the training data. This results
in approximately 380k sentence pairs after holdingout 5k pairs for evaluation purposes.

GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

Evaluations

4.1

Translation Similarity Search

95.92
96.09
95.86
96.64

Teacher: sentence-t5-large
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

87.78
90.80
87.02
91.60

Teacher: all-mpnet-base-v2
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

87.77
89.15
86.48
90.12

Table 2: Translation similarity search accuracy. Best
result is bolded.

ing since they specifically trained to maximize thecosine similarity between translation pairs and minimize similarity between non-pairs. There is no
significant difference in the performance between
the two base models. All the models performed
better when first trained on Modern Greek before
Ancient Greek.
4.2

Modern Greek The Modern Greek (EL) sentence pairs from Wikipedia are taken from the
OPUS project (Tiedemann and Nygaard, 2004).
We remove all duplicate pairs and pairs with very
short sentences (<10 characters), resulting in approximately 800k sentence pairs. This dataset contains a rich and diverse set of topics, including
historical topics which will hopefully transfer tothe Ancient Greek models. We compare all themodels with and without training on this data.

4

Accuracy

SimCSE

Semantic Textual Similarity

Sentence PairΣτωικοὶ ἀποφαίνονται σφαιροειδῆ τὸν κόσμον.
Stoics declare the world to be spherical.
Στωικός νομίζει ὅτι ἡ γῆ σφαίρα ἐστιν.
A Stoic thinks that the earth is a sphere.
ἐπὶ δὲ τοῦ ὄρους τῇ ἄκρᾳ Διός ἐστιν ναός.
On the top of the mountain is a temple of Zeus.
ὁ Ζεὺς οἰκεῖ ἐπὶ τὰ ὄρη ἐν ᾿Ολύμπῳ.
Zeus dwells on the mountains in Olympus.
Τὰ παιδία παίζουσιν ἐν τῇ ἀμμουδιᾷ.
The children are playing in the sand.
Τὰ παιδία ἀναπαύονται ἐν τῷ κήπῳ.
The children rest in the garden.
Σωκράτης εἶδεν ἓξ βόας.
Socrates saw six cows.
῾Ρώμουλος εἶδεν ἓξ οἰωνοὺς ὄρνιθας.
Romulus saw six birds of omen.

The first measure of the quality of the sentence embeddings is each model’s accuracy at choosing thecorrect English translation for each Ancient Greek
sentence from the 5k hold-out pairs. The scoreis computed as the percentage of sentence pairs
for which the embedding of source sentence si has
the closest cosine similarity to the embedding oftranslated sentence ti out of all the target sentences.
The accuracy is computed in both directions andaveraged. The results are reported in Table 2.
The SimCSE models perform on this task better
than the distillation models, which is not surpris-

Score
0.9

0.8

0.5

0.1

Table 3: Example pairs from STS evaluation dataset.
Scores are examples and not actual scores.

We compiled a dataset of Ancient Greek sentence pairs with gold scores to measure semantic
textual similarity in the range [0,1], with 0 representing completely unrelated meaning, and 1 representing full semantic equivalence. Each sentence
was given a corresponding English translation to17ModelGRC↔GRCEN↔EN

GRC↔EN

Average

75.68
74.85
77.83
78.27

77.58
78.30
78.82
79.11

76.30
76.40
77.21
77.76

76.52
76.52
77.95
78.38

87.54
89.33
85.37
88.37

84.02
86.37
82.56
85.45

84.58
86.84
83.43
86.24

87.60
88.77
87.07
88.24

84.68
86.28
84.53
85.92

84.86
86.63
85.13
86.45

SimCSEGRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

Teacher: sentence-t5-large
GRC mBERT (GRC)

GRC mBERT (EL,GRC)

GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

82.17
84.84
82.37
84.88

Teacher: all-mpnet-base-v2
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

82.30
84.84
83.80
85.18

Table 4: Spearman rank correlation ρ between the cosine similarity of sentence embeddings and gold labels for
STS dataset. Scores are reported as ρ × 100. Best results are bolded. There are twice as many GRC-EN pairs as
GRC-GRC pairs so their scores are not directly comparable.

allow for cross-lingual evaluation (See Table 3).
The gold scores for STS datasets are typically
produced by averaging the scores from many human annotators. However, for Ancient Greek itis difficult to find enough annotators to produce
high quality gold scores. Our solution is to use a
Cross-Encoder8 to produce the gold scores basedon the English translations of each pair. A CrossEncoder takes two sentences as input and produces
a similarity score in the range [0, 1] without theneed to encode the semantic properties of each sentence into a vector, and therefore performs better
than cosine similarity between embeddings (SeeFigure 3). With this setup, we measure how closely
each model can match the performance of the English Cross-Encoder. The accuracy of this method
depends on how closely the English translations
match the meaning of the Greek sentences. Therefore the English translations are reviewed by anexpert to ensure that they are literal and accuratetranslations of the Greek text.
Due to the need to manually verify the translations for each pair, the STS dataset is relatively small. The dataset consists of 165 Ancient
Greek sentences pairs, each having an English
translation: ((aGRC , aEN ), (bGRC , bEN )). The
GRC↔EN comparison can be performed two ways:
aGRC ↔ bEN and aEN ↔ bGRC for a total of 330
GRC↔EN comparisons, 165 GRC↔GRC comparisons, and 165 EN↔EN comparisons.

Bi-Encoder

Cross-Encoder

Score

Score

Cosine Similarity
Classiﬁer
Embedding

EmbeddingPooling

Pooling

BERT

BERT

Sentence A

Sentence B

BERT

Sentence A

Sentence B

Figure 3: We use a Cross-Encoder (right) to produce
STS gold scores which are used to evaluate our sentence
embedding models, which are Bi-Encoders (left).

The score for each model is computed as Spearman correlation between gold scores and the cosine
similarities between the sentence embeddings. The
results are reported in Table 4.
The models trained via knowledge distillation significantly outperform the SimCSE models,
showing that they have inherited the properties of
the teacher models for STS tasks. The models withthe st5 teacher have a small lead, which is expected since st5 was trained for STS tasks. All
the models improve slightly when first trained onModern Greek before Ancient Greek.
4.3

Semantic Retrieval

Sentence embeddings can be used for semantic
retrieval tasks by ranking a set of passage embeddings by cosine similarity with a query embedding.
Performing this process with our models on the
Greek sentences in the Perseus and First1KGreek

8
https://huggingface.co/cross-encoder/stsb-roberta-base18

Modelcorpora yields promising results. For example, thefollowing query is correctly answered by several
passages in the top 10 highest ranked passages:
Query: “Was Aristotle a student of Plato?”

GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)• ᾿Αριστοτέλης Πλάτωνος μαθητής· οὗτος τὴν διαλεκτικὴν συνεστήσατο. - Hyppolytus of Rome
Aristotle, a disciple of Plato — He established dialectics.

GRC XLM-R (EL,GRC)

mAP@20

26.61
18.08
21.50
29.56

15.33
10.84
9.86
15.08

• ἀλλὰ καὶ τοῖς Πλάτωνος ἐγκαλέσαι ἄν τις δόγμασιδι΄ ᾿Αριστοτέλην, ἀποφοιτήσαντα τῆς διατριβῆς αὐτοῦ ἐν καινοτομίαις. - Origen
But someone could also challenge certain doctrines ofPlato through Aristotle, who, upon completing his studies, departed from his teachings with innovations.

GRC XLM-R (GRC)

Teacher: sentence-t5-large
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (EL,GRC)

41.34
49.63
34.88
47.07

25.37
36.17
20.07
31.31

Teacher: all-mpnet-base-v2
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)• παρὰ
Πλάτωνι᾿Αριστοτέλης
φιλοσοφήσας
μετελθὼν εἰς τὸ Λύκειον κτίζει τὴν Περιπατητικὴναἵρεσιν. - Clement of Alexandria
After studying philosophy under Plato, Aristotle, having
come to the Lyceum, founded the Peripatetic school.

GRC XLM-R (EL,GRC)

63.60
69.87
53.84
60.13

44.97
53.00
36.42
44.36

Table 5: Recall@10 and mAP@20 scores for English
search queries and Ancient Greek passages. Best results
are bolded.

To quantify the performance of each model forsemantic retrieval, we compile a dataset of 40k
Greek passages from the Perseus and First1KGreek
corpora. We then produce 100 English queries(in the form of both phrases and questions) andassociate them with relevant passages. We measurerecall and mean average precision (mAP) for eachmodel. The scores are reported in Table 5.
The SimCSE models perform poorly, which is
expected since they were not trained for retrieval
tasks. The models with the mpnet teacher, which
was trained for semantic search, score highest by
a large margin. The models with the st5 teacher,which was trained for semantic textual similarity
tasks, perform better than the SimCSE models but
worse than the mpnet models. The models generally perform much better when trained on Modern
Greek. Perhaps this is because many of the queries
involve proper nouns for which Modern Greek data
gave additional training examples, or perhaps thestudent models benefited from the additional English examples to learn the vector space propertiesof the teacher. The GRC mBERT models consistently perform better than GRC XLM-R models.
Overall performance on this task was rather poor
even for the best models. An analysis of the top
ranked passages for each query revealed that passages about related topics often ranked above thedesired passages. In particular, it often confusedproper names, e.g. preferring passages about otherphilosophers for queries about Plato.
4.4

Recall@10

SimCSE

included in the training set, a text with many different translations is needed. The New Testament isa good candidate for this, since many translations
exist in different styles and eras of the English language. We take nine New Testament translations,
ranging from literal (NASB), archaic (KJV), andparaphrase (MSG), all fully aligned at the verselevel (7654 verses). There are no other Greek texts
that we are aware of that have this many translations available for comparison. We generate embeddings for each verse from the Greek text and thetranslations. We also generate an “average” translation for each verse by averaging the embeddingsof all the English translations. We take the cosine
similarity between the Greek embedding and eachtranslation and use it to compute the Mean Reciprocal Rank (MRR) across all verses, for each model:
M RR =

1 X 1|T | v∈T rankv

where T is a set of verses in a translation and rankv
is the rank of the translation for verse v. The results
are reported in Table 6.
The literal translations score highest, and thescore decreases the more non-literal the translations
become, with the MSG translation having the lowest score. Surprisingly, the archaic KJV translation
ranks highly, which is likely due to a high quantityof archaic English text in the training data. Thissuggests that the models are slightly biased to thisolder English translation style. Verses from two of
the translations (NKJV and NET) were included in
the training data. Despite being in the training data,

Translation Bias

To determine whether the models are biased towards certain translation styles, especially those19ModelKJVNKJV* NASB

ESV

RSV

NET*

NIV

NLT

MSG

Avg. Emb.

32.59
33.27
35.78
35.34

36.56
36.05
37.85
36.74

39.97
40.79
38.17
38.02

32.17
31.97
32.15
32.96

29.28
28.74
29.76
30.06

25.91
26.17
26.03
25.76

20.32
20.15
20.58
20.75

12.92
12.79
13.14
13.09

11.14
11.21
11.32
11.25

52.04
51.75
48.11
48.93

30.70
30.66
29.70
30.01

30.13
29.98
29.14
29.08

27.81
28.02
28.10
28.35

25.90
25.82
26.39
26.82

23.05
23.39
23.82
23.61

20.09
19.70
20.38
19.75

14.43
13.93
14.24
13.68

12.49
12.24
12.90
12.36

78.66
80.42
76.40
78.82

31.15
31.20
31.31
31.21

29.93
30.37
29.91
29.64

30.04
30.35
30.00
30.25

28.94
28.68
29.42
29.59

23.33
23.51
23.32
23.55

19.52
19.53
19.67
19.32

13.93
13.76
13.97
13.61

11.98
11.81
12.32
11.89

72.32
73.26
65.74
70.76

SimCSEGRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

Teacher: sentence-t5-large
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)

29.63
28.73
31.82
30.41

Teacher: all-mpnet-base-v2
GRC mBERT (GRC)

GRC mBERT (EL,GRC)
GRC XLM-R (GRC)

GRC XLM-R (EL,GRC)
*

31.76
30.42
37.25
33.08

Verses from the NET and NKJV were included in parallel training data.

Table 6: Mean Reciprocal Rank (MRR) ×100 of cosine similarity between Greek verses of the New Testament andEnglish translations, as well as MRR of per-verse averaged embedding of all the translations. Highest translation
MRR for each model is bolded. MRR of averaged embedding is underlined if it is higher than any of the translations.

there does not appear to be bias to the NET since
it consistently ranks lower than other translations.
The NKJV ranks highly, but does not consistently
outrank other literal translations. Interestingly, theaverage embedding of all the translations ranked
highest by a significant margin.

5

translation alignment.
Although contamination from modern languages
is not a big concern for the tasks in this paper, there
could be issues of anachronisms when searching
Ancient Greek texts with English. Furthermore,using texts from such a long chronological periodof the Greek language could introduce additional
lexical polysemy as Greek words changed in meaning over time. This could explain why the averaged
embedding of many translations had a higher MRR
than any individual translation source in Table 6,since the combination of many translations represents a higher degree of polysemy. In future work,
such historical polysemy could be measured bysampling translations of words from texts of different historical periods. This could help to determine
whether the high MRR of the averaged embedding
is a useful result or simply an artifact of a potentially high amount of polysemy in the training data.

Discussion and Future Work

Overall, the base models mBERT and XLM-R performed similarly except for the semantic retrieval
task where the mBERT-derived models have a sizeable lead. The reason for this is unclear, since thesemodels have different tokenizers, parameter counts,
and vocabularies. It is also unclear how much the
pre-training process affects the results. An area of
future research would be to investigate the effectof student model architecture, tokenizer, and pretraining on the ability of the student model to learn
from the teacher model.
The main limitation of using multilingual knowledge distillation to train sentence embedding models is that the embeddings produced are almost
entirely derived from English translations, which
could be undesirable if the goal is to study Ancient Greek text without any prior translator’s interpretation. Furthermore, the student model cannever fully replicate the performance of the teacher
model when transfering to another language, sincetranslated sentences are often not entirely semantically equivalent to their source sentences, especially when removed from the original context via

6

Conclusion

In this paper, we have shown that multilingual
knowledge distillation is an effective approach fortraining sentence embedding models for Ancient
Greek, in spite of the lack of available training datacompared to modern, high-resource languages. In
addition, we have produced a new dataset of parallel Ancient Greek and English sentences as well as
evaluation datasets for translation search, semantic
textual similarity, and semantic retrieval, which we
make publicly available.
20References

GREEK-BERT: The Greeks visiting Sesame Street.
https://arxiv.org/abs/2008.12014v2.

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python, 1st ed edition.
O’Reilly, Beijing ; Cambridge [Mass.].

Lei Liu and Min Zhu. 2023. Bertalign: Improved
word embedding-based sentence alignment for Chinese–English parallel corpora of literary texts. Digital Scholarship in the Humanities, 38(2):621–634.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–
8451, Online. Association for Computational Linguistics.

Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant,Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022.
Sentence-t5: Scalable sentence encoders from pretrained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022,
pages 1864–1874, Dublin, Ireland. Association for
Computational Linguistics.

Charlie Cowen-Breen, Creston Brooks, Johannes
Haubold, and Barbara Graziosi. 2023. Logion: Machine Learning for Greek Philology.

Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz.
2021. Morphology matters: A multilingual language
modeling analysis. Transactions of the Association
for Computational Linguistics, 9:261–276.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171–4186, Minneapolis, Minnesota. Association for
Computational Linguistics.

Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.
How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
3982–3992, Hong Kong, China. Association for Computational Linguistics.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conferenceon Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational
Linguistics.
Péter Halácsy, Andras Kornai, Viktor Nagy, László
Németh, and Viktor Trón. 2007. Parallel corpora
for medium density languages. In Recent Advancesin Natural Language Processing IV, pages 247–258.

Nils Reimers and Iryna Gurevych. 2020. Making
monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 4512–4525,
Online. Association for Computational Linguistics.

Valentin Hofmann, Janet Pierrehumbert, and Hinrich
Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT’s interpretationof complex words. In Proceedings of the 59th Annual
Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long
Papers), pages 3594–3608, Online. Association for
Computational Linguistics.

Frederick Riemenschneider and Anette Frank. 2023. Exploring Large Language Models for Classical Philology.
Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021.
A pilot study for BERT language modelling and morphological analysis for ancient and medieval Greek.
In Proceedings of the 5th Joint SIGHUM Workshop
on Computational Linguistics for Cultural Heritage,
Social Sciences, Humanities and Literature, pages
128–137, Punta Cana, Dominican Republic (online).
Association for Computational Linguistics.

Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd
Cook, Clément Besnier, and William J. B. Mattingly.
2021. The Classical Language Toolkit: An NLP
framework for pre-modern languages. In Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing:
System Demonstrations, pages 20–29, Online. Association for Computational Linguistics.

Weiting Tan, Kevin Heffernan, Holger Schwenk, andPhilipp Koehn. 2023. Multilingual representation
distillation with contrastive learning. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,
pages 1477–1490, Dubrovnik, Croatia. Association
for Computational Linguistics.

John Koutsikakis, Ilias Chalkidis, Prodromos
Malakasiotis, and Ion Androutsopoulos. 2020.

21

Jörg Tiedemann and Lars Nygaard. 2004. The OPUS
corpus - parallel and free: http://logos.uio.
no/opus. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA).
Ivan Yamshchikov, Alexey Tikhonov, Yorgos Pantis,
Charlotte Schubert, and Jürgen Jost. 2022. BERT in
plutarch’s shadows. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language
Processing, pages 6071–6080, Abu Dhabi, United
Arab Emirates. Association for Computational Linguistics.

A

Appendix: Training Details
Parameter

GRC mBERTGRC XLM-R

Batch Size
Learning Rate
LR Scheduler
EpochsWarmup Steps
Mask Percentage

140
2e-5
linear
10
2000
15%

128
2e-5
linear10
2000
15%

Table 7: Pre-training hyperparameters
Parameter

GRC mBERTGRC XLM-R

Batch Size
128
Learning Rate
2e-5
LR Scheduler
linearMax Seq. Length
128
Poolingmean
Embedding Dim.
768
Teacher: all-mpnet-base-v2

128
2e-5
linear
128
mean
768

Epochs (GRC)
15
Epochs (EL)3
GRC Warmup Steps 2000
EL Warmup Steps
2000
Teacher: sentence-t5-large

15
3
2000
8000

Epochs (GRC)
Epochs (EL)GRC Warmup Steps
EL Warmup Steps

30
6
2000
2000

30
6
2000
2000

Table 8: Knowledge distillation hyperparameters
Parameter

GRC mBERTGRC XLM-R

Batch Size
Learning Rate
LR Scheduler
Warmup Steps
Max Seq. Length
Epochs (GRC)
Epochs (EL)
Pooling
Embedding Dim.

82
2e-5
linear
2000
128
10
3
CLS
768

82
2e-5
linear
2000
128
10
3
CLS
768

Table 9: SimCSE hyperparameters

22

B

Appendix: Sentence Segmentation

For translation alignment, it is not necessary thateach segment be a sentence, since the alignment
process can handle 1-many, many-1 or many-tomany relations. The Greek texts in our corpus
contain punctuation, so we segment them by period(.), question mark (;), and raised dot (·). Some of
the Greek texts use a colon instead of a raised dot,
and in these cases we treat colons as raised dots.
For the English texts we first segment using the
NLTK sentence tokenizer (Bird et al., 2009) thenfurther subdivide these segments by semicolon (;)
and colon (:).

A Transformer-based Parser for Syriac Morphology
Martijn Naaijer♡1 Constantijn Sikkel♠ Mathias Coeckelbergs♣
Jisk Attema♢ Willem Th. Van Peursen♠♡University of Copenhagen, Denmark ♠Vrije Universiteit Amsterdam, The Netherlands♣Katholieke Universiteit Leuven, Belgium ♢Netherlands eScience Center, The Netherlands

Syriac training set. Therefore, we try to improvethe model’s prediction accuracy by adding Biblical
Hebrew data to the training process. Biblical
Hebrew is a Semitic language that is closely related
to Syriac, and the training set that we have for thislanguage is substantially bigger.
Since the late 1970s, the Eep Talstra Center for
Bible and Computer (ETCBC) of the Vrije
Universiteit Amsterdam has developed andmaintained a richly annotated dataset of theMasoretic Text of the Hebrew Bible. This dataset
contains a wealth of linguistic features on the levelsof words, phrases, clauses and larger text units.
More recently, ancient texts in Syriac have been
prepared in a similar way. However, a vast corpus
of Syriac texts is available, and we hope to develop
a faster approach to annotate these texts, because
annotating them manually is a labor-intensive task.
We have trained the Transformer model in fivedifferent ways, to see which approach gives thehighest accuracy on the Syriac test set: a modeltrained on Syriac data only, a model trained on amix of (vocalized or unvocalized) Hebrew and
Syriac data, and a model which is trained on(vocalized or unvocalized) Hebrew data first andtrained further on Syriac data.
A trained model can make predictions on “new”
Syriac texts, resulting in morphologicallysegmented texts. These results need to be correctedmanually, and these corrected results can beprocessed further in a rule-driven way to produce
the linguistic annotations. Therefore, training themodels is the first step in a longer pipeline.

Abstract
In this project we train a Transformer-basedmodel from scratch, with the goal of parsing themorphology of Ancient Syriac texts as
accurately as possible. Syriac is a low-resource
language, only a relatively small training set
was available. Therefore, the training set was
expanded by adding Biblical Hebrew data to it.
Five different experiments were done: themodel was trained on Syriac data only, it wastrained with mixed Syriac and (un)vocalized
Hebrew data, and it was trained first on(un)vocalized Hebrew data and then trained
further on Syriac data. The models trained on
Hebrew and Syriac data consistently
outperform the models trained on Syriac data
only. This shows that the differences between
Syriac and Hebrew are small enough that it isworth adding Hebrew data to train the model
for parsing Syriac morphology. Training
models with data from multiple languages is an
important trend in NLP, we show that thisworks well for relatively small datasets of
Syriac and Hebrew.

1 Introduction
In this paper we develop a morphological parser for
the Syriac language. The trained model is able tosegment graphical units into distinct words, it
segments the morphemes within a word, anddisambiguates morphemes and lexemes, all at thesame time.
Syriac is a Semitic language with a richmorphology. Therefore, to add linguistic
annotations to a text, it is better to encode thesmaller parts of a word (morphemes) rather thanthe complete words. A complication is that the
Syriac language is written without vowels, whichleads to the problem that a word can be parsed indifferent ways. Furthermore, we only have a small
1

2 State of the art
Between 2000 and 2020 a number of studies were
published in which Natural Language Processing(NLP) tasks for Semitic languages are described,
often dealing with part of speech tagging (e.g.,

Corresponding author: mna@teol.ku.dk.

23
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 23–29,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_003

Modern Hebrew: Bar Haim et al. 2008; Amharic:
Tachbelle et al. 2011; Arabic: Kübler, andMohamed 2012; Mishnaic Hebrew: Giovanetti etal. 2018). Other studies deal with morphologicalanalysis (Daya et al. 2004, Lembersky et al. 2014)
and segmentation (Zeldes 2018).
With the larger availability of digital (annotated)
Semitic texts and the advent of large, Transformerbased language models, there is an acceleration in
the development of models and tools for NLP tasks
for Semitic languages. A Large Language Model
which focuses on Modern Hebrew, is AlephBERT(Seker et al. 2021), which can be used for a numberof tasks, including segmentation, part of speechtagging, full morphological tagging, named-entity
recognition and sentiment analysis. A similar
model for Arabic, AraBERT, was developed byAntoun, Bali and Hajj (2021).
Relatively close to our research is a paper on
adding diacritics to consonantal Hebrew texts(Shmidman et al. 2020). It uses a combination of amachine learning (“several bi-LSTM based
modules”) and a rule-driven approach(“comprehensive inflection tables and lexicons”).
Koppel and Shmidman (2020) give an overview ofdevelopments in Machine Learning in relation to
the Hebrew language and its texts.
A list of NLP resources for Hebrew can be foundhere:
https://github.com/NNLPIL/Resources.
An important trend in NLP is the developmentof multilingual models. These are models that canbe used for a number of NLP tasks in various
languages. Some of these models are trained on
one language, like English, and they can be trained
further on other languages, but there are alsomodels that are trained from scratch on a numberof languages (Ruder 2020).

(MT) of the Hebrew Bible. The Hebrew output filecontains the morphologically parsed MT. The textof these datasets is based on the fifth edition of the
Biblica Hebraica Stuttgartensia3. The Syriac input
file contains some books from the Peshitta, atranslation of the Hebrew Bible in Syriac 4 (TerHaar Romeny and Van Peursen, 1966–) and somenon-biblical texts 5 . The Syriac input texts areunvocalized, but they contain some diacritics,
which can be found in the Syriac manuscripts.
Each line in a data file contains one verse, andthe text is represented in the ETCBC transcription.
The first line of the vocalized Hebrew dataset,
which is the first sentence of the Hebrew Bible,
looks as follows:

3 DataThis text contains the same consonants as the
vocalized text, but it misses the vowel signs.
Finally, the corresponding verse in themorphologically parsed output file looks asfollows:

Gen 1 1 B.:R;>CIJT B.@R@> >:ELOHIJM>;T HAC.@MAJIM W:>;T H@>@REY
This line contains four tab-separated fields, withthe following data: book, chapter, verse, and text.
In Hebrew script, the text, which means “In the
beginning God created the heaven and the earth”,
looks as follows:‫אשׁית בָּ ָ ֣רא ֱא ִהים אֵ ת הַ שָּׁ מַ יִ ם וְ אֵ ת הָ אָ ֶרץ‬֖ ִ ‫בְּ ֵר‬
All consonants, vowel signs and diacritics have avalue in the transcription, e.g., ‫ ב‬is transcribed withB, ‫ א‬with >, qametz is transcribed with @, shewa
with “:”, and dagesh with “.”. The transcription isread from left to right, unlike the text in Hebrew
script.
The same line, but taken from the unvocalized
dataset looks as follows:
Gen 1 1 BR>CJT BR> >LHJM >T HCMJM
W>T H>RY2

Our dataset consists of five files , which are basedon the ETCBC database. The Hebrew files that canserve as the input data for the model, contain
vocalized or unvocalized text of the Masoretic Text
2

3

The files can be found in the data folder of our GitHub
repository:
https://github.com/etcbc/ssi_morphology.
The raw input files are s2-in (Syriac), t-in_voc (vocalized
Hebrew), t-in_con (unvocalized Hebrew), the corresponding
parsed output files are s2-out (Syriac) and t-out (Hebrew).
In this repository one can also find the code.

For an electronic edition of the MT with all theannotations, see:https://github.com/ETCBC/bhsa.
4 A digitized version of the whole Peshitta can be foundhere: https://github.com/ETCBC/peshitta.
5
For the texts, see also:
https://github.com/ETCBC/linksyr/tree/master/data.

24

Gen 1 1 B-R>CJT/ BR>[ >LH(J(M/JM >T
H-CMJ(M/(JM W->T H->RY/:a

The alphabets of Syriac and Hebrew are identical,
also in the ETCBC transcription, except that the sin(‫ )שׂ‬is lacking in Syriac. The Syriac dataset contains
three different Syriac diacritics: dots below andabove the text, and seyame.
A limitation of the present dataset is that forevery word in the input, there is only one correctparsing in the output. In some cases, the text is
ambiguous, and a word could be parsed correctly
in different ways. A possible improvement of thedataset is to include alternative parsing options.

The output dataset contains the same consonantal
text as the input data, with a number of extra signs
which indicate the morphological structure of thewords:The dash (-) separates different words within agraphical unit.
A word can have different morphemes, which aremarked with special signs:
After “[“ follow verbal endings, and after “/”follow nominal endings.
“+” initializes a pronominal suffix.
Between exclamation marks, one finds the verbal
preformative, e.g., !J! in a 3rd person masculine
singular yiqtol, !T! in a 2nd person masculine
singular yiqtol or !! in a qal infinitive or imperative.
Between closing square brackets one finds theprefix that is characteristic for a verbal stem, e.g.
]HT] for hitpael, ]N] for niphal, etc.
“~” initializes a univalent final, for example, a ~H
is a locative he.

4 Data preparation
We start with texts that do not have any parsing,which means that a text has not been segmented inphrases, clauses, or sentences. All verses of a bookin the dataset are concatenated and split separately
in shorter sequences of n graphical units. n is oneof the required hyperparameters for training amodel. These shorter sequences are partly
overlapping and form a moving window. E.g., if
the text is:
BR>CJT BR> >LHJM >T HCMJM W>T H>RY

The ETCBC approach of encoding morphology
distinguishes between a paradigmatic form and arealized form of the morphemes. E.g., theparadigmatic form of the masculine plural marker
is JM (‫ים‬- in Hebrew script). In several places in theMT, it is spelled as M (‫)ם‬. Here the J (‫)י‬, which ispart of the paradigmatic form, is not written. This
is indicated in the encoding with an opening
parenthesis. E.g., in Genesis 17:20, one finds ‫נשיאם‬(“princes”), which has the morphological encoding
NFJ>/(JM, indicating that the J occurs in theparadigmatic plural form, but it is not realized. The
opposite can also occur. If a character occurs in thetext, but not in the paradigmatic form, it is preceded
by “&”.
In the morphological encoding, there are some
Latin letters preceded by a colon::a marks that a word is in absolute state.
:c marks that a word is in construct state.
:n marks the narrative vocalization of the waw.
:d marks the D-stem.
:u marks the u-a pattern of the passive.

and n is 5, the text will be split in the following
three training inputs:
BR>CJT BR> >LHJM >T HCMJM
BR> >LHJM >T HCMJM W>T>LHJM >T HCMJM W>T H>RYWhen all the texts are split in partly overlapping
sequences and a subset is selected randomly as
Syriac test set, a problem is that part of thesequences in the test set can also be found in the
training set, which means that training and test set
are not independent of each other. A possible
solution is to select a few complete books as test
set, but that leads to the problem that the languageof these books may not be representative of Syriac
in general. Therefore, we have used a different
solution. If n is 5, the texts of 5 consecutive verses
are grouped, and from all these groups of 5 verses,
the validation and test set are selected. With thisapproach, it is guaranteed that the texts are longenough to extract at least one sequence of 5graphical units, and they are short enough to split abook in many sequences, with the result that partsof the book can be found in the training, validationand test set, without overlap between these

The “=” sign is used to disambiguate consonantal
homographs, e.g., one distinguishes between KBD/(‫כָּבֵ ד‬, “heavy”), KBD=/ (‫כָּבֵ ד‬, “liver”), and KBD==/(‫כֹּ בֶ ד‬, “heaviness”).

25datasets. After this split, each sequence of 5 verses
is split further in the partly overlapping shorter
sequences of 5 graphical units. All short sequencesthat contain a case of ketiv/qere in the Hebrew
datasets are removed, because the consonantal textthat is written (the ketiv) and the morphologicalanalysis generally do not match. These words areindicated with a “*” in the data files.

Just like the encoder, the decoder consists of anumber of layers, one layer giving its output to thenext one.
The decoder of the transformer model starts witha start symbol and the representation of thesentence produced by the encoder, and from thatthe first word of the output after the start symbol is
generated. Then, the representation, the startsymbol and the first word together are fed to theencoder, after which the second word is generated.
This is done until a stop symbol is generated.
In the present implementation, various
hyperparameters can be tweaked, which can befound in the README of the GitHub repo. The
only thing that we vary in the experiments
described here are the number of epochs and thetraining datasets.
The model is trained from scratch, which makes
it possible to get a good impression of what thedifference is between a model trained on Syriac
data alone, and a model that is trained on Hebrew
and Syriac data.
In all our experiments, the number of heads inthe encoder is 8, and the number of encoder layers
and decoder layers is 3. The feedforward hidden
dimension is 512. During decoding we used beam
search, with a beam size of 3. The length of thepartly overlapping text sequences is 7 graphicalunits.

5 The model
The morphological analysis is approached here asa sequence to sequence (seq2seq) problem, forwhich we use a Transformer model 6 . The
Transformer is the state-of-the-art model fornumerous NLP tasks (Vaswani et al. 2017) and isalso the basis of Large Language Models like
ChatGPT and GPT4. The Transformer seq2seq
model has an encoder/decoder architecture. The
encoder consists of a stack of encoder layers, inwhich the output of one layer serves as the input of
the next one. Each layer consists of two
components: multi-head attention and afeedforward network. Fundamental for thetransformer model is the concept of self-attention,with which a word is related to all other words in atext sequence. In the self-attention mechanism, theembedding matrix of a sentence is multiplied withthree randomly initialized matrices WQ, WK, andWV, thus forming three new matrices Q (Query), K(Key) and V (Value). From these matrices, theattention matrix Z1 is calculated as follows:
𝑍 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥

𝑄𝐾
(𝑑 )

6 Results
The model was trained with five different training
strategies:𝑉

1. The model was trained on Syriac data.
2. The model was trained on a mix of unvocalized
Hebrew and Syriac data.
3. The model was trained on a mix of vocalized
Hebrew and Syriac data.
4. The model was trained first on unvocalized
Hebrew data (10 epochs), and after that trained
further on Syriac data.
5. The model was trained first on vocalized Hebrew
data (10 epochs), and trained further on Syriac
data.

Z1 has the index 1, because this is the first attention
head. There can be an arbitrary number of heads
that are concatenated:
𝑀𝑢𝑙𝑡𝑖ℎ𝑒𝑎𝑑 𝑎𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛= 𝐶𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒(𝑍 , 𝑍 , 𝑍 , … )𝑊
in which W0 is a new weight matrix.
After that, information of the word order in a
sentence is added using positional encoding. The
resulting matrix is fed to a feedforward network
consisting of two dense layers with ReLU
activation.

The approach of two experiments is called
transfer learning. In transfer learning, a model is

6

The code for the model can be found in the filemodel_transformer.py in the scr folder in the GitHub
repository.

26

trained first on a large dataset, after which themodel is trained further on a smaller dataset for a
specialized task. This is generally beneficial if
there is only a small training dataset available forthe specialized task, like in our case.
In all the experiments, we varied the number of
epochs in the main training loop (20, 25, 30, 35,
and 40 epochs).
We checked the accuracy of the predictions on
the Syriac test set, which is identical for eachexperiment. The accuracy is defined as thepercentage of graphical units that is predicted fully
correctly at a specific index of the test sequences.
These test sequences are partly overlapping, justlike the training sequences. Therefore, for mostwords in the test set multiple predictions are made.
The results can be found in figure 1, whichshows the results of the index with the highest
accuracy.

The Hebrew datasets consisting of 22946 verses
are substantially bigger than the Syriac datasets(5596 verses) we used. Therefore, training a modelwith Hebrew data takes substantially longer, which
may be a disadvantage for including this dataset,especially if one wants to optimize the model
further by tuning the hyperparameters. So, as isoften the case, there is a tradeoff between speed andperformance.

7 Error analysis
In the predictions on the test set, the model canmake different kinds of mistakes. We provide anotebook in the GitHub repository 7 , with which
each mistaken prediction is classified as one of six
error categories, with the goal of further improving
the model. The following kinds of mistakes are
distinguished:
0. Parse errors in the encoding. In this case, theprediction is ungrammatical according to theparsing conventions.
1. The consonantal form of the prediction and thetrue surface form differ.
2. Ungrammatical morpheme type combinations.
This is the case if there is, e.g., a combination ofverbal and nominal morphemes that do not match.
3. Unparadigmatic morphemes. In this case themodel predicts a morpheme that falls outside of the
ETCBC inventory of paradigmatic Syriac
morphemes.
4. Difference in number of analytical words withthe true form. In this case, the number of “-” signs
in the graphical unit is incorrect.
5. Difference in morphemes with the true form. In
this case, the analysis of the word is grammaticallycorrect, but not within the given context, there
could for instance be an incorrect number of “=”signs at the end of the lexeme.

Figure 1. The accuracy of predictions on the
Syriac test set with five different training
strategies.

The accuracy of the model trained on Syriac data
increases with the number of epochs from 87.3%
for 20 epochs to 89.3% for 30 or more epochs. The
accuracy of the predictions of the models trainedon Hebrew and Syriac data vary somewhat
between 89.8% (20 epochs) and 90.8% (35epochs), both achieved by the model trained
simultaneously on unvocalized Hebrew and Syriac
data.
The models trained on Hebrew and Syriac data
perform consistently better than the models trainedon Syriac data only. Even though the accuracy ofthe latter models is only 1-2% higher, this is quite
substantial, and it is hard to achieve this result bytuning hyperparameters.
7 This is the file evaluation_syriac.ipynb in the folder

badness_analysis.

27

The results are shown in figure 2. It shows theresults for the number of epochs with the highest
accuracy.

that were trained on Syriac data alone, models thatwere trained on (un)vocalized Hebrew and thentrained on Syriac, and models that were trained on(un)vocalized Hebrew and Syriac simultaneously.
The highest accuracy of the model trained on
Syriac data was 89.3%. The best model overall wastrained on unvocalized Hebrew and Syriac
simultaneously with an accuracy of 90.8%, which
outperforms the best “Syriac only” model with
1.5%.
Further improvements can possibly be achievedby optimizing the hyperparameters of the models,
but it is clear that adding Hebrew data to thetraining set helps with improving the performanceon the Syriac test set. The same effect may be
expected with a larger Syriac dataset, but as longas that dataset is relatively small, adding Hebrew
data is a good solution. Another way to expand thedataset is to use data augmentation, which we areconsidering for future experiments.
It has been shown in other tasks that a modeltrained on a variety of data can be very useful to be
trained further for specialized tasks. In our project
we see the same phenomenon. The experiment
could be broadened in various ways. One could for
instance use one of our models and train it further
on data from other languages than Hebrew and
Syriac, such as Akkadian or Arabic, or train models
to parse Syriac texts syntactically.

Figure 2. Frequencies of different types of errors
made by the five models.

In general, the models show similar patterns. For
every model, the most frequent type of error is 5,
which means that the parsing is grammaticallycorrect, but not in the given context. The error
types 0 and 2 hardly occur.
In most error categories, the model which wastrained on Syriac only has more errors than theother models. An important difference between thismodel and the other models is found in error type
1, indicating errors in the surface text, where themodel trained on Syriac has 2-3 times more errors
than the models trained on both Hebrew and Syriac
data. The consonantal text of the input and output
should be identical, and this is language
independent. This is a clear sign that adding the
Hebrew data helps here, simply because thevolume increases. The same may be true for theerror categories 3, 4, and 5. Here and there, theHebrew may help because a morpheme is the sameas in Syriac, but it is likely that it helps mostly
because it adds volume to the dataset, which helps
to make the model more consistent in analyzing
morphemes.

Acknowledgments
We thank Martin Ehrensvärd for proofreading themanuscript and the Netherlands eScience Centerfor their support in this project.

References
Shihadeh Alqrainy and Muhammed Alawairdhi. 2020.
Towards Developing a Comprehensive Tag Set for
the Arabic Language. Journal of Intelligent Systems
30:287–296.
https://doi.org/10.1515/jisys-2019-0256.
Wissam Antoun, Fady Baly, and Hazem Hajj. 2021.
AraBERT: Transformer-based Model for Arabic
Language Understanding. Proceedings of the 4thWorkshop on Open-Source Arabic Corpora andProcessing Tools, with a Shared Task on Offensive
Language Detection, 9–15.
https://aclanthology.org/2020.osact-1.2.

8 Conclusions
In this paper we trained a Transformer model fromscratch with the goal of analyzing Syriac
morphology. An important part of the research was
to see if adding Hebrew to the training set would
improve the accuracy of the predictions on the
Syriac test set. We compared results of the models

Roy Bar-Haim, Khalil Sima'an, and Yoad Winter.
2008. Part-of-speech tagging of Modern Hebrew
text. Natural Language Engineering 14(2):223–251.
https://doi.org/10.1017/S135132490700455X.

28

Ezra Daya, Dan Roth, and Shuly Wintner. 2004.
Learning Hebrew Roots: Machine Learnng with
Linguistic Constraints. Proceedings of the 2004
Conference on Empirical Methods in Natural
Language Processing, 357–364.

Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. Advances in Neural Information
Processing Systems, 6000–6010.
https://arxiv.org/pdf/1706.03762.pdf.
Amir Zeldes. 2018. A Characterwise Windowed
Approach to Hebrew Morphological Segmentation.
Proceedings of the 15th SIGMORPHON Workshop
on Computational Research in Phonetics,
Phonology,
andMorphology,
101–110.
http://dx.doi.org/10.18653/v1/W18-5811.

Emiliano Giovannetti, Davide Albanesi, Andrea
Bellandi, Simone Marchi, and Alessandra Pecchioli.
2018. Constructing an Annotated Resource for PartOf-Speech Tagging of Mishnaic Hebrew.
Proceedings of the Fifth Italian Conference on
Computational Linguistics CLiC-it 2018: 10–12
December 2018, Torino. Torino: Accademia
University.
https://doi.org/10.4000/books.aaccademia.3394.
Moshe Koppel and Avi Shmidman. 2020. Torah Studyand the Digital Revolution, a Glimpse of the Future.
https://thelehrhaus.com/commentary/torah-studyand-the-digital-revolution-a-glimpse-of-the-future.
Sandra Kübler and Emad Mohamed. 2012. Part ofspeech tagging for Arabic. Natural Language
Engineering 18(4):521–548.
https://doi.org/10.1017/S1351324911000325.
Gennadi Lembersky, Danny Shacham, and Shuly
Wintner. 2012. Morphological disambiguation of
Hebrew: A case study in classifier combination,
Natural Language Engineering 20(1):69–97.
https://doi.org/10.1017/S1351324912000216.
Amit Seker, Elron Bandel, Dan Bareket, Idan
Brusilovsky, Refael Shaked Greenfeld, and Reut
Tsarfaty. 2021. AlephBERT: A Hebrew Large PreTrained Language Model to Start-off your Hebrew
NLP Application.
https://arxiv.org/abs/2104.04052.
Avi Shmidman, Shaltiel Shmidman, Moshe Koppel,
and Yoav Goldberg. 2020. Nakdan: Professional
Hebrew Diacritizer. Proceedings of the 58th Annual
Meeting of the Association for Computational
Linguistics: System Demonstrations, Association
for Computational Linguistics 197–203.
https://doi.org/10.18653/v1/2020.acl-demos.23.
Martha Y. Tachbelie, Solomon T. Abate, and Laurent
Besacier. 2011. Part-of-Speech Tagging for UnderResourced and Morphologically Rich Languages–
The Case of Amharic. Conference on Human
LanguageTechnology
forDevelopment,
Alexandria,
Egypt,
2–5
May
2011.
https://www.cle.org.pk/hltd/pdf/HLTD201109.pdf.
Bas Ter Haar Romeny and Willem Th. Van Peursen(eds.). 1966–. The Old Testament in Syriac
according to the Peshiṭta Version. Leiden: Brill.
Sebastian Ruder. 2022. The State of Multilingual AI.
https://www.ruder.io/state-of-multilingual-ai.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz

29

Graecia capta ferum victorem cepit
Detecting Latin Allusions to Ancient Greek Literature
Frederick Riemenschneider
Anette Frank
Department of Computational Linguistic
Heidelberg University
69120 Heidelberg, Germany{riemenschneider|frank}@cl.uni-heidelberg.de
Abstract

The importance of intertextuality, especiallygiven the considerable attention it has received,
is beyond dispute. While there exists a plethora oftheoretical work exploring specific forms of intertextuality, our focus in this work is on the generaloccurrence of textual resemblances, specificallywithin Latin and Greek texts.
Traditionally, the identification of such parallels has largely relied on scholars’ close reading.
However, recent years have seen the developmentof statistical NLP tools – driven especially by the
Tesserae project (Coffee et al., 2012; Forstall et al.,2014) at the forefront of this movement – that areable to automatically uncover a considerable number of textual parallels. These approaches, however, typically rely on string-level parallels and are
grounded in carefully designed rules and scoring
functions. Notably, these systems are generally
restricted to detecting parallels in the same language, as they rely on identifying identical tokens
or stems.
Recently, the breakthrough in self-supervised
training of powerful pre-trained language models(PLMs) has also led to a surge of diverse PLMs
for Classical Philology (Bamman and Burns, 2020;
Yamshchikov et al., 2022; Mercelis and Keersmaekers, 2022; Singh et al., 2021; Riemenschneider
and Frank, 2023). In fact, two recent case studies in Bamman and Burns (2020) and Burns (2023)
have shown that contextualized embeddings produced by such models can indeed identify texts
bearing similar content. While a rigorous quantitative evaluation of these findings still remains to be
conducted, the perceived potential of using thesemodels for finding intertextual relations is clearlysparking widespread interest.
However, research into modern language analysis tasks has demonstrated that sentence embeddings derived solely from standalone BERTor RO BERTA-based models generate suboptimal
and inefficient embeddings. This insight led toIntertextual allusions hold a pivotal role in Classical Philology, with Latin authors frequently
referencing Ancient Greek texts. Until now,
the automatic identification of these intertextual references has been constrained to monolingual approaches, seeking parallels solely
within Latin or Greek texts. In this study, we
introduce SP HIL BERTA, a trilingual SentenceRO BERTA model tailored for Classical Philology, which excels at cross-lingual semantic
comprehension and identification of identicalsentences across Ancient Greek, Latin, and English. We generate new training data by automatically translating English texts into Ancient Greek. Further, we present a case study,
demonstrating SP HIL BERTA’s capability to facilitate automated detection of intertextual parallels. Our models and resources are available
at https://github.com/Heidelberg-NLP/ancient-language-models.

1

Introduction

The study of intertextuality and allusions to literary sources has a longstanding tradition in Classical Philology, highlighting complex interconnections between different literary works. During the
1960s, the concept of intertextuality was shaped by
a comprehensive theoretical framework developedby scholars such as Julia Kristeva, Ferdinand deSaussure, and Michail Bakhtin. The term “intertextuality” itself was introduced by Kristeva during
this pivotal era (Alfaro, 1996; Bendlin, 2006; Kristeva, 1986; Orr, 2003).
Intertextuality proves particularly crucial when
examining Roman literature’s relationship with Ancient Greek texts. Many Latin authors consciously
mirrored elements of Greek classics, making intertextuality an essential concept for understanding
this cultural literary exchange.1
1

Cf. Hutchinson (2013): “How Latin literature relates to
Greek literature is one of the most fundamental questions for
Latin literature, and for the reception of Greek.”

30
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 30–38,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_004

the creation of Sentence-BERT (SBERT) models(Reimers and Gurevych, 2019).
Among the latest language models introducedin the field of Classical Philology is P HIL BERTA(Riemenschneider and Frank, 2023), a RO BERTAbased model pre-trained on Ancient Greek, Latin,
and English language data. Building upon thismodel, we present SP HIL BERTA, a model tailoredto the discovery of intertextual parallels acrossLatin, Ancient Greek, and English texts.
In this work, our goal is to move away from systems relying on hand-crafted rules, and instead toemploy state-of-the-art tools for identifying intertextual relations that are easy to adapt to a wide
variety of languages from Classical Philology andbeyond. Most importantly, we probe the feasibility of uncovering intertextual parallels across languages, an area that has been largely neglected in
the automatic identification of intertextual allusions
until this point. This novel capability will considerably enlargen the space for new findings, by being
able to compare texts directly across languages.
We show that SP HIL BERTA is proficient in recognizing direct translations of sentences in Ancient Greek, Latin, and English, thereby demonstrating comprehensive cross-lingual competence.
Applying our model directly to texts of philological significance not only underlines its practicalapplicability but also highlights areas for improvement, suggesting promising avenues for future exploration.
In summary, our contributions are as follows:cantly enhances the performance of the system for Ancient Greek. While finding textual
allusions still requires philological expertise,
we present cases where the model identifies
passages linked to known allusive texts.

2

Related Work

Detecting Intertextual Allusions. Initiated in
2008, the Tesserae project (Coffee et al., 2012;
Forstall et al., 2014) has been instrumental in advancing the automatic detection of intertextuality
in Latin and Greek texts. Their open-source tools
have seen numerous enhancements and refinements
over the years.2
Existing research has explored matching words
or stems (Coffee et al., 2012) as well as methods
that focus on semantics (Scheirer et al., 2014). Additionally, techniques that combine both lexical andsemantic elements have been examined, where semantic understanding is established through wordembeddings (Manjavacas et al., 2019) or via the(Ancient Greek) WordNet (Bizzoni et al., 2014).
While the majority of preceding studies have concentrated on detecting text reuse in the Bible andvarious religious texts, Burns et al. (2021) focus on
Classical Latin literature.
However, to our knowledge, no efforts havebeen undertaken to automatically detect intertextual similarities across languages, specifically between Greek, Latin, and English texts. This lack islikely due to the inherent complications of inducing
cross-language mappings, a difficulty that arises
both with surface form-based strategies and withtechniques utilizing word embeddings. Notwithstanding, this gap is of significant importance, as
it overlooks the frequent appearance of such allusions, especially from Latin to Greek literature.

i) We introduce SP HIL BERTA, a multilingual
sentence transformer for Latin, Ancient Greek,
and English. To our knowledge, we are thefirst to apply this type of model to automatically detect passages of potential cross-lingual
allusions in Latin texts.
ii) To alleviate the scarcity of parallel sentencepairs for training SP HIL BERTA, we augment
the available resources by automatically translating English texts to Ancient Greek using an
existing multilingual T5 model pre-trained on
Ancient Greek, Latin, and English data.
iii) We conduct experiments on retrieving translations or similar sentences from textual passages in foreign-language texts, using crosslingual SP HIL BERTA sentence embeddings.
iv) Our experiments demonstrate that SP HIL BERTA is able to detect translations with highaccuracy and that data augmentation signifi-

Language Models for Classical Philology. Bamman and Burns (2020) and Mercelis and Keersmaekers (2022) introduced Latin BERT and
ELECTRA models, respectively. For Ancient
Greek, Singh et al. (2021) and Yamshchikov et al.
(2022) provided BERT models, initialized from
Modern Greek BERT and subsequently trainedon Ancient Greek data. Similarly, the UGARIT
project has successfully explored the usage of the
XLM-R model (Conneau et al., 2020) for Ancient
Greek and Latin texts (Yousef et al., 2022a,b), even2
https://tesserae.caset.buffalo.edu/b
log/about-tesserae/.

31though XLM-R has not been pre-trained on Ancient Greek texts. Recently, Riemenschneider andFrank (2023) have complemented the encoder-only
landscape with encoder-decoder models and developed trilingual models using Ancient Greek, Latin,
and English texts. Moreover, Kostkan et al. (2023)
and Burns (2023) have developed odyCy and latinCy, respectively, as dedicated spaCy pipelines3
for Ancient Greek and Latin.

the English teacher model, we build on the MPN ET model of Song et al. (2020), an encoder-onlymodel that has been pre-trained using a combination of masked language modeling and permuted
language modeling. Specifically, we use different
sentence transformer variants induced from MPN ET, as provided by the SBERT library (Reimers
and Gurevych, 2019). For the student model, we
experiment with initializing it from different multilingual models: XLM-R (Conneau et al., 2020), a
multilingual model based on RO BERTA that covers 100 languages, including Modern Greek and
Latin, in contrast to P HIL BERTA (Riemenschneider and Frank, 2023), a recent trilingual model thathas been pre-trained on Ancient Greek, Latin, andEnglish texts.
More detail about our models and the specificexperimental setup is provided in Section 5.

SBERT Embeddings. Reimers and Gurevych(2019) have shown that vanilla BERT embeddings
are not suitable for creating sentence embeddings,
and instead proposed the S(entence)-BERT models, which are based on Siamese and triplet network structures. Building on their work, Reimers
and Gurevych (2020) introduced a method to learn
multilingual sentence embeddings via multilingual knowledge distillation. This method realizes
knowledge transfer from a monolingual teacher
model to a student model, by training the student
model to align the original sentence and its translation to the same location in the embedding space.

3

4

As outlined in Section 3, the knowledge distillation
method of Reimers and Gurevych (2020) cruciallydepends on the availability of parallel sentences
between the relevant source and target languages –
here, the source language English for the teacher
model, and English, Ancient Greek, and Latin forour student model.
We collect this data from various sources: fromthe Perseus Digital Library,4 from parallel Bible
data,5 parallel English-to-Greek sentences from the
OPUS corpus (Tiedemann, 2012), and an extensive
collection of parallel English and Latin sentences
available on the Huggingface Hub.6 We refer tothe latter dataset as “Rosenthal”, named after its
associated account.
The Perseus project features a large collection ofAncient Greek and Latin texts, many of them withcorresponding translations. However, the alignment of the provided data is not always fine-grained
enough for our purpose. Therefore, we align individual lines with their corresponding translation,and discard lines that we cannot align successfully.
To generate additional parallel data for enhanced
knowledge transfer, we experiment with translating the English portions of the Rosenthal dataset,

Methodology

We closely follow Reimers and Gurevych’s (2020)
multilingual knowledge distillation recipe. Theirmethod requires a monolingual teacher model M
and parallel sentences in the given source language
and the target language(s) ((s1 , t1 ), ..., (sn , tn )).
The teacher trains a student model M̂ such thatM̂ (si ) ≈ M (si ) and M̂ (ti ) ≈ M (si ). For a givenmini-batch B, the mean-squared loss is minimized:
1 P
j∈B
|B|

h

i
(M (sj ) − M̂ (sj ))2 + (M (sj ) − M̂ (tj ))2 .

In other words, the student model is trained to map
a given sentence to the same vector across languages, i.e., the translation of a given sentence
should be mapped to the same vector as the source
sentence. Notably, this method is not restricted
to a bilingual setup. Instead, the student can betrained to map sentence vectors stemming frommultiple languages to the same vector, namely theone provided by the teacher model.
In our work, the teacher and student SBERTmodels to be used for cross-lingual knowledgetransfer will be initialized from strong transformer
language models for the respective languages. For
3

Parallel Data

4
https://github.com/PerseusDL/canonic
al-greekLit and https://github.com/Perseus
DL/canonical-latinLit.
5https://github.com/npedrazzini/paral
lelbibles/tree/main.
6
https://huggingface.co/datasets/grosenthal/latin_english_parallel.

https://spacy.io/.

32

Perseus
Bible
Opus
Rosenthal

English

Greek

Latin

3 743K897K
5K
3 428K

2 120K
128K
4K
2 370K†

384K
520K
—
2 095K

In order to transparently evaluate our models,
we first measure their ability to correctly detect
translations of a sentence. For each parallel dataset,
we hold out 1 000 sentences as test sets. Given aquery, i.e., the embedding of a specific sentencein the source language, we compute the cosine
similarity to the embeddings of all 1 000 sentencesin the target language.
Following Reimers and Gurevych (2020), we
measure the success of our models by determining translation accuracy: we count a translation to
be correctly identified if the model computes thehighest cosine similarity between the query and its
correct translation, and vice versa. This evaluates
the student model’s ability to align a source language sentence with an equivalent target language
sentence.
However, our primary interest is whether themodel can effectively link Ancient Greek and Latin
texts. Regrettably, the volume of parallel data
available in Ancient Greek and Latin is severely
constrained. Consequently, we utilize Bible data,
which is accessible in Ancient Greek, Latin, andEnglish. Again, we examine the model’s performance on 1 000 test sentences, given in Ancient
Greek or Latin. We ensure that the model has notencountered any of these sentences in its training
data, either in English or Latin, or in Ancient Greek.
In addition, we use the P HILTAEn→Grc -generated
Ancient Greek test set translations of the Rosenthal
corpus and compare them to their Latin originals.
We are aware that the task of identifying intertextual allusions poses a much greater challenge
than merely recognizing translations, as allusions
typically exhibit more subtlety and may extend beyond sentence or verse boundaries. However, weconsider this evaluation a transparent method forcomparing the effectiveness of different model configurations and an approximate measure to evaluate
the potential success of our models in identifying
intertextual allusions across languages.

Table 1: Dataset statistics (in number of words) of available parallel sentences across languages. The
Greek Rosenthal data marked with a dagger (†)
has been translated using P HILTAEn→Grc .7

which consists solely of English and Latin parallel data, into Ancient Greek. This required firstfine-tuning the multilingual P HILTA model7 on the
Perseus data to enable translation from English toAncient Greek. Subsequently, we used the trained
P HILTAEn→Grc model to translate the Rosenthal
dataset into Ancient Greek, thereby expanding it toa trilingual parallel dataset.
Table 1 provides the data statistics. Since partsof the corpora overlap, we deduplicate the data.

5

Experiments

Our first aim is to compare different model configurations. We test the following configurations:
We use the all• Teacher Model.
8-mpnet-base-v2
and the multi-qa-mpnet-base-dot-v19 model fromthe SBERT library (Reimers and Gurevych,2019) as teacher models. While the former isfine-tuned on a variety of tasks, the latter is
optimized for semantic search.
• Student Model. We compare the performance of XLM-R (Conneau et al., 2020) tothat of P HIL BERTA (Riemenschneider andFrank, 2023) when used as student models.
XLM-R serves as a well-established multilingual baseline.
• Data Augmentation. We evaluate whether
the automatic English-to-Greek translations
produced by P HILTAEn→Grc to extend the
Rosenthal dataset improve task performance.

Experiment Details. We train all models withthe exact same configurations. We fine-tune allmodels for 30 epochs, using a batch size of 32, theAdamW optimizer with a learning rate of 2e−5,
and 10 000 warmup-steps. The best-performing
model is selected based on the translation accuracy
derived from a total of 2 000 held-out validation
examples, comprised of 1 000 English-Greek and
1 000 English-Latin sentence pairs.

7
P HILTA (Riemenschneider and Frank, 2023) is a trilingual
encoder-decoder model based on T5 (Raffel et al., 2020) thatwas pre-trained on Ancient Greek, Latin, and English data.
8
https://huggingface.co/sentence-transformers/all-mpnet-base-v2.
9https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1.

33

TeacherStudent

all-mpnet-base-v2
all-mpnet-base-v2
multi-qa-mpnet
multi-qa-mpnet

XLM-R
P HIL BERTAP HIL BERTAP HIL BERTAP HILTAtranslations

Bible
En→La La→En

✗
✗
✗
✓

0.10
96.10
96.90
96.40

Perseus
En→La La→En

0.10
95.60
96.00
95.90

0.30
90.10
91.60
91.90

0.60
88.40
91.30
90.90

Rosenthal
En→La La→En
0.50
95.90
97.90
97.80

0.60
95.20
96.90
96.60

Table 2: Translation accuracy for various English-Latin test sets. Utilizing XLM-R as a student model leads to catastrophic results. It is crucial to substitute P HIL BERTA as the student model for successful model training.
Switching to the semantically-oriented multi-qa-mpnet from the broader all-mpnet-base-v2
provides further enhancements.
Teacher

Student

all-mpnet-base-v2
all-mpnet-base-v2
multi-qa-mpnet
multi-qa-mpnet

XLM-R
P HIL BERTAP HIL BERTAP HIL BERTAP HILTAtranslations
✗
✗
✗
✓

Bible
Perseus
Rosenthal
En→Grc Grc→En En→Grc Grc→En En→Grc† Grc† →En
0.20
96.50
97.80
98.30

0.20
96.50
97.70
98.00

0.30
89.50
89.80
91.10

0.10
87.40
88.80
90.50

0.30
93.39
92.29
96.80

0.10
92.49
86.99
94.29

Table 3: Translation accuracy for various English-Greek test sets. The Greek Rosenthal data has been translatedby P HILTA. We see the same trends as in Table 2. The enrichment of the training corpus with additional
P HILTA-translated content notably increases the performance for Ancient Greek.
Teacher

Student

all-mpnet-base-v2
all-mpnet-base-v2
multi-qa-mpnet
multi-qa-mpnet

XLM-R
P HIL BERTAP HIL BERTAP HIL BERTAP HILTA-translations
✗
✗
✗
✓

Bible
La→Grc Grc→La
0.10
96.10
96.50
96.70

0.10
95.60
96.69
96.90

Rosenthal
La→Grc† Grc† →La
0.20
83.97
84.97
92.08

0.20
83.67
82.57
91.68

Table 4: Translation accuracy for various Latin-Greek test sets. The Greek Rosenthal data has been translated by
P HILTA. We see similar trends as described in Tables 2 and 3.

34

6

Resultsels notably exhibit strong performance across bothdatasets. Again, utilizing the Greek Rosenthal dataconsiderably improves performance. These results
show that SP HIL BERTA can be efficiently utilized
in a scenario that solely involves Greek and Latin
texts, without necessitating the involvement of English texts.

We present our results for the different configurations in Tables 2 to 4. Specifically, we evaluate: i)the performance of different teacher models (themore general all-mpnet-base-v2 SBERTmodel in comparison to the multi-qa-mpnet
SBERT fine-tuned for semantic search), ii)different student models (XLM-R versus the
P HIL BERTA model), and iii) augmenting theparallel data for training SP HIL BERTA using
P HILTAEn→Grc -translated texts.
Employing XLM-R as the student model leads tocatastrophic performance. Specifically, the model
never surpasses the 1% mark in test set performance. We observed this trend consistently, regardless of the model configuration or the random
seed employed. This outcome is, to some degree,
to be expected, as XLM-R is not pre-trained on
Ancient Greek data. Still, it is surprising that
XLM-R performs so badly also on Latin data,
as its pre-training corpus contained a Latin portion. Moreover, the UGARIT project (Yousef et al.,2022a,b) has successfully adapted XLM-R to Ancient Greek. We hypothesize that the effectiveness of a broadly multilingual but unspecialized
model may be task-dependent, and continuing selfsupervised pre-training on Ancient Greek texts may
be required for XLM-R to adapt adequately. Thesefindings highlight the importance of initializing thestudent model with a model that is proficient in thetarget languages.
Initializing the student model with P HIL BERTA
yields strong performance, often surpassing 95%
translation accuracy.
Generally, employing
multi-qa-mpnet as a teacher model contributes to a slight performance improvement overall-mpnet-base-v2. Yet, when testing themodel on the Ancient Greek Rosenthal corpus,
using the multi-qa-mpnet teacher model results in a performance decline. Importantly, theGreek part of this dataset has been translated by
P HILTAEn→Grc , which could possibly have affected
the quality of the dataset. Indeed, while we see
this negative trend when testing on the generated
data, the inclusion of the P HILTA-generated Ancient Greek Rosenthal corpus as additional training data leads to a notable enhancement for the
Greek datasets, while the performance for Latintranslation retrieval remains largely unaffected.
The results for Latin-to-Greek and Greek-toLatin translations are shown in Table 4. Our mod-Case Study: The Aeneid and Homer’s
Odyssey

7

Examinations of the intertextual allusions in Virgil’s Aeneid to both the Iliad and the Odyssey havea long history, dating back to antiquity. Structurally,
the Aeneid’s initial six books mirror the narrativeof the Odyssey, while the concluding six books
correspond more closely to the Iliad.
In the second book of the Aeneid, the protagonist
Aeneas attempts to escape from the ravaged city ofTroy with his family. Tragically, his wife, Creusa,
is lost amidst the chaos. Creusa’s ghost consoles
him and bids him goodbye before receding into
thin air: “This speech uttered, while I wept andwould have said many a thing, she left me andretreated into thin air. Thrice there was I fain tolay mine arms round her neck; thrice the vision I
vainly clasped fled out of my hands, even as thelight breezes, or most like to fluttering sleep.”10
These verses mirror closely a scene in the Nekyiaof the Odyssey, where Odysseus meets his mother
Anticleia in the underworld: “So she spoke, and I
pondered in heart, and was fain to clasp the spiritof my dead mother. Thrice I sprang towards her,and my heart bade me clasp her, and thrice she
flitted from my arms like a shadow or a dream, andpain grew ever sharper at my heart.”11
To evaluate our model’s proficiency in identifying these intertextual allusions, we employ eachverse of the Aeneid passage (i.e., 5 verses) as aquery, which we then compare to the verse embeddings (approx. 11 000 verses) of the complete
Odyssey. Table 5 shows the three highest resultsfor each verse, according to our best-performing
model setup (teacher: multi-qa-mpnet; student: P HIL BERTA; additional P HILTA-generated
Rosenthal data).
We note that these verses do not share a direct
one-to-one relationship and they are not translations of each other, the scenario in which our model
10

Virgil, Aeneid, 2.790–794, translated by Mackail (1885).
Homer, Odyssey, 11.204–208, translated by Murray(1919).
1135

QueryResults

Haec ubi dicta dedit, lacrimantem et multa volentem
This speech uttered, while I wept and would have said many a thing,τῆς δ᾿ ἄρ᾿ ἀκουούσης ῥέε δάκρυα , τήκετο δὲ χρώς·and as she listened her tears flowed and her face meltedὣς φάτο , τῆς δ᾿ εὔνησε γόον , σχέθε δ᾿ ὄσσε γόοιο .
So she spoke , and lulled Penelope’s laments , and made her eyes to cease from weeping .
ὣς φάτο , τῇ δ᾿ ἄρα θυμὸν ἐνὶ στήθεσσιν ὄρινε .
So he spoke , and stirred the heart in her breast.

dicere deseruit , tenuisque recessit in auras .
[...said] , she left me and retreated into thin air .

ἡ μὲν ἄρ᾿ ὣς ἔρξασ᾿ ἀπεβήσετο δῖα θεάων ,Now when she had done this the fair goddess departed ,ἡ μὲν ἄρ᾿ ὣς εἰποῦσ᾿ ἀπέβη πρὸς δώματα καλά ,So saying , she departed to the fair palace .
ἡ μὲν ἄρ᾿ ἐς κρήνην κατεβήσετο καλλιρέεθρον[She] had come down to the fair-flowing spring [Artacia],

Ter conatus ibi collo dare bracchia circum:
Thrice there was I fain to lay mine arms round her neck;ὄπτ᾿ ἐν χερσὶν ἑλών, τά ῥά οἱ γέρα πάρθεσαν αὐτῷ.
he took in his hands roast meat and set it before them, [...] which they had set before himself as amess of honor.
τρὶς μέν μιν πελέμιξεν ἐρύσσεσθαι μενεαίνων ,Thrice he made it quiver in his eagerness to draw it,αὐτίκ᾿ ἔπειτα τρίαιναν ἑλὼν χερσὶ στιβαρῇσινstraightway took his trident in his mighty hands ,ter frustra comprensa manus effugit imago ,τρὶς δέ μοι ἐκ χειρῶν σκιῇ εἴκελον ἢ καὶ ὀνείρῳthrice the vision I vainly clasped fled out of my hands ,and thrice [she flitted] from my arms like a shadow or a dream ,τρὶς μὲν ἐφωρμήθην, ἑλέειν τέ με θυμὸς ἀνώγει,Thrice I sprang towards her, and my heart bade me clasp her,χερσὶ δὲ μή τι λίην προκαλίζεο, μή με χολώσῃς,
But with thy [hands] do not provoke me overmuch,par levibus ventis volucrique simillima somno .

ἡ δ᾿ ἔθεεν Βορέῃ ἀνέμῳ ἀκραέϊ καλῷ,even as the light breezes , or most like to fluttering sleep .

And she ran before the North Wind , blowing fresh and fair ,ὄρσας ἀργαλέων ἀνέμων ἀμέγαρτον ἀυτμήν ,when he had roused a furious blast of cruel windsἐς πνοιὰς ἀνέμων . ἡ δ᾿ ἐξ ὕπνου ἀνόρουσε
into the breath of the winds . And [she] started up from sleep

Table 5: Top 3 predictions of our best-performing SP HIL BERTA model (teacher: multi-qa-mpnet; student:
P HIL BERTA; additional P HILTA-generated Rosenthal data) when queried over the whole Odyssey. We
mark corresponding cross-lingual concept pairs with individual colors.

36

Acknowledgements

was trained. Even so, we observe that verse 793(“thrice the vision I vainly clasped fled out of myhands”) is correctly paired with the best corresponding Greek verse (“and thrice she flitted frommy arms like a shadow or a dream”). In the majority of cases, our model accurately captures crucial
concepts, such as weeping, departing, triplicity,
wind, and sleep, linking them reasonably to different verses. However, our verse-to-verse mapping,which precludes longer texts, results in a lack of acohesive concept of extended intertextually alluding passages.
Still, our case study demonstrates the proficiencyof our models in recognizing sentence structures
and translating them to a different language (as in“this speech uttered” → “so she spoke”), and in
identifying common topics or concepts across languages, even locating verses where multiple relevant concepts exist within the same verse (“thrice”,“the vision”, “out of my hands” → “thrice”, “ashadow or a dream”, “from my arms”).
Despite these remarkable results, our case study
also reveals the need for a more sophisticatedretrieval mechanism that extends beyond verse
boundaries to search for broader patterns. Yet, already in the present state, our SP HIL BERTA model
can serve as a useful tool for automatic first-pass
exploration of potential cross-lingual intertextual
allusions, and in this way can support philologists
in the search for intertextual allusions.

8

We thank Nina Stahl for the insightful discussions
we shared on the Greek and Latin parallel passages.

References
María Jesús Martínez Alfaro. 1996. Intertextuality:
Origins and development of the concept. Atlantis,
18(1/2):268–285.
David Bamman and Patrick J Burns. 2020. Latin bert:
A contextual language model for classical philology.
arXiv preprint arXiv:2009.10053.
Andreas Bendlin. 2006. Intertextuality. http://dx.doi.org.ubproxy.ub.uni-heidelberg.de/10.1163/1574-9347_bnp_e525570.
Accessed: 28 June 2023.
Yuri Bizzoni, Federico Boschetti, Harry Diakoff, Riccardo Del Gratta, Monica Monachini, and Gregory
Crane. 2014. The making of Ancient Greek WordNet.
In Proceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC’14),
pages 1140–1147, Reykjavik, Iceland. European Language Resources Association (ELRA).
Patrick J Burns. 2023.
Latincy:
Synthetic
trained pipelines for latin nlp. arXiv preprint
arXiv:2305.04365.
Patrick J. Burns, James A. Brofos, Kyle Li, Pramit
Chaudhuri, and Joseph P. Dexter. 2021. Profilingof intertextuality in Latin literature using word embeddings. In Proceedings of the 2021 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pages 4900–4907, Online. Association for
Computational Linguistics.

Conclusion

We introduce SP HIL BERTA, a multilingual
P HIL BERTA-derived sentence transformer model,
specifically adapted to Classical Philology. Ourmodel represents a pioneering effort in detecting
intertextual allusions between Ancient Greek and
Latin texts, which is characteristic of many Roman
writers who used Greek literature for juxtaposition.
SP HIL BERTA displays impressive performanceacross various datasets, confidently identifying direct translations among English, Latin, and Ancient
Greek. We have illustrated that SP HIL BERTA
holds strong potential in revealing intertextual allusions; however, additional research is needed to
fully exploit the model’s capabilities. Our multilingual SP HIL BERTA and the similarity-driven retrieval settings built upon it offer, for the first time,
the option to study intertextuality cross-linguallyon a broader scale in original Classical Literature.

Neil Coffee, Jean-Pierre Koenig, Shakthi Poornima,
Christopher W Forstall, Roelant Ossewaarde, andSarah L Jacobson. 2012. The tesserae project: intertextual analysis of latin poetry. Literary and linguistic computing, 28(2):221–228.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–
8451, Online. Association for Computational Linguistics.
Christopher Forstall, Neil Coffee, Thomas Buck, Katherine Roache, and Sarah Jacobson. 2014. Modeling thescholars: Detecting intertextuality through enhanced
word-level n-gram matching. Digital Scholarship inthe Humanities, 30(4):503–515.

37

G. O. Hutchinson. 2013. Greek to Latin: Frameworks
and Contexts for Intertextuality. Oxford University
Press.

Frederick Riemenschneider and Anette Frank. 2023. Exploring large language models for classical philology.
In Proceedings of the 61st Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 15181–15199, Toronto, Canada.
Association for Computational Linguistics.

Jan Kostkan, Márton Kardos, Jacob Palle Bliddal
Mortensen, and Kristoffer Laigaard Nielbo. 2023.
OdyCy – a general-purpose NLP pipeline for Ancient
Greek. In Proceedings of the 7th Joint SIGHUM
Workshop on Computational Linguistics for Cultural
Heritage, Social Sciences, Humanities and Literature,pages 128–134, Dubrovnik, Croatia. Association for
Computational Linguistics.

Walter Scheirer, Christopher Forstall, and Neil Coffee.
2014. The sense of a connection: Automatic tracingof intertextuality by meaning. Digital Scholarship inthe Humanities, 31(1):204–217.
Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021.
A pilot study for BERT language modelling and morphological analysis for ancient and medieval Greek.
In Proceedings of the 5th Joint SIGHUM Workshop
on Computational Linguistics for Cultural Heritage,
Social Sciences, Humanities and Literature, pages
128–137, Punta Cana, Dominican Republic (online).
Association for Computational Linguistics.

Julia Kristeva. 1986. Word, dialogue, and novel. In The
Kristeva reader, pages 34–61. Columbia University
Press.
John William Mackail. 1885. The Aeneid of Virgil,volume 36. Macmillan.
Enrique Manjavacas, Brian Long, and Mike Kestemont.
2019. On the feasibility of automated detection ofallusive text reuse. In Proceedings of the 3rd JointSIGHUM Workshop on Computational Linguistics
for Cultural Heritage, Social Sciences, Humanities
and Literature, pages 104–114, Minneapolis, USA.
Association for Computational Linguistics.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. arXiv preprint
arXiv:2004.09297.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and
Evaluation (LREC’12), pages 2214–2218, Istanbul,
Turkey. European Language Resources Association(ELRA).

Wouter Mercelis and Alek Keersmaekers. 2022. An
ELECTRA model for Latin token tagging tasks. In
Proceedings of the Second Workshop on Language
Technologies for Historical and Ancient Languages,
pages 189–192, Marseille, France. European Language Resources Association.

Ivan Yamshchikov, Alexey Tikhonov, Yorgos Pantis,
Charlotte Schubert, and Jürgen Jost. 2022. BERT in
plutarch’s shadows. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language
Processing, pages 6071–6080, Abu Dhabi, United
Arab Emirates. Association for Computational Linguistics.

A. T. Murray. 1919. Homer. The Odyssey with an English Translation. Harvard University Press, London.
Mary Orr. 2003. Intertextuality: Debates and contexts.
Polity Press.

Tariq Yousef, Chiara Palladino, Farnoosh Shamsian,Anise d’Orange Ferreira, and Michel Ferreira dos
Reis. 2022a. An automatic model and gold standard
for translation alignment of Ancient Greek. In Proceedings of the Thirteenth Language Resources and
Evaluation Conference, pages 5894–5905, Marseille,
France. European Language Resources Association.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. 2020. Exploring thelimits of transfer learning with a unified text-to-texttransformer. Journal of Machine Learning Research,
21(140):1–67.

Tariq Yousef, Chiara Palladino, David J. Wright, andMonica Berti. 2022b. Automatic translation alignment for Ancient Greek and Latin. In Proceedings of
the Second Workshop on Language Technologies for
Historical and Ancient Languages, pages 101–107,
Marseille, France. European Language Resources
Association.

Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
3982–3992, Hong Kong, China. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making
monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 4512–4525,
Online. Association for Computational Linguistics.

38

Larth: Dataset and Machine Translation for Etruscan
Gianluca Vico and Gerasimos Spanakis
Department of Advanced Computing Sciences / Paul-Henri Spaaklaan 1
Maastricht University / Maastricht, The Netherlands
g.vico@student.maastrichtuniversity.nl
jerry.spanakis@maastrichtuniversity.nl

Abstract2008). However, only a few of them have a significant length to be considered complete. Other
ancient languages used in similar areas and periods
in history, such as Latin and Ancient Greek, havemore resources, thus, making natural language processing techniques and tools easier to develop forthese languages.
The contribution of this paper is threefold: First,
we build a corpus of Etruscan inscriptions usable
for natural language processing. We use as a starting point existing academic resources for this language exist, and we try to create our corpus bothby manual and automatic work. Second, we focus
on the machine translation task from Etruscan toEnglish. We evaluate whether neural models canbe trained with this data and if they can outperform
less data-hungry models. Finally, we investigate
if it is possible to exploit any similarity between
Etruscan and Latin or Ancient Greek to improve
the aforementioned model.
In Section 2, we introduce state-of-the-art techniques relevant to this paper. Then, in Sections 3
and 4 we explain the methods used to work on thedata and the model used. Section 5 and Section 6
illustrate the experiments and compare the different
techniques. Finally, Section 7 concludes the paper.

Etruscan is an ancient language spoken in Italy
from the 7th century BC to the 1st century AD.
There are no native speakers of the language at
the present day, and its resources are scarce, as
there exist only around 12,000 known inscriptions. To the best of our knowledge, there areno publicly available Etruscan corpora for natural language processing. Therefore, we propose
a dataset for machine translation from Etruscan
to English, which contains 2891 translated examples from existing academic sources. Some
examples are extracted manually, while others
are acquired in an automatic way. Along withthe dataset, we benchmark different machinetranslation models observing that it is possible to achieve a BLEU score of 10.1 with asmall transformer model. Releasing the dataset
1
can help enable future research on this language, similar languages or other languages
with scarce resources.

1

Introduction

Etruscan (ISO 639-3 code: ett) is a language
spoken in the Etruria region (modern-day centre
Italy) from the 7th century BC to the 1st century AD(Wallace, 2008). It is written right to left using theEtruscan alphabet, derived from the Greek alphabet(Wallace, 2008). The predominant word order inthis language is mostly subject-object-verb (Wallace, 2008). This pattern is similar to Latin, but
distinguishing it from other languages like English,
where the words follows the subject-verb-object
order. It has 5 cases (accusative, nominative, genitive, dative and locative), two numbers (singularand plural) and takes into consideration animacy
and gender (Wallace, 2008).
Only a small number of inscriptions in this language survived up to the present day: an estimated
12,000 inscriptions have been recovered (Wallace,
1

The
data
and
code
are
available
https://github.com/GianlucaVico/
Larth-Etruscan-NLP.git

2

Literature review

The Etruscan Texts Project (ETP) (Wallace et al.,2004) is a digital Etruscan corpus which contains 369 inscriptions. The project is based on
Etruskische Texte (Rix and Meiser, 1991) andis used in the book Zihk Rasna (Wallace, 2008).
Another digital Etruscan work is the Corpus Inscripionum Etruscarum Plenissimum (CIEP) (Hill,
2018), based on the Corpus Inscriptionum Etruscarum (CIE) (Pauli, 1893).
Similar works exist for Latin and Ancient Greek,
like I.PHI (Sommerschield et al., 2021) and Perseus(Crane, 1985). In addition, toolkits like CLTK(Johnson et al., 2021) offer natural language pro-

here:

39
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 39–48,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_005

cessing for these languages. Projects that aim to increase the resources available for low-resource languages may also include ancient languages, like the
Tatoeba Translation Challenge (Tiedemann, 2020).
It has Latin and Ancient Greek datasets, however,it does not include Etruscan.

and BLEU are based on word n-grams, while chrF (Popović, 2015) uses the F-score of matching
character n-grams.

The machine translation task can be solved via
neural machine translation (Sutskever et al., 2014a),which involves training neural networks that taketexts from the source language and generate thetranslation in the target language. Popular architectures include Long short-term memory (LSTM)(Hochreiter and Schmidhuber, 1997) and transformers (Vaswani et al., 2017). These models
are sequence-to-sequence (Sutskever et al., 2014b),
meaning they take a sequence as input and generatea sequence of possibly different lengths as output.
One approach is to feed word or word pieces to themodel like in T5 (Raffel et al., 2020) or Bahdanau
et al. (2014). Yang et al. (2016) and Ling et al.
(2015) show that it is possible to work directly on
characters, while other models (Shahih and Purwarianti, 2019 and Bansal and Lobiyal, 2020) use ahybrid approach by working on both the character
and word sequences.

First, we collect a dataset containing Etruscan texts.
The main sources used are CIEP (Hill, 2018), ETP(Wallace et al., 2004), and the book "Zikh Rasna: A
Manual of the Etruscan Language and Inscriptions"(Wallace, 2008), which cites "Etruskische Texte"(Rix and Meiser, 1991). It is possible to extract
Etruscan inscriptions and their translations whereavailable from ETP and Zikh Rasna. In addition,
we extract the date and location of the inscriptions.
Also, Zikh Rasna contains a list of Etruscan words
and proper names used to make a dictionary. From
CIEP, we extract only the inscriptions and the translations. However, the inscriptions are often incomplete or noisy due to the structure of CIEP itself
and the limitation of the PDF extracting software(PyMuPDF, McKie and Liu, 2016). We make twodatasets. The first, ETP, uses data from ETP and
ZIkh Rasna, while the second ETP+CIEP, adds
the data from CIEP.
After removing strings that are in the wrong language, the text is normalised. CIEP and ETP use
two different transcription conventions. Also, Etruscan uses several symbols as word separators (".
", "·", ":", ".."), which are converted to white space

Besides neural networks, other approaches include rule-based models, such as dictionary models, which translate the text based on explicit rules,
and statistical models (Koehn, 2010).
By using the transformer architecture, Ithaca(Assael et al., 2022) is able to perform textual
restoration and geographical and chronological attribution of ancient Greek inscriptions. The model
consists of a sparse self-attention encoder (Zaheer
et al., 2021) that takes as input the characters andthe words of the input text, and then three feedforward blocks generate the output for each task.
Other examples of transformer models working on
ancient languages are the multi-language translation model Opus-MT (Tiedemann and Thottingal,
2020), tested on the Latin → English split of the
Tatoeba dataset, or the language model Latin-BERT(Bamman and Burns, 2020).

3

Data

3.1

Etruscan(" "). Table 1 illustrates how the Etruscan alphabet
is transcribed by ETP and by us (Larth). Note thatthe transcription is not reversible.
In the end, we obtain 7139 Etruscan texts (561from ETP and 6578 from CIEP). Among these, atranslation is available for only 2891 inscriptions(239 from ETP and 2652 from CIEP). Also, thevocabulary built from ETP contains 1122 words,of which 956 with a translation. Each word isalso described by 54 binary grammatical features(e.g., plural, active, passive, ...). The type of text is
not included in the dataset, however, ETP lists on
their website mostly proprietary and funerary texts(Wallace et al., 2004) (137 and 104 out of 369).
Since the data is limited, we perform data augmentation. Many inscriptions contain proper nouns,
so we use the dictionary we built to replace themwith other proper nouns with the same grammaticalfeatures. The substitution is done simultaneouslyon the Etruscan and English texts in order to keep
the translations correct, as shown in Figure 1. Also,

Translation models can be evaluated by using
various metrics. Papineni et al. (2002) proposes
BLEU: this metric considers the average matching
precision of n-grams between the reference text andthe machine-translated text. Another metric is TER(Snover et al., 2006), which measures the quality ofthe translation based on the number of edits needed
to change the system text to the reference one. TER
40

Etruscan
a
b
C
d
e
F
z
h
T
i
k
l
m
n
⊞
o
S
p
q
r
s
t
V
X
f
P
v

ETP
a
b
c
d
e
v
z
h
T
i
k
l
m
n
+
s
o
σ, σ́
p
q
r
s, ś, ς, ς́
t
u
‰s
F
X
f

Larth
a
b
c
d
e
v
z
h
th
i
k
l
m
n
s
o
s, sh
p
q
r
s, sh, s, sh
t
u
sh
ph
kh
f

ETT: larisal • clan
ENG: son of laris

ETT: arnthal • clan

ENG: son of lartth

ENG: son of arnth...
Figure 1: Example of data augmentation by replacing
proper names. The name is replaced both in the Etruscan
text and the English translation.

1985). In this case, we also remove all diacriticalmarks and transliterate the text to Latin. In this way,
all the languages used share the same alphabet.

4

Machine Translation

We compare different models for machine translation on the BLEU metric but chr-F and TER metrics are also reported. The metrics are computed by
SacreBLEU (Post, 2018). Higher BLEU and chr-F
and lower TER indicate a better-performing model.
Moreover, we evaluate the case where we use only
ETP and ETP+CIEP for training and testing themodels.

Table 1: Texts from ETP are already transliterated, but
CIEP transliteration is sometimes ambiguous. We further reduce the number of symbols by using a subset of
the Latin alphabet.

4.1

Random Model

The output of this model does not depend on theEtruscan inputs, but only on the training translations. It assumes that the length of the translations
follows a normal distribution whose parameters are
estimated from the training data. Then, it samples
English tokens from the training distribution. The
experiment is repeated 10 times with random splitsof the dataset in training and testing data. The
resulting metrics are then averaged.

inscriptions can be damaged, so parts of the words
cannot be read and the translation models have toeither discard those words or rely only on the remaining characters. So, we generate more training
samples by damaging more words. We assume
that the damage occurs at the beginning or end of
the words with a set probability. Also, we assume
the number of damaged characters follows a geometric distribution. In this way, for instance, theword "clan" can stay unchanged or it might become"–an", "cla-", "-l–".
3.2

ETT: larthal • clan

4.2

Dictionary-based Model

The second model is a dictionary-based model
based on the vocabulary provided in Zihk Rasna(Wallace, 2008). The model assumes that eachword has one meaning and one translation. Moreover, it does not rearrange the word order and it
does not consider the grammar of the source language or the target language. This model splits the
input text into word tokens. Then, for each token, it
searches for the exact match in the dictionary. If a

Latin and Ancient Greek

Models introduced later in the paper use Latin or
Ancient Greek documents. Tatoeba eng-lat (Tiedemann, 2020) is used to train the Latin model. The
text is normalised and non-Latin characters are removed. For Ancient Greek, we use Perseus (Crane,41Eng.:

Ett.:

<pad>

<pad>

<pad>

<pad>

<eng1><ett1><eng2><ett2>

Figure 2: The first approach for the n-gram model.
<eng n> indicates English tokens, while <et n>are Etruscan tokens; <pad> is the padding token.
The example shows P (<eng1>|<pad><pad><et1>) andP (<eng2>|<pad><et1><et2>). The context is made up
of Etruscan trigrams.

<pad><eng1><eng2>

Ett.:

<pad>

<pad>

<ett1><ett2>

not contain this information. Therefore, we test themodels as if the sequences were aligned.
IBM1 does not consider the word order. IBM2
introduces the word order, while IBM3 takes also
into consideration that a word can be translated
into zero or more words. IBM4 and IBM5 can also
reorder the output words. Moreover, IBM4 andIBM5 also need the part-of-speech (POS) tags ofboth the source and target sequences. POS tags areinferred from the grammatical features listed in the
dictionary. For Etruscan, these are obtained by amanually annotated list of words, while the English
sequences are tagged by NLTK perceptron tagger.

N-gram and Naïve Bayes Models

Then, we try to translate Etruscan taking into
consideration the previous n tokens.
Themodel estimates the probability distribution
P(engi |etti , etti−1 , ...etti−n ), where engi and etti
are tokens at position i. This is done either directlyfrom the training data or as a Naïve Bayes modelwith the following expression:P(engi |etti , etti−1 , ...etti−n ) ∝nY∝ P(engi )P(etti−j |engi ) (1)

4.5

j=0

The model assumes that one nth Etruscan token istranslated into the single nth English token. Figure
2 shows how the sequences are aligned and which
Etruscan context is used for each English token.
A second N-gram model also includes theprevious English tokens in the context by computing P(engi |etti , ...etti−n , engi−1 , ..., engi−n−1 )as shown in Figure 3. When the probability
distribution is estimated directly, we consider thecase when the word order is taken into account andwhen it is not. We use beam search to generate theoutput.
4.4<pad>

Figure 3: the second approach for the n-gram model.
<eng n> indicates English tokens, while <ett n> are
Etruscan tokens; <pad> is the padding token. The
example shows P (<eng1>|<pad><ett1>, <pad><pad>)and P (<eng2>|<ett1><ett2>, <pad><eng1>). The context is made up of Etruscan and English bigrams.

match is found, it adds the translation to the output;
otherwise, the token is ignored.
4.3

Eng.:

IBM Models

Next, we compare our models to existing ones. To
do so, we consider the IBM models (Koehn, 2010)
from the NLTK package (Bird and Loper, 2004).
They are a series of 5 models with increasing complexity. These models consider the alignment between the source strings and the target strings, however, the Etruscan-English pairs we are using do
42Transformer Models - LarthFinally, we propose a transformer model, Larth.
The encoder is based on Ithaca (Assael et al., 2022).
It takes both the characters and the words as input
and concatenates their embeddings. Then, the sequence is encoded with a BigBird attention block(Zaheer et al., 2021). The character and word sequences are aligned so that they have the samelength. To do so, we test two approaches: we either
extend the word sequence by repeating the word tokens or by adding space tokens as shown in Figure
4.
The decoder uses the encoded and the translated word sequences as input. First, it applies
self-attention to the translated sequence, and thenit computes the cross-attention between the translation and the encoded inputs. A feed-forward layer
generates the output. Figure 5 illustrates this architecture.
First, we train the model from scratch on Etruscan → English. Then, the model is initiallytrained for Latin → English or Ancient Greek →Repeated word tokens

LogitsChar: <v> <i> <n> <u><m><_> <t> <h> <i> <c>Linear

Word: <vinum><vinum><vinum><vinum><vinum> <_> <thic> <thic> <thic> <thic>

Layer
Normalisation

Space tokens
Char: <v> <i> <n> <u><m><_> <t> <h> <i> <c>

Layer
Normalisation

Word: <vinum> <_> <_> <_> <_> <_> <thic> <_> <_> <_>xN

Figure 4: Example of how the character and word sequence are aligned. The string vinum thic means wine
and water.

Dataset
ETP+
CIEP
ETP

BLEU
0.059
(0.0174)
0.324(0.064)

chr-F
9.263
(0.295)13.970(1.150)

BigBirdSelf Attention

Positional Concatenate
Embedding++TER
194.977(10.676)133.878(11.877)

Char
Embedding

Word
Embedding

K

Q

Layer
NormalisationxN
Self Attention+

Output Words
Decoder

Figure 5: Transformer architecture used to translate
Etruscan to English. The encoder imitates Ithaca’s torso.
For both the encoder and the decoder, one attention
block is used.

DatasetETP+CIEPETPCIEPETP (Suffix)

BLEU
0.167
4.505
0.000
1.605

chr-F
9.120
40.771
1.896
37.669TER
89.79968.13598.672
82.666

Table 3: Results of the dictionary-based model whentested on the different sets. ETP (Suffix) is the model
tested on ETP with the suffix tokenizer.

Experiments

In this section, we compare different machine translation models trained on Etruscan data. The models
are compared on the BLEU score.

lations.
We compare two tokenizers for Etruscan: thefirst uses white spaces to split the tokens, and thesecond also separates the suffixes from the root.
The list of suffixes is also obtained from Zikh Rasna
and the tokenizer recognises 178 suffixes. Table
3 shows the results of this model when translating
Etruscan.
If we consider the example "itun turuce venel
atelinas tinas dlniiaras" with the reference translation "venel atelinas dedicated this vase to the sonsof tinia", this model predicts "this dedicated venel
atelina tinia". If we use the suffix tokenizer the prediction is "this for him dedicated three this venel
laris atelina shows".

Random ModelFirst, we run the random model on the EtruscanEnglih data. The dataset is split into 80 % fortraining and 20 % for testing. Only English labels
are used for the training. Each experiment is repeated 10 times with random dataset splits. Table 2reports the mean scores and the standard deviationof the models with different combinations of thedatasets.
5.2

Cross
AttentionV

Output Word
Embedding

English and later fine-tuned on the original task
Etruscan → English.
Moreover, we investigate the effect of using boththe character and the word sequence by training
with only one of the sequences and the effect ofdata augmentation The model uses beam searchwhen generating the output sequences, but we use
one beam when evaluating during the training for
efficiency. Sequences are truncated at 256 tokens
due to memory and computational resources.

5.1

Layer
Normalisation

Char
Word
Sequence
Sequence
EncoderTable 2: Performance of the random model on the different Etruscan datasets. The table reports the mean value
and the standard deviation of the metrics.

5

Feed Forward

Dictionary-based Models

From the book Zikh Rasna is possible to build a
dictionary containing 821 vocables and their trans43

Context: ETT - Word order: No
N-gram BLEU chr-F
TER
0.406
7.727
92.6051(0.163) (0.867) (0.960)
0.006
3.249
98.0352(0.001) (0.752) (0.821)
0.001
2.523
98.5533(0.001) (0.753) (0.821)
Context: ETT - Word order: Yes
N-gram BLEU chr-F
TER
0.405
7.727
92.6051(0.163) (0.867) (0.960)
0.005
3.211
98.0132(0.005) (1.004) (1.089)
0.001
2.523
98.531
3(0.001) (0.748) (0.870)

Context: ETT-ENG - Word order: No
N-gram BLEU chr-F
TER
0.218
3.059
92.9021(0.018) (0.301)(1.160)
0
0
100
2
(0)
(0)
(0)
0
0
100
3
(0)
(0)
(0)
Context: ETT-ENG - Word order: Yes
N-gram BLEU chr-F
TER
0.447
5.360
92.1051(0.211) (0.856)(1.117)0.000
0.370
99.7052(0.000) (0.167)(0.346)0.000
0.357
99.6903(0.000) (0.097)(0.297)

Table 4: Mean scores and their standard deviation (inparenthesis) of the n-gram models that use only theEtruscan texts.

Table 5: Mean scores and their standard deviation (inparenthesis) of the n-gram models that use the Etruscan
texts and the English translations. When the scores are
zero is because the models immediately predict the endof-sequence (EOS) token.

5.3

N-gram and Naive Bayes models

Similarly to the random models, 80% of the data
is used for training, while the remaining 20% is
for testing. The dataset is ETP. Each experiment isrepeated 10 times with different random splits.
With the N-gram models, we compare models
with a context size of 1, 2 and 3 that use only
Etruscan or both Etruscan and English as context
and whether they consider the word order. Out-ofvocabulary (OOV) tokens are handled with additive smoothing. We use 8 beams when generating
the output sequence, however, this is equivalent togreedy search when the context uses only Etruscan.
Table 4 shows the results of the models that use
only the Etruscan sequence, while Table 5 shows
the models that also use the English translations.
For the Naive Bayes models, we only use a context size of 2 and 3, and the models always consider
the word order. Table 6 reports the results.
5.4

NContext

2

Ett.

3

Ett.

2

Ett.-Eng.

3

Ett.-Eng.

BLEU
0.160
(0.023)0.146(0.030)0.055(0.048)0.055(0.048)chr-F
12.609(1.009)12.708(0.921)9.547(1.821)9.954(2.103)TER
101.482(1.251)103.867(1.220)101.522(0.851)103.038(1.005)

Table 6: Mean scores and their standard deviation (inparenthesis) of the Naïve Bayes models.

ETP are tested on ETP as shown in Tables 7 and 8.
As an example, IBM3 translates "eca shuthic
velus ezpus clensi cerine" as "this funerary vel etspus son constructed", while the reference translation is "this funerary monument belongs to vel
etspu it is constructed by his son".

IBM models

5.5

We split 80 % of the data for training and 20 %for testing. Moreover, we use the previously built
dictionary as training data. No alignment information is given to the model, but IBM4 and IBM5
receive a dictionary that maps words to POS tags.
We assume that words can only have one tag.
IBM3, IBM4, and IBM5 are trained only withthe dictionary data. Models trained on ETP+CIEP
are tested on ETP+CIEP, while models trained on

Transformer Models - LarthThe model is trained for Etruscan → English translation with ETP+CIEP and with ETP only. The
models are tested on the same split of the dataset.
Due to the small size of the dataset, 95 % of thedata is used for training.
The optimizer is RAdam (Liu et al., 2019), withan initial learning rate of 0.002 and 250 warmup
steps. We use a reverse square root learning sched44

ModelIBM1
IBM2
IBM3(*)
IBM4(*)
IBM5(*)ETP+CIEPBLEU chr-F
0.402
19.744(0.183) (1.178)
0.392
19.450(0.183) (1.383)
0.105
8.629
(0.046) (1.148)
0.105
8.627
(0.046) (1.148)
0.105
8.631
(0.046) (1.147)

TER
89.213(0.693)89.551(0.487)91.052(1.507)91.052(1.507)91.063(1.516)Model
repeatspace
repeat+unk
repeat+name
Modelrepeat
space

Model
IBM1
IBM2
IBM3(*)
IBM4(*)
IBM5(*)

TER
144.5
274.8
189.1
615.9
TER
137
124.7

Table 9: Larth trained from scratch for Etruscan →English. Repeat and space indicate how the characterand the word sequence are aligned. +name is trainedwith data augmented by changing names, while +unk isaugmented by deleting characters.

Table 7: Performance of the IBM models on the
ETP+CIEP dataset. (*): IBM3, IBM4 and IBM5 aretrained only with the dictionary.

ETP
BLEU chr-F
2.187
37.363(0.596) (2.011)
2.104
36.721(0.449) (2.098)
2.482
39.393(0.513) (2.229)
2.482
39.391(0.514) (2.228)
2.481
39.416(0.513) (2.235)

ETP+CIEPBLEU chr-F
10.1
15.11
5.201
16.9
2.8
14.8
1.004
12.2
ETP
BLEU chr-F
9.053 17.24
5.784 15.88

ETP+CIEP
InputsBLEU chr-F
char0.9694 14.42
word
2.776 13.49
char+word
10.1
15.11
ETP
InputsBLEU chr-F
char
0.1431 11.22
word
7.679 18.48
char+word 9.053 17.24

TER
73.917(2.163)74.334(2.090)71.270(2.456)71.270(2.456)71.331(2.415)

TER
254.8
99.88
144.5
TER
528.1
131.6
137

Table 10: Larth trained from scratch for Etruscan →English with only the character or the word sequence orboth as input.

Table 8: Performance of the IBM models on the ETP
dataset. (*): IBM3, IBM4 and IBM5 are trained onlywith the dictionary.

tienas" as "i am the tomb" while the reference translation is "i am the tomb of avele metienas". Notethat in this example "the tomb" is implied and not
mentioned explicitly.
When trained on ETP+CIEP, we have "e cashuthi anes cuclnies" translated as "this tomb" butthe reference is "this is the tomb of ane cuclnies".
In this case "the tomb" is mentioned, but the model
misses the name of the owner, which is also mentioned.

ule. The loss function is cross-entropy, and thebatch size is 32. We set the label smoothing to 0.1.
We first try to train from scratch and with different alignment techniques. The BLEU, chr-F andTER scores are shown in Table 9. We use data
augmentation with ETP+CIEP with the sequencesaligned by repeating the word tokens, however, we
do not use it on ETP due to the decrease in performance.
Next, we train the same architecture with only
the word sequence or only the character sequence.
The results are shown in Table 10.
When training the same model with the Latin andGreek data, it achieved, respectively, BLEU/chrF/TER of 0.4968/5.01/151.4 and 0.12/6.186/107.3.
Then, we fine-tune those models with Etruscan asshown in Table 11.
Larth trained on ETP translates "mi aveles me-

6

Results & Discussion

Figure 6 and Figure 7 compare the scores of themodels presented in the previous Section. Compared to the random model, the dictionary-basedmodel shows higher BLEU and chr-F scores andlower TER scores except when tested only on CIEP.
This suggests that CIEP is noisier than ETP andthat the dictionary is not suited for CIEP.
The N-gram models perform better than random
45

Data
Lat+ETP+CIEP
Grc+ETP+CIEP
Lat+ETP
Grc+ETP

BLEU
0.1965
1.011
0.293
2.037

chr-F
2.195
8.148
3.784
6.04

TER
351.6
215.3
654.4
164

Table 11: Larth trained with Latin (Lat) or Ancient
Greek (Grc) and then fine-tuned on Etruscan.

only when using unigrams as context. With longer
n-grams, the performance decrease until the model
only predicts the EOS token. We can make similar
observations for Naïve Bayes models.

Figure 6: Comparison of the models with the best BLEU
scores on ETP+CIEP. One model from each type isselected.

IBM models are able to perform better than random. When trained on ETP+CIEP, simpler models
work better. This, again, might depend on the noisein CIEP. IBM3 works better on ETP despite being trained only with the dictionary. Adding POS
information (IBM4 and IBM5) does not improve
the results. However, on ETP the dictionary-basedmodel still performs better than the IBM models.
Larth is able to achieve a better BLEU score thanthe previous models on both ETP and ETP+CIEP.
However, it needs to use both the character andword sequences and the word tokens are repeated toalign the two sequences, whereas the other models
only use the word tokens. Using the space token toalign the sequences decrease the performance, but
the BLEU score is still higher than the dictionarybased model. A similar observation can be madefor the model using only the word sequence. Using
data augmentation or only the character sequences
reduces the performance that is still higher thanrandom.

Figure 7: Comparison of the models with the best BLEU
scores on ETP. One model from each type is selected.

7Discussion

In this paper, we present a dataset for Etruscan →English machine translation. Although the dataset
is not very big, we show that it is possible to train
statistical and transformer models. Given the unexplored nature of Etruscan language, the fact thattrained models perform better than random is an
important first step for this language. Moreoever,
we demonstrated that Larth performs better thanthe IBM models when trained on the available data.
However, our model does not provide any explanation about the generated translation neither it
guarantees whether it is correct. Our model’s performance also depends on the dataset itself, which
does not contain any bibliographic information or
the reasoning that the original authors used to translate the inscriptions. Future work includes delivering a cleaner and more complete version of thedataset and the inclusion of additional metadata,such as bibliographic information, more accurate
location, or interesting graphical information (e.g.
the direction of the inscription).

Fine-tuning from Latin and Ancient Greek always performs worse than the dictionary-basedmodel. This may depend on the small size of themodel that is not able to adapt.
As for chr-F and TER, the dictionary model andIBM models perform better than Larth. These twomodels can only output tokens from the training set
and ignore unknown tokens. Thus, they can generate longer sequences of correct characters (highchr-F) and the errors are mainly for unknown tokens or from English tokens that are not directly
present in the Etruscan texts like articles (low TER).
Whereas, Larth uses tokens that can be word pieces
and it still generates a translation for unknown tokens.
46

References

40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA.
Association for Computational Linguistics.

Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita
Chatzipanagiotou, Ion Androutsopoulos, Jonathan
Prag, and Nando de Freitas. 2022. Restoring andattributing ancient texts using deep neural networks.
Nature.

Carl Pauli. 1893. Corpus Inscriptionum Etruscarum. J.
A. Barth.
Maja Popović. 2015. chrF: character n-gram F-score
for automatic MT evaluation. In Proceedings of theTenth Workshop on Statistical Machine Translation,
pages 392–395, Lisbon, Portugal. Association for
Computational Linguistics.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. CoRR, abs/1409.0473.
David Bamman and Patrick J. Burns. 2020. Latin BERT:
A contextual language model for classical philology.
CoRR, abs/2009.10053.

Matt Post. 2018. A call for clarity in reporting BLEU
scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186–
191, Belgium, Brussels. Association for Computational Linguistics.

Mani Bansal and D.K. Lobiyal. 2020. Word-characterhybrid machine translation model. In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), pages 270–274.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu. 2020. Exploring the limitsof transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1).

Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages
214–217, Barcelona, Spain. Association for Computational Linguistics.

Helmut Rix and Gerhard Meiser. 1991. Etruskische
Texte. A. G. Narr.
Khaidzir Muhammad Shahih and Ayu Purwarianti. 2019.
Combining word and character vector representation
on neural machine translation. In 2019 Fourth International Conference on Informatics and Computing(ICIC), pages 1–4.

Gregory R. Crane. 1985. Perseus digital library.
Jeff Hill. 2018. Corpus inscriptionum etruscarum plenissimum.
Sepp Hochreiter and Jürgen Schmidhuber. 1997.
Long Short-Term Memory. Neural Computation,
9(8):1735–1780.

Matthew Snover, Bonnier Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A studyof translation edit rate with targeted human annotation. In Proceedings of Association for MachineTranslation in the Americas.

Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd
Cook, Clément Besnier, and William J. B. Mattingly.
2021. The Classical Language Toolkit: An NLP
framework for pre-modern languages. In Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing:
System Demonstrations, pages 20–29, Online. Association for Computational Linguistics.

Thea Sommerschield, Yannis Assael, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita
Chatzipanagiotou, Ion Androutsopoulos, Jonathan
Prag, and Nando de Freitas. 2021. I.PHI dataset:
ancient greek inscriptions. https://github.
com/sommerschield/iphi.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014a.
Sequence to sequence learning with neural networks.
CoRR, abs/1409.3215.

Philipp Koehn. 2010. Statistical Machine Translation,
1st edition. Cambridge University Press, UnitedStates.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014b.
Sequence to sequence learning with neural networks.
CoRR, abs/1409.3215.

Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W.
Black. 2015. Character-based neural machine translation. CoRR, abs/1511.04586.

Jörg Tiedemann. 2020. The Tatoeba Translation Challenge – Realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference
on Machine Translation, pages 1174–1182, Online.
Association for Computational Linguistics.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu
Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han.
2019. On the variance of the adaptive learning rate
and beyond. CoRR, abs/1908.03265.

Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT — Building open translation services for theWorld. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), Lisbon, Portugal.

Jorj X. McKie and Ruikai Liu. 2016. PyMuPDF.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the47

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Proceedings of the 31st International
Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY,
USA. Curran Associates Inc.
Rex Wallace. 2008. Zikh Rasna: A Manual of the Etruscan Language and Inscriptions. Beech Stave Press.
Rex Wallace, Michael Shamgochian, and James Patterson. 2004. Etruscan texts project.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2016.
A character-aware encoder for neural machine translation. In Proceedings of COLING 2016, the 26th
International Conference on Computational Linguistics: Technical Papers, pages 3063–3070, Osaka,
Japan. The COLING 2016 Organizing Committee.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey,
Joshua Ainslie, Chris Alberti, Santiago Ontanon,
Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang,and Amr Ahmed. 2021. Big bird: Transformers forlonger sequences.

48

Evaluation of Distributional Semantic Models of Ancient Greek:
Preliminary Results and a Road Map for Future Work
Saskia Peels
Silvia Stopponi
Nilo Pedrazzini
CLCG, University of
CLCG, University of
The Alan Turing Institute /Groningen, The Netherlands
Groningen, The Netherlands
University of Oxford
s.peels@rug.nl
s.stopponi@rug.nl
United Kindgom
npedrazzini@turing.ac.uk
Barbara McGillivrayKing’s College London
United Kindgom
barbara.mcgillivray@kcl.ac.uk
Abstract

map for further, wider evaluation. We exploit thelayered nature of AGREE to assess at different
levels four DSMs, and discuss results not only interms of model comparison, but mostly in termsof best evaluation strategies, suggesting various
precision- and recall-based options. On that basis,in Section 6 we propose a road map for a morecomprehensive evaluation campaign, which would
involve training a wider range of models, including
dependency-based embeddings (see, among others,
Padó and Lapata 2007; Levy and Goldberg 2014;
Lapesa and Evert 2017; Lenci et al. 2022), alreadypreliminarily tested in Stopponi et al. (2024a), andstudying their behaviour with respect to a numberof metrics. Specifically, we propose to assess thedifference in performance between syntactic embeddings trained on manually tagged and on automatically tagged treebanks. We plan to evaluate theDSMs, trained with different parameters, against
the full version of AGREE, including both humanand machine-generated judgements. We also suggest alternative ways to use the data collected for
AGREE and possible evaluation metrics.

We evaluate four count-based and predictive distributional semantic models of Ancient
Greek against AGREE, a composite benchmarkof human judgements, to assess their ability toretrieve semantic relatedness. On the basis of
the observations deriving from the analysis of
the results, we design a procedure for a largerscale intrinsic evaluation of count-based andpredictive language models, including syntactic
embeddings. We also propose possible waysof exploiting the different layers of the whole
AGREE benchmark (including both humanand machine-generated data) and different evaluation metrics.

1

Malvina Nissim
CLCG, University of Groningen
The Netherlands
m.nissim@rug.nl

Introduction

The application of Natural Language Processing to
the study of Ancient Greek semantics is an emerging research area which has proven to be a fruitful
avenue for our understanding of the Ancient Greek
language and culture. Previous work has focusedon the training of Distributional Semantic Models (DSMs) on Ancient Greek corpora (Boschetti,
2009; Rodda et al., 2017, 2019; McGillivray et al.,2019; Perrone et al., 2021a), a task enabled by the
relatively large quantity of extant texts available forthis language. DSM evaluation is a necessary stepto properly assess the usefulness of applying thesemodels to large-scale studies of Ancient Greek, but
is made particularly challenging by the lack of native speakers and, compared to modern languages,
a limited number of experts available.
This paper offers an evaluation of DSMs forAncient Greek against the newly created AGREE
benchmark (Stopponi et al., 2024b) and a road2

Previous work

Few resources exist as gold standards for the evaluation of DSMs on Ancient Greek. Vatri and Lähteenoja (2019) contains the manual annotation of
the senses of the lemmas μῦς, ἁρμονία, and κόσμος
(Vatri and McGillivray, 2018) and was used in Perrone et al. (2021a) and Perrone et al. (2021b) toevaluate models for semantic change detection.
Rodda et al. (2019) evaluated count-based DSMs
for Ancient Greek against benchmarks obtained
49

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 49–58,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_006

from an ancient lexicon, a modern dictionary ofsynonyms, and the computational lexicon Ancient
Greek WordNet (Boschetti et al., 2016). The data
they released represent the first benchmark for theevaluation of Ancient Greek DSMs.1 Reusing
preexisting resources, as they did, allows incorporating in the evaluation the semantic knowledge ofreal speakers of Ancient Greek (as in the case of theancient lexicon) and to leverage the semantic knowledge of highly specialized experts, from resourcesthat can be the product of years of work. Thisdata collection seems less biased by the aims of theresearch, however it also has downsides. Lexical
resources, compiled by humans, can suffer fromidiosyncrasies, for example being biased by theinterests and language taste of their author, and if
the author is not alive anymore, it is not possible toget explanations about specific choices. Moreover,ancient resources can reflect ideas of semantic relationships between words (e.g. word similarity) thatare different from the contemporary conceptualization, as also noticed by Rodda et al. (2019, 6–8)
and discussed in Stopponi et al. (2024b).

3

the largest possible manually-annotated treebank,created from a collation of the available corpora.

4

The AGREE Benchmark

The AGREE benchmark contains pairs of lemmas
semantically related to 36 selected ‘seed’ lemmas(12 nouns, 12 adjectives, and 12 verbs), for a totalof 638 lemma pairs.4 The judgements were collected via questionnaires distributed to a large number(> 50) of academic scholars of Ancient Greek. The
final benchmark, AGREE, incorporates a mix ofexpert-elicited pairs and expert-assessed, machinegenerated pairs. The machine-generated items arepairs of [seed lemma - nearest neighbour], withnearest neighbours extracted from Word2Vec models (Mikolov et al., 2013) that underwent expert
judgement and were assessed as highly related.
For the experiments reported in this paper, weonly use the human-elicited portion of the benchmark: AGREE-task1. This portion can be further
divided into the subset of pairs that were proposedby one expert only, and the subset of pairs thatwere proposed by more than one annotator, under
the assumptions that the latter might be cases of astronger relatedness, and/or higher frequency.

Training Data for DSMs of Ancient
Greek

The largest corpus of Ancient Greek, the Thesaurus
Linguae Graecae (Pantelia, 2022), containing morethan 110 million tokens,2 is only accessible throughthe web interface. However, scholars can use anumber of open-access machine-readable Ancient
Greek corpora, containing different ranges of text
types.3 Some corpora are annotated, for example
with lemma, POS, and syntactic information. The
Diorisis Ancient Greek Corpus (Vatri and McGillivray, 2018), a portion of which was used as training
data for the study presented in this paper, contains
10,206,421 automatically lemmatized and POStagged tokens. But many corpora with syntactic
annotation also exist: an overview of the most often
used treebanks for Ancient Greek is in Table 1.
As the case of GLAUx shows (see Table 1),
automatic parsing allows for the creation of larger treebanks, even if the syntactic annotation is
expected to be less accurate. We thus plan to trainsyntactic embeddings on two corpora, GLAUx and5

Evaluation of DSMs of Ancient Greek

5.1

Procedure

For this study we evaluated two count-based andtwo predictive DSMs trained on a portion of the
Diorisis corpus (Vatri and McGillivray, 2018), merging text from the Archaic, Classical and Hellenistic periods, since the AGREE benchmark (andespecially the pairs proposed by experts) is particularly suited to the evaluation of models trained on
texts from those periods (Stopponi et al., 2024b).
The lemmatized version of Diorisis was used, toreduce the impact of word sparsity. Stop word filtering was performed, according to the list also usedin Rodda et al. (2019)5 . Stop word filtering reduced
the size of the corpus from 5,768,916 to 2,960,459
tokens. The four models were evaluated against
AGREE-task1, by comparing the top 5, 10, 15 (k)nearest neighbours of each of the 36 seed lemmas
in the benchmark with the lemmas related to thesame seed in AGREE-task1. The nearest neighbours extracted from the models were compared1

https://zenodo.org/record/3552763#.YfAItOrMKWA
2
https://wiki.digitalclassicist.org/
Thesaurus_Linguae_Graecae
3
A review of most available open-access corpora for Ancient Greek is in Keersmaekers (2021, 40).

4https://zenodo.org/record/8027490.
https://figshare.com/articles/
dataset/Ancient_Greek_stop_words/9724613,
by A. Vatri.
550TreebankN. tokens

Manual annotation

Texts

Ancient Greek Dependency Treebank(Perseus, Bamman and Crane, 2011)ca. 550K*

yes

Literary, full list at http://
perseusdl.github.io/
treebank_data/

PROIEL Treebank (Haug and Jøhndal,
2008)

ca. 250.5K

yes

The Greek New Testament, Histories(Herodotus), Chronicles (Sphrantzes)

Gorman Trees (Gorman, 2020)ca. 240K*

yes

Literary prose, full list at https://perseids-publications.
github.io/gorman-trees/

Pedalion Trees (Keersmaekers et al.,2019)

ca. 300K

yes

Literary, full list at https://perseids-publications.
github.io/pedalion-trees/

Harrington Treebanks (Harrington, 2018)ca. 18K*

yes

Nicene Creed; Book of Susanna (Septuaginta), Verae historiae (Lucian of
Samosata), Vita Aesopi

PapyGreek (Vierros and Henriksson,
2021)

ca. 44K

syntactic layer only

PapyriAphthonius (Yordanova, 2018)ca. 7K*

yes

Progymnasmata (Aphtonius)

ca. 11,860K

noLiterary, papyrological, epigraphical.
A sample was released at https://perseids-publications.
github.io/glaux-trees/

GLAUx corpus (Keersmaekers, 2021)

Table 1: Some available treebanks for Ancient Greek. If the size of the treebank is followed by a *, it is taken fromKeersmaekers et al. (2019, 110). The size of the PapyGreek treebanks has been calculated by summing up all the‘word’ elements in the XML files.

to: all the lemmas in AGREE-task1, the lemmas in
AGREE-task1 proposed by more than one expert,
and the lemmas in AGREE-task1 proposed by only
one expert. Precision and recall were adopted as
evaluation metrics and defined as follows:

size = 30, window = 5, min_count = 5,
negative = 20.
5.3

Results

The average precision and recall are reported inTable 2. We immediately see that recall is generoverlap model’s near. neighb. and benchmark ally low. This can be explained by the fact that
Precision@K =kthere are on average 14 neighbours per lemma6 in
AGREE-task1, so that the denominator in recall@k
near. neighb. model also in benchmark
Recall@K =is generally larger than the numerator when k = 5
n. related lemmas benchmark
or k = 10. The recall consequently increases (on5.2 Models
average) if k also increases, while the opposite hapThe models selected for evaluation are two
pens for precision, which increases if k decreases.
Word2Vec models, one SGNS and one CBOW, andTaking into account recall for k < 15 makes thus
two count-based models. The matrices of the count- little sense, since it is never possible to achieve fullbased models were weighted with PPMI and one
recall when the lemmas related to a certain seed
one of the two dimensionality reduction was per- in the benchmark are more than the extracted kformed with Singular Value Decomposition (SVD). nearest neighbours. Conversely, it is theoretically
The two count-based models were built by using
possible to achieve 100% precision if all the exthe software provided by the LSCDetection repos- tracted k-nearest neighbours are also in the benchitory (Schlechtweg et al., 2019) with window = 5
mark. The higher precision with smaller values ofand the following other parameters: k = 1 andk seems to confirm that the closest neighbours inalpha = 0.75 for PPMI, 300 dimensions andthe semantic space are actually more strictly related
gamma = 0.0 for SVD. The two Word2Vec mod- to the seed lemma, while the strength of the seedels were trained with the Gensim library (Řehůřek
6
and Sojka, 2010) and the following parameters:
Min. = 6, max. = 24, standard deviation = 4.43.
51

k

Precision

Recall

5
10
15

0.20
0.16
0.13

0.06
0.09
0.11

μαχία, ‘alliance’, Λακεδαιμόνιος, ‘Spartan’, andπολεμέω, ‘fight’) and it is πόλεμος, ‘war’ for the
PPMI, which also returns συμμαχία, ‘alliance’,Φίλιππος, ‘Philip’, πολεμέω, ‘fight’, and πρεσβεία, ‘embassy’. We notice that both models return συμμαχία, ‘alliance’ among their first 5 neighbours. This word was not proposed by the experts
in the first phase of data collection for the AGREE
benchmark, but is however semantically related toεἰρήνη, ‘peace’. More in general, we deem all the
top 5 nearest neighbours returned by both models as acceptable results, since they all are relatedto εἰρήνη, ‘peace’; the two models just differ inresults from one other, as well as from the benchmark. Of course, there are also cases in which
the overlapping lemma(s) are the same between
models. One example is μέγας, ‘big’, for which
there are 15 related lemmas in AGREE-task1.7
Both the CBOW and the PPMI model have precision 0.2 (1/5) and recall 0.07 (1/14) with k = 5,
and the lemma overlapping with the AGREE-task1
benchmark is the same for both models, μέγεθος,‘greatness’. Again, the extracted nearest neighbours that are not in the benchmark are not necessarily unrelated to the seed μέγας, ‘big’. The
CBOW model also returns τηλικοῦτος, ‘of such anage/so large’, ἄξιος, ‘weighing as much/worthy’,ῥοπή, ‘weight’, and ὑπερβάλλω, ‘surpass/exceed’,
while the PPMI model also returns ἐλάσσων, ‘smaller’, ἴσος, ‘equal’, ἄρος, ‘use/profit’, and πολύς,‘many’. Except from ἄρος, ‘use/profit’, they all
relate to μέγας, even if, intuitively,with a different
strength and with different types semantic relations.
The internal layering of the benchmark AGREEtask1, which accounts for the number of experts
who proposed a specific lemma, allows for otherobservations (Table 4). On average, the lemmas
returned by only one expert (AGREE-task1-only1
in 4) are more (13.02 per seed lemma) than those
returned by several experts (AGREE-task1-more1,
4.69 per seed). We could hypothesize that the relatedness among the latter may be stronger or moreevident, since more than one expert independently
had proposed the same lemmas as related to the relevant seed word. When we evaluate against lemma
pairs proposed by more than one expert higher pre-Table 2: Average precision and recall calculated against
the whole AGREE-task1 benchmark and divided by k.

ModelPrecision

Recall

SGNS
CBOW
SVD
PPMI

0.11
0.15
0.16
0.22

0.06
0.08
0.09
0.12

Table 3: Average precision and recall calculated against
the whole AGREE-task1 benchmark, divided by model.

neighbour relationship declines for neighbours thatare further away from the seed.
Model architecture also has an impact, withcount-based performing better than predictive models. This is in line with what is observed by Lenciet al. (2022). Moreover, the model without dimensionality reduction performs better than the one
to which SVD was applied, as shown in Table 3.
Further, Word2Vec CBOW seems to perform better
than Word2Vec SGNS. However, parameter optimization was not performed for this preliminary
study, and a limited number of model architectures
was tested. In future, larger evaluation will probably give a better picture of the differences between
count-based and predictive models.
For example, for the seed lemma εἰρήνη, ‘peace’,
there are 9 related lemmas in AGREE-task1:
πόλεμος, ‘war’, σπονδή, ‘drink-offering/treaty’,ἥσυχος, ‘quiet’ (adj.), ἡσυχία, ‘quiet, silence’(noun), σπένδω, ‘make a drink-offering’, μάχη,‘battle’, γαληνός, ‘calm’, πολιτεία, ‘citizenship’,συγγραφή, ‘writing’, ὁμολογέω, ‘agree’, νίκη,‘victory’, ὄλβος, ‘happiness’, γαλήνη, ‘stillness’,and φιλία, ‘friendship’. Both the CBOW and
the PPMI model have precision 0.2 with k = 5,
i.e. among the first 5 nearest neighbours returned
there is one that is also in AGREE-task1. The
recall is 0.07 (1/14). The overlapping lemma
is σπονδή, ‘drink-offering/treaty’ for the CBOW
model, (which also returns as the other four nearest
neighbours διάλυσις, ‘separating/ending’, συμ-7They are μικρός, ‘small’, ὅρκος, ‘oath’, βασιλεύς,‘king’, θαῦμα, ‘wonder’, θεός, ‘god’, μακρός, ‘long’, ὀλίγος, ‘little’, βραχύς, ‘short’, μέγεθος, ‘greatness’, αὐξάνω,‘increase’, μεγαλοψυχία, ‘greatness of soul’, ἥρως, ‘hero’,γίγας, ‘giant’, καλός, ‘beautiful’, and μεγαλοφροσύνη,‘greatness of mind’.

52

Benchmark subset

Prec

Rec

AGREE-task1
AGREE-task1-more1
AGREE-task1-only1

0.16
0.09
0.07

0.09
0.05
0.04

ὁράω, ‘see’, both with average precision 0.4. The
lowest precision, 0, is achieved with the seed lemmas ἀκτή, ‘headland’, κλυτός, ‘renowned’, ναίω,‘dwell’, ῥῆσις, ‘speech’, σῆμα, ‘sign/mark’, andτεύχω, ‘make/build’, all with average precision 0.
Nevertheless, as we already observed, a low precision does not necessarily correspond to bad results(i.e. unrelated lemmas), even if it is true that someof the nearest neighbours returned by the models
to these are unrelated or intuitively less strictly
related to the seed lemmas. Moreover, a higher
precision seems to correspond to higher-frequency
words, while the lemmas yielding the lowest precision also have a low frequency in the corpus.8 In
Table 7 the average precision and recall for eachlemma are reported, calculated against the whole
AGREE-task1 and with k = 15. Note that changing
the value of k the order of the seed lemmas, ranked
by precision, also changes.

Table 4: Average precision and recall calculated against
different subsets of the AGREE-task1 benchmark. The
results with the three values of k were averaged.

cision and recall scores are observed, possibly suggesting that pairs proposed by more experts are
more closely related to their seed lemma, and possibly more frequent. This is particularly true for
the PPMI model, which achieves an average of
0.22, 0.14, and 0.09 precision, and an average of
0.12, 0.07, and 0.05 recall against, respectively, thewhole AGREE-task1, the pairs proposed by morethan one expert, and the pairs proposed by only
one expert (the results are averaged across the threevalues of k). This is observed when averaging theresults of all models, but it does not necessarily
hold for each model. The CBOW model, for example, achieves a higher precision against the setof pairs proposed by only one expert than against
those proposed by more experts. Both Word2Vec
models instead achieve the same precision and recall on both subsets of AGREE-task1. The results
discussed until now are summarised in Table 5.
Another dimension of the benchmark is the partof-speech (POS) of the seed lemmas. In Table 6 we
see that evaluating against pairs including an adjective seed lemma the highest precision is achieved,
followed by noun seeds and verb seeds. The recall
is higher when evaluated against pairs including
adjective or noun seeds. However, the differencesin precision and recall are very small.
Finally, dividing the results by lemma reveals
a great variety in precision and recall among thedifferent lemmas. For example, with k = 5 thehighest precision is achieved. The average precision per lemma calculated against the whole
AGREE-task1 is 0.20, with standard deviation 0.16.
There is indeed a large variability between the average precision against the “best” and the “worstperforming” lemmas. Those yielding the highest
precision are some nouns and adjectives: ἅρμα,‘chariot’, average precision 0.6; ψευδής, ‘false’,
0.55; ἐλεύθερος, ‘free’, 0.45; πατήρ, ‘father’, 0.45;
and ἄγριος, ‘wild’, 0.45. However, they are immediately followed by verbs, ἔρχομαι, ‘go’ and6Road Map for Future WorkWe plan a larger evaluation including more modelarchitectures, different parameters and different
evaluation metrics, with the aim of understanding the differences between model types, rather
than finding the ‘best’ model (see also Lenci et al.,2022), and evaluation adequacy. More investigation is needed to understand whether the difference
between count-based and predictive models trainedon Ancient Greek lies in the quality of results (i.e.,
if some architectures actually return less relevant
nearest neighbours), or only in the kind of relationships they capture. Further experiments will also
concern dependency-based embeddings.
Moreover, this extended study will exploit thefull dataset produced for the AGREE benchmark,including the second part of the dataset, not used
for the current evaluation. Since in the second
phase of the data collection the experts assigned relatedness scores to human- and machine- generated
lemma pairs, these items items allows ranking thelemma pairs according to their degree of relatedness, and thus for a more nuanced evaluation.
8
The frequency in the subcorpus of the mentioned“best performing” lemmas is: ἅρμα: 541, ψευδής: 1048,ἐλεύθερος: 940, πατήρ: 5685, ἄγριος: 348, ἔρχομαι: 5251,
ὁράω: 4987, while the frequency of the mentioned “worstperforming lemmas is: ἀκτή: 177, κλυτός: 142, ναίω: 283,
ῥῆσις: 48, σῆμα: 213, τεύχω: 255.

53

Precision

RecallBench. subset

k

PPMI

SVD

CBOW

SGNS

PPMI

SVD

CBOW

SGNS

Tot. prec.

Tot. rec.

Tot. pairs

AGREE-task1

all k

0.22

0.16

0.15

0.11

0.12

0.09

0.08

0.06

0.16

0.09

638

k=5

0.28

0.19

0.19

0.14

0.08

0.06

0.05

0.04

0.20

0.06

k = 10

0.22

0.16

0.14

0.11

0.12

0.09

0.08

0.07

0.16

0.09

k = 15

0.17

0.13

0.11

0.09

0.15

0.11

0.10

0.08

0.13

0.11

all k

0.14

0.09

0.07

0.06

0.07

0.05

0.04

0.03

0.09

0.05

k=5

0.19

0.11

0.08

0.07

0.06

0.03

0.02

0.02

0.11

0.03

k = 10

0.13

0.10

0.06

0.05

0.08

0.06

0.04

0.03

0.09

0.05

k = 15

0.10

0.08

0.06

0.04

0.09

0.07

0.05

0.04

0.07

0.06

all k

0.09

0.07

0.08

0.06

0.05

0.04

0.04

0.03

0.07

0.04

k=5

0.09

0.08

0.11

0.07

0.03

0.02

0.03

0.02

0.09

0.02

k = 10

0.09

0.07

0.08

0.06

0.05

0.04

0.04

0.03

0.07

0.04

k = 15

0.07

0.06

0.06

0.04

0.06

0.05

0.05

0.04

0.06

0.05

AGREE-task1-more1

AGREE-task1-only1

169

469

Table 5: Average precision and recall calculated against different subsets of the AGREE-task1 benchmark, divided
by model type and by k. The recall for values of k lower than 15 has been reported for completeness, but it has
limited usefulness (see above). The column ’Tot. pairs’ contains the total number of pairs in the relevant subsets.

POS
A
N
V

Precision

Recall

0.18
0.15
0.15

0.09
0.09
0.08

Sampling (SGNS) and the Continuous-Bag-ofWords (CBOW);
5. two ‘syntax-filtered’ models (Padó and Lapata,
2007; Lapesa and Evert, 2017; Lenci et al.,2022), a SGNS one but using direct dependency
between tokens to extract co-occurrences rather
than mere token windows and one trained using
the SuperGraph approach described in Al-Ghezi
and Kurimo (2020). The latter method consists
in using dependency relations between tokens
to generate graph structures for every sentence
in a treebank, before merging all graphs into
one SuperGraph. The SuperGraph then serves
as input to Node2Vec (Grover and Leskovec,2016), a modification of the SGNS architecture
which enables the training of word representations starting from nodes in a graph.

Table 6: Average precision and recall calculated against
the whole AGREE-task1 benchmark and divided by
POS of the seed lemmas.

6.1

Models

We will test a selection of popular DSMs belonging
to the first two generations defined by Lenci et al.
(2022), i.e. count-based models (PPMI and GloVe)and predictive models (Word2Vec and FastText).
In particular, we will test:
1. two count-based models trained by using Positive Pointwise Mutual Information (PPMI) as
association measure,9 with and without dimensionality reduction with the Singular Value Decomposition (SVD);

Contextual models will not be included, instead.
Even if some work exists on the training of contextual models of Ancient Greek (Singh et al., 2021;
Keersmaekers and Mercelis, 2021; Yamshchikov
et al., 2022; Riemenschneider and Frank, 2023)(despite the fact that contextual models requirehuge quantities of training data (Lenci et al., 2022,
1274)), the only existing evaluation datasets forsemantic models of Ancient Greek (Rodda et al.,2019 and Stopponi et al., 2024b) were created

2. GloVe (Pennington et al., 2014));
3. FastText (Bojanowski et al., 2017);
4. the two architectures of word2vec (Mikolov
et al., 2013), the Skip-gram with Negative
9

About association measures, see Evert et al. (2008).

54

LemmaPrecision

Recallἅρμα, ‘chariot’

0.32

0.22

ὁράω

0.30

ναῦς, ‘ship’

Lemma

Precision

Recallεἰρήνη, ‘peace’

0.10

0.11

0.24

᾿Αθηναῖος, ‘Athenian’

0.08

0.08

0.27

0.25

νόστος, ‘return’

0.08

0.07

χρυσός, ‘gold’

0.27

0.27

παλαιός, ‘old’

0.08

0.07

ἄγριος, ‘wild’

0.23

0.17

ζεύγνυμι, ‘yoke’

0.08

0.09

ἐλεύθερος, ‘free’

0.23

0.17

μέγας, ‘big’

0.08

0.08

ἔρχομαι, ‘go’

0.23

0.19

μῦθος, ‘word/story’

0.07

0.07

πατήρ, ‘father’

0.22

0.30

ἀκτή, ‘headland’

0.07

0.07

ψευδής, ‘false’

0.20

0.18

μοχθέω, ‘labour’

0.07

0.07

κακός, ‘bad’

0.17

0.12

Σάμος, ‘Samos’

0.07

0.06

οἰκέω, ‘inhabit’

0.17

0.11

ἄλκιμος, ‘brave’

0.05

0.04

αὐξάνω, ‘increase’

0.17

0.14

ῥῆσις, ‘speech’

0.05

0.04

ὀρφανός, ‘orphan’

0.17

0.14

τέμνω, ‘cut’

0.03

0.03

πόντος, ‘sea’

0.15

0.12

κλυτός, ‘renowned’

0.02

0.01

φιλέω, ‘love’

0.15

0.15

λείπω, ‘leave/quit’

0.02

0.01

αἴθω, ‘light up’

0.13

0.10

τεύχω, ‘make/build’

0.02

0.01

πρέσβυς, ‘old man, elder’

0.13

0.11

ναίω, ‘dwell’

0.00

0.00

ἑνδέκατος, ‘eleventh’

0.13

0.11

σῆμα, ‘sign/mark’

0.00

0.00

Table 7: Average precision and recall calculated against the whole AGREE-task1 benchmark and with k = 15,
divided by seed lemma. The lemmas are ranked by average precision.

6.2

for the evaluation of static (type-based) embeddings. Although type-based embeddings can beobtained from contextualized token embeddings,
e.g. by averaging the model representations ofeach word (see the discussion in Lenci et al., 2022,
1290–1291), their superiority over type embeddings obtained from static DSMs has been questioned (Lenci et al., 2022, 1289–1293). This evaluation will thus be limited to the evaluation of static
embeddings, leaving the training and evaluationof contextual embeddings for future work.10 All
the models will be trained with two different context windows, e.g. 5 and 10. According to thelarge-scale evaluation of Lenci et al. (2022), modelarchitecture and context window size are the twoparameters that significantly affect model performance (especially model architecture). We thus concentrate on testing of these two.

Dependency-based embeddings

Ancient Greek syntactic embeddings obtained with
the SuperGraph method have already been compared with window-based models by Stopponi et al.
(2024a), clearly suggesting that the former capture functional rather than topical similarity, as
had already been shown at least since Levy andGoldberg (2014) on the basis of English models. Given this ontological difference between
the two, an open question, then, is whether syntactic embeddings should be evaluated on a parwith traditional count-based and word2vec models, namely whether there are arguments for using the same benchmark to judge the quality ofmodels regardless of whether syntactic information is integrated in their training or not. Previous large-scale comparisons of dependency-based
and window-based DSMs suggested that the latter,when fine-tuned, generally outperform the former
in most downstream tasks (Kiela and Clark, 2014;
Lapesa and Evert, 2017). Given the generally
greater computational costs associated with dependency parsing and the extraction of syntactic
collocates (i.e. tokens with a direct dependency

10
It should be noted that the training corpus of Lenci et al.
(2022, 1279) are English texts from the Web. Their conclusions could thus not entirely apply to Ancient Greek, alanguage with a different syntax and morphology.

55

relation), it has been questioned whether the training of dependency-based embeddings is justifiable
after all. However, there is evidence, at least as
far as high-resource languages such as English areconcerned, that dependency-based embeddings outperform window-based models in a limited but coherent number of tasks. This has been shown to beconsistently the case, for instance, of categorization tasks, namely grouping lexical items into semantically coherent categories (Rothenhäusler and
Schütze, 2009; Lapesa and Evert, 2017; Lenci et al.,2022), as well as thematic fit estimation, namely
evaluating the typicality of the argument of a verb
given a thematic role (e.g., agent or patient) (Baroni
and Lenci, 2010; Chersoni et al., 2017). Different
tasks such as categorization and synonymity tests
present, in many ways, the same ontological differences occurring between dependency- and windowbased models as a whole. This alone would seem
to warrant the training of different models (and, asa result, the development of different evaluation
methods) depending on the task at hand. Classic
distributional semantic models (i.e. window-based)
are generally fined-tuned to capture attributional
similarity (Turney, 2006), namely the number of
attributes, or properties, shared by the referents of
two given words. As pointed out by Baroni andLenci (2010), words that share many collocates willshow a high attributional similarity since common
collocates can be seen as a proxy for some of theattributes that the two words denote. Pairs such as
dog-puppy will then have a high attributional similarity but not necessarily a high relational similarity(Turney, 2006), which in turns refers to sharing
similar semantic relations to their nearest neighbours. In Baroni and Lenci’s 2010 example, thepair dog-tail will be more similar to car-wheel than
it is to dog-animal, even though attributionally thatis clearly not the case.

better suited to evaluate window-based DSMs.
6.3

Evaluation Metrics

We observed above how precision and recall only
provide an absolute evaluation against the benchmark, capturing whether the words in the benchmark are returned by the models or not, but they do
not allow us to take into account the strength of thesemantic relationship between lemmas. Moreover,
only the first k neighbours returned by the modelare evaluated, while there is no information about
how close to the seed lemma in a semantic space
the related lemmas in the benchmark are which
are not among the first k neighbours. Furthermore,
the use of recall in this kind of evaluation can beproblematic when the number of k is lower than
the number of pairs in the benchmark.
To overcome these limitations, we plan to include additional evaluation strategies. One option
is to use the evaluation items that were rated on a 0100 relatedness scale (AGREE-task2), to calculate
for each seed lemma the correlation between: (i)the scores assigned to pairs including that lemmain the benchmark; (ii) the cosine distances between
the same word pairs in a semantic space. The scores
can also be used to rank the items, and a correlation
can be calculated between ranks and cosine distances. Taking into account degrees of relatedness
may be a more adequate way to evaluate models
on a phenomenon such as semantic relatedness.
Another possibility is to exploit the information
about the number of raters who proposed the words
collected in the first phase (AGREE-task1), for example by giving greater weight to pairs suggested
by multiple raters. However, this will first require a
deeper investigation on the nature of the pairs proposed by one versus several experts, and the impact
this might have on evaluation. Relatedly, frequency
should also be considered to verify the ways andextent to which precision and recall are impacted
by high-frequency items (both the human-elicited
ones and those returned by the models).

Building on the preliminary observation made in
Stopponi et al. (2024a) about the relational, rather
than attributional, similarity captured by Ancient
Greek dependency-based models, we thus plan totest different Ancient Greek models on different
tasks depending on the kind of similarity the modelis trained to capture. Categorization and thematic
fit task, for example, can be set up with the help of
the richly annotated resources for the language (e.g.
the verbal semantic annotation in the PROIEL treebank) for dependency-based models, in addition tosimilarity judgement tasks, which may be instead

7

Conclusion

We presented and discussed the results of an evaluation of four Distributional Semantic Models ofAncient Greek, two count-based and two predictive models. The gold standard was a subset of
the AGREE benchmark, AGREE-task1, including
pairs of related lemmas proposed by experts ofAncient Greek. The evaluation showed that count56

based models achieved higher precision and recall
on AGREE-task1, and higher precision and recall
were also achieved on average when evaluating
against pairs of related lemmas proposed by morethan one expert. Another important finding was
the great difference in performance between different lemmas. We also presented a plan for amore extended evaluation, including more modelarchitectures, parameters, and evaluation metrics.
This evaluation will take into account different degrees of relatedness between lemmas and allow fora better understanding of the differences between
DSMs of Ancient Greek and of the possible impact of such differences on computational studies
in Ancient Greek lexical semantics.

semantics? In IWCS 2017 - 12th International Conference on Computational Semantics - Long papers.
Stefan Evert et al. 2008. Corpora and collocations. Corpus linguistics. An international handbook, 2:1212–
1248.
Vanessa B Gorman. 2020. Dependency treebanks ofAncient Greek prose. Journal of Open Humanities
Data, 6(1):1.
Aditya Grover and Jure Leskovec. 2016. node2vec:
Scalable feature learning for networks. CoRR,
abs/1607.00653.
Matthew Harrington. 2018. Perseids project. treebanked
commentaries at Tufts University.
Dag TT Haug and Marius Jøhndal. 2008. Creatinga parallel treebank of the old Indo-European Bible
translations. In Proceedings of the second workshop
on language technology for cultural heritage data
(LaTeCH 2008), pages 27–34.

Acknowledgements
This work was partially supported by the Young
Academy Groningen through the PhD scholarshipof Silvia Stopponi.

Alek Keersmaekers. 2021. The GLAUx corpus: methodological issues in designing a long-term, diverse,
multi-layered corpus of Ancient Greek. In Proceedings of the 2nd International Workshop on Computational Approaches to Historical Language Change
2021, pages 39–50, Online. Association for Computational Linguistics.

References
Ragheb Al-Ghezi and Mikko Kurimo. 2020. Graphbased syntactic word embeddings. In Proceedings ofthe Graph-based Methods for Natural Language Processing (TextGraphs), pages 72–78, Barcelona, Spain(Online). Association for Computational Linguistics.

Alek Keersmaekers and Wouter Mercelis. 2021. Improving morphological analysis of Greek withtransformer-based approaches: First results with
ELECTRA.

David Bamman and Gregory Crane. 2011. The Ancient Greek and Latin dependency treebanks. In
Language technology for cultural heritage: Selected papers from the LaTeCH Workshop Series, pages
79–98. Springer.

Alek Keersmaekers, Wouter Mercelis, Colin Swaelens,and Toon Van Hal. 2019. Creating, enriching and valorizing treebanks of Ancient Greek. In Proceedingsof the 18th International Workshop on Treebanks and
Linguistic Theories (TLT, SyntaxFest 2019), pages
109–117, Paris, France. Association for Computational Linguistics.

Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for
corpus-based semantics. Computational Linguistics,
36(4):673–721.

Douwe Kiela and Stephen Clark. 2014. A systematic
study of semantic vector space model parameters. In
Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC),
pages 21–30, Gothenburg, Sweden. Association for
Computational Linguistics.

Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information. Transactions of the Association for Computational Linguistics, 5:135–146.
Federico Boschetti. 2009. A Corpus-based Approach toPhilological Issues. University of Trento.

Gabriella Lapesa and Stefan Evert. 2017. Large-scaleevaluation of dependency-based DSMs: Are they
worth the effort? In Proceedings of the 15th Conference of the European Chapter of the Association
for Computational Linguistics: Volume 2, Short Papers, pages 394–400, Valencia, Spain. Association
for Computational Linguistics.

Federico Boschetti, Riccardo Del Gratta, and Harry
Diakoff. 2016. Open Ancient Greek WordNet
0.5. ILC-CNR for CLARIN-IT repository hosted at Institute for Computational Linguistics ’A.
Zampolli’, National Research Council, in Pisa.
http://hdl.handle.net/20.500.11752/ILC-56 (accessed
4 July 2022).

Alessandro Lenci, Magnus Sahlgren, Patrick Jeuniaux,
Amaru Cuba Gyllensten, and Martina Miliani. 2022.
A comparative evaluation and analysis of three generations of distributional semantic models. Language
resources and evaluation, 56(4):1269–1313.

Emmanuele Chersoni, Enrico Santus, Philippe Blache,and Alessandro Lenci. 2017. Is structure necessary
for modeling argument expectations in distributional

57

Omer Levy and Yoav Goldberg. 2014. Dependencybased word embeddings. In Proceedings of the 52nd
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pages 302–308,
Baltimore, Maryland. Association for Computational
Linguistics.

Martina A Rodda, Marco SG Senaldi, and Alessandro
Lenci. 2017. Panta Rei: Tracking Semantic Changewith Distributional Semantics in Ancient Greek.
IJCoL. Italian Journal of Computational Linguistics, 3(3-1):11–24.
Klaus Rothenhäusler and Hinrich Schütze. 2009. Unsupervised classification with dependency based word
spaces. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages
17–24, Athens, Greece. Association for Computational Linguistics.

Barbara McGillivray, Simon Hengchen, Viivi Lähteenoja, Marco Palma, and Alessandro Vatri. 2019. A
computational approach to lexical polysemy in Ancient Greek. Digital Scholarship in the Humanities,
34(4):893–907.

Dominik Schlechtweg, Anna Hätty, Marco Del Tredici,and Sabine Schulte im Walde. 2019. A wind ofchange: Detecting and evaluating lexical semanticchange across times and domains. In Proceedingsof the 57th Annual Meeting of the Association for
Computational Linguistics, pages 732–746, Florence,
Italy. Association for Computational Linguistics.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.
Sebastian Padó and Mirella Lapata. 2007. Dependencybased construction of semantic space models. Computational Linguistics, 33(2):161–199.

Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021.
A pilot study for BERT language modelling and morphological analysis for ancient and medieval Greek.
In 5th Joint SIGHUM Workshop on Computational
Linguistics for Cultural Heritage, Social Sciences,
Humanities and Literature, co-located with EMNLP
2021, pages 128–137. Association for Computational
Linguistics.

Maria C Pantelia. 2022. Thesaurus Linguae Graecae
Digital Library. University of California, Irvine.
Jeffrey Pennington, Richard Socher, and Christopher
Manning. 2014. GloVe: Global vectors for word
representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar.
Association for Computational Linguistics.

Silvia Stopponi, Nilo Pedrazzini, Saskia Peels-Matthey,
Barbara McGillivray, and Malvina Nissim. 2024a.
Natural language processing for Ancient Greek:
Design, advantages, and challenges of language models. Diachronica.

Valerio Perrone, Simon Hengchen, Marco Palma, Alessandro Vatri, Jim Q Smith, and Barbara McGillivray.
2021a. Lexical semantic change for Ancient Greek
and Latin. In N. Tahmasebi, L. Borin, A. Jatowt,
Y. Xu, and S. Hengchen, editors, Computational approaches to semantic change, pages 287–310. Language Science Press.

Silvia Stopponi, Saskia Peels-Matthey, and Malvina
Nissim. 2024b. AGREE: A new benchmark for theevaluation of distributional semantic models of Ancient Greek. Digital Scholarship in the Humanities.
Peter D. Turney. 2006. Similarity of semantic relations.
Computational Linguistics, 32(3):379–416.

Valerio Perrone, Simon Hengchen, Marco Palma, Alessandro Vatri, Jim Q. Smith, and Barbara McGillivray.
2021b. Lexical semantic change for Ancient Greek
and Latin: Computational approaches to semanticchange. In Nina Tahmasebi, Lars Borin, Adam
Jatowt, Yang Xu, and Simon Hengchen, editors, Computational approaches to semantic change, volume 6,
chapter 9, pages 287–310. Language Science Press.

A. Vatri and B. McGillivray. 2018. The Diorisis Ancient
Greek Corpus: Linguistics and literature. Research
Data Journal for the Humanities and Social Sciences,3(1):55 – 65.
Alessandro Vatri and Viivi Lähteenoja. 2019. Ancient
Greek semantic annotation datasets.

Radim Řehůřek and Petr Sojka. 2010. Software
Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on
New Challenges for NLP Frameworks, pages 45–50,
Valletta, Malta. ELRA. http://is.muni.cz/
publication/884893/en.

Marja Vierros and Erik Henriksson. 2021. PapyGreek
Treebanks: A dataset of linguistically annotatedGreek documentary papyri. Journal of open humanities data.
Ivan Yamshchikov, Alexey Tikhonov, Yorgos Pantis,
Charlotte Schubert, and Jürgen Jost. 2022. BERT in
plutarch’s shadows. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language
Processing, pages 6071–6080, Abu Dhabi, United
Arab Emirates. Association for Computational Linguistics.

Frederick Riemenschneider and Anette Frank. 2023. Exploring large language models for classical philology.
arXiv preprint arXiv:2305.13698.
Martina A Rodda, Philomen Probert, and Barbara McGillivray. 2019. Vector space models of Ancient
Greek word meaning, and a case study on Homer.
Traitement Automatique Des Langues, 60(3):63–87.

Polina Yordanova. 2018. Treebank of Aphtonius, Progymnasmata.

58

Latin Morphology through the Centuries:
Ensuring Consistency for Better Language Processing
Federica Gamba and Daniel Zeman
Charles University, Faculty of Mathematics and Physics,
Institute of Formal and Applied Linguistics, Prague, Czechia
gamba,zeman@ufal.mff.cuni.cz

Abstract

while Dante Alighieri’s work available in UDantebelongs to XIV century. In addition to that, Perseus
and PROIEL include classical texts (I BC - IV AD),as well as the Vulgate (IV century). A level of spatial variability can be observed too; for instance,
LLCT includes texts written in Tuscany, Italy, andsome features typical of the Romance languages
are already emerging.

This paper focuses on the process of harmonising the five Latin treebanks available in Universal Dependencies with respect to morphological annotation. We propose a workflow thatallows to first spot inconsistencies and missing
information, in order to detect to what extent
the annotations differ, and then correct the retrieved bugs, with the goal of equalising the annotation of morphological features in the treebanks and producing more consistent linguistic
data. Subsequently, we present some experiments carried out with UDPipe and Stanza inorder to assess the impact of such harmonisation on parsing accuracy.

1

In addition to the aforementioned levels of variability, and besides variation in size, Latin treebanks also differ in terms of annotation choices, inspite of the UD work towards consistency. Thisissue can be doubly problematic: first with respect
to UD itself, as the annotation is expected to beconsistent across and within languages; secondly,in light of the fact that the quality of data may affectthe results of any experiment or linguistic investigation carried out on those data. Gamba and Zeman(2023), investigating parsing performances, alreadyobserve this as regards the syntactic layer of these
data. Nevertheless, what has been observed withrespect to syntax does not necessarily apply to morphological features as well, and the extent to whichinconsistent morphological annotation affects parsing performances thus remains unclear.

Introduction

In Universal Dependencies (de Marneffe et al.,2021) five treebanks are available for Latin:1 Index
Thomisticus Treebank (ITTB; Passarotti, 2019),
Late Latin Charter Treebank (LLCT; Cecchini
et al., 2020b), Perseus (Bamman and Crane, 2011),
PROIEL (Haug and Jøhndal, 2008), UDante (Cecchini et al., 2020a). These treebanks differ on multiple levels. First, they cover different domains:a shallow distinction can be made between poetry (found in Perseus and, less, in UDante) andprose (all treebanks), but it can be further specified in terms of specific genre included. For
instance, ITTB encompasses philosophical texts,
while LLCT consists of charters, representing an
instance of documentary genre. Additionally, thehistory of Latin, spoken for over two millennia,
entails a substantial diachronic variation, as thelanguage gradually evolved over time. Indeed, thefive Latin treebanks include data that differ substantially in this respect. Already considering theMedieval treebanks alone, we can observe how
wide the covered time range is: ITTB encompasses
Medieval texts dating back to XIII century, LLCT
features Early Medieval charters (VIII-IX century),
1

For this purpose, we first propose a harmonisation of the morphological features of the five treebanks, and thereafter assess its impact on models
predicting morphology, as well as syntactic parsers.
Section 2 presents some related work and the motivation behind our study. Section 3 features an
overview of the harmonisation process, while in
Section 4 we describe the strategy designed to spotinconsistent or missing annotations. Section 5 highlights the main harmonising interventions, whose
impact on parsing accuracy is assessed in Section 6.
Finally, Section 7 concludes the paper and suggests
future research directions.

See https://universaldependencies.org/.

59
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 59–67,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_007

2Related Work and Motivation

treebank maintainers to refine morphologicalannotation. Towards the latter goal, we develop a UDapi (Popel et al., 2017) block fordetecting required and allowed morphologicalfeatures in Latin treebanks. The Latin block
was inspired by a similar block for Czech
and we will contribute it to the official UDapi
repository; it can be adapted to any other language by modifying the template according tolanguage-specific features.

Any NLP task is likely to show degraded performance when a model is applied to data thatdiffer from training data. It has been observed
several times that this issue is particularly prominent in (morpho-)syntactic parsing of Latin texts.
The issue is strongly intertwined with Latin intralinguistic variability, as the language has undergone
a number of significant changes by spreading overa period of more than two millennia and acrossEurope. In order to investigate genuine linguistic
diversity, first and foremost the impact of divergent
annotation styles has to be ruled out. To perform
any experiment that exploits data, we need thosedata to be consistent. Harmonising such discrepancies would allow for the isolation of the impactthat annotation choices have, so that actual intralinguistic variability can emerge and be examined.
The issue of Latin variability has been addressedin the two EvaLatin campaigns (Sprugnoli et al.,2020; Sprugnoli et al., 2022), aiming to evaluate
NLP tools for Latin. In particular, EvaLatin has
been focusing on lemmatisation, morphologicalanalysis and POS tagging. However, Latin diversity has been observed several times already before,in light of the behaviour of parsing accuracy, which
was far from being homogeneous. See, for instance,
Passarotti and Ruffolo (2010), Ponti and Passarotti(2016), Passarotti and Dell’Orletta (2010). Several studies have also been addressing the issue ofinconsistent annotations. Dickinson and Meurers(2003), Volokh and Neumann (2011), Ambati et al.
(2011), de Marneffe et al. (2017), Aggarwal and Zeman (2020), and Aggarwal and Alzetta (2021) areonly some of the methods that have been proposed
to detect inconsistencies in treebanks. Gamba and
Zeman (2023) present a harmonisation of dependency relations in Latin treebanks, yet without intervening at the level of morphological features.
Their harmonisation highlighted several levels of
inconsistencies and proved to lead to substantial
improvements in terms of parsing accuracy. We
investigate whether similar improvements can beachieved by also addressing inconsistencies in morphological annotation.

• Investigating the impact of harmonised morphological features in parsing, by assessing if
and to what extent they affect accuracy scores.
A comparison of two parsers, UDPipe (Straka
et al., 2016) and Stanza (Qi et al., 2020) is
proposed.

3

Overview of the Harmonisation Process

The focus of the harmonisation process is exclusively on morphological features.
We define the workflow to detect inconsistencies and missing features as follows. First, we run
the UDapi block on the input data, with the goalof spotting features which are either required but
missing, or not allowed. As output, the trees thatfeature either of these two kinds of inconsistencies
are stored in a html file, where those bugs areprominently highlighted (see Figure 1). In light ofthe output html file, we build Python scripts that
address and fix the observed bugs.
We employ the harmonised version of the fivetreebanks, as made available by Gamba and Zeman(2023), as input. Nevertheless, differently fromwhat was done for syntactic harmonisation, we donot strictly follow UDante annotation. This choice
is justified by the fact that we observe a considerable difference in the set of morphological features
employed in UDante – predominantly – and theother treebanks (ITTB and LLCT) maintained by
the same developers, i.e. the team at Università
Cattolica del Sacro Cuore in Milan, Italy, as opposed to the two remaining treebanks (Perseus and
PROIEL) out of the five available for Latin. We
thus decide to define two levels of coherence:

The output of the present study is two-fold:• lower level (default): only information which
can be considered somehow core, or more essential, is required. For instance, all pronouns
must have a PronType, and all verbs musthave VerbForm and Aspect.

• Producing a new version of the treebanks, harmonised at the level of morphological features,
to be potentially contributed to the UD official
release or to serve as an inspiration for other60

if re.match(r'^(VERB|AUX)$', node.upos):rf = ['VerbForm', 'Aspect']af = {'VerbForm': ['Inf', 'Fin',,→'Part', 'Conv'],'Aspect': ['Imp', 'Inch', 'Perf',,→'Prosp']}if node.feats['VerbForm'] not in,→['Part', 'Conv']:
rf.append('Tense')
af['Tense'] = ['Past', 'Pqp',,→'Pres', 'Fut']
if node.upos == 'VERB' or (node.upos,→== 'AUX' and node.lemma !=,→'sum'):
rf.append('Voice')af['Voice'] = ['Act', 'Pass']
if node.feats['VerbForm'] == 'Fin':
rf.extend(['Mood', 'Person',,→'Number'])af['Mood'] = ['Ind', 'Sub',,→'Imp']
af['Person'] = ['1', '2', '3']
af['Number'] = ['Sing', 'Plur']elif node.feats['VerbForm'] ==,→'Part':
rf.extend(['Gender', 'Number',,→'Case'])
af['Number'] = ['Sing', 'Plur']if,→node.misc['TraditionalMood'],→!= 'Gerundium' else ['Sing'],→
af['Gender'] = ['Masc', 'Fem',,→'Neut'] if,→
node.misc['TraditionalMood'],→!= 'Gerundium' else ['Neut']
af['Case'] = ['Nom', 'Gen',,→'Dat', 'Acc', 'Voc', 'Loc',,→'Abl']
af['Degree'] = ['Abs', 'Cmp']
if node.misc['TraditionalMood'].
,→
startswith('Gerundi'):af['Voice'] = ['Pass']
af['Aspect'] = 'Prosp'
elif node.feats['VerbForm'] ==,→'Conv':
rf.extend(['Case', 'Gender',,→'Number'])af['Case'] = ['Abl', 'Acc']
af['Gender'] = ['Masc']
af['Number'] = ['Sing']
af['Voice'] = ['Act']
elif node.feats['VerbForm'] ==,→'Inf':
af['Tense'].remove('Pqp')• higher level: additional information, such as
InflClass, is expected and allowed. Thislevel of validation can be applied only to a
subset of the Latin treebanks.
By default, the block operates at the lower level,
but a parameter can be supplied to UDapi, which
will trigger the more detailed features.
Morphologically harmonised treebanks and harmonisation scripts are available on GitHub,2 while
the block is available in UDapi GitHub repository. Moreover, we are ready to contribute theharmonised treebanks to the official UD release.

4

The markFeatsBugs Block

The markFeatsBugs block is structured as follows. For each UPOS tag, a set of required features
is first defined. (Note that the official UD validator3
has some limited ability to check permitted UPOSfeature combinations, but not to enforce requiredfeatures.) Additional features that are permitted
but not required are listed, and for each permitted
feature the set of its permitted values is defined. Unlike in the official UD validator, the conditions fora feature-value to be permitted or required are notlimited to whole UPOS categories. For example,
the UD validator knows that the Person feature isallowed for verbs and auxiliaries; but we further restrict it to finite forms, i.e., the feature VerbForm
must be present and its value must be Fin.
The set of allowed features is then expanded
to include additional feature-value pairs that may
be found in UDante, ITTB or LLCT (higher levelof detail). Eventually, the block checks for eachnode whether its morphological features are permitted and if every node has all the required features.
If not, invalid and missing features are explicitly
marked with a transparent label allowing to easily
distinguish them, and saved in the Bug attribute in
the MISC column of the CoNLL-U file. It can belater used in filters and highlighted in the data. The
code snippet in Script 1 provides an example, although not exhaustive, of the block section concerning verbs and auxiliaries, in compliance to what has
been implemented in the treebanks among all the
proposals illustrated in Cecchini (2021). Script 2

Script 1: Portion of the block that partially exemplifies
how morphological features are checked for the verbal
system: rf stands for ‘required features’, af stands for‘allowed features’.

2
https://github.com/fjambe/
Latin-variability/tree/main/morpho_
harmonization (commit 2d14807).
3
https://github.com/
UniversalDependencies/tools/blob/master/
validate.py.

illustrates the expansion of the feature-value sets tothe higher level, applicable to only three treebanks.
UDante is used as reference to select those features.
61if self.flavio:
af['Compound'] = ['Yes']af['Variant'] = ['Greek']
af['NameType'] = ['Ast', 'Cal',,→'Com', 'Geo', 'Giv', 'Let',,→'Lit', 'Met', 'Nat', 'Rel',,→'Sur', 'Oth']af['InflClass'] = ['Ind', 'IndEurA',,→'IndEurE', 'IndEurI', 'IndEurO',,→'IndEurU', 'IndEurX']

TraditionalTense is extended to finite forms
as well, for the purpose of consistency and in linewith UDante. As far as finite forms are concerned,
auxiliaries occurring in ITTB require some intervention as well. Unlike in the other treebanks, suchforms (e.g. sum ‘they are’) do not present Aspect,
Mood, Person and Tense. For the sake of consistency, we annotate them with respect to thosefeatures, assigning the relevant value.
Overall, the examinations of bugs highlighted by
the block confirms what has been already noted in
Gamba and Zeman (2023) with respect to Perseus
and PROIEL status: their level of annotation detail is remarkably lower in comparison to ITTB,
LLCT and UDante. An outstanding example isprovided by PronType, which is a key feature forpronouns and determiners. Often missing in particular in Perseus, it is systematically added during
the harmonisation process.
Additionally, the block can also serve as a tool
to spot isolated errors. Whenever such errors arehighlighted, we proceed to correct them.
Table 1 presents a quantitative overview of the
major interventions applied.

Script 2: Richer, more detailed morphological features
as allowed by the relevant parameter if set to 1.

The most representative example is InflClass,4
which reflects the original endings of the ProtoIndo-European stems. InflClass has not been
added everywhere in UDante, therefore – whenhigher-level validation is turned on – it is only considered as allowed, instead of required.

5

Harmonisation Examples

Three treebanks have been harmonised to thehigher level of detail (as defined in the previous sections): LLCT, ITTB and UDante. The remaining
two treebanks (Perseus and PROIEL) have been
harmonised to the lower level because the highlevel annotation is not available for them.
The harmonisation process derives transparently
from the feature constraints in the UDapi block. It
would not be helpful to discuss every constraint indetail here (and if necessary, the reader can referdirectly to the source code of the block); nonetheless, we want to discuss some interesting examples regarding verbs and auxiliaries. There is amore general issue raised by Cecchini (2021), who
proposes a reorganisation of Latin non-finite verbal features towards a higher degree of universality. In accordance with their proposal,5 we reannotate all gerund and gerundive forms as participles (VerbForm=Part) with Aspect=Prosp.
Traditional terminology used in grammars, i.e.
gerund and gerundive, is saved in the MISC
field as TraditionalMood=Gerund andTraditionalMood=Gerundive to prevent
loss of information and allow linguistic researchbased on traditional categories. Similarly, supine
forms are reannotated as VerbForm=Conv with
Aspect=Prosp and TraditionalMood=
Sup. The use of TraditionalMood and6

Impact on Parsing

To evaluate the significance of the harmonisation
process of morphological features, we try to investigate its impact on parsing accuracy. Therefore, we train new models for every morphologically harmonised treebank. The models aretrained on the same data, but in the first case
UDPipe 1.2 is used, while for the second one
we choose to employ Stanza. With both Stanza
and UDPipe we train the parser model on predicted lemmas and tags. Indeed, through Stanza’s
prepare_depparse_treebank.py script,6
the trained POS tagging model is used to retag
the training data before training the parser. Similarly, for UDPipe7 we train a parsing model that
relies on lemmas, UPOS tags and features as generated by the tagger. We use pretrained fastText
embeddings8 (Grave et al., 2018) and training hyperparameters as used for syntactic harmonisationin Gamba and Zeman (2023). For UDPipe, these
hyperparameters correspond to the optimised ones
6

https://stanfordnlp.github.io/stanza/training_and_evaluation.html.
7
https://ufal.mff.cuni.cz/udpipe/1/
users-manual#model_training_parser.
8
Available at https://fasttext.cc/docs/en/
crawl-vectors.html.

4
https://universaldependencies.org/la/feat/InflClass.html.
5With the only exception of the VNoun feature, which has
eventually not been introduced in UDante.

62Figure 1: Example of the html file highlighting bugs found in the data.
AspectGender_N

ITTB
26,243
2,655

LLCT
4,596
9,746

Perseus
4,344
1,037

PROIEL
35,42011,756

UDante-Gender_V

1,514

1,834

30

3,899

-

Gerund(ive)s
Mood
Number_V
NumForm=Word

2,740
20,26921,783
2,029

1,855
1,834
2,415

91
30
162

1,046
322
1,671

142

PersonPerson_P

20,269--1,346

15,887-

PronType

24,82521,062

3,105

31,023

21

Tense
Voice
Voice_NO

51,096
2,591-

10,988
1,855
4,113

1,277
216
369

9,430
1,064
7,848

-

notes
Aspect is added.
Gender is added, corrected or deleted (nominal only).
Gender is added, corrected or deleted (verbal only).
Interventions on gerunds and gerundives.
Mood is added.
Number is added (verbal only).
NumForm=Word is added to numerals likeviginti ‘twenty’.
Person is added to verbs.
Person in pronouns is either added, if missing, or deleted, if not relevant.
PronType is either added, if missing, orcorrected.
Tense is either added, corrected or deleted.
Voice is added when missing.
Voice is deleted when not relevant.

Table 1: Count of harmonising interventions.

made available for reproducible training by Straka
and Straková (2019) when available (ITTB, Perseus
and PROIEL), and to parameters inspired by thosein the case of LLCT and UDante.9
We then evaluate the parsing model on morphologically harmonised test data for each treebank
and compare results to the accuracy scores obtainedwith parsing models trained on data that underwent
a harmonisation process only at syntax level.10
Tables 2 and 3 report results obtained with UDPipe, in terms of Labeled Attachment Score (LAS)and Unlabeled Attachment Score (UAS) (Buchholz
and Marsi, 2006), whereas Tables 6 and 7 presents
analogous scores as obtained by the model trainedwith Stanza. Scores highlighted in blue denote
an increase, while scores highlighted in red pinpoint decreased results. Accuracy is measured withthe evaluation script11 designed for the CoNLL
2018 Shared Task on Multilingual Parsing fromRaw Text to Universal Dependencies (Zeman et al.,2018), which takes into consideration main dependency relations only and not subtypes.
First and foremost, a clarification is necessary.
As explained earlier, the treebanks are not forced
all to the same set of features: LLCT, ITTB and
UDante have some extra features that are not foundin Perseus and PROIEL. It would be possible toremove these extra features for the sake of parsing
evaluation but we chose to keep them. One canthus expect somewhat worse results when applying
models from one of these treebank groups to test
data from the other group.
As illustrated in the tables, the results do notshow any clear pattern and, overall, the improvements are neither widespread nor substantial. A
closer look at the scores reveals that UDPipe shows
improved accuracy scores in less than half of thecases, and in general performs worse than Stanza,with the gap being almost around 10% on average. Improvements obtained with models trainedon UDPipe are never substantial and, in general,very hard to interpret. Stanza seems to allow forsome additional remark. We first want to examine
distinctly the two groups that correspond to the two
possible values of the discussed parameter. The

9
LLCT: learning_rate=0.02, transition_system=swap, transition_oracle=static_lazy, structured_interval=8.
UDante: learning_rate=0.01, transition_system=projective,
transition_oracle=dynamic, structured_interval=8.
10
In both cases parsing models are trained on predictedtags.
11
https://github.com/
UniversalDependencies/tools/blob/master/eval.py.

63

ITTB
LLCT
Perseus
PROIEL
UDanteittb.udp
LAS
UAS
79.86% 83.11%
35.50% 45.63%
44.14% 55.57%
49.37% 58.58%
46.89% 57.28%

llct.udp
LAS
UAS
38.62% 50.59%
91.84% 93.20%
32.60% 45.50%
36.67% 48.72%
31.85% 44.31%

perseus.udp
LAS
UAS
44.16% 53.86%
32.64% 42.66%
43.73% 57.28%
45.23% 54.41%
34.51% 45.73%

proiel.udp
LAS
UAS
45.19% 55.56%
35.81% 47.55%
40.36% 53.25%
70.02% 75.16%
35.50% 47.64%

udante.udp
LAS
UAS
53.51% 63.06%
30.86% 41.31%
42.13% 54.23%
41.85% 53.13%
48.24% 57.99%

Table 2: UDPipe LAS and UAS before morphological harmonisation. Columns correspond to trained models, rows
to test data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.udp
LAS
UAS
81.01% 84.05%
34.76% 44.55%
42.89% 53.76%
49.96% 58.89%
46.31% 56.18%

llct.udp
LAS
UAS
39.31% 51.29%
91.57% 92.72%
31.52% 44.65%
36.84% 49.02%
31.20% 43.72%

perseus.udp
LAS
UAS
45.22% 55.37%
32.12% 41.00%
47.76% 57.33%
45.16% 54.51%
34.20% 45.60%

proiel.udp
LAS
UAS
44.42% 54.10%
36.55% 48.25%
39.99% 51.96%
70.24% 75.59%
35.76% 46.51%

udante.udp
LAS
UAS
53.66% 62.40%
32.69% 42.44%
41.49% 53.36%
41.80% 52.72%
47.99% 57.44%

Table 3: UDPipe LAS and UAS after morphological harmonisation. Columns correspond to trained models, rows totest data.

7

LLCT model obtains lower accuracy scores only
on Perseus, which presents a more coarse-grained
morphological annotation, but not on any of thetreebanks belonging to the same class. A similar
remark could be made about the ITTB model; thelower scores obtained on ITTB test data, despite
being coloured in red, are probably not significant.
Nevertheless, this reasoning does not hold true forthe model trained on UDante, which incongruously
performs best on Perseus and PROIEL. On theother hand, the PROIEL model is the only one
showing improvements on all test data; despite notbeing substantial in most of the cases, a +3% increase can be observed when the model is used toparse LLCT data.
All the discussion so far concerns syntactic parsing, which is only indirectly affected by the consistency of morphological annotation. So the natural
next question is about the impact of the harmonisation on prediction of morphology. Both UDPipeand Stanza predict morphological annotation together with syntax. Tables 4 and 8 show accuracyof feature prediction (percentage of correct words,
whereas a word is correct if all its feature-value
pairs have been predicted correctly). Each accuracy
is computed before and after harmonisation, shownin the same table. Here we see a clear improvement
in all experiments where a model is applied to datafrom different treebank; and for ITTB and PROIEL,
the improved consistency led to improvement also
in the in-domain experiment. The improvement isfurther confirmed in Tables 5 and 9, which show
the MLAS scores (Zeman et al., 2018), combining
morphology and syntax.

Conclusion and Future Work

The paper presents the harmonisation process that
we carried out, with respect to morphology, on
the five Latin UD treebanks. We first defined an
UDapi block for Latin, listing which morphological features a token should possess. Such lists offeatures are defined based on UPOS tags. Subsequently, we corrected the retrieved inconsistencies– consisting in either missing or not allowed features – via Python scripts. As a result, we produced
morphologically harmonised versions of the Latin
treebanks that were previously harmonised syntactically (Gamba and Zeman, 2023). We contributed
the script to investigate Latin features, possibly
reusable by anyone working on Latin treebanks,
and we described a workflow that can be replicated and applied to potentially any other language,
provided that language-specific information is supplied within the template. In the second part ofthe paper, we presented some parsing experiments
carried out with UDPipe and Stanza. By comparing syntactic attachment scores before and after
morphological harmonisation, we observed the absence of a clear pattern that would allow to explain
results; on the other hand, morphological accuracy
clearly improved. The coexistence of a coarsegrained and a fine-grained level of consistency inannotation partially explains the outcome of theparsing experiments, that however must not discourage from pursuing an ever-growing harmonisation of linguistic resources in terms of annotation
choices. Intra- and inter-resource consistency isa key factor to exploit data, whether it comes to
64

ITTB
LLCT
Perseus
PROIEL
UDanteittb.udp
beforeafter
93.57% 93.91%
52.38% 60.39%
52.54% 65.25%
46.47% 69.98%
58.30% 64.99%

llct.udp
beforeafter
55.41% 63.72%
95.89% 95.86%
46.74% 55.10%
45.12% 56.83%
47.47% 54.90%

perseus.udp
beforeafter
53.76% 69.09%
50.53% 60.36%
72.03% 71.11%
61.16% 69.11%
44.60% 59.29%

proiel.udp
beforeafter
54.50% 78.02%
54.45% 67.45%
69.45% 76.26%
87.19% 88.87%
48.30% 69.57%

udante.udp
beforeafter
62.67% 70.68%
50.59% 58.68%
45.12% 57.77%
40.35% 59.81%
74.84% 74.67%

Table 4: Comparison of UDPipe accuracy scores on morphological features. Columns correspond to trained models,rows to test data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.udp
beforeafter
69.97% 71.64%
10.41% 13.14%
15.37% 21.98%
16.14% 29.49%
18.87% 21.32%

llct.udp
beforeafter
15.10% 17.23%
85.76% 85.50%
8.68%
12.80%
10.81% 15.58%
8.62%
10.15%

perseus.udp
beforeafter
15.24% 22.90%
6.49%
11.38%
28.60% 28.89%
19.00% 25.21%
8.94%
13.53%

proiel.udp
beforeafter
18.68% 30.85%
11.07% 17.04%
23.59% 29.45%
56.42% 58.07%
11.97% 19.43%

udante.udp
beforeafter
25.39% 29.04%
7.52%
8.93%
10.70% 17.59%
11.47% 18.82%
25.90% 25.46%

Table 5: Comparison of UDPipe MLAS scores. Columns correspond to trained models, rows to test data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.mdl
LAS
UAS
88.60% 90.55%
40.84% 52.66%
57.68% 67.85%
62.34% 71.27%
56.62% 67.27%

llct.mdl
LAS
UAS
45.63% 58.74%
94.61% 95.81%
40.80% 53.88%
46.76% 59.92%
39.67% 52.97%

perseus.mdl
LAS
UAS
50.55% 61.47%
37.82% 47.50%
58.41% 68.22%
55.03% 65.25%
39.53% 52.98%

proiel.mdl
LAS
UAS
51.16% 60.72%
40.97% 53.24%
47.30% 58.68%
80.57% 84.36%
41.27% 52.41%

udante.mdl
LAS
UAS
63.78% 72.96%
43.64% 56.09%
52.98% 64.06%
52.61% 63.91%
57.92% 67.60%

Table 6: Stanza LAS and UAS before morphological harmonisation. Columns correspond to trained models, rows
to test data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.mdl
LAS
UAS
88.29% 90.28%
42.18% 54.50%
59.00% 69.00%
62.33% 71.27%
58.24% 68.42%

llct.mdl
LAS
UAS
46.93% 60.21%
94.91% 96.08%
39.82% 53.34%
48.17% 61.25%
40.39% 53.84%

perseus.mdl
LAS
UAS
50.02% 60.22%
38.10% 48.50%
59.43% 68.97%
55.56% 64.81%
39.73% 52.47%

proiel.mdl
LAS
UAS
52.86% 62.13%
42.48% 56.08%
47.97% 59.36%
81.25% 84.91%
41.41% 52.74%

udante.mdl
LAS
UAS
64.87% 72.91%
42.43% 54.97%
54.26% 65.17%
54.37% 64.41%
57.40% 66.79%

Table 7: Stanza LAS and UAS after morphological harmonisation. Columns correspond to trained models, rows totest data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.mdl
beforeafter
95.70% 96.15%
56.92% 63.95%
57.29% 72.49%
49.88% 75.90%
62.47% 69.85%

llct.mdl
beforeafter
57.07% 66.19%
96.89% 96.81%
48.66% 57.23%
48.31% 60.97%
48.56% 56.32%

perseus.mdl
beforeafter
55.19% 72.91%
53.53% 65.33%
78.02% 77.86%
66.57% 75.95%
45.89% 63.42%

proiel.mdl
beforeafter
52.14% 79.97%
57.07% 71.87%
70.01% 79.51%
90.91% 92.72%
46.22% 70.64%

udante.mdl
beforeafter
66.22% 75.34%
55.73% 63.47%
49.75% 64.63%
44.53% 67.10%
79.39% 79.30%

Table 8: Comparison of Stanza accuracy scores on morphological features. Columns correspond to trained models,rows to test data.

ITTB
LLCT
Perseus
PROIEL
UDanteittb.mdl
beforeafter
78.97% 80.74%
12.22% 17.67%
22.63% 35.20%
22.23% 41.32%
25.06% 29.95%

llct.mdl
beforeafter
16.56% 19.07%
89.46% 90.04%
11.57% 16.92%
14.86% 22.74%
12.21% 14.77%

perseus.mdl
beforeafter
19.45% 27.87%
9.12%
16.63%
38.86% 40.21%
27.64% 35.92%
10.64% 17.37%

proiel.mdl
beforeafter
22.13% 40.05%
15.98% 24.25%
31.33% 38.66%
68.49% 71.23%
13.45% 25.40%

udante.mdl
beforeafter
33.14% 39.59%
12.59% 18.02%
16.25% 27.29%
17.17% 30.61%
35.96% 35.32%

Table 9: Comparison of Stanza MLAS scores. Columns correspond to trained models, rows to test data.

65

linguistic research or any other application.

References

In light of the slight improvement that resulted inparsing accuracy from the harmonisation process,we do not plan on further developing the harmonisation of treebanks. The higher degree of consistency in treebank annotation, i.e. the availabilityof more homogeneous data, allows now to investigate the actual reasons for variability in parsing.
Syntactic constructions evolving over time may beinspected, as well as other factors that may affectparsing results on data that differ from training
data – as already problematised several times, e.g.
by Passarotti and Dell’Orletta (2010), Passarotti
and Ruffolo (2010), Ponti and Passarotti (2016).
Variation in time is most probably expected to bea relevant factor, and it is strongly connected totwo other relevant variables, i.e. space and domain.
Consider, for instance, the Late Latin Charter Treebank: while featuring early medieval Latin (VIIIIX century), not as late as ITTB (XIII century) andUDante (XIV century) Latin varieties, the treebank
does not include literary texts yet charters written in Tuscany, Italy. The gradual developmentof Latin towards Romance languages, exemplified
by evolving syntactic constructions and changes inword endings, can already be observed in the treebank (Cecchini et al., 2020c). Variation in terms ofgenre appears to be relevant also with respect to the
distinction between poetry and prose. With Latin
treebanks encompassing mostly literary data, such
distinction cannot be overlooked. Indeed, Latin poetry is strongly affected by prosody and metre: thesequence of short and long syllables in words, as
defined by prosodic rules, together with the specificstructure of the selected metre, rigidly determine
possible sequences of words. As a result, the natural word order is unsettled, and the position of aword in the verse (and, hence, in the sentence) ismostly defined by the way its short and long syllables follows one another. This whole mechanism,highly affecting word order, entails a high degreeof non-projectivity, and would need to be further
inspected.

Akshay Aggarwal and Chiara Alzetta. 2021. Atypicalor underrepresented? A pilot study on small treebanks. In Proceedings of the Eighth Italian Conference on Computational Linguistics, CLiC-it 2021,
Milan, Italy, January 26-28, 2022, volume 3033 of
CEUR Workshop Proceedings. CEUR-WS.org.
Akshay Aggarwal and Daniel Zeman. 2020. Estimating
POS annotation consistency of different treebanks
in a language. In Proceedings of the 19th International Workshop on Treebanks and Linguistic Theories, pages 93–110, Düsseldorf, Germany. Association for Computational Linguistics.
Bharat Ram Ambati, Rahul Agarwal, Mridul Gupta,
Samar Husain, and Dipti Misra Sharma. 2011. Error
detection for treebank validation. In Proceedingsof the 9th Workshop on Asian Language Resources,
pages 23–30, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
David Bamman and Gregory Crane. 2011. The Ancient
Greek and Latin Dependency Treebanks. In Language technology for cultural heritage, pages 79–98.
Springer.
Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X
shared task on multilingual dependency parsing. In
Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages
149–164, New York City. Association for Computational Linguistics.
Flavio M Cecchini, Rachele Sprugnoli, Giovanni
Moretti, and Marco Passarotti. 2020a. UDante: First
Steps Towards the Universal Dependencies Treebank
of Dante’s Latin Works. In Seventh Italian Conference on Computational Linguistics, pages 1–7.
CEUR Workshop Proceedings.
Flavio Massimiliano Cecchini. 2021. Formae reformandae: for a reorganisation of verb form annotation in
Universal Dependencies illustrated by the specificcase of Latin. In Proceedings of the Fifth Workshop
on Universal Dependencies (UDW, SyntaxFest 2021),
pages 1–15, Sofia, Bulgaria. Association for Computational Linguistics.
Flavio Massimiliano Cecchini, Timo Korkiakangas, andMarco Passarotti. 2020b. A New Latin Treebank
for Universal Dependencies: Charters between Ancient Latin and Romance Languages. In Proceedings of the 12th Language Resources and Evaluation
Conference, Marseille, France. European Language
Resources Association.

Acknowledgements

Flavio Massimiliano Cecchini, Timo Korkiakangas, andMarco Passarotti. 2020c. A new Latin treebank for
Universal Dependencies: Charters between Ancient
Latin and Romance languages. In Proceedings of the
Twelfth Language Resources and Evaluation Conference, pages 933–942, Marseille, France. European
Language Resources Association.

This work was supported by the Grant No. 2016819X (LUSyD) of the Czech Science Foundation(GAČR).
66

Marie-Catherine de Marneffe, Matias Grioni, Jenna
Kanerva, and Filip Ginter. 2017. Assessing the annotation consistency of the Universal Dependencies corpora. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017),
pages 108–115, Pisa,Italy. Linköping University Electronic Press.

Martin Popel, Zdeněk Žabokrtský, and Martin Vojtek.
2017. Udapi: Universal API for Universal Dependencies. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages
96–101, Gothenburg, Sweden. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, andChristopher D. Manning. 2020. Stanza: A python
natural language processing toolkit for many humanlanguages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics:
System Demonstrations, pages 101–108, Online. Association for Computational Linguistics.

Marie-Catherine de Marneffe, Christopher Manning,
Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics,
47(2):255–308.
Markus Dickinson and W Detmar Meurers. 2003. Detecting inconsistencies in treebanks. In Proceedings
of TLT, volume 3, pages 45–56.

Rachele Sprugnoli, Marco Passarotti, Flavio Massimiliano Cecchini, Margherita Fantoli, and Giovanni
Moretti. 2022. Overview of the EvaLatin 2022 evaluation campaign. In Proceedings of the Second
Workshop on Language Technologies for Historicaland Ancient Languages, pages 183–188, Marseille,
France. European Language Resources Association.

Federica Gamba and Daniel Zeman. 2023. Universalising Latin Universal Dependencies: a harmonisationof Latin treebanks in UD. In Proceedings of the
Sixth Workshop on Universal Dependencies (UDW,GURT/SyntaxFest 2023), pages 7–16, Washington,
D.C. Association for Computational Linguistics.

Rachele Sprugnoli, Marco Passarotti, Flavio Massimiliano Cecchini, and Matteo Pellegrini. 2020.
Overview of the EvaLatin 2020 evaluation campaign.
In Proceedings of LT4HALA 2020 - 1st Workshop
on Language Technologies for Historical and Ancient Languages, pages 105–110, Marseille, France.
European Language Resources Association (ELRA).

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning
word vectors for 157 languages. In Proceedings ofthe Eleventh International Conference on Language
Resources and Evaluation (LREC 2018), Miyazaki,
Japan. European Language Resources Association(ELRA).

Milan Straka, Jan Hajič, and Jana Straková. 2016. UDPipe: Trainable pipeline for processing CoNLL-U
files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of
the Tenth International Conference on Language
Resources and Evaluation (LREC’16), pages 4290–
4297, Portorož, Slovenia. European Language Resources Association (ELRA).

Dag TT Haug and Marius Jøhndal. 2008. Creating a
Parallel Treebank of the Old Indo-European Bible
Translations. In Proceedings of the Second Workshop
on Language Technology for Cultural Heritage Data
(LaTeCH 2008), pages 27–34.
Marco Passarotti. 2019. The Project of the Index
Thomisticus Treebank. Digital Classical Philology,
10:299–320.

Milan Straka and Jana Straková. 2019. Universal dependencies 2.5 models for UDPipe (2019-12-06).
LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.

Marco Passarotti and Felice Dell’Orletta. 2010. Improvements in parsing the index Thomisticus treebank. revision, combination and a feature model formedieval Latin. In Proceedings of the Seventh International Conference on Language Resources and
Evaluation (LREC’10), Valletta, Malta. European
Language Resources Association (ELRA).

Alexander Volokh and Günter Neumann. 2011. Automatic detection and correction of errors in dependency treebanks. In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies, pages
346–350, Portland, Oregon, USA. Association for
Computational Linguistics.

Marco Passarotti and Paolo Ruffolo. 2010. Parsing
the Index Thomisticus Treebank. Some Preliminary
Results. In 15th International Colloquium on Latin
Linguistics, pages 714––725. Innsbrucker Beiträge
zur Sprachwissenschaft.

Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast,
Milan Straka, Filip Ginter, Joakim Nivre, and SlavPetrov. 2018. CoNLL 2018 shared task: Multilingualparsing from raw text to Universal Dependencies. In
Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association
for Computational Linguistics.

Edoardo Maria Ponti and Marco Passarotti. 2016. Differentia compositionem facit. a slower-paced and reliable parser for Latin. In Proceedings of the TenthInternational Conference on Language Resources
and Evaluation (LREC’16), pages 683–688, Portorož,Slovenia. European Language Resources Association(ELRA).

67

Cross-Lingual Constituency Parsing for Middle High German:
A Delexicalized Approach1

Ercong Nie1,2Helmut Schmid1
Hinrich Schütze1,2Center for Information and Language Processing (CIS), LMU Munich, Germany
2
Munich Center for Machine Learning (MCML), Germany
nie@cis.lmu.de

Abstract

the analysis of ancient languages that are no longer
spoken. Its significance extends beyond linguistic
analysis, serving as a building block for various
natural language processing (NLP) applications,such as information extraction (Jiang, 2012; Jiang
and Diesner, 2019), sentiment analysis (Li et al.,2020), question answering (Hermjakob, 2001), etc.
However, ancient languages lack large labeled andunlabeled corpora (Assael et al., 2022) and treebanks suitable for parser training are seldom available. This scarcity of resources can be attributed
to two reasons. Firstly, ancient languages usually
have a dearth of digital text resources. Secondly,
the construction of a treebank for an ancient language requires substantial linguistic expertise andmanual effort. Nonetheless, the continuity in theprocess of language evolution gives rise to linguistic similarities between ancient languages and theircorresponding modern counterparts (Parravicini
and Pievani, 2018). Cross-lingual transfer techniques (Ruder, 2019; Lauscher et al., 2020) aretrained on high-resource languages and require little or no annotated data from low-resource target
languages. They can effectively be applied to languages with similar sentence structure and word
order. Hence, they can be a viable solution to thischallenge.

Constituency parsing plays a fundamentalrole in advancing natural language processing (NLP) tasks. However, training an automatic syntactic analysis system for ancient languages solely relying on annotated parse data is
a formidable task due to the inherent challenges
in building treebanks for such languages. It demands extensive linguistic expertise, leading toa scarcity of available resources. To overcome
this hurdle, cross-lingual transfer techniques
which require minimal or even no annotateddata for low-resource target languages offer a
promising solution. In this study, we focus
on building a constituency parser for Middle
High German (MHG) under realistic conditions, where no annotated MHG treebank isavailable for training. In our approach, weleverage the linguistic continuity and structuralsimilarity between MHG and Modern German(MG), along with the abundance of MG treebank resources. Specifically, by employing
the delexicalization method, we train a constituency parser on MG parse datasets and perform cross-lingual transfer to MHG parsing.
Our delexicalized constituency parser demonstrates remarkable performance on the MHG
test set, achieving an F1-score of 67.3%. It
outperforms the best zero-shot cross-lingual 1
baseline by a margin of 28.6% points. These
encouraging results underscore the practicality
and potential for automatic syntactic analysis inother ancient languages that face similar challenges as MHG.

1

In this work, we focus on building a constituency
parser for Middle High German (MHG). MHG isa historical stage of the German language that was
spoken between 1050 and 1350. It is the linguistic
predecessor of Modern German (MG). Both languages have many similarities in word formationand grammatical features, e.g., similar word orderpatterns and inflectional systems (Salmons, 2018).
The availability of MHG parse trees is extremely
limited. The Deutsche Diachrone Baumbank (German Diachronical Treebank, DDB) (Hirschmannand Linde, 2023) comprises merely around 100
manually annotated parse trees, encompassing lessthan 3000 tokens. These resources are far from

Introduction

Constituency parsing, which involves analyzing thegrammatical structure of sentences and identifying
the hierarchical relationships between words, plays
a crucial role in linguistic research, especially for1

As is prevalent in the realm of multilingual NLP, theterm “zero-shot cross-lingual” in this context pertains to atransfer learning method where we finetune the model withtask-specific data in a source language and test on the target
language directly (Sitaram et al., 2023).

68
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 68–79,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_008

TrainingInference
S

S
NP

OutputVP

PPER VVFIN

Output
ADV

NP

VP

PRON

VVFIN

NP

ART

Delexicalized Model

Delexicalized Model1

Decoder

Decoder

NN

Cross-lingualtransfer…

Encoder

POS Embebdding

Delexicalized
Input

POS Embebdding
Delexicalized
Input

PPER VVFIN ADVPDS VVFIN ART

NN

TagMapper 3
DDS VVFIN DIART NA

Delexicalization
Er
(He

…

Encoder

MHGTagger 2

liest auchreads too)

diu
(This

MG

ist
is

ein muoter
a mother)

MHG

Figure 1: Overview of the cross-lingual delexicalized parsing system for MHG. In the training, the delexicalized
parsing model is trained on the delexicalized MG trees. The trained parser is subsequently applied to MHG sentences.
The delexicalized parsing system for MHG consists of three key modules: (1) Delexicalized parsing model trainedon delexicalized MG trees, (2) MHG POS tagger, and (3) Tag mapper.

what is required to train an automatic syntactic analysis system, and are only suitable for use as test
sets. On the other hand, there is an abundance oftreebank resources available for MG, in particular
the Tiger Treebank (Smith, 2003). Hence, we capitalize on the structural similarity between MHG
and MG, as well as the rich MG treebank resourcesin order to develop a cross-lingual delexicalized
constituency parsing model that we can directly
apply to MHG sentences.

ploy a POS mapper to replace the MHG tags by thecorresponding MG tags, ensuring the uniformity ofthe model’s inputs across the two languages, which
is a prerequisite of the delexicalization method.
The experimental results show that our delexicalized constituency parser substantially outperforms
all other zero-shot cross-lingual parsing baselines,
achieving an F1-score of 67.3% on the MHG parse
test set.
The delexicalization method is particularly wellsuited for languages which (1) lack treebank resources, (2) possess sufficient annotated data fortraining POS taggers, and (3) exhibit syntactic similarities with a high-resource language. Our investigation of this realistic scenario shows the feasibilityof automatic syntactic analysis for an ancient language.

In the delexicalized approach, the parsing model
operates on part-of-speech (POS) sequences rather
than token sequences. We accomplish this by training a cross-lingual parser using POS sequencesfrom high-resource source languages as input. Subsequently, we utilize this trained parser to directly
parse POS sequences of low-resource target languages (McDonald et al., 2011).
In our work, we first train a delexicalized constituency parsing model on a delexicalized MG
treebank. In order to parse MHG sentences withthis model, we need to annotate them first with the
POS tags used in the MG treebank. To this end,
we train a POS tagger on an MHG corpus which
has been manually annotated using a POS tag set
similar, but not identical to the MG tag set. We em-

The subsequent sections of this paper are organized as follows. In Sec. 2, we discuss related work.
Sec. 3 gives an overview of our research languages
and the available corpora. The delexicalization
method employed in our approach is detailed inSec. 4. Sec. 5 describes our experimental setup,
and in Sec. 6, we analyze the results. We conclude
in Sec. 7.
692

Related Work

tage of the newly developed pretrained language
models (PLMs) and use BERT (Devlin et al., 2019)
to compute the span representations, resulting inenhanced performance. Kitaev and Klein (2020)
improve the runtime complexity of constituency
parsing to linear time by reducing parsing to tagging.

Cross-Lingual Transfer Learning The fundamental principle underlying cross-lingual transfer
is that the processing of source and target languages
uses a shared input representation, which can beeither discrete or continuous. The delexicalization
method is based on a shared discrete input representation, i.e., POS tags. Other discrete representation
types include glossed words (Zeman and Resnik,2008) and grounding texts in multilingual knowledge bases (Lehmann et al., 2015). Continuous
cross-lingual representation spaces emerged withadvancements in neural networks. Typical examples are cross-lingual word embeddings (Ammar
et al., 2016) and sentence embeddings (Artetxe andSchwenk, 2019).
The emergence of massively multilingual transformers (Devlin et al., 2019; Conneau et al., 2020),
which are jointly pretrained on multilingual corpora, introduces a novel pattern of zero-shot crosslingual transfer learning. In this paradigm, a pretrained multilingual model is finetuned on a downstream NLP task dataset of a source language. The
finetuned multilingual model is then directly applied to target language data for the same task (Ket al., 2019; Hu et al., 2020; Liu et al., 2022; Nieet al., 2023).

Cross-Lingual Constituency Parsing There has
been relatively limited scholarly attention dedicated to cross-lingual constituency parsing in recent studies, especially for target languages situated in low-resource settings, such as MHG. Kitaev et al. (2019) have employed the multilingual
BERT model to train a single parser with parameters shared across languages. They jointly finetune
the multilingual BERT on 10 languages utilizing acommon BERT backbone, but the model contains
distinct MLP span classifiers for each language toaccommodate the different tree labels. However,their approach necessitates the availability of treebanks of all the encompassed languages as training
datasets. Kaing et al. (2021) undertake a comprehensive series of experiments to validate the efficacy of delexicalization techniques for zero-shotcross-lingual constituency parsing. Additionally,
their study underscores significance of typologicalaffinity in the source language selection. We build
upon these investigations and apply their findings
to the zero-shot parsing of MHG within a practicalcontextual framework.

Neural Constituency Parsing Recent advancesin constituency parsing have witnessed a growing
emphasis on harnessing neural network representations, making a shift from the previously prominentrole of grammars, whose relevance has gradually diminished. Cross and Huang (2016) propose a spanbased constituency parsing system specifically designed to leverage the powerful representation capabilities of the bidirectional long short-term memory(LSTM) networks (Hochreiter and Schmidhuber,
1997). In this method, an input sentence is represented as a set of spans, and each span is assigned ascore. The best-scoring parse tree is computed using dynamic programming techniques. They combine smaller spans into larger spans until the entire
sentence is covered. Subsequently, several variations of the span-based method have been proposed,
e.g. approaches replacing the inference algorithmwith chart-based methods (Stern et al., 2017), using character-level representations instead of wordlevel representations (Gaddy et al., 2018), and replacing LSTMs with self-attention modules (Kitaev
and Klein, 2018). Kitaev et al. (2019) take advan-

Constituency Parsing on Historical German
There is a notable scarcity of syntactically annotated corpora for historical German. In instanceswhere annotated treebanks are absent, approaches
such as rule-based, unsupervised, or zero-shotcross-lingual methods can be employed for constituency parsing, For instance, Chiarcos et al.
(2018) have created a rule-based shallow parser for
MHG. Recent advancements in the construction ofsuch corpora encompass:• German Diachronical Treebank (DDB): asmall yet syntactically deeply annotated corpus, comprising three subcorpora of different stages of German, i.e., Old High German,
Middle High German and Early New HighGerman (Hirschmann and Linde, 2023). The
construction of the DDB corpus is oriented
towards the Tiger Corpus (Smith, 2003), oneof the largest German treebanks.
• UP Treebank of Early New High German
70

(ENHG): a syntactically annotated corpus of
ENHG containing 21,432 sentences consisting of 600,569 word tokens based on the Reference Corpus of ENHG (Demske, 2019).

forms. Texts in ReM have been digitized andrichly annotated, e.g., with POS, morphologicaland lemma features. The morphological annotation uses the HiTS tag set (Dipper et al., 2013),
a tag set for historical German, derived from the
Stuttgart-Tübinger Tag Set (STTS) for modern German texts (Schiller et al., 1995). Although the ReM
corpus provides rich morphologically annotatedtext data for MHG, the availability of syntacticallyannotated data for MHG is severely limited, with
only approximately 100 MHG parse trees includedin the DDB treebank. In contrast, the treebank
resources for MG are abundant. The Tiger Treebank (Brants et al., 2002), for instance, consists ofapproximately 40,000 sentences of German newspaper text, taken from the Frankfurter Rundschau.

• Corpus of Historical Low German (CHLG):
a Penn-style treebank of Middle Low German (Booth et al., 2020)
Contemporary work on historical German parsing
based on previously mentioned corpora includes endeavors such as cross-dialectal parsing for ENHG
based on CHLG (Sapp et al., 2023).

3

Languages and Corpora

The ancient language which we study in this paper is Middle High German (MHG). MHG andModern German (MG) are stages of the same Germanic language family, representing different historical periods. MHG emerged during the Middle
Ages in the German-speaking regions of Central
Europe. It was primarily used in literary and administrative contexts and played an important rolein medieval literature, including epic poems suchas the Nibelungenlied and Minnesang (courtly love
poetry) (Salmons, 2018).

4

Methods

In our work, we focus on developing a constituency
parser for MHG. In the previous section, we reviewed annotated resources available for MHG and
MG. Basically, we have ample treebank resources
for MG and plenty of POS-tagged texts for MHG,
whereas the treebank resources for MHG are extremely limited. Given the resource availability
for MG and MHG along with the linguistic connection between the two languages, employing across-lingual constituency parsing approach utilizing delexicalization proves to be an effective solution. As Figure 1 shows, the delexicalized modelis trained on the delexicalized inputs of MG. In theinference stage, the delexicalized parser is directly
applied to MHG POS sequences. The delexicalization method requires that MHG and MG sharethe same set of POS tags. The final constituency
parser for MHG (the right side of Figure 1 comprises three modules: (1) the delexicalized parser,(2) the MHG POS tagger, and (3) the POS mapper
from MHG to MG. In the next section, we describe
the delexicalized parsing system in more detail.

Linguistic Considerations of MHG MHG has
a phonetic system that included a set of vowel andconsonant sounds. The pronunciation and soundpatterns differ from those of MG, but some MHG
words are still recognizable in MG. MHG has amore complex grammatical system, such as a more
extensive case system with different noun and adjective declensions. Besides, verb conjugation has
more intricate forms and patterns (Jones and Jones,
2019). In terms of orthography, the spelling andwriting conventions of MHG are different from
MG. For example, ü, the umlaut of u, is usually
written iu in MHG. The transition from MHG to
MG was a gradual process, occurring over several
centuries. MG can be considered the linguistic
descendant of MHG, with linguistic changes anddevelopments shaping the language over time.

4.1

Delexicalized Parser

Our delexicalized MHG parser is based on the
Berkeley neural parser (Benepar) (Kitaev andKlein, 2018), a span-based parser using selfattention. As illustrated in Figure 1, Benepar has
an encoder-decoder architecture which combinesa chart decoder with a sentence encoder based on
self-attention. The sentence encoder computes contextualized representations for all word positions
and combines them to form span representations.

MHG Corpora Resources During the MHG period, the amount of textual material that survives
to the present increases markedly. The Reference
Corpus of Middle High German (ReM) (Klein
et al., 2016) encompasses a large collection of nonliterary and non-religious texts. ReM is a corpus
of diplomatically transcribed and annotated texts
of MHG with a size of around 2 million word
71

bank contain additional semantic information, suchas edge labels, and special structures, such as coreference indices and trace nodes. We remove all ofthem during delexicalization.
In the Tiger treebank, the label of each preterminal node contains not only the POS tag, butalso morphological features, such as case, number, gender. During delexicalization, we overwrite
the word at the leaf node with this extended POStag, but only keep the POS information in the label of the preterminal node. This means that the
input of our delexicalized parser contains information about morphological features. Figure 2 shows
an example of the delexicalization for a MG sentence. As shown the edge labels, e.g., “NK” areremoved and the tokens are replaced by the POS
tag combined with morphological features, e.g.,“ART.Nom.Pl.Fem”, where “ART” (determiner)is the POS tag, and “Nom.Pl.Fem” denotes themorphological information with case being nominative, number being plural, and gender being feminine.

TOP
S-TOP
NP-SB

VAFIN-HD-3.Pl.Pres.Ind

AP-PD

sind

PTKA-MO ADJD-HD-Pos

ART-NK-Nom.Pl.Fem NN-NK-Nom.Pl.Fem
Die

$.

Bänke

zu

(a) Original MG Parse

.

klein

TOP
S
NP
ART

$_
VAFIN

NN

ART.Nom.Pl.Fem NN.Nom.Pl.Fem

VAFIN.3.Pl.Pres.Ind

AP
PTKA

$_

ADJD

PTKA ADJD.Pos(b) Delexicalized MG Parse

Figure 2: An example illustrating the delexicalization
process of a MG tree.

From the span representations, the parser computes
label scores, which are subsequently used to incrementally construct a tree using a chart parsing
algorithm (Sakai, 1961).
According to Kaing et al. (2021), Benepar exhibits two key features which are advantageous forcross-lingual transfer. Firstly, it employs a selfattentive encoder that effectively captures global
context information and exhibits less sensitivity toword order. Secondly, the parser independently
scores each span without considering the label decisions of its children or parent. This means that a
failure in label prediction for a certain span does
not strongly impact the label prediction for otherspans (Gaddy et al., 2018). Consequently, the prediction errors resulting from local syntax variations
between two languages have a limited effect on theoverall prediction.
While our delexicalized parser adopts the samearchitecture to Benepar, there exist distinctions in
the inputs of the two. Specifically, Benepar istrained on parse trees with words, whereas ourdelexicalized parser operates on POS sequences
as inputs, i.e. tree strings devoid of words. Therefore, the delexicalized version of the MG treebank
is required to train the delexicalized parser. For the
MHG parsing in the inference, we feed the delexicalized model with the POS sequences of MHG
sentences.
4.2

MHG POS Tagger For the delexicalization of
MHG sentences, we need a POS tagger for MHG.
We use the RNNTagger of Schmid (2019) for thispurpose, which annotates MHG sentences with
POS tags as well as morphological features andhas been trained on the ReM corpus. RNNTagger uses deep bidirectional LSTMs with characterbased word representations.
4.3

Tag Set Mapping

The Tiger Treebank uses the STTS tag set, whereas
the MHG version of the RNNTagger and the ReM
corpus on which it was trained employ the HiTS
tag set. Due to this discrepancy, we cannot directly
use the POS labels from RNNTagger as input tothe delexicalized parser. HiTS, for example, hasseparate tags for definite (DDART) and indefinite
articles (DIART), whereas STTS uses the tag “ART”
for both of them. Since the delexicalization methoddemands that the source and target languages sharethe same tag set, we have to map the MHG tags
to the MG . The small MHG treebank that we usefor evaluation purposes uses STTS and requires nomapping.
The mapping process involves two dimensions.
Firstly, we map the morphological features of
MHG to those of MG. Secondly, we map the POS
tags of MHG to those of MG primarily based on a
mapping dictionary. Table 2 shows a selected part

Delexicalization for MG and MHG

Delexicalization for MG We use the Tiger Treebank to train the delexicalized parsing model on
MG parse trees. The parse trees in the Tiger Tree72

Tiger
DDB
ReM

Type
Treebank
Treebank
POS-tagged corpus

Language
MG
MHG
MHG

Size
50,474 trees
96 trees
2,269,738 tokens

Usage
Parser training
Parser evaluation
POS tagger training

Table 1: Overview of the datasets.
MHG Tag
CARDD
DDA
DDART
DIA
DIART
DID
NA
VAPS

DDB The German Diachronic Treebank(DDB) (Hirschmann and Linde, 2023) consistsof a limited number of 100 parse trees for MHG.
Due to the small data size, we utilize the DDB
treebank solely for the cross-lingual evaluationof the delexicalized parser. To prepare the DDB
treebank for evaluation, we perform preprocessing
steps, including converting it to the format of thePenn Treebank and removing incomplete parsetrees and parse trees with mostly Latin words. We
also removed numbers and periods which formed
the first token of a parse tree and corrected a few
more minor problems. At the end, we had 96
sentences for evaluation purposes.

MG Tag
CARD
PDAT
ART
PIAT
ART
PDAT
NN
ADJD.Pos

Table 2: Representative mapping pairs in the mapping
dictionary.

of the POS tag mapping dictionary. It should be
noted that our mapping is not flawless due to certain challenges. For instance, the composite word
in MHG “enerde (on earth)” is separated into “auf ”
and “Erde” in MG and are tagged as “APPR|NA”.
In the DDB treebank, such composite words areannotated with two separate tags combined with “|”in the DDB treebank. However, for simplification
purposes, our mapping only retains the first part of
the tag, leading to a loss of information.

5

ReM The Reference Corpus for Middle HighGerman (ReM) (Klein et al., 2016) is an extensive
collection of texts written in MHG. This corpus
encompasses approximately 2.3 million tokens andprovides comprehensive linguistic annotations, including POS tags, morphological analysis, lemma
features, and more. The ReM corpus has been usedby Schmid (2019) to train the MHG version of his
RNNTagger which annotates MHG texts with POS
tags and morphological features.

Experiments

We begin by training Benepar on the delexicalized
Tiger treebank for MG. Then we annotate the sentences of the small DDB treebank for MHG with
RNNTagger and map the HiTS tags that it returns
to STTS tags. Finally, we parse the POS tag sequences with the trained parser.
5.1

5.2

Baselines

We evaluate the performance of our proposed delexicalized MHG parser which is based on the Benepar
parser (Kitaev and Klein, 2018), and compare itwith the cross-lingual transfer performance of theoriginal Benepar without using the delexicalization
method and other parsing approaches that incorporate pretrained language models, which have shownpromising results in various NLP tasks.

Datasets

In our experiments, we utilize the following three
corpora (see also Table 1).
Tiger Treebank The delexicalized parser istrained on the Tiger Treebank (Smith, 2003), which
comprises a total number of 50,474 parse trees
for MG. We use a version of the Tiger Treebank
which has been converted to the Penn Treebank
format (Marcus et al., 1993). We delexicalize theTiger corpus and divide it into a training set and adevelopment set. The first 47,474 parse trees in the
Tiger corpus comprise the training set and the last
3,000 parse trees comprise the development set.

Vanilla Benepar The vanilla Benepar model istrained directly on the original training set of the
Tiger Treebank for MG without delexicalization.
After training, the parser is directly used to parse
the MHG sentences as token sequences. This allows us to compare the performance of the delexicalized MHG parser with the vanilla Benepar
model, highlighting the impact of delexicalization
on cross-lingual parsing performance.
73Baselines
Vanilla Benepar
Tetra-gBERT
Tetra-mBERT
Our proposed method
Dexparser

RecallMG
MHG

Precision
MG
MHG

FScore
MG
MHG

MG

CM
MHG

84.18
86.31
60.68

34.41
23.20
19.69

87.57
88.19
65.61

44.40
29.53
23.25

85.84
87.24
63.15

38.77
25.98
21.32

45.80
51.70
21.35

0.00
3.12
0.00

81.39

64.72

84.89

70.19

83.10

67.34

39.03

12.50

Table 3: Main results of the cross-lingual parsing transfer performance of different parsers. CM refers to “complete
match”. gBERT refers to the pretrained German BERT and mBERT refers to the multilingual version BERT. The
best value of each column is indicated in bold.

Tetra-Tagging with PLMs Tetra-tagging (Kitaev and Klein, 2020) is a technique for reducing
constituency parsing to sequence labeling. In thisapproach, special parsing tags are predicted in parallel using a PLM, and then merged into a parse tree.
In our experiment, we use the pretrained German
BERT model (Chan et al., 2020) and the multilingual BERT model (Devlin et al., 2019) available
on the HuggingFace website (Wolf et al., 2020).
We start by finetuning these models on the Tiger
Treebank using the Tetra-tagging technique. Subsequently, we evaluate their performance on the
MHG parse test set.
5.3

predicted parse trees which were exactly identical to the respective reference parse trees.
As is the standard practice, the evaluation disregards POS labels and punctuation.
5.4

Training Setup

For training the delexicalized parser, we adopt thesame hyperparameter settings as described in (Kitaev and Klein, 2018). The encoder architecture
consists of a character-level bidirectional LSTM
neural network. We configure the encoder with adimension of 1024, utilizing 8 layers, 8 attention
heads, and a dimension of 64 for the key, query,and value. The size of the feedforward layer is set
to 2048, and the character embedding dimension
is 64. The batch size is set to 32, the learning rate
is 5e-5, and the maximum sequence length of theencoder is 512. We use the random seed 10 for
training. We conduct all our experiments using aserver with 8 GPUs with 11GB RAM (NVIDIA
GeForce GTX 1080 Ti).

Evaluation

Following Kitaev and Klein (2018), we use thethe standard evalb measures (Sekine and Collins,
1997; Collins, 1997) for the parser quality evaluation. evalb is a software tool that provides
metrics to assess the accuracy and similarity ofparsed sentences against reference or gold standard
parse trees, including precision, recall, F1 score,and complete match.
• Precision measures the proportion of predicted constituents in the generated parse tree
which are also contained in the reference parsetree. It quantifies the accuracy of the parser incorrectly identifying constituents.

6

Results and Analysis

6.1

Main Results

Table 3 shows the parsing performance of different
cross-lingual parsers. Notably, our proposed parser
attains the highest scores across all metrics for
MHG, demonstrating that the delexicalized parser
possesses superior cross-lingual parsing performance on MHG. Our delexicalized parser demonstrates substantial advantages in parsing MHG,
achieving an impressive increase of almost 30%
points in F1 score. Besides, it achieves comparable results on MG. In terms of the baselines, theVanilla Benepar and the Tetra-gBERT parser bothachieve relatively high recall and precision for MG
but have noticeably lower values for MHG. The
Tetra-mBERT parser exhibits lower values for both
recall and precision for both MG and MHG. It is• Recall measures the proportion of constituents in the reference parse tree whichwere predicted by the parser in the generated
parse tree. It quantifies the parser’s ability
to generate all the constituents present in thereference parse tree.
• F1 Score is the harmonic mean of precisionand recall.
• Complete Match measures the proportion of74

Recall
66.18
64.72
59.16
48.66

Delexicalized parser using gold tags
- using predicted tags
- without mapping- without morphological information

Precision
71.17
70.19
68.82
65.38

FScore
68.59
67.34
63.63
55.8

CM
14.58
12.50
7.29
9.28

Table 4: The MHG parsing results with delexicalized parser in the ablation study.

worth noting that the parsing performance of thedelexicalized model on the source language MG
is surpassed by the two strong baselines, Vanilla
Benepar and Tetra-gBERT. This outcome is expected as the delexicalization process diminishes
the semantic information present in the input sequences. However, the trade-off of the performanceloss in MG leads to a big leap in the cross-lingualparsing performance for MHG.
Our delexicalized constituency parser exhibits
outstanding performance on the MHG test set, attaining an impressive F1-score of 67.3%. This
substantial improvement outperforms the best zeroshot cross-lingual baseline by a considerable margin of 28.6%. Although there is a slight decline
in the parsing performance for MG, the tradeoff proves worthwhile considering the substantial
gains achieved in parsing MHG. This emphasizes
the effectiveness of the delexicalized approach infacilitating cross-lingual transfer and highlights
its potential for parsing ancient and historical languages like MHG.
6.2

by the cross-lingual parser without mapping themfrom HiTS to STTS. This finding highlights theindispensability of mapping from MHG to MG for
maintaining satisfactory parsing performance. The
results underscore the significance of aligning the
tag sets between MHG and MG to ensure effective
cross-lingual parsing and emphasize the necessityof this mapping process in our approach.
Morphological Information The inclusion ofmorphological markers provides the neural modelwith valuable additional information for parsing
MHG sentences. In our experiments, we augment
the delexicalized MHG sequences with morphological information, such as case, gender, number,and more. The outcomes of the ablation study
clearly indicate that removing this morphologicalinformation from the delexicalized input sequences
obviously impairs parsing performance. Specifically, this exclusion leads to a noticeable decline in
the F1 score, amounting to a reduction of 11.5%.
6.3

Case Study

Figure 3 shows two MHG trees generated by ourdelexicalized parser and the corresponding gold
standard trees for comparison. This case study
reveals that the delexicalized parser demonstrates
relatively accurate predictions of constituents whencompared to the reference trees, especially for short
MHG sentences. Some prediction errors in constituents stem from the intricacy and the ambiguityof the MHG grammar, as exmplified by the case of“her” in Example 2. From a linguistic perspective,
determining whether “her” functions as an adverb(ADV) or a separated verb prefix (PTKVZ) poseschallenges. However, in longer and more complex sentences, e.g., the sentence in Example 1, theparser typically maintains a high level of accuracy
locally while occasionally struggling to accurately
determine the overall structure of the entire sentence. Besides, the presence of noise in the ancient
texts is another factor that can impact the effectiveness of the cross-lingual parsing for MHG. Overall,the qualitative analysis provides further evidenceof the effectiveness of the delexicalized parser for

Ablation Study

We now examine how the parsing performancechanges (i) as we replace predicted POS tags withgoldstandard POS tags, (ii) as we use the original
HiTS tags instead of mapping them to STTS tags,and (iii) as we remove the morphological featuresfrom the parser input. Table 4 presents the resultsof our ablation study.
Goldstandard POS Tags We observe that the fscore of the delexicalized parser increases by 1.3%
points when it processes gold standard POS tag
sequences instead of POS tag sequences predicted
by RNNTagger. This finding underscores the quality of the POS tags predicted by RNNTagger. We
loose very little performance due to POS tagging
errors.
Tag Set Mapping Table 4 demonstrates a noticeable decline in parsing performance from 67.34%
to 43.43% in terms of F1 score when the delexicalized MHG sequences are directly processed
75

Example 2

Example 1Goldstandard

Prediction

Figure 3: Two examples of the trees generated by our delexicalized parser compared to the reference parses.

MHG, emphasizing its ability to accurately predict constituents, especially in shorter sentences.
While challenges may arise in handling longer andmore complex sentences, the delexicalized parser
showcases promising results, contributing to theadvancement of MHG parsing.

7

where a POS tagger for the target language and a
related language with a treebank exist.

Acknowledgements
We extend our sincere gratitude to the anonymous
reviewers for their invaluable contributions and constructive feedback that have greatly enriched thequality and scope of this paper. This work was
supported by China Scholarship Council (CSC).

Conclusion

In conclusion, our study presents an effective crosslingual constituency parsing approach for ancientlanguages, specifically focusing on the parsing of
Middle High German (MHG) sentences. Throughthe utilization of delexicalization and and the similarities between MHG and Modern German (MG),
we have developed a delexicalized parser based on
the rich treebank resources of MG, which demonstrates remarkable performance in parsing MHG
sentences. Our experimental results showcase theefficacy of the delexicalized approach, outperforming existing baselines and achieving substantial
improvements in parsing accuracy. These findings
highlight the practicality and promise of our approach for parsing historical and ancient languages,
addressing the challenges posed by limited annotated data and linguistic variations.

References
Waleed Ammar, George Mulcaire, Miguel Ballesteros,
Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association
for Computational Linguistics, 4:431–444.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zeroshot cross-lingual transfer and beyond. Transactionsof the Association for Computational Linguistics,
7:597–610.
Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita
Chatzipanagiotou, Ion Androutsopoulos, Jonathan
Prag, and Nando de Freitas. 2022. Restoring andattributing ancient texts using deep neural networks.
Nature, 603(7900):280–283.

Limitations

Hannah Booth, Anne Breitbarth, Aaron Ecay, andMelissa Farasyn. 2020. A Penn-style treebank ofMiddle Low German. In Proceedings of the 12th Language Resources and Evaluation Conference, pages
766–775, Marseille, France. European Language Resources Association.

One limitation of our study is the need for further
improvement in the robustness of the delexicalized
parsing method, particularly when applied to ancient texts. By addressing this limitation, we canfurther enhance the applicability of our approachto a wider range of ancient languages and ensure
more reliable parsing results. Besides, our proposed method is only applicable to the scenario

Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The tiger treebank. In Proceedings of the workshop on treebanks
and linguistic theories, volume 168, pages 24–41.

76

Branden Chan, Stefan Schweter, and Timo Möller. 2020.
German’s next language model. In Proceedings ofthe 28th International Conference on Computational
Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 999–1010,
New Orleans, Louisiana. Association for Computational Linguistics.
Ulf Hermjakob. 2001. Parsing and question classification for question answering. In Proceedings of
the ACL 2001 Workshop on Open-Domain Question
Answering.

Christian Chiarcos, Benjamin Kosmehl, Christian Fäth,and Maria Sukhareva. 2018. Analyzing Middle HighGerman syntax with RDF and SPARQL. In Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018),
Miyazaki, Japan. European Language Resources Association (ELRA).

Hagen Hirschmann and Sonja Linde. 2023. Deutsche
diachrone baumbank (version 1.0). HumboldtUniversität zu Berlin.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long
short-term memory. Neural computation, 9(8):1735–
1780.

Michael Collins. 1997. Three generative, lexicalisedmodels for statistical parsing. In 35th Annual Meeting of the Association for Computational Linguistics
and 8th Conference of the European Chapter of the
Association for Computational Linguistics, pages 16–
23, Madrid, Spain. Association for Computational
Linguistics.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International
Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages
4411–4421. PMLR.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–
8451, Online. Association for Computational Linguistics.

Jing Jiang. 2012. Information extraction from text. Mining text data, pages 11–41.
Ming Jiang and Jana Diesner. 2019. A constituency
parsing tree based method for relation extraction fromabstracts of scholarly publications. In Proceedings of
the Thirteenth Workshop on Graph-Based Methods
for Natural Language Processing (TextGraphs-13),
pages 186–191, Hong Kong. Association for Computational Linguistics.

James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system andprovably optimal dynamic oracles. In Proceedings ofthe 2016 Conference on Empirical Methods in Natural Language Processing, pages 1–11, Austin, Texas.
Association for Computational Linguistics.

Howard Jones and Martin H Jones. 2019. The Oxford
Guide to Middle High German. Oxford University
Press.

Ulrike Demske. 2019.
Referenzkorpus frühneuhochdeutsch: Baumbank. UP. Universität Potsdam: Institut für Germanistik (https://hdl. handle.
net/11022/0000-0007-EAF7-B).

Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan
Roth. 2019. Cross-lingual ability of multilingual bert:
An empirical study. ArXiv, abs/1912.07840.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171–4186, Minneapolis, Minnesota. Association for
Computational Linguistics.

Hour Kaing, Chenchen Ding, Masao Utiyama, Eiichiro
Sumita, Katsuhito Sudoh, and Satoshi Nakamura.
2021. Constituency parsing by cross-lingual delexicalization. IEEE Access, 9:141571–141578.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention andpre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
pages 3499–3505, Florence, Italy. Association for
Computational Linguistics.

Stefanie Dipper, Karin Donhauser, Thomas Klein, Sonja
Linde, Stefan Müller, and Klaus-Peter Wegera. 2013.
Hits: ein tagset für historische sprachstufen desdeutschen. Journal for Language Technology andComputational Linguistics, 28(1):85–137.

Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedingsof the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 2676–2686, Melbourne, Australia. Association
for Computational Linguistics.

David Gaddy, Mitchell Stern, and Dan Klein. 2018.
What’s going on in neural constituency parsers? ananalysis. In Proceedings of the 2018 Conference of
the North American Chapter of the Association for77Nikita Kitaev and Dan Klein. 2020. Tetra-tagging:Word-synchronous parsing with linear-time inference.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6255–
6261, Online. Association for Computational Linguistics.

putting an old-fashioned debate in its historical perspective. Topoi, 37(2):279–287.
Sebastian Ruder. 2019. Neural Transfer Learning for
Natural Language Processing. Ph.D. thesis, National
University of Ireland, Galway.
Itiroo Sakai. 1961. Syntax in universal translation. In
Proceedings of the International Conference on Machine Translation and Applied Language Analysis,
National Physical Laboratory, Teddington, UK.

Thomas Klein, Klaus-Peter Wegera, Stefanie Dipper,
and Claudia Wich-Reif. 2016. Reference Corpusof Middle High German (1050–1350) (Version 1.0).
Rheinische Friedrich-Wilhelms-Universität Bonn,
Ruhr-Universität Bochum.

Joseph Salmons. 2018. A history of German: What
the past reveals about today’s language. Oxford
University Press.

Anne Lauscher, Vinit Ravishankar, Ivan Vulić, andGoran Glavaš. 2020. From zero to hero: On thelimitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Association for Computational Linguistics.

Christopher Sapp, Daniel Dakota, and Elliott Evans.
2023. Parsing early New High German: Benefits andlimitations of cross-dialectal training. In Proceedingsof the 21st International Workshop on Treebanks and
Linguistic Theories (TLT, GURT/SyntaxFest 2023),
pages 54–66, Washington, D.C. Association for Computational Linguistics.

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch,
Dimitris Kontokostas, Pablo N Mendes, SebastianHellmann, Mohamed Morsey, Patrick Van Kleef,
Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia.
Semantic web, 6(2):167–195.

Anne Schiller, Simone Teufel, and Christine Thielen.
1995. Guidelines f ur das tagging deutscher textcorpora mit stts. Universität Stuttgart, Universität
Tübingen, Germany.

Yuncong Li, Cunxiang Yin, and Sheng-hua Zhong.
2020. Sentence constituent-aware aspect-category
sentiment analysis with graph attention networks.
In Natural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC
2020, Zhengzhou, China, October 14–18, 2020, Proceedings, Part I 9, pages 815–827. Springer.

Helmut Schmid. 2019. Deep learning-based morphological taggers and lemmatizers for annotating historicaltexts. In Proceedings of the 3rd international conference on digital access to textual cultural heritage,
pages 133–137.
Satoshi Sekine and Michael Collins. 1997. Evalb
bracket scoring program. URL: http://www. cs. nyu.
edu/cs/projects/proteus/evalb.

Yongkang Liu, Shi Feng, Daling Wang, and Yifei Zhang.
2022. MulZDG: Multilingual code-switching framework for zero-shot dialogue generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 648–659, Gyeongju,
Republic of Korea. International Committee on Computational Linguistics.

Sunayana Sitaram, Monojit Choudhury, Barun Patra,
Vishrav Chaudhary, Kabir Ahuja, and Kalika Bali.
2023. Everything you need to know about multilingual LLMs: Towards fair, performant and reliable
models for languages of the world. In Proceedingsof the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 6: Tutorial Abstracts), pages 21–26, Toronto, Canada. Association
for Computational Linguistics.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann
Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational
Linguistics, 19(2):313–330.

George Smith. 2003. A brief introduction to the tiger
treebank, version 1. Technical report, Universität
Potsdam.

Ryan McDonald, Slav Petrov, and Keith Hall. 2011.
Multi-source transfer of delexicalized dependency
parsers. In Proceedings of the 2011 Conference on
Empirical Methods in Natural Language Processing,pages 62–72, Edinburgh, Scotland, UK. Association
for Computational Linguistics.

Mitchell Stern, Jacob Andreas, and Dan Klein. 2017.
A minimal span-based neural constituency parser.
In Proceedings of the 55th Annual Meeting of the
Association for Computational Linguistics (Volume
1: Long Papers), pages 818–827, Vancouver, Canada.
Association for Computational Linguistics.

Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich
Schütze. 2023. Cross-lingual retrieval augmented
prompt for low-resource languages. In Findings of
the Association for Computational Linguistics: ACL
2022, Toronto, Canada. Association for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,

Andrea Parravicini and Telmo Pievani. 2018. Continuity and discontinuity in human language evolution:

78

Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations, pages 38–45, Online. Association
for Computational Linguistics.
Daniel Zeman and Philip Resnik. 2008. Cross-language
parser adaptation between related languages. In Proceedings of the IJCNLP-08 Workshop on NLP for
Less Privileged Languages.

79Can Large Language Model Comprehend Ancient Chinese?
A Preliminary Test on ACLUE
Yixuan Zhang Haonan Li
Mohamed bin Zayed University of Artificial Intelligence, UAE
haonan.li@mbzuai.ac.ae

Abstract

2020; Scao et al., 2022; Touvron et al., 2023; Muennighoff et al., 2022; Zeng et al., 2023). These models encapsulate extensive knowledge and sophisticated reasoning capabilities. Notably, the emergence of ChatGPT (OpenAI, 2023) and Chineseoriented LLMs such as ChatGLM (Zeng et al.,2023), has accentuated their remarkable ability incomprehending and generating modern language.
However, due to the lack of ancient language benchmarks, the abilities of LLMs in handling ancientlanguage remains largely unexplored.

Large language models (LLMs) have showcased remarkable capabilities in understanding
and generating language. However, their ability
in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE,
an evaluation benchmark designed to assess
the capability of language models in comprehending ancient Chinese. ACLUE consists of
15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference
and knowledge. Through the evaluation ofeight state-of-the-art LLMs, we observed a noticeable disparity in their performance between
modern Chinese and ancient Chinese. Amongthe assessed models, ChatGLM2 demonstrates
the most remarkable performance, achieving
an average score of 37.4%. We have made ourcode and data public available.1

1

We present the Ancient Chinese Language Understanding Evaluation (ACLUE), an evaluation
benchmark consisting of 15 tasks. These tasks arederived from a combination of manually curated
questions from publicly available resources, andautomatically generated questions from classicalChinese language corpora. The range of questions
span from the Xia dynasty (2070 BCE) to the Ming
dynasty (1368 CE), covering a broad temporal
range. Similar to the well-established LLM benchmarks such as ARC (Clark et al., 2018) and MMLU
(Hendrycks et al., 2021), ACLUE adopts multiplechoice question format for all tasks. This ensures
simplicity and uniformity in evaluating models, accommodating variations in different training or finetuning procedures and prompting methodologies.

Introduction

The study of ancient languages provides valuable
insights into the past civilizations’ thoughts, languages, societies, and histories (Zhiming, 1990;
Woodard, 2008; Bouchard-Côté et al., 2013). Ancient China, as one of the oldest civilizations, hasleft a significant impact on contemporary societies
including Japan, Korea, and Vietnam. However,existing research in ancient Chinese language processing have primarily focused on specific time
periods or genres (Yan et al., 2016; Xie et al., 2019;
Liu et al., 2020; Hu et al., 2021; Tian et al., 2021).
Typically, the previously proposed models require
customized fine-tuning for particular tasks.
Recently, the significant advancements made inlarge language models (LLMs) underscore theirremarkable proficiency across a range of tasks,
showcasing their potential in performing various
tasks without the need for fine-tuning (Brown et al.,1

In our preliminary experiments, we assessed theperformance of 8 advanced LLMs, where the Chinese LLM ChatGLM2 demonstrates the best performance with an average accuracy of 37.4%, slightly
surpassing ChatGPT. However, considering thebaseline accuracy of 25% from random guessing
and the average accuracy of around 50% achievedby the same models on contemporary modern Chinese benchmarks such as AGIEval (Zhong et al.,2023) and CMMLU (Li et al., 2023), we believe
there is still ample room for improvement in the
proficiency of existing LLMs in understanding ancient Chinese.

https://github.com/isen-zhang/ACLUE

80
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 80–87,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_009

2

ACLUE Benchmark

2.2

Sentence segmentation is a task that involves
choosing the correct segmentation of a given sentence. Since ancient Chinese lacks punctuation
marks, accurate sentence segmentation becomes
crucial for analyzing syntax and semantics of asentence. We create the task by sampling sentences from the Classical-Modern Chinese Corpus,2 which provides labeled sentence segmentation. To create false options, we manipulate theoriginal punctuation marks by moving, adding, ordeleting them.
Couplet prediction involves predicting the most
likely second line of a Chinese couplet based ona given first line. Chinese couplet, also known
as “对联” (duì lián), is a traditional form of poetic
expression consisting of two lines of verse. The twolines are expected to match in terms of meaning,
rhyme, and other poetic elements. We constructthis task using a couplet dataset.3
Poetry context prediction is a task constructedusing the Chinese-poetry corpus.4 The objective of
this task is to select the most likely next or previous
sentence given a specific sentence from a poem.

ACLUE consists of 15 tasks that encompassing
lexical, syntactic, semantic, inference, and generalknowledge of ancient Chinese. The details of thetasks are provided in Appendix A, where basic
statistics can be found in Table 2, and examples ofeach task are listed in Table 3. The questions covera wide range of genres, including poetry, prose,classical novels, couplets, historical records, andbiographies, spanning the period from 2070 BCE to1368 CE. Among the 15 tasks, 8 were automaticallygenerated using existing corpora or datasets, 5 werecollected from freely available standard tests, and2 were directly sourced from other work. Eachtask includes 100 to 500 questions, exceeding thenumber required for testing a human participant.
ACLUE serves as an evaluation suite for LLMs
ability in understanding ancient Chinese withouttask-specific fine-tuning. To ensure fair comparison among different models trained with varying
approaches, all tasks are formatted into multiplechoice questions with four choices, of which only
one is correct. The task details and dataset construction process are elaborated in this section.
2.1

Syntactic and Semantic Tasks

2.3

Inference

Poem quality estimation task is constructed basedon dataset proposed by Yi et al. (2018), which consists of 173 Chinese quatrains, with each one being
rated for fluency, coherence, and meaningfulness
on a scale of 0 to 5 by human expert. We randomly
select four poems and create questions asking models to identify the best or worst poem based on a
specific criterion. To ensure clear distinctions, we
maintain a minimum score differences of 2 between
the correct option and the other options. The task
aims to evaluate the ability of models to compare
the quality of Chinese quatrains.
Reading comprehension is based on the AGIEval
dataset (Zhong et al., 2023). It contains a subset
of Chinese Gaokao questions. We select questions
that contains ancient Chinese text from this subset.
Poetry sentiment analysis involves predicting thesentiment of an entire poem or parts of a poem,
determining whether it is positive, neutral, or negative. We utilize a dataset proposed by Shao et al.
(2021), which contains 5,000 poems. Each poem

Lexical Tasks

We create three lexical tasks using the ancient Chinese corpus, which includes over 50,000 word
sense annotations and 3,000 named entity annotations (Shu et al., 2021).
Polysemy resolution aims to understand the different senses or meanings of words. Two types ofquestions are created: one asks which characterin a given sentence carries a particular meaning,
while the other requires identifying the meaning ofa character within the sentence.
Homographic character resolution focuses onrecognizing homographic characters in ancient Chinese texts. Homographic characters, also known as“通假字” (tōng jiǎ zì) in Chinese, are substitutionsof characters in ancient Chinese texts with others
that have similar pronunciation or appearance.
Named entity recognition focuses on identifying
named entities (e.g., names of people, places, dynasties, etc.) in ancient Chinese texts. Two types ofquestions are created: one type asks for the specificentity type of a given entity within a contextual sentence, while the other type asks in which context a
Chinese word represents an entity.

2
https://github.com/NiuTrans/
Classical-Modern
3
https://github.com/wb14123/
couplet-dataset
4
https://github.com/chinese-poetry81

MOSS (OpenLMLab, 2023). Details about thesemodels are introduced in Appendix C.
For models optimized to function as chatbots,
such as ChatGPT and ChatGLM, we generate output and use regular expressions to extract the answer key. For other models, we directly obtain
the probability of the next tokens after the promptand selected the one with the highest probability
among the answer keys (i.e., ‘A’,‘B’,‘C’,‘D’). We
employ both zero-shot (do not provide examples)and in-context five-shot (provide few examples)
evaluation. An example of evaluation instance isshown in Figure 1.

以下是关于 古代文学知识 的单项选择题，请直接给出正确答
案的选项。
Here are some multiple-choice questions about Ancient Chinese
literature , please provide the correct answer choice directly.
题目：下列诗句中，属于杜牧咏史诗的是：
Question: Among the following lines of poetry, the one that belongs
to Du Mu’s historical poem is:
A. 旧时王谢堂前燕，飞入寻常百姓家In former times, the swallows in front of the halls of Wang and Xie
flew into the homes of ordinary people
B. 长空澹澹孤岛没，万古销沉向此中
The vast sky engulfed the desolate island, and for eternity it sank into
this place.
C. 千寻铁锁沉江底，一片降幡出石头
Thousands of chains sank to the bottom of the river, and a stone
emerged with a descending flag
D. 三百年间同晓梦，钟山何处有龙盘For three hundred years, the same dream awakened at dawn, where on
Zhongshan Mountain can a dragon coil
答案是： (Answer:)

3.1
Figure 1: An examples from ACLUE. English translations are provided for better readability.

Table 1 shows the zero-shot performance of allmodels. The five-shot results are similar to thezero-shot results, suggesting that models can comprehend the task without additional demonstrations.
Overall, the Chinese model ChatGLM2 demonstrates the best performance, with an average accuracy of 37.4%. Moreover, its performance on
almost all tasks is above the random guessing(25%). The multilingual model ChatGPT achievesa slightly lower accuracy of 36.9%, compared to
ChatGLM2, yet it maintains relatively consistent
performance in terms of standard deviation.
Regarding specific tasks, we have several findings: (1) BLOOMZ exhibits exceptional performance in couplet prediction (T5), achieving an
accuracy of 60.2%. This accuracy is nearly double that of most other models, possibly due to
BLOOMZ’s training set, xP3, having overlaps with
our data source. Similar, ChatGLM2 may havebeen exposed to the original texts used for sentence segmentation (T4) and poetry appreciation(T9), which explains its proficient performance inthese tasks. (2) All models face challenges in thehomographic character resolution (T2), with performance close to random guessing. This issue
likely arises because the auto-regressive training
objective does not emphasize understanding of homographic concepts. (3) Reading comprehension(T8) poses a considerable challenge for all models due to the extreme long length of the question (nearly 1,000 tokens on average). Specifically,
BLOOMZ, LLaMA, and Baichuan are significantly
affected, exhibiting lower performance on this task
compared to their average across other tasks. Thisobservation suggests that these models may lackadequate support for processing very long input.

and its individual sentences are labeled with finegrained sentiment categories, including negative,
implicit negative, neutral, implicit positive, andpositive sentiments. We merge implicit negativeand implicit positive labels with their respective
categories to address ambiguity.
Poetry appreciation is manually curated from
openly accessible online resources.
2.4

Knowledge-intensive Tasks

Ancient Chinese knowledge tasks cover various
subjects, including ancient Chinese medical, ancient Chinese literature, traditional Chinese culture, and ancient Chinese phonetics. To create
these tasks, we collected relevant questions fromvarious online open resources. Additionally, we
extracted a subset of questions from the CMMLU
dataset (Li et al., 2023), which consist of questions
at the high-school level in current Chinese education. This selection allows us to form the tasks ofbasic ancient Chinese.

3

Experiment

To provide an overview of the language abilityof existing open-sourced LLMs on ancient Chinese, we assess 8 models including 4 multilingual
models: ChatGPT (OpenAI, 2023), LLaMA (Touvron et al., 2023), Falcon (Almazrouei et al.,2023), BLOOMZ (Muennighoff et al., 2022), and4 Chinese models: ChatGLM (Du et al., 2022),
Baichuan,5 ChatGLM2 (Zeng et al., 2023), and5

Results

https://github.com/baichuan-inc/baichuan-7B

82

LexicalModel
ChatGLM2
ChatGPTBLOOMZ
ChatGLM
FalconBaichuan
LLaMA
MOSS

Syntactic

Semantic

Inference

Knowledge

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

T11

T12

T13

T14

T15

45.4
41.8
45.2
39.6
40.4
31.6
36.4
30.6

24.4
20.6
22.4
19.4
28.8
26.4
22.2
27.6

34.8
41.2
35.6
39.4
21.2
22.0
26.4
25.8

46.4
43.0
32.2
36.6
32.6
33.0
33.0
24.0

39.8
45.4
60.2
37.2
37.2
37.2
29.6
30.0

24.6
27.4
27.2
23.4
31.4
27.8
29.6
25.0

28.3
39.7
31.5
30.8
36.9
30.3
31.5
29.8

29.7
39.6
17.8
32.7
22.8
16.8
18.8
27.7

42.7
38.8
26.2
30.1
31.1
25.2
24.3
21.4

52.6
47.8
45.2
43.8
43.8
38.2
41.8
30.8

28.9
29.3
29.7
29.3
30.5
27.3
24.5
26.5

50.7
43.4
44.1
36.8
30.1
36.0
23.5
22.1

34.6
34.6
39.3
30.8
30.3
37.0
29.4
24.6

43.8
33.8
44.4
40.6
36.9
41.9
29.4
22.5

35.0
27.0
29.0
27.0
26.0
31.0
31.0
26.0

Overall37.4±8.936.9±7.635.3±10.733.2±6.6
32.0±6.0
30.8±6.5
28.8±5.626.3±3.0

Table 1: Zero-shot average accuracy of all models. The overall results are averaged (with standard deviation) over
all tasks. T1: Polysemy resolution, T2: Homographic character resolution, T3: Named entity recognition, T4:
Sentence segmentation, T5: Couplet prediction, T6: Poetry context prediction, T7: Poetry quality estimation, T8:
Reading comprehension, T9: Poetry appreciation, T10: Poetry sentiment analysis, T11: Basic ancient Chinese, T12:
Traditional Chinese culture, T13: Ancient Chinese medical, T14: Ancient Chinese literature, T15: Ancient Chinesephonetics.

generated50

et al., 2019; Qu et al., 2022), Classic Chinese poem
generation (Yi et al., 2017; Yang et al., 2018; Guoet al., 2019; Xie et al., 2019; Zhao et al., 2022; Maet al., 2023), and ancient Chinese sentence segmentation (Han et al., 2018; Hu et al., 2021), as wellas general language model pre-training (Tian et al.,2021). However, many of these studies focus on
specific types or literary formats that were popularduring specific time periods.
Recently, large language models have demonstrated remarkable language understanding andgeneration capabilities (Brown et al., 2020; Scaoet al., 2022; Almazrouei et al., 2023). Researchers
have began to evaluate these LLMs based on theirperformance across a wide range of tasks (Touvron
et al., 2023; Muennighoff et al., 2022; OpenAI,2023). However, the absence of a comprehensive
evaluation benchmark poses a challenge in assessing the performance of LLMs in ancient language
understanding. Existing ancient Chinese evaluation datasets either have a narrow focus on specifictasks, limiting the scope of evaluation, or require
model fine-tuning prior to evaluation. In contrast,
ACLUE provides a natural support for evaluation
under zero-shot and in-context learning settings,
making it more compatible with LLMs.

collected

250lexicalsyntatic semantic inference inference knowledge
ChatGLM2-6B

ChatGPT

Figure 2: The performance of ChatGPT and ChatGLM2
on ACLUE of different categories.

Based on data origin, we divide the tasks into
two categories: auto-generated and manually collected. In Figure 2, we compare the performanceof ChatGPT and ChatGLM2, the best multilingual
and Chinese models, respectively. We find thatwhile ChatGLM2 exhibits superior overall performance on ACLUE, its dominance only observedin the auto-generated syntactic tasks and collected
knowledge categories. More comparison results
are provided in Appendix B.
In terms of data quality and reliability, autogenerated questions within ACLUE were slightly
less intricate than collected questions, but the difference was not significant. This suggests that theauto-generated questions hold reasonable potential
for effectively evaluating models’ grasp of ancient
Chinese language.

4

5

Conclusion

We propose ACLUE, the first evaluation benchmark for ancient Chinese language understanding. Our preliminary evaluation of 8 large language models reveals that, despite their exceptional
performance in modern language understanding,they struggle with even basic tasks in ancient Chinese. Through analysis, we illustrate that the autogenerated questions possess similar difficulty levels
to those found in actual school tests.

Related WorkA lot of research has been conducted on various
aspects of ancient Chinese language processing,encompassing topics such as ancient Chinese tomodern Chinese translation (Liu et al., 2020), Chinese couplets generation (Yan et al., 2016; Yuan
83

References

Dayiheng Liu, Kexin Yang, Qian Qu, and Jiancheng
Lv. 2020. Ancient-modern chinese translation with a
new large training dataset. ACM Trans. Asian Low
Resour. Lang. Inf. Process., 19(1):6:1–6:13.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru,
Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine
Noune, Baptiste Pannier, and Guilherme Penedo.
2023. Falcon-40B: an open large language modelwith state-of-the-art performance.

Jingkun Ma, Runzhe Zhan, and Derek F. Wong. 2023.
Yu sheng: Human-in-loop classical chinese poetry
generation system. In Proceedings of the 17th Conference of the European Chapter of the Association
for Computational Linguistics. EACL 2023 - System
Demonstrations, Dubrovnik, Croatia, May 2-4, 2023,
pages 57–66. Association for Computational Linguistics.

Alexandre Bouchard-Côté, David Hall, Thomas L. Griffiths, and Dan Klein. 2013. Automated reconstruction of ancient languages using probabilistic models of sound change. Proc. Natl. Acad. Sci. USA,110(11):4224–4229.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika,
Adam Roberts, Stella Biderman, Teven Le Scao,
M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev,
Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward
Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning. CoRR,
abs/2211.01786.

Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020.
Language models are few-shot learners. In Advancesin Neural Information Processing Systems 33: Annual Conference on Neural Information Processing
Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,
Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR,
abs/1803.05457.

OpenAI. 2023. Gpt-4 technical report.
OpenLMLab. 2023. Moss.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding,
Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm:General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 320–335.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow,
Ruxandra Cojocaru, Alessandro Cappelli, Hamza
Alobeidli, Baptiste Pannier, Ebtesam Almazrouei,
and Julien Launay. 2023. The RefinedWeb dataset
for Falcon LLM: outperforming curated corpora
with web data, and web data only. arXiv preprint
arXiv:2306.01116.

Zhipeng Guo, Xiaoyuan Yi, Maosong Sun, Wenhao Li,
Cheng Yang, Jiannan Liang, Huimin Chen, Yuhui
Zhang, and Ruoyu Li. 2019. Jiuge: A humanmachine collaborative chinese classical poetry generation system. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August
2, 2019, Volume 3: System Demonstrations, pages
25–30. Association for Computational Linguistics.

Qian Qu, Jiancheng Lv, Dayiheng Liu, and Kexin Yang.
2022. Coupgan: Chinese couplet generation via
encoder-decoder model and adversarial training under global control. Soft Comput., 26(15):7423–7433.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie
Pavlick, Suzana Ilic, et al. 2022. BLOOM: A 176bparameter open-access multilingual language model.
CoRR, abs/2211.05100.

Xu Han, Hongsu Wang, Sanqian Zhang, Qunchao Fu,
and Jun S. Liu. 2018. Sentence segmentation for classical chinese based on LSTM with radical embedding.
CoRR, abs/1810.03479.

Yizhan Shao, Tong Shao, Minghao Wang, Peng Wang,and Jie Gao. 2021. A sentiment and style controllable approach for chinese poetry generation. In
CIKM ’21: The 30th ACM International Conferenceon Information and Knowledge Management, Virtual
Event, Queensland, Australia, November 1 - 5, 2021,
pages 4784–4788. ACM.

Dan Hendrycks, Collin Burns, Steven Basart, Andy
Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language
understanding. In 9th International Conference on
Learning Representations, ICLR 2021, Virtual Event,Austria, May 3-7, 2021. OpenReview.net.

Lei Shu, Yiluan Guo, Huiping Wang, Xuetao Zhang,and Renfen Hu. 2021. 古汉语词义标注语料库的构建及应用研究(the construction and applicationof Ancient Chinese corpus with word sense annotation). In Proceedings of the 20th Chinese National
Conference on Computational Linguistics, pages 549–
563, Huhhot, China. Chinese Information Processing
Society of China.

Renfen Hu, Shen Li, and Yuchen Zhu. 2021. Knowledge representation and sentence segmentation of ancient chinese based on deep language models. Journal of Chinese Information Processing, 35(4):8–15.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai
Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask
language understanding in chinese.

Huishuang Tian, Kexin Yang, Dayiheng Liu, andJiancheng Lv. 2021. Anchibert: A pre-trained model

84

for ancient chinese language understanding and generation. In International Joint Conference on Neural
Networks, IJCNN 2021, Shenzhen, China, July 18-22,
2021, pages 1–8. IEEE.

Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma,
Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan
Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.
GLM-130b: An open bilingual pre-trained model. In
The Eleventh International Conference on Learning
Representations.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, Aurelien Rodriguez, Armand Joulin, Edouard
Grave, and Guillaume Lample. 2023. Llama: Openand efficient foundation language models. arXiv
preprint arXiv:2302.13971.

Jiaqi Zhao, Ting Bai, Yuting Wei, and Bin Wu. 2022.
Poetrybert: Pre-training with sememe knowledge forclassical chinese poetry. In Data Mining and Big
Data - 7th International Conference, DMBD 2022,
Beijing, China, November 21-24, 2022, Proceedings, Part II, volume 1745 of Communications in
Computer and Information Science, pages 369–384.
Springer.

Roger D Woodard. 2008. The ancient languages ofEurope. Cambridge University Press.
Zhuohan Xie, Jey Han Lau, and Trevor Cohn. 2019.
From shakespeare to li-bai: Adapting a sonnet model
to chinese poetry. In Proceedings of the The 17th
Annual Workshop of the Australasian Language Technology Association, ALTA 2019, Sydney, Australia,
December 4-6, 2019, pages 10–18. Australasian Language Technology Association.

Bao Zhiming. 1990. Language and world view in ancient china. Philosophy East and West, 40(2):195–
219.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang,
Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen,
and Nan Duan. 2023. Agieval: A human-centricbenchmark for evaluating foundation models. CoRR,
abs/2304.06364.

Rui Yan, Cheng-Te Li, Xiaohua Hu, and Ming Zhang.
2016. Chinese couplet generation with neural network structures. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for
Computer Linguistics.

A

Data details

The Table 2 listed the Chinese, category, and originof the tasks in ACLUE, and the Table 3 provides
examples for each task.

Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic chinese poetry generation
via unsupervised style disentanglement. In Proceedings of the 2018 Conference on Empirical Methods
in Natural Language Processing, Brussels, Belgium,
October 31 - November 4, 2018, pages 3960–3969.
Association for Computational Linguistics.

B

Further analysis

The performance comparison of all LLMs on different data origins is illustrated in Figure 3. Evaluating the LLMs’ performance on auto-generated
questions versus manually collected questions in
ACLUE, we found that while the generated questions were less intricate than the collected ones, thedifference was not significant. This indicates a comparable level of difficulty between the two types ofquestions. Among all the models, only ChatGLM2
demonstrated better performance on collected questions compared to auto-generated questions, which
may indicate exposure to the original question texts
used in ACLUE.

Xiaoyuan Yi, Ruoyu Li, and Maosong Sun. 2017. Generating chinese classical poems with RNN encoderdecoder. In Chinese Computational Linguistics andNatural Language Processing Based on Naturally
Annotated Big Data - 16th China National Conference, CCL 2017, - and - 5th International Symposium,
NLP-NABD 2017, Nanjing, China, October 13-15,
2017, Proceedings, volume 10565 of Lecture Notes
in Computer Science, pages 211–223. Springer.
Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Wenhao Li.
2018. Automatic poetry generation with mutual reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing, pages 3143–3153, Brussels, Belgium.

C

Shengqiong Yuan, Luo Zhong, Lin Li, and Rui Zhang.
2019. Automatic generation of chinese coupletswith attention based encoder-decoder model. In 2ndIEEE Conference on Multimedia Information Processing and Retrieval, MIPR 2019, San Jose, CA,
USA, March 28-30, 2019, pages 65–70. IEEE.

Models being Evaluated

BLOOMZ is derived from BLOOM throughfine-tuning on a crosslingual task mixture(xP3), which is an instruction-following dataset.
BLOOMZ exhibits competitive performance withmodels that have a larger number of parametersacross various non-generation tasks.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,
Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,

85

Task
Named entity recognition
Polysemy resolution
Homographic character resolution
Sentence segmentation
Couplet prediction
Poetry context prediction
Poetry sentiment analysis
Poem quality estimation
Ancient Chinese medical
Ancient Chinese literature
Traditional Chinese culture
Poetry appreciation
Basic ancient Chinese
Reading comprehension
Ancient Chinese phonetics

Total Q.

Avg. len

Task (zh)Category

Origin

500
500
500
500
500
500
500
406
211
160
136
103
249
101
101

138
116
137
210
62
77
60
118
38
44
59
258
52
982
50

古汉语命名体识别
古文单字多义
通假字
古文断句
对联预测
古诗词上下句预测诗词情感分类
古诗词质量评估
医古文
古代文学知识
国学常识古诗词曲鉴赏
基础古汉语知识
古文阅读理解古音学

lexicallexicallexicalsyntatic
semantic
semantic
inference
inference
knowledgeknowledgeknowledge
inference
knowledge
inference
knowledge

generated
generated
generated
generated
generated
generated
generated
generatedcollected
collected
collected
collected
collected
collected
collected

Table 2: ACLUE task overview. We list the total number of questions (Total Q.), average question length counted in
Chinese characters (Avg. len), task names in Chinese, task type, and data origin type.

Baichuan-7b is an open-source large-scale pretrained model developed by Baichuan Intelligence.
Built on the Transformer architecture, it adopts
the same model design as LLaMA. This 7-billionparameter model was trained on approximately 1.2
trillion tokens using proprietary Chinese-English
bilingual corpora, with optimization focused on
Chinese.

generated

MOSS-SFT-16B
LLaMA-65B
Baichuan-7B
Falcon-40B
ChatGLM-6B
BLOOMZ-7B
ChatGPT
ChatGLM2-6B
0collected

25

ChatGLM-6B is bidirectional dense model pretrained using the General Language Model (GLM)
algorithm developed by Tsinghua University. It
supports bilingual (Chinese and English) language
processing. ChatGLM is a version of GLM thathas been supplemented with supervised fine-tuning,
feedback bootstrap, and reinforcement learningwith human feedback, specifically optimized for
Chinese question answering (QA) and dialogue
tasks.

Figure 3: The performance comparison of LLMs on
ACLUE across different data origins.

ChatGLM2-6B is the second generation of ChatGLM. It uses the hybrid objective function of GLM,
and has undergone pre-training with 1.4T bilingual
tokens and human preference alignment training. It
offers enhanced performance and an expanded context length of 32K. With efficient inference using
Multi-Query Attention technology, it achieves efficient inference with higher speed and lower memory usage.

Falcon-40B is a decoder-only model created by
TII and trained on 1,000B tokens of RefinedWeb(Penedo et al., 2023) data. Due to the high qualityof its training data, Falcon-40B performs competitively with LLaMA-65B on various benchmarks.

LLaMA-65B is an auto-regressive language
model proposed by Meta. It incorporates several structural improvements over the vanilla transformer and is trained on a mixture of publicly available data sources. LLaMA has demonstrated comparable or even superior performance to models
that are ten times its size.

MOSS is an open-source Chinese language
model proposed by Fudan University. It matches
ChatGPT in terms of training scale and alignment
techniques. MOSS-SFT is initialized with CodeGen and further pre-trained on 100B Chinese tokens and 20B English tokens. The SFT (supervised fine-tuned) version of MOSS-SFT enables
the model to follow instructions in multi-turn dialogues.

ChatGPT is a GPT model developed by OpenAIand fine-tuned using reinforcement learning fromhuman feedback (RLHF). As a commercial product, specific details about its model size, training
data, and training process are not disclosed.
86

ID

Task

ExampleT1单字多义Polysemy resolution

下列选项中对“此神农之所以[长]，而尧舜之所以章也。”这句话中的“长”字理解正确的是( )A. 首领
B.排行第一, 长子
C.长处, 专长D. 长大，成年T2

通假字
Homographic character resolution

列选项中[]内的“红”字是通假字的是( )A. 吾已食禄，又夺园夫女[红]利。
B. 晓看[红]湿处，花重锦官城
C. [红] 芳满院参差折，绿醑盈杯次第衔。
D. 竹缘浦以被绿，石照涧而映[红] 。

T3

命名体识别
Named entity recognition

下列选项中[]内的“阳”字代表了地名的是( )A. 夫人授兆丹书真文、月中玉。
B. 令飞升上造洞[阳]之宫。
C. 今朝日[阳]里，梳落数茎丝。
D. 晓发碧水[阳]，暝宿金山寺。

T4

古文断句
Sentence segmentation

以下选项断句正确的是( )A. 史记/曰/秦使武安君白起攻赵/赵发兵拒秦/秦大破赵於长平/
B. 史记/曰秦/使武安君白起攻赵/赵发兵拒秦/秦大破赵於长平C. 史记/曰/秦使武安君白起攻赵/赵发兵拒秦秦大/破赵於长/平D. 史/记曰秦使武安君白起攻赵/赵发兵拒秦秦大/破赵於长平

T5对联Couplet prediction“兔去龙来，交替人间春好景”的下联最可能是( )A. 莺歌燕舞，和谐社会岁祥光。
B. 香遗书案，传家苦读育春风。
C. 赛龙夺锦，万人江岸闹端阳。
D. 情牵天下，凭谁设榻效陈蕃。

T6

古诗词上下句预测
Poetry context prediction“何秣候明，便可一横江。”的上一句是( )A. 江藉草作寒食，雨梨花思故。
B. 孰知文有忌，情至自生哀。
C. 樽前唱醉翁曲，歌花舞催。
D. 千村落呼客，山南北花吹香。

T7

古诗词质量评估
Poem quality estimation

下列古诗词前后文连贯性最差的是( )A. 阴雨难侵牖|春虫足哺儿|年年秋报喜|牛女有佳期
B.富贵良非愿|林泉毕此生|酒因随量饮|诗或偶然成
C.久不闻山歌|南风五月多|牧童呼伴侣|吹笛下西坡
D.今日骐阁|当年鹦鹉洲|寄书愁不达|书达得无愁T8

古文阅读理解Reading comprehension

下列对原文有关内容的理解和分析，表述不正确的一项是( )谢贞，字元正，陈郡阳夏人，晋太傅安九世孙也。父蔺，正员外郎，... ... 察因启曰：“贞有一子年六岁。”即有敕长给衣粮。（节选自《陈书·列传第二十六》，有删改）。【注】惠连：谢惠连，南朝宋文学家。
A. 谢贞天性聪慧，小时候读过不少典籍，有的读过就能背诵，有的粗通大意；他八岁时写的诗就深得长辈称赞。
B. 谢贞受府长史周确委托，为他撰写辞让都官尚书的表文。陈后主读过之后，怀疑该表文不是周确亲笔所作。
C. 谢贞非常孝顺，小时候祖母因病难以进食，他便也不进食；父亲去世他悲痛欲绝，之后，
奉养母亲未曾间断。
D. 母亲去世后，谢贞一心守丧，极度悲痛，骨瘦如柴，令人叹息。他忧病而死后，后主下令长期供他儿子吃穿。

T9古诗词曲鉴赏
Poetry appreciation

下列对这首诗的赏析，不正确的一项是( )《幽居初夏》陆游。湖山胜处放翁家，槐柳阴中野径斜。水满有时观下鹭，草深无处不鸣蛙。箨龙已过头番笋，木笔犹开第一花。叹息老来交旧尽，睡来谁共午瓯茶。
A. 首句“湖山”二字总冒全篇，勾勒环境，笔力开张，巧妙地从山光水色中引出“幽居”。
B. 首句概言“湖山胜处”，颔联写湖，是近处宽处静景；颈联写庭院周围，是远处细处动态。
C. 诗中写放翁心中郁结与柳宗元《小石潭记》中写“以其境过清”时的心境相似。
D. 本诗前三联写景，尾联结情，景情相衬，描写与抒情紧密关联，脉络清晰。

T10

诗词情感分类
Poetry sentiment analysis

古诗词“庭前芍药妖无格|池上芙蕖净少情|唯有牡丹真国色|花开时节动京城“的整体情感是( )A. 积极的B. 消极的
C. 中性的D. 无法判断T11

国学常识
Basic ancient Chinese"近朱者赤，近墨者黑"所蕴含的道理和下列哪句话最相似？( )A. 青出于蓝，而胜于蓝。
B. 蓬生麻中，不扶而直。
C. 公生明，偏生暗。
D. 三天打鱼两天晒网

T12

古汉语知识
Traditional Chinese culture

下列句中，含有双宾语的一句是( )A. 夫何之有?
B. 重之而之。

T13

医古文
Ancient Chinese medical

以下除（）之外，都有病愈之义。
A. 已
B. 起
C. 性
D. 差

T14

古代文学知识
Ancient Chinese literature

杜甫《春望》中的“感时花溅泪，恨别鸟惊心”所反映的是( )A. 早年的读书和漫游生活。
B. 困居长安十年时的感受。
C. “安史之乱”时的国恨家愁。
D. 晚年漂泊西南的客旅生活。

T15

古音学
Ancient Chinese phonetics

下列字在古代的声母、调类、等和开合口标注错误的是( )A. 温（影母平声二等开）
B. 权（群母平声三等合）
C. 空（溪母平声一等合）D. 狂（群母平声三等合）C. 兔不可得，而身宋笑。

Table 3: ACLUE tasks examples.

87

D. 甚矣，汝之不惠!

Unveiling Emotional Landscapes in Plautus and Terentius Comedies: A
Computational Approach for Qualitative Analysis
Davide Picca
University of Lausanne / Switzerland
davide.picca@unil.ch
Caroline Richard
EDITTA, Sorbonne Université / France
caroline.richard@sorbonne-universite.fr
Abstract

2019a], Lemlat enhancements [Passarotti, 2019b],
and Index Thomisticus Treebank adaptation [Passarotti, 2019c]. However, emotion recognition inclassical languages remains relatively untouched.
Essential contributions include Sprugnoli et al.
[2020a]’s work on Latin sentiment lexicons andsentiment analysis in Latin poetry [Sprugnoli et al.,2020b]. Studies on other classical languages, like
Greek, also offer valuable insights [Yeruva et al.,2020, Pavlopoulos et al., 2022].
Even with these developments, the complexity of
Latin’s grammar and emotion portrayal makes thisa challenging, yet fertile field. A blend of advanced
Natural Language Processing (NLP) techniques
and a robust understanding of the language’s heritage are key to unlocking this potential, promisingmore profound insights into emotion recognitionin classical languages.

This ongoing study explores emotion recognition in Latin texts, specifically focusing on
Latin comedies. Leveraging Natural Language
Processing and classical philology insights, theproject navigates the challenges of Latin’s intricate grammar and nuanced emotional expression. Despite initial challenges with lexicontranslation and emotional alignment, the work
provides a foundation for a more comprehensive analysis of emotions in Latin literature.

1

Introduction

Emotion recognition in text, extensively applied tomodern languages, has scarcely targeted classicallanguages like Latin, despite its rich historical andcultural data [Alswaidan and Menai, 2020, Gatley, 2023, Korolova et al., 2019]. Recognizing
emotions in Latin texts could illuminate classicalliterature, historical documents, and the evolutionof emotional expression.
Latin’s intricate grammar, extensive vocabulary,
and ancient emotional nuances present unique challenges, requiring sophisticated NLP techniques andcultural understanding [Buzassyova, 2016, GruberMiller and Mulligan, 2022]. The limited availability of large, annotated Latin corpora further complicates traditional machine learning applications[Strapparava and Mihalcea, 2008].
This paper addresses these hurdles and explores
emotion recognition in Latin texts. We propose
a novel method combining NLP techniques andclassical philology, extending emotion recognition
techniques to Latin language analysis [Pang andLee, 2008].

2

3

Methodology

3.1

Research Design

The research design for this study commences witha quantitative phase, employing NLP techniquessuch as tokenization and lemmatization, in conjunction with a lexicon-based approach for emotion recognition. It is worth noting that our discourse analysis deviates from conventional norms
by adopting a character-based perspective, facilitating the exploration of play dynamics throughemotional trajectories Vandersmissen [2019]. Inalignment with this perspective, we segment andindex the texts according to speaker metadata, thus
facilitating an individualized character analysis.
After the quantitative phase, we integrate a qualitative analysis, studying selected Latin texts tounderstand language and emotion, and devising
an emotion coding scheme based on study principles. We then merge quantitative and qualitative
results, comparing computational and manual analyses. These findings address the research questions,

State-of-the-Art

Remarkable strides have been made in computational linguistics and Latin language analysis, including lexicon development [Passarotti, 2016],
Medieval Latin Charters annotation [Passarotti,
88

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 88–95,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_010

deepening our understanding of the issue. This approach sets the foundation for our larger project:
developing an emotion lexicon for Latin studies,
reducing modern language bias to ensure authentic
emotional data extraction.
3.2

meticulously dissecting and interpreting a Latin
corpus, which has been subdivided based on speakers’ speeches. We implement lemmatization to integrate and enrich the Latin lexicon, reducing word
variations to their fundamental form or lemma. It
is noteworthy that the lemma detection rate in bothstages is approximately 15%, a figure that doubles
when excluding lexicon with no emotional connotations.
After the lemmatization stage, the Latin corpus,
comprising XML files each representing a Latin literary work, is extracted and parsed. Post-extraction,
the speeches are evaluated for emotional content
using the prepared NRCLex instance3 .
Contrary to the assumption that comedic works
primarily harbor positive emotions, our preliminary
findings reveal a varied emotional terrain, highlighting the complexity within comedic plays. Correlations are observed between specific emotional
expressions and character archetypes. However,inherent limitations in the lexicon applied, suchas automatic English to Latin translation, alignment of emotions with lemmas based on contemporary English perspectives, and authenticity of
employed lemmas, warrant caution. These aspects
may engender potential misinterpretations of ancient emotions [Rosenwein, 2010, Konstan, 2016],
along with lexical discrepancies, hindering word
recognition in the lemmatized corpus, as exemplified by the non-recognition of the verb metuo (tofear).
Despite these limitations, the project provides aroadmap for future exploration in this growing field.
Efforts are directed towards refining the lexicon
and methodology to enhance the assessment of
the emotional spectrum within Latin comedy 4,
and to advance the broader objective of creatinga Latin-specific emotion lexicon, thus enhancing
data authenticity and minimizing modern language
biases.

Data Collection

The data for this study was collected from the
Perseus Digital Library1 . Our attention centerson investigating the genre of Latin comedy, with a
specific emphasis on Plautus and Terentius’s works.
These plays were designated for observation owing to their applicability and donations to the Latin
comedy genre and to their relative completeness(indeed most comic plays have been lost or are
known by fragments).
The digitization of these texts was already completed by the Perseus Digital Library [Smith et al.,2000], which has undertaken extensive efforts todigitize and preserve classical texts. The data collection process involved downloading the relevant
files from the GitHub repository and processing
them using a Python script. This script extracts the
text content from the XML files, along with the associated metadata. The extracted data is then saved
in a structured format (CSV) for further analysis.
3.3

Emotion detection on Latin comedies

This ongoing, exploratory project is centered on theinvestigation of Latin comedy through the application of emotion recognition theory and technology,
a realm that promises significant insight into narrative structures and character developments within
the genre. Comedy offers a structured medium
to analyze the emotional nuances in speeches andcompare them with the genre’s inherent traits andexpectations. Our analysis is directed towards a
selected corpus of 26 extant works by the influential Roman playwrights Plautus and Terentius,utilizing advanced computational tools to discerna spectrum of emotions—specifically anger, anticipation, disgust, fear, joy, sadness, surprise, andtrust—embedded in these texts.
The research methodology unfolds in threestages: (1) the integration and expansion of lexical databases, (2) the construction of a lemmatized
lexicon utilizing resources from the National Research Council Canada (NRC) for Latin2 , and (3)3.4

Qualitative evaluation of emotion
recognition

The computational analysis has yielded substantial insights regarding the emotional strategies employed by Plautus and Terentius to captivate theiraudiences. It validates the pronounced prevalenceof joy, thereby affirming the comedic essence ofthe genre. Furthermore, a notable correlation exists

1
The
datasetisfreelyavailableathttps://github.com/PerseusDL
2https://saifmohammad.com/WebPages/NRC-EmotionLexicon.htm

3
GitHubrepository
available
https://github.com/CarolineRichard/ENCODEM.git

89

here:

between the emotion displayed and the character
archetypes, suggesting that Plautus meticulously
crafted character personalities and roles to evoke
specific emotional responses from his audience.
This endeavor has thus substantially broadened ourcomprehension of the intricate interplay between
emotion and language within Roman New Comedy.
Initial examinations reveal that comedies adhere tocertain core emotional motifs, with surprise emerging as the most predominantly depicted and universally shared sentiment. This finding aligns with theinherent narrative logic of comedies, wherein theplot revolves around unforeseen twists, deceptive
maneuvers, and mistaken identities. Emotions suchas anger and fear also figure prominently in the
narrative landscape. Certain characters appear to
be consistently characterized by these dual emotions, such that one is seldom portrayed without
the other. This pattern is discernible in characterslike Simon from Andria or Antiphon from Stichus.
The characters embodying these paired emotions
often assume pivotal roles in the narrative, such as
the adulescens (young man) or the senex (old man)as shown in Figure 4 in the Appendix.
The intricate interplay between the dual roles inthe drama manifests itself through the core dynamic
tension between fear and anger.
From an emotional perspective, characters canbe dichotomized into two groups:

and anger, whereas others predominantly display
surprise. This pattern is accentuated by the character discrepancies between the pater lenis (gentle father) and the pater durus (harsh father), as
observed in the Heautontimoroumenos (refer to
Lhostis [2019] and Figure 2 in the Appendix forfurther details).
Across various plays, there is a discernible consistency in the characterization: characters predominantly characterized by anger and fear; thoselargely exhibiting surprise, and others manifesting
a diverse emotional range. Within this last group,
joy tends to be the most prevalent emotion. For
example, the narrative of the Mostellaria revolves
around two young men primarily associated withanger and fear, while the characters of Father Teuropides and Philematia the freed courtesan is dominated by surprise (See Figure 1 in the Appendix).
Other characters display a blend of emotions.
The emotional distribution among charactersdoes not necessarily correspond to their degree of
involvement in the narrative arc. For example, in
Mostellaria, despite the central role of the character Trianon, his emotional display is subdued andnot polarized. However, this appears more alignedwith a distribution based on the characters’ roles
within the dramatic schema: characters in conflicttend to display polarised emotions, whereas supporting or ancillary characters exhibit a more varied, non-polarised emotional range. This pattern isdiscernible in plays such as Stichus, Poenulus andMostellaria, among others.

• those who experience a broad emotional spectrum.
• others who are defined by single or dual predominating emotions.

Quantitative research underscores recurring patterns in the dramatic construction of comedy andstock characters, specifically in the works of Plautus and Terentius, which deftly employ complex
emotion networks to enhance the dynamicity of
their plays. Unexpectedly, each play exhibits aunique global emotion network thereby suggesting that each play dynamic is distinct, regardlessof their stereotypical characters and plots. Thisemotion-centric interaction is integral in shaping
audience reception. The balance struck between
standard emotional archetypes, such as the pater
durus’ anger, and an innovative emotional dynamic
indicates the nuanced comical effects.

The bifurcation of emotional responses can be
attributed to the alignment of specific characterswith particular emotional types. For instance, as
shown in Figure 3 in the Appendix, a majorityof the slaves typically display a limited array of
emotions, commonly fear, anger, or joy.
In a similar vein, the parasitus character is primarily associated with emotions of anger or fear,with a scant expression of other emotions. Stereotypical emotions in comedy may hint at social representation trends. A character’s social status might
correlate with the type and complexity of emotions
they express. However, even within this framework, emotional responses exhibit significant variations within the same character archetype. For
instance, within the demographic of elderly men,certain characters are solely associated with fear

Traditional analysis, which emphasizes plot progression and dramatic dynamics, may overlook
these emotional nuances. Therefore, this study
advocates an alternate perspective that emphasizes
the emotional interaction between characters.
90

4

Future WorksJohn Gruber-Miller and Bret Mulligan. Latin vocabulary knowledge and the readability of latintexts: A preliminary study. New England Classical Journal, 2022. doi: 10.52284/necj.49.1.
article.gruber-millerandmulligan. URL https://dx.doi.org/10.52284/necj.49.1.
article.gruber-millerandmulligan.

This proposal seeks to build upon our initial analysis of character dialogues, with the objective of developing an enhanced lexicon, rooted in the foundational NRC-Emolex model. This process includes
meticulous data sanitization and augmentation ofemotional markers. After refining the model, thenext step is to study two plays, comparing manual and automatic emotional annotations, to further
improve emotion recognition.
The indispensable preliminary discourse analysis provides a foundational understanding, vital to
the formulation of a specialized emotion lexicon
for Latin textual studies. By doing so, the proposal
aims to reinforce the efficiency of emotion detection and bolster the reliability and authenticity ofthe extracted emotional data.
The complexity of emotional semiotization necessitates that we do not solely depend on specificemotion-related lemmas, given their inherent instability and context-dependence Micheli [2014].
Utilizing emotional markers derived from a Latin
corpus, along with phraseological characteristics,
will enrich our lexicon via a thematic, rather thanstrictly lexical approach. This will enable a more
precise comprehension of emotions, allowing foran accurate assessment of the emotion network anda nuanced understanding of emotional representation.
The proposed project, thus, marks a significant
step towards an exhaustive tool for deep investigation of emotions within Latin literature. This innovative endeavour is set to amplify our understanding of the emotional dimensions present withinthese foundational texts.

Konstan.
Their emotions and ours: A single
history?
L’Atelier du Centre de recherches
historiques, 16, 2016.
doi: https://doi.org/
10.4000/acrh.6756. URL http://journals.
openedition.org/acrh/6756. Online; accessed 27-June-2023.
Nataliia Korolova, Oksana Koshchii, and Valentyna Myronova. The latin language as a universal culturalcode. Journal of History Culture and Art Research,
8:278–290, 2019.
Nathalie Lhostis. Le langage de la sagesse dansl’heautontimoroumenos de térence. Vita Latina, 199(1):171–195, 2019. doi: 10.3406/vita.2019.1909.
URL https://www.persee.fr/doc/vita_
0042-7306_2019_num_199_1_1909.
Raphaël Micheli. Les émotions dans les discours: modèle d’analyse, perspectives empiriques. Champs
linguistiques. de Boeck Duculot, Louvain-la-Neuve,2014. ISBN 978-2-8011-1738-5.
B. Pang and L. Lee. Opinion mining and sentiment
analysis. Foundations and Trends in Information
Retrieval, 2(1-2), 2008.
Marco Passarotti. Building a word formation lexicon
for latin. In LREC, 2016. URL https://www.
aclweb.org/anthology/L16-1681.pdf.
Marco Passarotti. Annotating medieval latin charters.
In Proceedings of the NoDaLiDa 2019
Workshop on Processing Historical Language,
2019a.
URL https://www.aclweb.org/
anthology/W19-4718.pdf.

References

Marco Passarotti. Enhancing the latin morphologicalanalyser lemlat with an onomasticon. In Proceedings of the NoDaLiDa 2019 Workshop on Processing
Historical Language, 2019b. URL https://www.
aclweb.org/anthology/W19-4719.pdf.

N Alswaidan and MEB Menai. A survey of state-ofthe-art approaches for emotion recognition in text.
Knowledge and Information Systems, 2020. doi: 10.
1007/s10115-020-01449-0. Query date: 2023-06-21
21:34:37.

Marco Passarotti. Converting the index thomisticus
treebank into universal dependencies. In Proceedings of the NoDaLiDa 2019 Workshop on Processing
Historical Language, 2019c. URL https://www.
aclweb.org/anthology/W19-4720.pdf.
J Pavlopoulos, A Xenos, and D Picca. Sentiment analysis of homeric text: The 1st book of iliad. Proceedings of the Thirteenth Language Resources and
Evaluation Conference, 2022. URL https://aclanthology.org/2022.lrec-1.765/.

Ludmila Buzassyova. The ’phonetic complex’ in renaissance latin grammar petrus ramus’s dichotomies
and their reflections in two vernacular grammaticaltexts. Graeco-Latina Brunensia, 21(2):81–98, January 2016. doi: 10.5817/GLB2016-2-8.

Barbara Rosenwein. Problems and Methods in the History of Emotions. Passions in Context I. International Journal for the History and Theory of Emotions, 1, January 2010.

Jane Gatley. Cultural capital, curriculum policy andteaching latin. British Educational Research Journal,
49:174–185, 2023.

91David A Smith, Jeffrey A Rydberg-Cox, and Gregory R
Crane. The perseus project: A digital library for thehumanities. Literary and Linguistic Computing, 15(1):15–25, 2000.
Rachele Sprugnoli, Marco Passarotti, Daniela Corbetta,
and Andrea Peverelli. Creating, evaluating and extending sentiment lexicons for latin. In Proceedingsof The 12th Language Resources and Evaluation Conference, pages 3078–3086, 2020a.
Rachele Sprugnoli, Marco Passarotti, Daniela Corbetta,
and Andrea Peverelli. Odi et amo: Creating, evaluating and extending sentiment lexicons for latin. In
Proceedings of the 12th Conference on Language
Resources and Evaluation (LREC 2020), pages 3078–
3086. European Language Resources Association(ELRA), 2020b.
C. Strapparava and R. Mihalcea. Learning to identify
emotions in text. In Proceedings of the 2008 ACM
symposium on Applied computing, 2008.
Marc Vandersmissen. Discours des personnages feminins chez Seneque. Collection Latomus, 2019. URL
https://www.academia.edu/38169453/
Discours_des_personnages_f%C3%
A9minins_chez_S%C3%A9n%C3%A8que_Approches_logom%C3%A9triques_et_
contrastives_dun_corpus_th%C3%A9%
C3%A2tral. ISBN: 9789042937970.
Vijaya Kumari Yeruva, Mayanka Chandrashekar, Yugyung Lee, Jeff Rydberg-Cox, Virginia Blanton, andNathan A Oyler. Interpretation of sentiment analysis in aeschylus’s greek tragedy. In Proceedings of
LaTeCH-CLfL 2020, pages 138–146, 2020.

92

A

Appendix: Figures

93Figure 1: Emotional Distribution in Mostellaria

Figure 2: Emotional Landscape in the Heautontimoroumenos (Terentius)

94

Figure 3: Representation of slaves’ Emotional Spectrum

Figure 4: Emotions of seneces in Plautus and Terentius’ drama95

Morphological and Semantic Evaluation of Ancient Chinese
Machine Translation
Kai Jin1, Dan Zhao1, Wuying Liu2,3✉
1. School of Foreign Languages, Qilu University of Technology, 250353, Jinan, Shandong, China
2. Shandong Key Laboratory of Language Resources Development and Application, Ludong
University, 264025, Yantai, Shandong, China
3. Center for Linguistics and Applied Linguistics, Guangdong University of Foreign Studies,
510420, Guangzhou, Guangdong, China{jk@qlu.edu.cn, dzhao639@gmail.com, wyliu@ldu.edu.cn}

mode for translation. At the same time, the qualityof MT has been a subject of great interest andconcern for both the MT and translation fields.
Researchers, institutions, and conferences arecontinuously conducting studies in this area, andvarious evaluation metrics for MT have been
proposed.
There have also been studies on MT of ancient
texts. Some researchers have made algorithmic
improvements specifically tailored for translating
ancient texts (Gutherz et al. 2023; Park et al. 2020;
Zhang et al. 2019; Zhou & Liu 2022). Researchers
have also conducted evaluations of the quality of
MT for ancient texts (Yao et al. 2013; Yang et al.
2021; Yousef et al. 2022). However, research on
MT for ancient texts, including ancient Chinese
texts, remains relatively scarce.
This paper primarily focuses on MT quality ofancient Chinese texts, and the subsequentdiscussions will concentrate on this specificdomain. Compared to modern Chinese, ancient
Chinese has its own unique characteristics. Firstly,
ancient Chinese employs complex and distinctive
grammatical structures, including syntax, word
order, and rhetoric, among other aspects. These
structures differ significantly from modern Chinese.
MT struggles to accurately capture and parse theintricate grammatical relationships embedded inancient Chinese texts. Secondly, ancient Chinese
texts often employ rhetorical devices such asallusions, symbolism, and metaphors, which
involve rich cultural connotations and backgrounds.
These allusions and cultural nuances are often
challenging for non-Chinese MT systems tocomprehend, leading to translation errors or theloss of the original essence and aesthetic appeal.
Thirdly, ancient Chinese texts often exhibit
polysemy and ambiguity, where a single word orphrase may have multiple interpretations andAbstract
Machine translation (MT) of ancient
Chinese texts presents unique challenges
due to the complex grammatical structures,
cultural nuances, and polysemy of thelanguage. This paper focuses on evaluating
the translation quality of different platforms
for ancient Chinese texts using The
Analects as a case study. The evaluation isconducted using the BLEU, LMS, and ESS
metrics, and the platforms compared
include three machine translation platforms(BaiduTranslate,
Bing
Microsoft
Translator, and DeepL), and one language
generation model ChatGPT that can engage
in translation endeavors. Results show that
Baidu performs the best, surpassing theother platforms in all three metrics, while
ChatGPT ranks second and demonstrates
unique advantages. The translations
generated by ChatGPT are deemed highly
valuable as references. The study
contributes to understanding the challenges
of MT for ancient Chinese texts andprovides insights for users and researchers
in this field. It also highlights theimportance of considering specific domain
requirements when evaluating MT systems.

1

Introduction

Machine translation (MT) has been a prominentarea of research and development in artificial
intelligence since the 1950s. Over the years, it has
undergone significant advancements, evolvingfrom rule-based methods, statistical methods, andmore recently, neural network-based learning
methods. As the quality of MT continues toimprove and the demand for translation work
steadily increases, more and more translators are
adopting the “machine translation + post-editing”
96

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 96–102,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_011

meanings. MT systems find it challenging toaccurately select and judge among these complex
semantic relationships, often leading to
mistranslationsorinaccuracies.
The
aforementioned characteristics pose significant
challenges for MT of ancient Chinese texts.
This study aims to evaluate the translation
quality of different platforms for ancient Chinese
texts. Through this evaluation, we can gain insights
and understanding in dealing with the complexities
of ancient language and culture, contribute to theadvancement in the field of natural language
processing, and provide a supplement to MT
quality assessment applications. Furthermore,these evaluation results will help users gain
insights into the performance of different platforms,

allowing them to identify potential issues andlimitations.

2

Experiment design

This study takes the Chinese classic The Analects1
as the research text and compares three classic
human-translated versions and four versions
generated by four platforms. Three MT quality
evaluation metrics are used as evaluation criteria toassess the translation quality of the four platforms.
For each human-translated text and each machinetranslated text, quality scores are calculated
individually. Then, the mean scores are calculated
for each platform. The scheme is illustrated in
Figure 1.

Figure 1: Research scheme.
1

2.1

The Analects is one of the most influential texts
in Chinese ancient philosophy and culture,
regarded as a masterpiece in Chinese literature. Its
impact extends not only within China but also
across the globe, and it has been translated into
multiple languages, generating significant
influence worldwide (Li & Li 2013). As a

Texts

In this study, we select The Analects as the sample
ancient Chinese texts and its three translations as
the reference human translations to compare withMT.

The original Chinese title is “论语” (lunyu), and it has several different English versions. In this paper, apartfrom referring to specific translators, we use “The Analects” to refer to the book.
1

97

foundational work of Confucian thought, The
Analects has garnered the largest number ofEnglish translations among classical Chinese texts.
We have compiled the original text of The
Analects into a corpus, consisting of a total of 1,153
sentences.
We have also selected three highly influential
versions of The Analects for our study, translated
respectively by Tomson (辜鸿铭) (Tomson 2011),
James Legge (Legge 2016) and Ezra Pound (Pound
1933). In 1861, James Legge, a missionary fromthe London Missionary Society, published the first
English translation of The Analects in Hong Kong.
Legge extensively studied the commentaries on
The Analects from previous generations and used
Victorian English in his translation, striving forfaithfulness and comprehensiveness. Initially,
Legge had a less favorable portrayal of Confucius
in his translation. In contrast, Ezra Pound, who
identified himself as a Confucian, aimed totransform the world through his translation ofConfucian classics (Wang 2004). Pound’stranslation was published in 1951. Despite hislimited proficiency in Chinese, Pound heavily
relied on Legge’s translation as a reference but also
recognized its imperfections, leading him to makesignificant modifications in his own translation.
Pound also emphasized linguistic conciseness (Wei
2005). Another noteworthy translation was by
Tomson, published in 1898, which marked theearliest independent Chinese translation of The
Analects. Tomson had a strong command of
multiple languages, a solid linguistic foundation,and extensive knowledge. His English translationof The Analects gained wide recognition in theWestern world. Tomson believed that Legge’s
translation often fell short of accurately or fully
conveying the original meaning. Thus, Tomson’stranslation aimed to elucidate the cultural elements
missing in the Western context, enabling readers toachieve a more comprehensive understanding(Meng et al. 2012).
In conclusion, while the translations by thesethree individuals are interconnected, they exhibit
distinct characteristics in terms of vocabulary, style,and expression. As highly influential versions, they
excel in terms of faithfulness, intelligibility, andelegance in their language. For the aforementioned
reasons, we have selected these three translations2
3

as reference translations for the purpose ofcomparing and evaluating machine-translated texts.
2.2

Platforms

The platforms selected for this study include:
Baidu Translate (“Baidu” for short), Bing
Microsoft Translator (“Bing” for short), DeepL,
and ChatGPT 2 . The former three are dedicatedonline MT systems, while the last one is aconversational generation system based on largescale language models.
Given that the source text in our research is inancient Chinese, it is essential for us to select atleast one representative MT platforms from China.
Baidu is one of the biggest and most influential MT
platforms in China. According to industry reports
and market data, Baidu consistently ranks first interms of usage among Chinese MT platforms 3 .
Therefore, Baidu has become our top choice as a
MT platform developed in China.
For MT platforms outside of China, we havechosen Bing and DeepL. Both platforms are widely
recognized and highly regarded for their usage andperformance worldwide. Based on our extensive
translation practice, we have observed that DeepL’s
translations occasionally exhibit noticeable
differences in vocabulary and even sentence
structure compared to other MT platforms.
ChatGPT is a language generation model thatpossesses the capability to comprehend andproduce natural language text, encompassing
translation tasks as well. While its primary utilitylies in dialog and text generation, it can, to a certain
extent, engage in translation endeavors. This
attribute permits viable comparisons withconventional MT systems under specificcircumstances. Recently, ChatGPT’s performancein translation tasks has gained increasing attention
and recognition. Although there is currently limited
research on the translation quality of ChatGPT,
some researchers have already drawn the
conclusion that “ChatGPT has already become agood translator.” (Jiao et al. 2023) Based on ourobservation, we have also found that thetranslations generated by ChatGPT exhibit
differences from the three MT platforms. It is worth
noting that each generation of translation by
ChatGPT can vary, and the translation can also be
adjusted based on the given prompts. Therefore, toChatGPT-3.5 version is utilized in this study.
Information source: http://bjx.iimedia.cn/app_rank, last accessed 2023/6/27.

98

ensure relatively reliable experimental results, weonly select the first-generation translation
produced by ChatGPT without adding any otherprompts than “Translate… into English.”a greater morphological similarity between thesentences.
𝐿𝑀𝑆(𝑟, 𝑐) = 1 −3

Evaluation metrics

𝐿𝐷(𝑟, 𝑐)𝑀𝑎𝑥(𝐿𝑒𝑛(𝑟), 𝐿𝑒𝑛(𝑐))(2)

For a specific application scenario involving amachine-translated text collection (C) consisting ofm sentences and the corresponding n sets of humanreference translations (R), we evaluate using thearithmetic mean 𝐿𝑀𝑆, as shown in equation (3), of
the LMS metric. In this experiment, thegetLevenshteinDistance library function fromorg.apache.commons.lang3.StringUtils is used.

The evaluation metrics adopted in this research
include Bilingual Evaluation Understudy (BLEU)(Papineni et al. 2002), Levenshtein-distance-based
Morphological Similarity (LMS) and Pretrainedmodel-based Embedding Semantic Similarity(ESS).
3.1 BLEU
In 2002, IBM proposed the BLEU metric, which
has become the de facto standard for evaluating
MT quality. This metric is based on the mechanicalmorphological evaluation method using n-gramgrammar. In this paper, the BLEU referred to is
BLEU4.

𝑛

𝑚

1
𝐿𝑀𝑆 =
∑ ∑ 𝐿𝑀𝑆(𝑟𝑖𝑗 , 𝑐𝑖 )𝑛𝑚

(3)

𝑗=1 𝑖=1

3.3 ESS𝑛

𝐵𝐿𝐸𝑈 =

𝑚1∑ ∑ 𝐵𝐿𝐸𝑈(𝑟𝑖𝑗 , 𝑐𝑖 )𝑛𝑚

To address the challenge of handling synonymousand morphologically variant expressions, we
introduce the ESS metric as a semantic similarity
evaluation index. This metric maps the humanreference translation (r) and machine-translated
candidate (c) to embedding vectors in a pre-trainedmodel (Peters at al. 2018). Specifically, we obtain
the embedding vectors (vr) for the referencetranslation and (vc) for the candidate translation.
Then, we calculate the cosine similarity between
vectors vr and vc in the embedding vector space(Reimers & Gurevych, 2019), representing theembedding semantic similarity between thereference and candidate translations as ESS(r, c).
According to the definition of this metric, theembedding semantic similarity values between two
sentences is within [-1, 1]. To further normalize
these values so that ESS(r, c)∈[0, 1], we apply a
proportional scaling transformation.
For a specific application scenario involving amachine-translated text collection (C) consisting ofm sentences and the corresponding n sets of humanreference translations (R), we evaluate using thearithmetic mean 𝐸𝑆𝑆, as shown in equation (4), of
the ESS metric. In this study, the all-roberta-largev1 pre-trained model4 was used.

(1)

𝑗=1 𝑖=1

For a specific application scenario involving amachine-translated text collection (C) consisting ofm sentences and the corresponding n sets of humanreference translations (R), we evaluate using thearithmetic mean 𝐵𝐿𝐸𝑈, as shown in equation (1),of the BLEU metric.
3.2 LMSTo evaluate the morphological similarity between
sentences, we introduce the LMS metric. This
metric is based on the edit distance proposed by theSoviet mathematician Vladimir Levenshtein in
1965. The edit distance refers to the minimum
number of editing operations required to transform
one string into another, including substitution,
insertion, and deletion. Let LD(r, c) represent theedit distance between a human referencetranslation (r) and a machine-translated candidate(c). The equation (2) represents the LMS. In theequation, length (r) represents the length of thereference translation and length (c) represents thelength of the candidate translation. The LMS value
ranges from 0 to 1, where a higher value indicates4

https://huggingface.co/sentence-transformers/all-roberta-large-v1

99

𝑛

𝐸𝑆𝑆 =

three metrics BLEU, LMS and ESS, and the results
are shown in Table 1. The highest value obtained
when comparing the texts from different platforms
to the same human translator is highlighted in bold.
We can see that, except for one LMS value fromDeepL, all the other highest values belong to Baidu.

𝑚1∑ ∑ 𝐸𝑆𝑆(𝑟𝑖𝑗 , 𝑐𝑖 )𝑛𝑚

(4)

𝑗=1 𝑖=1

4

Experiment results and analysis

First, we compare each human-translated text witheach machine-translated text respectively under the
Platforms

Human translators

𝑩𝑳𝑬𝑼
0.10590.49010.539
0.04690.02510.09050.06210.03560.10490.04740.02530.0907

TomsonJames Legge
Ezra PoundTomsonJames Legge
Ezra PoundTomsonJames Legge
Ezra PoundTomsonJames Legge
Ezra Pound

Baidu

Bing

DeepL

ChatGPT

Metrics
𝑳𝑴𝑺
0.28570.3731
0.53820.29870.239
0.38680.33230.2656
0.37310.29960.24080.3878𝑬𝑺𝑺
0.84940.91620.95160.81090.84680.85970.84080.8611
0.83210.81170.84780.8606

Table 1. BLEU, LMS and ESS results of human-translated texts and machine-translated texts.

Then, we calculate the mean of the three values
for each platform under each metric, resulting in
the evaluation results for the translation quality ofeach platform. The results are shown in Table 2. It
can be observed that Baidu has the best
performance under all the metrics, with the BLEU
value significantly surpassing the other threeplatforms.
Platforms
Baidu
Bing
DeepL
ChatGPT

𝑩𝑳𝑬𝑼
0.37830.05420.05450.0675𝑳𝑴𝑺
0.39900.30780.30910.3234𝑬𝑺𝑺
0.90570.83910.84000.8446

Table 2. Evaluation results of the four platforms.

In this study, Baidu outperforms other platforms
in translating ancient Chinese texts into English.
Aside from technical architecture and optimized
algorithms, the most possible reason is that Baidu,
as a Chinese company, may have access to a larger
and more diverse dataset of bilingual EnglishChinese texts compared to other platforms,
allowing their models to learn from a wide range oflanguage patterns and translation examples. Thisextensive training data contributes to improving theaccuracy and fluency of it Chinese-English
translations.

5

Other findings: ChatGPT’s
advantages

From Table 2 we can observe that ChatGPT has thesecond-best performance in translation, right after
Baidu. Also, we have found that ChatGPT has also
demonstrated exceptional advantages in certain
aspects.
(1) Semantic interpretation
In ancient Chinese, the phenomenon ofpolysemy, where a single word has multiple
meanings, is very common. MT must makeaccurate judgments regarding the selection of theintended meaning for such words. These judgments
often rely on the contextual information provided
by the surrounding text. However, when thecontext does not provide sufficient information,
MT systems often make errors in determining thecorrect meaning. On the other hand, ChatGPT
demonstrates superior performance in thedisambiguation of polysemous words, excelling inmaking accurate sense selection judgments even incases where context is limited.
Example 1
Source Text (“ST” for short): 君子不器。
Baidu: A gentleman is not a weapon.
100Bing: A gentleman is not a weapon.
DeepL: A gentleman is not a tool.
ChatGPT: A noble person is not defined by theirpossessions or material objects.
In this context, “器” refers to “utensil.” Thissentence conveys the idea that a noble person (君子) should have a broad and inclusive mindset,
rather than being limited to a specific role orfunction like a utensil (器具). However, Baidu andBing mistakenly interpret “器” as “weapon” (武器)and DeepL “tool” (工具).
Example 2
ST: 诲女知之乎！
Baidu: Teach women to know!
Bing: The girl knows!
DeepL: Teach your daughter to know!
ChatGPT: Have you learned to understand this?
In ancient Chinese, the character “女” (female)is often used as a phonetic loan character,
representing the pronoun “汝” (thou). In Example
2, Baidu, Bing, and DeepL do not correctly identify
the true intended meaning of “女,” while ChatGPT
do. We have found that, in our research, thetranslation of all instances of the character “女” asa phonetic loan character in The Analects exhibits
the same pattern across different MT translations.
(2) Fault tolerance
During the process of data compilation, we havediscovered that there were several errors in theoriginal text (These errors were meticulously
rectified, ensuring the source text’s accuracy prior
to commencing the MT task). When the text witherrors is input into the platforms, Baidu, Bing and
DeepL generate translations based on the faulty
text. However, ChatGPT often demonstrate theability to recognize and identify these errors andgenerate translations based on the correctcharacters. ChatGPT possesses this capability thatis not present in traditional MT systems.
Example 3
ST: 邦君之妻，君称之曰夫人，夫人自称日小童……
Baidu: Bangjun’s wife, whom you call Madame,
and whose wife calls herself Rixiaotong…
Bing: The wife of the prince, the prince is called
Mrs. Wang, and the lady calls herself Ri
Xiaotong…
DeepL: The wife of the king of the state, the kingcalled it the wife, the wife called herself the day
child…

ChatGPT: The wife of the ruler of a state iscalled “Madam” by the ruler, and she refers toherself as “Little Child”…
In Example 3, the underlined character “日”(pronounced as “ri”, meaning the sun or day) in thesource text should be “曰” (say), and “小童” that
follows means “little child”. This sentence meansthat the madam calls herself “小童.” Both Baidu
and Bing transliterate this character mistakenly and
DeepL interprets “日” as “day,” while ChatGPT
recognizes this fault.
Example 4
ST: 恭而无礼则劳，慎而无礼则蒽……
Baidu: Respectful but impolite leads to labor,
cautious but impolite leads to anthracene…
Bing: If you are respectful and rude, you will
work, if you are cautious and rude, you will be
rude…
DeepL: Respectful but no ritual is labor, prudent
but no ritual is anthracite…
ChatGPT: If one is respectful without propriety,it leads to laboriousness. If one is cautious without
propriety, it leads to timidity…
“蒽” (anthraquinone) in the original text is a
faulty character which should be “葸” (afraid,
bashful). Both Baidu and DeepL take “蒽” directly,
and Bing simply omits it. ChatGPT recognizes thefault and translates it into “timidity,” which aligns
with the intended meaning of the original text.

6

Conclusion

Based on the evaluation of translation quality of theChinese classic The Analects by Baidu, Bing,
DeepL, and ChatGPT using the BLEU, LMS, and
ESS metrics, we have found that among the four
platforms, Baidu, as a MT platform developed in
China, performs the best in handling ancient
Chinese texts. Its scores in all three metrics are
significantly higher than the other three platforms.
ChatGPT, as a general-purpose language model,
ranks second among the four, and has demonstrated
unique advantages, and the translations it produces
are highly valuable as references. It is worth
mentioning that in this study, the translations
generated by ChatGPT were done without any
prompts (except for the one mentioned in 2.2) oradjustments. We plan to discuss in our future
research the translation quality of ChatGPT byincorporating prompts for adjusting the translationof ancient Chinese into English.
101

Acknowledgments
The research is supported by the Scientific
Research Project of China National Committee for
Terminology in Science and Technology (No.
YB2021027), the Art Science Key Project ofShandong Provincial Association for Science ofArts & Culture (L2022Z06170304), the New
Liberal Arts Research and Reform Practice Projectof Ministry of Education of China (No.
2021060049), the Postgraduate Education andTeaching Reform Research Project of Shandong(No. SDYJG21185), the Key Project ofUndergraduate Teaching Reform Research of
Shandong (No. Z2021323), the Science andTechnology Program of Guangzhou (No.
202201010061), and the Humanity and Social
Science Research Project of Ministry of Educationof China (No. 20YJAZH069).

References
Kishore Papineni, Salim Roukos, Todd Ward, et al.
2002. Bleu: A Method for Automatic Evaluation of
Machine Translation. In Proceedings of the 40th
Annual Meeting of the Association for
Computational Linguistics, 311-318.
Gai Gutherz, Shai Gordin, Luis Sáenz, et al. 2023.
Translating Akkadian to English With Neural
Machine Translation. PNAS nexus, 2(5): pgad096.
Chanjun Park, Chanhee Lee, Yeongwook Yang, et al.
2020. Ancient Korean Neural Machine Translation.
IEEE Acces, 8: 116617-116625.
Zhiyuan Zhang, Wei Li, Qi Su. 2019. AutomaticTranslating Between Ancient Chinese andContemporary Chinese with Limited Aligned
Corpora. In Natural Language Processing andChinese Computing: 8th CCF International
Conference, NLPCC 2019, Dunhuang, China,
October 9–14, 2019, Proceedings, Part II 8.
Springer International Publishing, 157-167.
Chengbin Zhou, Zhongbao Liu. 2022. MachineTranslation of Ancient Chinese Text Based on
Transformer of Semantic Information Sharing.
Technology Intelligence Engineering, 8(06): 114127.
Zhenjun Yao, Xuhong Zheng, Pengtao Xu, et al. 2013.
An Exploration of Phrase-Based SMT for English
Translation of Tao Te Ching. Shandong Foreign
Language Teaching, 34(03): 109-112.

Chinese Translation. Neural Computing andApplications, 33: 3855-3867.
Tariq Yousef, Chiara Palladino, David J. Wright, et al.
2022. Automatic translation alignment for ancient
Greek and Latin. In Proceedings of the Second
Workshop on Language Technologies for
Historical and Ancient Languages, 101-107.
Gang Li, Jinshu Li. 2013. On the English Translationof The Analects: A Survey. Journal of Social
Science of Hunan Normal University, 42(01).
Tomson. (Trans.). 2011. The Discourse and Sayings ofConfucius. Yunnan People’s Publishing House,
Kunming, China.
James Legge. (Trans.). 2016. Confucian Analects.
Liaoning People’s Publishing House, Shenyang,
China.
Ezra Pound. (Trans.). 1933. Confucian Analects. Peter
Owen Limited, London, UK.
Hui Wang. 2004. A Comparison of Confucian Analects
Translated by James Legge and Ezra Pound.
Foreign Language and Literature, (05), 140-144.
Wangdong Wei. 2005. A Multi-Perspective
Comparison of Three Translations of Lun Yu:
From James Legge, Ezra Pound to Edward
Slingerland. Chinese Translators Journal, (03), 5257.
Jian Meng, Tao Qu, Yang Xia. 2012. English
Translation of Chinese Classics under Adaptation
Theory – Reflections on the English Translation of
Lunyu by Gu Hong-ming. Foreign Language
Research, (03), 104-108.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, et al.
2023. Is ChatGPT a Good Translator? A
Preliminary
Study.
arXiv
preprint
arXiv:2301.08745.
Kishore Papineni, Salim Roukos, Todd Ward, et al.
2002. Bleu: A Method for Automatic Evaluation of
Machine Translation. Proceedings of the 40th
Annual Meeting of the Association for
Computational Linguistics, 311-318.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, et al.
2018. Deep Contextualized Word Repretations.
arXiv:1802.05365.
Nils Reimers, Iryna Gurevych. 2019. Sentence-Bert:
Sentence Embeddings Using Siamese BertNetworks. arXiv preprint arXiv:1908.10084.

Kexin Yang, Dayiheng Liu, Qian Qu, et al. 2021. An
Automatic Evaluation Metric for Ancient-Modern102

A tailored Handwritten-Text-Recognition System for Medieval Latin
Esteban Garces Arias1♠
Gilary Vera Nuñez1♢
2♣
Alexander Häberlin2,3♡Matthias Schöffel
Matthias Aßenmacher1,4♠
1
Department of Statistics, LMU, Munich, Germany
2
Bavarian Academy of Sciences, BAdW, Munich, Germany
3
Universität Zürich, Zurich, Switzerland
4
Munich Center for Machine Learning (MCML), LMU, Munich, Germany

Philipp Koch1♢
Christian Heumann1♠♢{philipp.koch,gi.vera}@campus.lmu.de♠♣

matthias.schoeffel@badw.de♡alexander.haeberlin@sglp.uzh.ch{esteban.garcesarias,chris,matthias}@stat.uni-muenchen.de

Abstract
The Bavarian Academy of Sciences and Humanities aims to digitize its Medieval Latin
Dictionary. This dictionary entails record cards
referring to lemmas in medieval Latin, a lowresource language. A crucial step of the digitization process is the Handwritten Text Recognition (HTR) of the handwritten lemmas foundon these record cards. In our work, we introduce an end-to-end pipeline, tailored to the medieval Latin dictionary, for locating, extracting, and transcribing the lemmas. We employ
two state-of-the-art (SOTA) image segmentation models to prepare the initial data set for
the HTR task. Furthermore, we experimentwith different transformer-based models andconduct a set of experiments to explore the capabilities of different combinations of vision
encoders with a GPT-2 decoder. Additionally,
we also apply extensive data augmentation resulting in a highly competitive model. The bestperforming setup achieved a Character Error
Rate (CER) of 0.015, which is even superior
to the commercial Google Cloud Vision model,
and shows a more stable performance.

1

Figure 1: Record card from the MLW data set.

Introduction

The Medieval Latin Dictionary (MLW)1 deals with
Latin texts that were created between 500 and 1280
in the German-speaking region. The foundations
for this project have been developed from 1948 onwards and since then, the dictionary has been continuously published in individual partial editions
since 1959. The basis of the dictionary consists of
50 selected texts that have been fully transcribed
onto DIN-A6 record cards (cf. Fig. 1) constituting about 40% of the note material. Later, another
2,500 texts were excerpted and transcribed manually onto DIN-A6 record cards, using a typewriter.
In addition, there are so-called "index cards", atype of record card, that helps to uncover often
1

In German: Mittellateinisches Wörterbuch (MLW)

hundreds of additional references. In total, it isestimated that 1.3 million reference points havebeen recorded for the MLW. These record cards
were sorted alphabetically by the first letter of the
keyword (lemma), and serve as the foundation for
creating a dictionary. Around 200,000 record cards
have been scanned and annotated with their respective lemma. The accurate extraction and transcription of the lemma present a challenge, which isfurther compounded by the limited resources available for medieval Latin.
Our contributions are as follows: (1) We present
a novel end-to-end HTR pipeline specifically designed for detecting and transcribing handwritten
medieval Latin text. Notably, it surpasses commercial applications currently considered SOTA
for related tasks. (2) We train a lemma-detection
model without relying on human-annotated bounding boxes. (3) We conduct extensive experiments
to compare various vision encoders and evaluate
the effectiveness of data augmentation techniques.

2

Related Work

We provide an overview for HTR, which is themain challenge of this work. For object detection,
which is an intermediate step of this work, we refer
to Zaidi et al. (2021) for a detailed overview.
Connectionist Temporal Classification (CTC)
103

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 103–110,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_012

(Graves et al., 2006) is a technique in which aneural network – initially a Recurrent Neural Network (RNN) but other networks might also be used
(Chaudhary and Bali, 2022) – is trained to predict a
matrix of conditional transition probabilities. The
input image, represented as a vector representation
through a Convolutional Neural Network (CNN),
is fed to the network, and for each input (i.e. the activation maps of the CNN) the network predicts thecharacter. CTC, combined with CNNs and RNNs,
often yielded competitive results, such as shown by
Puigcerver (2017) and Bluche and Messina (2017).
Furthermore, approaches applying only CNNs and
CTC also exist (Chaudhary and Bali, 2021, 2022).
Easter2.0 achieved competitive results on IAM(Marti and Bunke, 2002), a frequently used HTR
data set consisting of English handwritten text.
A recent work that achieved SOTA results on IAM
is TrOCR (Li et al., 2022), based on the transformer(Vaswani et al., 2017), consisting of a vision encoder and a text decoder. This deviated from previous approaches where primarily CNNs and RNNs
were used. This development is closely linked tothe emergence of the transformer in the vision domain (Dosovitskiy et al., 2021; Bao et al., 2022).
Barrere et al. (2022) introduce another transformer
model also using CTC, with the main difference
to TrOCR being a different embedding techniquefor visual features based on a CNN. The results
have also been shown to be competitive on the
IAM data set. Diaz et al. (2021) compare encoderdecoder models’ performance on HTR, using different models in the encoder and decoder parts, e.g.
a transformer encoder plus a CTC-based decoder.
Furthermore, they found that enriching this architecture with a language model yields SOTA resultson IAM. The TrOCR framework has already beensuccessfully applied to historical data akin to ourtask. Ströbel et al. (2022) fine-tune a TrOCR instance to handwritten Latin from the 16th century(Stotz and Ströbel, 2021, referred to as Gwalther),achieving competitive results.

3

Data

observe lemmas from a length of one character up
to 19 characters, with an average length between
five and six characters. A total of 2,420 lemmas(69%) appear on ten record cards or less; 854 lemmas (24.4%), on 10 to 100 record cards, and just
233 lemmas (6.6%) on more than 100 record cards.
1,123 lemmas (36.7%) only occur on one card.

4

Lemma Extraction Pipeline

4.1

Visual Detection

Since the lemmas are always located in the upper left corner, but not annotated with their exact locations, training a custom object detection
model for extraction is not feasible. In order tostill retrieve the locations of the bounding boxes
for some lemmas, we use the One For All (OFA)transformer (Wang et al., 2022), fine-tuned on RefCOCO (Kazemzadeh et al., 2014). To ensure thequality of the extracted lemma, we experimentwith multiple prompts and examine their results(cf. Appendix A). After obtaining a training data
set of 20,000 instances, we train a YOLOv8 model(Jocher et al., 2023) based on the You Only Look
Once (YOLO) architecture (Redmon et al., 2016).
The model predictions from our YOLO model, arethen subject to two post-processing steps to ensure
the quality of the images:
For 17,674 images (15.42% of the data), the model
predicted multiple bounding boxes. We visually
examined the cases and found that other handwritten text was often recognized as a lemma, sometimes scattered throughout the record cards (cf. Fig.
5, Appendix B). We further visually examined 202
cases where no bounding box was detected, stemming mostly from machine writing or scanningerrors. For some images that follow the standard
layout of the record cards, the model also failed.
We disregard this set constituting less than 0.2% of
the entire data set.
Taking all aspects into account, we introduce tworules to determine the appropriate bounding box:
(1) choose the largest bounding box in (2) the upper left quarter of the entire image. The result after
applying these rules is displayed in Figure 6 (Appendix B). The final data set consists of 114,451
samples, exhibiting a difference of the 202 samples
to the initial 114,653 image-label pairs. We makeour data available on HuggingFace.2

Our data set comprises 114,653 images, holding
3,507 distinct lemmas. All images are in RGB,
but not uniform in size. The information on thecorresponding lemma is available on the image
level. Most lemmas start with the letter "s", followed by a large number of lemmas starting withthe letters "m", "v", "t", "u", "l", and "n". We
1042

https://huggingface.co/misoda

4.2

5.2

HTR Model

We use a transformer as the main model akin to
TrOCR. For the encoder, we consider three different architectures, while we use GPT-2 (Radford
et al., 2019) as a decoder model for all setups. All
models are trained from scratch, although we use
pre-trained image processors for the encoder models and train a tokenizer for our custom alphabet.
Tokenizer We use a customized byte-level BPE tokenizer (Sennrich et al., 2016) (trained on the labelsfrom our data) for the dictionary’s vocabulary.
Vision Encoders We consider three different encoder architectures, namely Vision Transformer(ViT) (Dosovitskiy et al., 2021), Bidirectional
Encoder representation for Image Transformers(BEiT) (Bao et al., 2022), and Shifted Window
Transformer (Swin) (Liu et al., 2021).
Text Decoder We use the GPT-2 (Radford et al.,2019) architecture, a decoder-only transformer,
which we train from scratch, i.e., we do not use
the pre-trained weights since we deal with a specific task in a low-resource language setting.
Implementation Details We use the HuggingFace transformers library (Wolf et al., 2020) and
PyTorch (Paszke et al., 2019) to train the HTR
pipeline. Our codebase, containing all scripts (experiments and training) is available via GitHub3 ,and the final model is on pypi.4 All the experiments
were conducted using a Tesla V100 GPU (16 GB).

5

Experiments

5.1

Standard Training Settings

After shuffling the data, we randomly split it intoa train (85% – 97,283 samples) and a test (15%
– 17,168 samples) set. In the train split, 94.53%(3,315) of the lemmas are present. For all training procedures, we use the AdamW optimizer(Loshchilov and Hutter, 2019) and did not engage
in hyperparameter tuning. Further details are reported in Appendix C. For standard training, themodel is trained using a data set that includes the
cut images from the record cards as input and theirrespective lemmas as the labels to be predicted. We
train each of the models for a total of 5 epochs.
We assess the model performance using the CER,
which is computed by summing up edit operations
and dividing by the label length. To account for
the varying length, we further utilize the weighted
CER.
3
4

https://github.com/slds-lmu/mlw-htr
https://pypi.org/project/mlw-lectiomat/

Data Augmentation

To increase the diversity of the training data, weapply random rotation, blurring, or modifications
related to color perception. For the augmentation
setting, we increase the number of epochs to 20(compared to 5 for the standard training). We
use three different augmentation pipelines, one ofwhich is randomly chosen with p = 13 .
Pipeline A applies blurring and modifications to
sharpness. The intensity of these modifications
is determined randomly and can range from no
modification to higher intensity. Pipeline B alters
brightness, contrast, saturation, sharpness, and hue.
The specific alterations for each instance are again
determined randomly, also including the possibilityof no modifications at all. Pipeline C combines
the modifications from the previous two. In addition to the described techniques, all augmentation
pipelines include random masking, where rectangles of the images are blackened, and random rotation within a range of -10 to 10 degrees.
Decoder Pre-Training We experiment with decoder pre-training (10 epochs) on a corpus of theconcatenated lemmas to incorporate prior knowledge about medieval Latin. After pre-training, we
combine it with the encoder and continue training
for 20 epochs as described in Section 5.1, using thesame augmentation techniques outlined before.
5.3

Experimental Results

The main results of our work are reported in Table 1. The BEiT+GPT-2 architecture achieved thebest results in case of the standard training regime,
exhibiting a CER of 0.258, followed by Swin+GPT2 (0.349) and ViT+GPT-2 (0.418). Applying theaugmentation pipelines notably improves model
performance compared to the standard training for
all three models. The best model with augmentation is Swin+GPT-2, achieving a CER of 0.017.
As for the other two models, the CER is 0.073 for
ViT+GPT-2 and 0.110 for BEiT+GPT-2.

Standard+ Data Augmentation+ Decoder Pre-Training

ViT

Swin

BEiT

0.418
0.073
0.049

0.349
0.017
0.018

0.258
0.110
0.114

Table 1: CERs for different encoder configurations.

Pre-training of the decoder does, on average, notlead to further improvement. ViT+GPT-2 is the
exception, for which the CER drops to 0.049. We
105

observe no improvements for the other models. To
summarize, the best results are achieved when using a Swin+GPT-2 model with data augmentations,reaching a CER value of 0.017.

To conclude, our best model exhibits a weighted
CER of 0.0153, while GCV only reaches 0.1045.
Overall, our model correctly predicts 97,09% of alllemmas, while GCV only does so for 78.26%.

5.4

5.6

Ablation Study

To investigate the impact of data augmentation, we
perform three ablations, removing individual steps
from the augmentation pipelines. To quantify theindividual effects of each augmentation technique,we train the model without a specific augmentation
method and report the resulting CER.
Swin+GPT-2 (Full augmentation pipelines)

0.017

w/o masking augmentation
w/o rotation augmentation
w/o color augmentation

0.015
0.021
0.017

Table 2: CER-Results of different model configurations.

Excluding the masking step from the pipeline leads
to an actual improvement of model performance
while excluding random rotations or color-related
augmentations results do not (cf. Tab. 2).
5.5

Google Cloud Vision Comparison

Performance of other HTR systems

Table 3 illustrates the CERs of other systems on
different HTR data sets. Ströbel et al. (2022) use
Gwalther, while all other papers evaluate their systems on IAM. Our model achieves the lowest CER.
However, it must be considered that we did notevaluate the same data set, which makes a directcomparison impossible. In contrast to the othertransformer-based models, our best model uses
Swin as an encoder.
Model

CER

Data set

Architecture

Ours (Best)0.0153

MLW

Transformer

TrOCR Large (Ströbel et al., 2022)
TrOCR Large (Li et al., 2022)
EASTER2.0 (Chaudhary and Bali, 2022)
Light Transformer (Barrere et al., 2022)
Self-Att.+CTC+LM (Diaz et al., 2021)

0.02550.02890.06210.05700.0275

Gwalther
IAM
IAM
IAM
IAM

Transformer
Transformer
CNN+CTC
CNN+Transformer
Trf.+CTC+ LM

Table 3: Performance of contemporary HTR systems.

6

Conclusion and Outlook

Since the record cards include much more information than the one we extracted, we recommend
further research into various extraction techniques.
With the recent publication of Segment Anything
Model, Kirillov et al. (2023) introduce a model thatmight be able to extract further features from therecord cards with much higher accuracy.
We present a novel end-to-end pipeline for the Medieval Latin dictionary. Our library includes animage-detection-based model for lemma extracFigure 2: Comparison of Swin+GPT-2 to GCV.
tion and a tailored HTR model. We experimentwith training different configurations of transformTo compare the results of our model, we decided
to use Google Cloud Vision (GCV) a highly com- ers using the ViT, BEiT, and Swin encoders while
using a GPT-2 decoder. Employing data augmenpetitive HTR model, which has proven effective
in practical applications (Thammarak et al., 2022). tation, our best model (Swin+GPT-2) achieves a
CER of 0.015. The evaluation of the results exGCV often predicts extra characters and/or suffixes
hibits a weaker performance on longer lemmas
that are not part of the true lemma, which is why
and on lemmas that appear less frequently in the
we post-process the predictions by GCV for a fairtraining data. Further experiments with generative
comparison by deleting extra characters and words
models to produce synthetic data (not reported inafter the first word or after a ’-’ or a ’(’. Figure
2 shows the comparison of our model with GCV. the paper) were not successful, however, we recommend further research into this direction. To
The violin plots of the (unweighted) CERs show
conclude, our approach presents a promising HTR
a concentration of the CER values around 0 for
solution for Medieval Latin. Future research canboth models. For our model, the most extreme
build upon our work, and explore its generalizabilvalues are at a CER of 3, for GCV the maximum
ity to other languages and data sets by making use
is nearly twice as high and we observe an overall
higher standard deviation compared to our model. of our pip-installable Python package.
106

Limitations
Our approach has several limitations that can beaddressed to improve its efficiency further. There
are issues regarding the data set (cf. Sec. 3) thatmight be reflected in the model’s performance. As
discussed in Section 3, some lemmas are stroked
out partially or entirely, introducing a notable noise
to the data. Further, handwritten comments or otherannotations have been added to some of the record
cards, and some images are not correctly labeled,which might have distorted the recognition capabilities of our model.
Since our pipeline was mostly trained on datafrom the S-series of the dictionary, many words
starting with other letters were not seen by themodel during training. Therefore, the performanceof the proposed approach, when applied to otherseries, remains somewhat uncertain. As elaborated
in section 6, the model tends to perform weaker on
unseen lemmas. Further, there are indications thatthe model might perform worse on longer lemmas.
The lemma-detection model (YOLOv8) is notguaranteed to predict the correct bounding box forthe lemma consistently. Errors at this early stageof the pipeline may severely impact the result. Although the failure rate for the training dataset inwhich no bounding box was predicted is close tozero, the problem can still appear during inference.
We did neither experiment with the initial
TrOCR architecture nor did we fine-tune a pretrained TrOCR instance for this task. However, theresults of Ströbel et al. (2022) suggest a strong performance of TrOCR. Thus, we recommend training
it on the MLW data set.

Ethics Statement

Acknowledgements
We would like to thank the three anonymous referees for their constructive comments and suggestions, which helped improve this paper considerably. We wish to thank the Bavarian Academy of
Sciences for providing us with the guidance andrequired access to the handwritten material. This
work has been partially funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research
Foundation) as part of BERD@NFDI - grant number 460037581.

References
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.
2022. Beit: Bert pre-training of image transformers.
Killian Barrere, Yann Soullard, Aurélie Lemaitre, andBertrand Coüasnon. 2022. A light transformer-basedarchitecture for handwritten text recognition. In
Document Analysis Systems, pages 275–290, Cham.
Springer International Publishing.
Théodore Bluche and Ronaldo Messina. 2017. Gated
convolutional recurrent neural networks for multilingual handwriting recognition. In 2017 14th IAPR
International Conference on Document Analysis andRecognition (ICDAR), volume 01, pages 646–651.
Kartik Chaudhary and Raghav Bali. 2021. Easter:
Simplifying Text Recognition using only 1dConvolutions.
Proceedings of the Canadian Conference on Artificial Intelligence.
Https://caiac.pubpub.org/pub/fm5sy88o.
Kartik Chaudhary and Raghav Bali. 2022. Easter2.0:
Improving convolutional models for handwritten text
recognition.
Daniel Hernandez Diaz, Siyang Qin, Reeve Ingle, Yasuhisa Fujii, and Alessandro Bissacco. 2021. Rethinking text line recognition models.
Alexey Dosovitskiy, Lucas Beyer, Alexander
Kolesnikov, Dirk Weissenborn, Xiaohua Zhai,
Thomas Unterthiner, Mostafa Dehghani, Matthias
Minderer, Georg Heigold, Sylvain Gelly, Jakob
Uszkoreit, and Neil Houlsby. 2021. An image
is worth 16x16 words: Transformers for image
recognition at scale.

We affirm that our research adheres to the ACL
Ethics Policy. This work involves the use of publicly available data sets and does not involve humansubjects or any personally identifiable information.
We declare that we have no conflicts of interest
that could potentially influence the outcomes, in- Alex Graves, Santiago Fernández, Faustino Gomez, andJürgen Schmidhuber. 2006. Connectionist temporal
terpretations, or conclusions of this research. Allclassification: Labelling unsegmented sequence data
funding sources supporting this study are acknowlwith recurrent neural networks. In Proceedings ofedged. We have made our best effort to document
the 23rd International Conference on Machine Learning, ICML ’06, page 369–376, New York, NY, USA.
our methodology, experiments, and results accuAssociation for Computing Machinery.
rately and are committed to sharing our code, data,and other relevant resources to foster reproducibil- Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023.
ity and further advancements in research.
YOLO by Ultralytics.
107Sahar Kazemzadeh, Vicente Ordonez, Mark Matten,
and Tamara Berg. 2014. ReferItGame: Referring to
objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 787–
798, Doha, Qatar. Association for Computational
Linguistics.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi
Mao, Chloe Rolland, Laura Gustafson, Tete Xiao,
Spencer Whitehead, Alexander C. Berg, Wan-Yen
Lo, Piotr Dollár, and Ross Girshick. 2023. Segment
anything.

Phillip Benjamin Ströbel, Simon Clematide, Martin
Volk, and Tobias Hodel. 2022. Transformer-based
htr for historical documents.
Karanrat Thammarak, Prateep Kongkla, Yaowarat
Sirisathitkul, and Sarun Intakosum. 2022. Comparative analysis of tesseract and google cloud vision
for thai vehicle registration certificate. International
Journal of Electrical and Computer Engineering,
22:1849–1858.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need.

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan
Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, andFuru Wei. 2022. Trocr: Transformer-based opticalcharacter recognition with pre-trained models.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei,
Zheng Zhang, Stephen Lin, and Baining Guo. 2021.
Swin transformer: Hierarchical vision transformer
using shifted windows.
Ilya Loshchilov and Frank Hutter. 2019. Decoupledweight decay regularization.
Urs-Viktor Marti and H. Bunke. 2002. The iamdatabase: An english sentence database for offline
handwriting recognition. International Journal on
Document Analysis and Recognition, 5:39–46.
Adam Paszke, Sam Gross, Francisco Massa, Adam
Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca
Antiga, Alban Desmaison, Andreas Köpf, Edward
Yang, Zach DeVito, Martin Raison, Alykhan Tejani,
Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An
imperative style, high-performance deep learning library.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai
Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren
Zhou, and Hongxia Yang. 2022. Ofa: Unifying architectures, tasks, and modalities through a simple
sequence-to-sequence learning framework.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing.
Syed Sahil Abbas Zaidi, Mohammad Samar Ansari,
Asra Aslam, Nadia Kanwal, Mamoona Asghar, andBrian Lee. 2021. A survey of modern deep learning
based object detection models.

Joan Puigcerver. 2017. Are multidimensional recurrent
layers really necessary for handwritten text recognition? In 2017 14th IAPR International Conference
on Document Analysis and Recognition (ICDAR),
volume 01, pages 67–72.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Dario Amodei, Ilya Sutskever, et al. 2019. Language
models are unsupervised multitask learners. OpenAI
blog, 1(8):9.
Joseph Redmon, Santosh Divvala, Ross Girshick, andAli Farhadi. 2016. You only look once: Unified,real-time object detection.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words withsubword units.
Peter Stotz and Phillip Ströbel. 2021. bullingerdigital/gwalther-handwriting-ground- truth: Initial
release.

108

Appendix
A

Annotatong the Bounding Boxes

This Appendix holds the details of the Visual Detection part of the pipeline, described in Section
4.1, and the challenges we were confronted with.
A.1

The Task(a) First Prompt

To annotate the bounding boxes, the model is provided with a prompt describing the lemma and theimage. The model then returns a bounding box forthe requested object, which is the lemma in ourcase. Different prompts are described in Table 4.
Prompt 1
Prompt 2
Prompt 3
Prompt 4

Length: 1-5:
Other:
Length: 1-6:
Other:

Cursive text upper left
Handwritten cursive word upper left
Blue drawing in the upper left
Handwritten cursive word upper left
Blue drawing in the upper left
Handwritten cursive word upper left(b) Second Prompt

Table 4: Different prompts used for OFA.

A.2

Assumption about Bounding Boxes

Since we do not have any ground truth about thebounding boxes, we rely on heuristics to verify
the correctness of the boxes. One such heuristic isthe assumed linear relationship between the lemma
length and the bounding box’s width. While theheight of the boxes is assumed to be similar acrossinstances, the lemma length must significantly impact the bounding box’s width. To verify the resultsof the annotation process, we use box plots to visualize the relationship between lemma length andwidth (cf. Fig. 3a – 3d).
A.3

Initial Implementation and Results

We use the RefCOCO-OFA model5 and modify it
four our purposes. Prompt one (cf. Tab. 4) is used
to obtain the lemmas for all images.
After running the model on the first instanceswith Prompt 1, we find that the relationship between the box’s width and the lemma length does
not look as expected. Figure 3 illustrates this problem. Investigating the short lemmas, we observethat the model often fails to annotate the record
cards appropriately. Often other textual objects areannotated, or the bounding box stretches throughout the entire record card.
5(c) Third Prompt

Huggingface: OFA-Base-RefCOCO(d) Fourth and final Prompt

Figure 3: Box-Plots for the width of the bounding boxes
based on the lemma’s length.

A.4

Two Different Prompts for Shorter andLonger Lemmata

After different experiments, Prompt 2 turned out
to work appropriately for shorter lemmas, but was,however, not suitable for longer ones. To combine
the strength of both prompts, we apply a conditional prompt based on the length of the lemmausing different cut-offs (5 or 6 characters). We findthat using Prompt 4 is the best-suited approach.
The analysis of the relationship between the bound109

ing box widths and the length of the lemma fordifferent prompts can be seen in Figure 3.

B

YOLO: Training and Inference

B.1

Training Results

C.1

Standard Training

Parameter
Seed
Optimizer
Epochs
Decoder
Encoder
Batch Size (Train & Test)

Value
42
AdamW
5
GPT-2
{BEIT, Swin, ViT}
64

Table 5: Parameters for the standard training.

C.2

Parameter
Seed
Optimizer
Epochs
Decoder
Encoder
Batch Size (Train & Test)

Figure 4: YOLO Training Results.

B.2

Training with Augmentation

Multiple Lemmas Detected by YOLO

Value
42
AdamW
{5, 20}
GPT-2{BEIT, Swin, ViT}
64

Table 6: Parameters for training with augmentation.

C.3

Natural Language Generation
Parameter
Max Length
Early Stopping
No Repeat Ngram Size
Length Penalty
Number of Beams

Figure 5: All bounding boxes from instances where
YOLO has detected more than one bounding box.

Value
32
True
3
2.0
4

Table 7: Parameters for natural language generation.

C.4

Decoder Pre-Training
Parameter
Seed
EpochsBatch Size (Train & Test)

Value
42
10
192

Table 8: Parameters for pre-training of the decoder.

Figure 6: Bounding boxes of all instances to which therule largest bounding box in the upper left corner wasapplied to.

C

Training details

We used the defaults from transformers (4.26.1),
if not reported otherwise.
110

Evaluating Existing Lemmatisers on Unedited Byzantine Greek
PoetryColin Swaelens1 , Ilse De Vos2 and Els Lefever1
1
LT3, Language and Translation Technology Team
Department of Translation, Interpreting and Communication, Ghent University
2
Department of Linguistics, Ghent University
9000 Ghent, Belgium{colin.swaelens, i.devos, els.lefever}@ugent.be
Abstract
This paper reports on the results of a comparative evaluation of four existing lemmatizers, all pre-trained on Ancient Greek
texts, on a novel corpus of unedited, Byzantine Greek texts. The aim of this study isto get insights into the pitfalls of existinglemmatisation approaches as well as the
specific challenges of our Byzantine Greek
corpus, in order to develop a new lemmatizer that can cope with its peculiarities.
The results of the experiment show an accuracy drop of 20% on our corpus, which
is further investigated in a qualitative error
analysis.

1

Introduction

parchment, omissions of words due to sloppiness or fatigue of the scribe or funky orthography due to phonetic changes. Although nosubstantial HTR-based corpus is available for
Greek, two online available corpora do offer
the unedited texts from manuscripts: the Trismegistos (Depauw and Gheldof, 2014) project
and the Database of Byzantine Book Epigrams(DBBE) (Ricceri et al., 2023). Both Trismegistos and DBBE do store the edited as well as
the unedited version of texts found in papyri
and manuscripts, respectively. The DBBE
provides Byzantine1 book epigrams, which are
metrical paratexts as they are written in themargin, next to (παρά, para) the main textof a manuscript. The literal transcription ofthese poems are stored as so-called Occurrences, which are linked to a normalised version called Type.
Our aim is to develop a linguistic annotation
pipeline for the latter, unedited Greek texts.
The differences between Ancient and Medieval
Greek are thoroughly described by Swaelens
et al. (Forthcoming 2023), the features relevant for this work are elaborated upon in Section 3. A new approach for part-of-speech tagging and morphological analysis was developed(Swaelens et al., 2023), as the existing techniques are not capable of handling the idiosyncrasies these unedited texts display. Before
diving into the development of the last stepof the pipeline, i.e. the lemmatizer, we wanted
to evaluate existing systems for lemmatisationon our gold standard of unedited, Byzantine
Greek texts.

If Ancient Greek is considered a low-resourced
language, Byzantine Greek is even lowerresourced.
What Ancient and Byzantine
Greek have in common, is that their texts havebeen continuously copied by hand until theend of the 15th century. So when we read,for instance, Plato’s Apology, we read a collation of a philologist who aspires to reconstruct Plato’s original 4th -century text basedon the existing Medieval manuscripts; basedon but not copied from these manuscripts, as
linguistic inconsistencies or orthographic mistakes are adapted to fit the dialect in which
the text was conceived. Existing NLP tools
for historical Greek are developed for this variant of Greek, that was edited to perfection.
However, because of a growing researchinterest and progress in optical character
recognition (OCR) and, even more relevant, handwritten text recognition (HTR) 2 Related Research(e.g. Tsochatzidis et al. 2021; Platanou et al.
The first lemmatizer for Greek was developed2022; Ströbel et al. 2022), more and more
unedited Greek texts will become available. by Packard (1973), as part of the first lin1
These unedited texts contain, among other
Byzantine and Medieval will be used as synonyms
things, lacunae due to a damaged piece ofto refer to the period from the 5th until 15th century.
111
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 111–116,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_013

guistic annotation pipeline. In order to perform morphological analysis, first suﬀixes are
removed to retrieve the stem of every token. Then, a dictionary made by Packard, issearched with a binary search algorithm to findthe matching stem. Based on this dictionary
search, the algorithm returns the lemma thatis linked to the matching stem. If multiple
lemmas are possible, a philologist is needed todiscern which lemma was the correct one.
In 2003, the biggest online resource of Greek
texts, the Thesaurus Linguae Graecae (TLG)(Pantelia, 2022) started their lemmatization
project. Few details on the methodology areprovided in the paper, except that the TLG
digitised and extracted a large number of headwords from dictionaries2 . The authors, however, claim that the lemmatizer is capable ofrecognising automatically 98.3% of all tokens
in the TLG.
RNN Tagger (Schmid, 2019) was developed
as the combination of a morphological tagger
and lemmatizer for historical texts. Schmid
has made use of a character-based bi-LSTM
network to cope with – systematic – spellingvariations and improve tagging accuracy. The
lemmatizer is also based on a recurrent neural network, making use of the dl4mt machinetranslation system (He et al., 2016). In his experiments, Schmid did also train and test his
tagger on the Ancient Greek Dependency Treebank, which resulted in a tagging accuracy of
91.29%.
The Classical Language Toolkit (CLTK), isan NLP framework developed for pre-modern
languages (Johnson et al., 2021). This framework stores several lemmatizers, among whicha back-off lemmatizer (Burns, 2020) thatmakes use of several, sequenced lemmatizers.
CLTK’s default lemmatizer for Ancient Greek
makes use of the Stanza (Qi et al., 2020)
lemmatization algorithm, that has been pretrained on the PROIEL treebanks (Haug and
Jøhndal, 2008). This algorithm consists of adictionary-based lemmatizer combined with aneural sequence-to-sequence lemmatizer. On
the encoder’s output of this combination, an
additional classifier is added to cope with,among other things, lowercasing. The authors,
however, did not provide an accuracy score of

how well the algorithm performs on Ancient
Greek.
Burns’ back-off lemmatizer, which is included in the CLTK, is a sequence of five lemmatizers. The token first passes a dictionarybased lemmatizer to tag frequently occurring,indeclinable words; then it passes through aunigram-model lemmatizer that is based on
training data of the Ancient Greek and Latin
Dependency Treebanks (Celano, 2019); third
in the sequence is a rule-based lemmatizer thatmakes use of regular expressions; the fourthlemmatizer is a variation of the previous, regular expression-based lemmatizer that factors inprincipal-part information; finally, the token
is passed through another dictionary-basedlemmatizer making use of Morpheus’ (Crane,1991) lemma dictionary. Should none of theselemmatizers output a proper lemma, the token itself is returned as lemma. Vatri and
McGillivray (2020) report an accuracy of 91%
on poetry and 93% on prose.
Where CLTK’s default lemmatizer disambiguates ambiguous tokens based on frequency, the GLEM lemmatizer (Bary et al.,2017) makes use of part-of-speech information to disambiguate. Even more interesting, is that GLEM provides a lemma for outof-vocabulary words. This is achieved by
combining a dictionary-based approach witha memory-based machine learning algorithm,
called FROG (Bosch et al., 2007). If the tobe-tagged word occurs only once in the lexicon, consisting of the PROIEL and Perseus(Celano, 2019) corpora, GLEM returns the lexicon’s lemma; if not, the word is consideredambiguous and FROG is applied. If several
lemmas are possible, GLEM evaluates whether
there is exactly one match with the part-ofspeech tag predicted by the FROG algorithm
and the lexicon. If so, the lemma is assigned; if
there are several possible or no matching partof-speech tags, frequency information is used
to assign a lemma from the lexicon.

The interest in lemmatizing Greek has increased, proved by Keersmaekers and Van Hal(2022) and de Graaf et al. (2022). That is,both articles discover how corpora which cannot be lumped together with classical, literary Greek prose, could be lemmatized. Keers2
https://stephanus.tlg.uci.edu/history.php
maekers and Van Hal, on the one hand, aim to112

lemmatize the papyri texts stored in Trismegistos, de Graaf et al., on the other hand, look
into lemmatizing Greek inscriptions. Just likethe unedited texts we want to tag, these corpora had some peculiarities that deviate fromthe polished, classical Greek on which the existing lemmatizers are based.
Although several other lemmatizers do exist, they are not part of this assessment because they are either not freely available or do
not disambiguate ambiguous word forms. We
did not test TreeTagger (Schmid, 1994) since
the parameter files3 do not contain any information on lemmas. Neither Morpheus (Crane,1991) nor Eleuxis4 have been part of our comparison as neither of those disambiguate ambiguous tokens.

3

Comparative Experiment

To evaluate the lemmatizers described in Section 2, we annotated about 10,000 tokens fromthe DBBE occurrences (Swaelens et al., 2023).
The DBBE occurrences are the literal transcription, viz. without any editing, of the textthat is found in a manuscript. As alreadymentioned in Section 1, these occurrences arelinked to edited, normalised versions called
Types, as shown in Example 1. Example 1ashows the occurrence, the text as it is found in
the manuscript Vat.gr.19085 , Example 1b theType to which the Occurrence is linked and its
translation (translated by the authors) is given
in Example 1c.
(1)a.

ὧς περ᾽ ξἔνη χἔρον|τες ἡδἧνπ(ατ)ρίδα
DBBE Occurrence 17870b.

Ὥσπερ ξένοι χαίρουσιν ἰδεῖν πατρίδα
DBBE Type 2820c.

Just like travellers rejoice upon seeing their homeland

3
Available at https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/
4
https://outils.biblissima.fr/en/
eulexis-web/index.php
5This book epigram is situated on f.118v and online consultable via https://digi.vatlib.it/view/
MSS_Vat.gr.1908/0121

This example displays one of the main characteristics of the Greek found in manuscripts
before they are edited: orthographic inconsistencies. Since the itacism – a phonetic shiftthat turned η, ι, υ, ει and οι into the phoneme/i/ – has made its introduction in the 3rd century, quite some orthographic inconsistencies
are to be found in the manuscripts. In Example 1a both the first syllable, ἡδ- (the stem of
the word), and the second, -ἧν (the suﬀix indicating the Greek infinitive), are affected by theitacism. This makes the word ἰδεῖν almost unrecognisable, which is why we hypothesise thata dictionary-based approach might be put at
a disadvantage.
For our comparative study, all lemmatizers
discussed in Section 2, CLTK, GLEM, RNN
Tagger, and the Stanza tagger, are used tolemmatize our gold standard containing 10,000
tokens of unedited, Byzantine Greek text. Before feeding the data to the lemmatizers, we removed all redundant white spaces and deleted
all punctuation.

4 Results
The results of the comparitive experiments areshown in Table 1. First of all, We observe a
general accuracy drop of 20% or more compared to the results of the lemmatizers on Ancient, edited Greek. This was expected, because our data is very challenging. Second, thesequential back-off lemmatizer comes out best,
performing almost 7% better than the Stanzalemmatizer, which performed worst. To gain
more insight in the results of the tested lemmatizers,we performed a qualitative analysisof the system output, which revealed some tendencies of the problems related to our corpus.
Lemmatizer
StanzaRNN Tagger
GLEM
CLTK

Accuracy
64.99%
66.67%
70.82%
71.69%

Table 1: Performance of existing lemmatizers on
Byzantine Greek poetry.

This comparative study uncovered an encoding problem in our test set: the transcriptions of the manuscripts stored in DBBE makeuse of multiple unicode characters for identi113

cal characters. The acute accent, for example,
is present in the DBBE as two different unicode characters. That is, the ί in πατρίδα(Example 1) has two different unicode representations within the DBBE corpus. Consequently, every deviation from the unicodecharacter that is stored in DBBE or its annotations has been evaluated as incorrect. Whatis more, the Stanza lemmatizer outputs unicode characters that are different from those
CLTK and RNN Tagger output.
The diachronic and/or diatopic alterations
that are inherent to the Greek language, hinders the evaluation of the taggers as well.
Verbs whose stem ends in a velar occlusive,
have a lemma that ends either in -ττω (theclassic, Athenian variant), or -σσω (other dialects’ variant). The token φύλαττε (keepguard) has been annotated as coming fromthe lemma φυλάττω, while all lemmatizers returned φυλάσσω as lemma. Although this is acorrect prediction, it was considered as incorrect by the automatic evaluation. In this same
category belongs the alteration between ι andυ, observable in the – identical – words βίβλος
and βύβλος (papyrus roll).6 The alteration ofa word’s final consonant, is the last example
that fits within this category. The preposition ἐκ (out) is written as ἐξ when followed
by a vowel. Again, these double forms caused
unjust penalties in the lemmatizers’ output.
In order to cope with these inconsistencies,
we harmonised the different outputs, mainly
caused by the unicode difference between thetonos and oxia accent (Tauber, 2019). The
new lemmatisation results, however, show aminor impact of the encoding problems andinconsistencies, resulting in improvements of
only 0.04 to 0.6 %, which makes no difference
for the final ranking of the tested lemmatizers.
The lemmatizers also have a hard time assigning the correct lemma to a verb in the perfect tense. This might be due to the very low
presence of this tense in general in Greek. It
is, however, surprising that the back-off lemmatizer cannot extract and match the stemof, e.g., πεφευγώς (having fled) to its lemmaφεύγω (to flee). What is even more surprising,is that GLEM did not even return a lemma of
6This alteration is not to be confused with theitacism; this alteration is already attested before theitacism appeared.

this quite frequent word, while it was statedthat GLEM could output lemmas it had neverseen before.
A GLEM-specific remark is how much thislemmatizer is affected by the absence of theiota subscriptum7 in, e.g., the dative case. In
DBBE this iota is sometimes written, now underneath the vowel, then next to it, and sometimes not written. Not once did it correctly
lemmatize τω as a form of the article ὁ, whileτῷ has been lemmatized correctly. The iota
adscriptum is not yet part of the test set.

5 Conclusion & Future Research
As a last step in the development of our new
annotation pipeline that cannot only handle
classical Greek texts but also unedited, Byzantine texts, we are exploring the field of lemmatizing Greek. We compared four freely available lemmatizers that are capable of copingwith ambiguity: CLTK back-off lemmatizer,
GLEM, RNN Tagger and the Stanza lemmatizer. The back-off lemmatizer performed best,
which might be attributed to the fact that it
combines five different lemmatizers. The error analysis provided us with useful insights,
which we will take into account while developing our own lemmatizer for Byzantine Greek.
At the moment of writing, we are looking
into a cascaded system that combines a rulebased module with a dictionary look-up as
a first step. In addition, a machine-learningcomponent will be developed to handle all tokens that cannot be lemmatized by the first
part. We are investigating several possible algorithms, going from a decision tree model toa neural approach. Furthermore, we will needto cope with the abundance of unicode characters and provide a mapping to evaluate our
output correctly. We also need to develop a
strategy to deal with the alterations that are
inherent to the language to make evaluationeasier and more correct, namely a mapping of(1) the five ways to write the /i/ sound, (2) the
iota subscriptum or adscriptum and (3) forms
like -σσω/-ττω. Finally, we will experimentwith the presence or absence of diacritics andtheir possible impact on the machine learning.
7

When a long vowel is followed by a iota, ι /j/, theiota is written either underneath (subscriptum) or next
to that vowel (adscriptum).

114

References
Corien Bary, Peter Berck, and Iris Hendrickx. 2017.
A memory-based lemmatizer for ancient greek.
In Proceedings of the 2nd International Conference on Digital Access to Textual Cultural Heritage, DATeCH2017, page 91–95, New York, NY,
USA. Association for Computing Machinery.
Antal van den Bosch, Bertjan Busser, Sander Canisius, and Walter Daelemans. 2007. An efficient memory-based morphosyntactic tagger
and parser for dutch. LOT Occasional Series,
7:191–206.
Patrick J Burns. 2020. Ensemble lemmatization
with the classical language toolkit. Studi e Saggi
Linguistici, 58(1):157–176.

Alek Keersmaekers and Toon Van Hal. 2022. In
search of the flocks: How to perform onomasiological queries in an Ancient Greek corpus?
In Proceedings of the Second Workshop on Language Technologies for Historical and Ancient
Languages, pages 73–83, Marseille, France. European Language Resources Association.
David W. Packard. 1973. Computer-assisted morphological analysis of Ancient Greek. In COLING 1973 Volume 2: Computational And Mathematical Linguistics: Proceedings of the International Conference on Computational Linguistics.
Maria C. Pantelia. 2022. Thesaurus Linguae Graecae, A Bibliographic Guide to the Canon of
Greek Authors and Works. University of California Press, Berkeley.
Paraskevi Platanou, John Pavlopoulos, and Georgios Papaioannou. 2022. Handwritten paleographic Greek text recognition: A century-based
approach. In Proceedings of the Thirteenth
Language Resources and Evaluation Conference,
pages 6585–6589, Marseille, France. European
Language Resources Association.

Giuseppe G.A. Celano. 2019. The Dependency
Treebanks for Ancient Greek and Latin, pages
279–298. De Gruyter Saur, Berlin, Boston.
Gregory R. Crane. 1991. Generating and Parsing
Classical Greek. Literary and Linguistic Computing, 6(4):243–245.
Evelien de Graaf, Silvia Stopponi, Jasper K. Bos,
Saskia Peels-Matthey, and Malvina Nissim. 2022.
AGILe: The first lemmatizer for Ancient Greek
inscriptions. In Proceedings of the Thirteenth
Language Resources and Evaluation Conference,
pages 5334–5344, Marseille, France. European
Language Resources Association.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason
Bolton, and Christopher D. Manning. 2020.
Stanza: A Python natural language processing
toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System
Demonstrations.

Mark Depauw and Tom Gheldof. 2014. Trismegistos: An interdisciplinary platform for ancientworld texts and related information. In Theory and Practice of Digital Libraries – TPDL
2013 Selected Workshops, pages 40–52, Cham.
Springer International Publishing.

Rachele Ricceri, Klaas Bentein, Floris Bernard,
Antoon Bronselaer, Els De Paermentier, Pieterjan De Potter, Guy De Tré, Ilse De Vos, Maxime
Deforche, Kristoffel Demoen, Els Lefever, AnneSophie Rouckhout, and Colin Swaelens. 2023.
The database of byzantine book epigrams
project: Principles, challenges, opportunities.
Journal of Data Mining & Digital Humanities.

Dag TT Haug and Marius Jøhndal. 2008. Creating a parallel treebank of the old indo-european
bible translations. In Proceedings of the second
workshop on language technology for cultural heritage data (LaTeCH 2008), pages 27–34.

Helmut Schmid. 1994. Probabilistic part-of-speechtagging using decision trees. In Proceedings ofthe International Conference on New Methods in
Language Processing, Manchester, UK.

Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai
Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Duallearning for machine translation. Advances inneural information processing systems, 29.
Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, and William
J. B. Mattingly. 2021. The Classical Language
Toolkit: An NLP framework for pre-modern languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 20–29, Online. Association for Computational Linguistics.

Helmut Schmid. 2019. Deep learning-based morphological taggers and lemmatizers for annotating historical texts. In Proceedings of the 3rdInternational Conference on Digital Access to
Textual Cultural Heritage, DATeCH2019, page
133–137, New York, NY, USA. Association for
Computing Machinery.
Phillip Benjamin Ströbel, Martin Volk, Simon
Clematide, Raphael Schwitter, Tobias Hodel,
and David Schoch. 2022. Evaluation of HTR
models without ground truth material. In
Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4395–
4404, Marseille, France. European Language Resources Association.

115

Colin Swaelens, Ilse De Vos, and Els Lefever. 2023.
Medieval social media: Manual and automatic
annotation of byzantine Greek marginal writing.
In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 1–9, Toronto,
Canada. Association for Computational Linguistics.
Colin Swaelens, Ilse De Vos, and Els Lefever. Forthcoming 2023. Linguistic annotation of byzantine
book epigrams. Language Resources and Evaluation.
James K. Tauber. 2019. Character Encoding ofClassical Languages, pages 137–158. De Gruyter
Saur, Berlin, Boston.
Lazaros Tsochatzidis, Symeon Symeonidis, Alexandros Papazoglou, and Ioannis Pratikakis. 2021.
Htr for greek historical handwritten documents.
Journal of Imaging, 7(12).
Alessandro Vatri and Barbara McGillivray. 2020.
Lemmatization for ancient greek: An experimental assessment of the state of the art. Journalof Greek Linguistics, 20(2):179 – 196.

116

Vector-Based Stylistic Analysis on Ancient Chinese Books: Take the Three
Commentaries on the Spring and Autumn Annals as an Example
Yue Qi 1 ,Liu Liu 1 *, Bin Li 2 , Dongbo Wang 1
College of Information Management, Nanjing Agricultural University,Nanjing,China
2
School of Chinese Language and Literature, Nanjing Normal University,Nanjing,China
liuliu@njau.edu.cn
1

Abstract"Poetic Sage". For the study of stylistics, the traditional paradigm generally starts from vocabulary,
Commentary of Gongyang, Commentary of Gurhetoric, sentence patterns, etc., with the help of
liang, and Commentary of Zuo are collectively
examples, and forms an interpretive logic that iscalled the Three Commentaries on the Spring
now called "close reading". Corresponding to thisand Autumn Annals, which are the supplement
and interpretation of the content of Spring andis the "distance reading" after the rise of digital
Autumn Annals with value in historical and literhumanities. With the help of many computationalary research. In traditional research paradigms,
methods such as lexical statistics, quantitative linscholars often explored the differences between
guistics, and natural language processing, the study
the Three Commentaries within the details inof textual style has increasingly inclined towards recontexts. Starting from the view of Stylistic
sults with precision value. This research paradigm,
Analysis, this paper examines the differences inor computational humanities, provides new explothe language style of the Three Commentaries
through the representation of language, whichration perspectives for studying stylistics.
takes the methods of deep learning. SpecifiThis study focuses on the style of ancient books
cally, this study vectorizes the context at wordin computational humanities. Compared with tradiand sentence levels. It maps them into the sametional methods, the advantage of the computational
plane to find the differences between the use
humanities lies in quantification, which is basedof words and sentences in the Three Commenon data and computation to obtain objective andtaries. The results show that the Commentaryof Gongyang and the Commentary of Guliang
verifiable conclusions. The study of stylistics underare relatively similar, while the Commentary
this paradigm also presents a variety of technicalof Zuo is significantly different. This paper
and theoretical frameworks due to the intersection
verifies the feasibility of deep learning methof fields, forming a developing trend of mutual inods in stylistics study under computational hutegration. This study of the style of ancient books
manities. It provides a valuable perspective
depends on multi-level observations from Chinese
for studying the Three Commentaries on thecharacters to vocabulary and sentences. RepresenSpring and Autumn Annals.
tation learning can also provide more comprehen1 Introduction
sive quantification for the style analysis of ancient
books.
Style is an additional component in the process oflanguage expression and expression. It changes
This research focuses on the Three Commendue to the social era and environment in whichtaries on the Spring and Autumn Annals and relanguage is used and in various forms due to thelated ancient books. The Three Commentaries areuser’s expression habits and intentions. This char- the most important classics among ancient Chiacteristic has received longstanding attention fromnese books and have also received much attention
stylistics. Among the study of ancient Chinese clas- in computational humanities. On the other hand,sics, the Spring and Autumn Annals was known as
stylistic studies on the difference between the Three"having profound meaning in simple words." andCommentaries have also gained much attention.
the Historical Records were called "Li Sao without
Specifically, this study will take Hong Ye’s Index
rhyme." These are classic summaries of ancienton Spring and Autumn Annals and the Three ComChinese books. The language style can also bementaries as the data source and use text represenused to compare and analyze authors, such as Li
tation learning in deep learning to examine the style
Bai and Du Fu honored as "Poetic Immortal" anddifferences between the Three Commentaries. As
117
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 117–121,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_014

an essential content of computational humanities,
this study will provide a compelling computational
research idea and reference for studying style inancient books.

2

Related Research

Annals as the research object. It aims at texts ofdifferent levels to explore the language style differences formed in writing the three biographies.
This is significant for exploring ancient books in
the Spring and Autumn Periods.

3

Style Comparison With Text

The Three Commentaries on the Spring and AuRepresentation
tumn Annals revolve around the history of the Lu
Words and sentences are important objects in stylisState recorded in the Spring and Autumn Period
in terms of content and ideological system. Still, tics research; different from the measurement ofword frequency in traditional diagrams, this study
there are apparent differences in the writing andlanguage style focus. Scholars often draw rele- uses representation learning models in deep learning to automatically obtain the vectorized reprevant conclusions based on a careful reading of theThree Commentaries(Chen, 2021), which have sig- sentation of words and sentences to acquire knowlnificantly contributed to the development of histo- edge about vocabulary and sentence style. Specifically, Word2vec and Sentence-BERT were chosen
riography but need more accurate, verifiable, andreproducible digital indicators to prove it. More- respectively for vectorizing words and sentences
over, the entry point for investigation is single, of- in the Three Commentaries, and mapping scatter
points with dimensionality reduction was used forten only starting from a specific problem, needing amacroscopic inspection from a global perspective. the style comparison.
As one of the research directions of ancient Chi3.1 Model Introduction
nese text mining, the metrological research of oldbooks has the characteristics of mature technology Word2vec is a neural network language model thatcan capture semantic information between contexts,
and diverse perspectives. According to the different
map each word into a word vector, and mine conproperties of the research objects, it can be divided
into different research levels, such as vocabulary, nections between words(Mikolov et al., 2013). The
Word2vec model contains two models for trainsentences, and text. The measurement research ofing word vectors: the CBOW (Continuous Bagancient books based on vocabulary includes wordof-Words Model) model and the Skip-gram model.
segmentation(Huang et al., 2015), part-of-speechtagging(Zhang et al., 2021), named entity recogni- The former uses N words before and after the feature word to predict the word, and the latter usestion(Liu and Wang, 2018), keyword extraction(Qinthe word’s context to predict N words before and afand Wang, 2020), etc. In the quantitative research
conducted around the sentence level, taking sen- ter. The Word2vec model adds contextual analysis
to the context, which makes the semantic analysis
tence segmentation (Zhao et al., 2022), sentenceclassification(Liu et al., 2013), and sentence extrac- more abundant.
Sentence BERT (SBERT for short) is a sention(Zhou et al., 2021) as examples, it is possible totence
vector computing model proposed (Reimers
explore the implicit features and inter-sentence reand Gurevych, 2019), which maps text into Veclationships of sentences in ancient books. Research
tor space in sentence units. One vector can repreat the chapter level includes research on automatic
summarization(Xu et al., 2022), bibliographic in- sent the semantics expressed by a sentence in the
text. Compared with the BERT model, SBERT canformation measurement(Tong et al., 2021), etc. In
better generate sentences. The Embedding vector
summary, studying computational humanities inancient books has extended research in various di- enables the vectorized expression to carry moresemantic information.
rections at different levels and has gradually formeda mature research paradigm. However, there is still
3.2 Word Vectorization Mappinga gap in the study of the style and style of ancient
books, and there needs to be more research that
As was written in the name, the Three Commentakes ancient books as the main body and uses
taries were all commentaries on Spring and Auquantitative analysis to observe the differences intumn Annals, which provided detailed descriptions
digital indicators of ancient books. This study takesof historical events of the State of Lu in the Spring
the Three Commentaries on the Spring and Autumn
and Autumn period. Their themes and contents118

Figure 1: The mapping of words in the Three Commentaries

Figure 2: The mapping of single-occurrence words in
the Three Commentaries

in the upper right corner of Figure 1, with a relatively clear and intuitive boundary between theCommentary of Gongyang and the Commentary of
Guliang. Based on this, from the perspective ofwords, even though the content is similar, the threebiographies still have differences in language style.
The Commentary of Gongyang and the Commentary of Guliang are identical in language style andpreferred word definition. At the same time, theCommentary of Zuo has a unique narrative stylethat prefers supplementary historical events.
Single-occurrence words refer to the words
that occurred only in one of the Three Commentaries.Compared with some more general words,
these words that only appear in certain commentaries can better reflect the language habits in theprocess of writing the book. On the map of singleoccurring words, there is almost no overlapping
part, which conforms to the definition concept ofsingle-occurring words, which can explain thatbased on similar content, the language styles of theThree Commentaries are different in terms of words.
And the distribution of each commentaries point is
consistent with that shown in Figure 2. The above
image also reflects the orange dots near points (4,1).
The distribution of single-occurrence words deepens the accuracy and credibility of the above picturefrom the side.The distribution of single-occurrencewords deepens the accuracy and credibility of theabove picture from the side, which confirms that
the Three Commentaries not only show differencesin the overall language style but also have different
habits in the use of single-occurrence words.

are similar to a certain extent, but their language
styles are different. Based on this, it can be considered that the differences shown in the mapping on the scatterplot are more due to the differences in the language styles of the Three Commentaries, rather than the "fixed collocation" between
words, that is, the differences caused by different
recorded content. We use the word-segmented text
to train the Word2vec model and generate words
into multi-dimensional word vectors. To map in
3.3 Sentence Vectorization Mapping
two-dimensional space, PCA is used to reduce thedimensionality of word vectors and map them on
The process of generating sentence vectors is to
the graph through different colors.The distribution
use the text after the sentence to train the SBERTof words and single-occurrence words of the threemodel, and each generated vector represents a senbiographies is shown in Figure 1 and Figure 2.
tence. Similar to the word vector dimensionality
Each dot represents a word in the scatter plot, reduction method, PCA is used to reduce the diand the three colors correspond to the Three Com- mensionality of the sentence vector so that it canbe presented on a two-dimensional graph. The dismentaries. For example, a blue dot represents a
tribution of sentence vectors is shown in Figure 3.
word in the Commentary of Zuo. The numberof points determines the depth of the color at thecoordinate position. Since a point represents a
Each point in Figure 3 represents a sentence,
word, the distance between points represents theand the distance between points represents the sendegree of similarity between the two words. It
tence’s similarity. In the sentence vector, it is obcan be observed that the three biographies have
served that the dispersion of the sentence vector isa slight overlap near the point ( 0, 0 ) in Figureslightly smaller than that of the word vector, and
1. In addition, the blue word points representing
the overlapping area is larger around the point (0,0).
the Commentary of Zuo are mainly concentrated
Most of Figure 3’s color blocks are composed of119Figure 3: The distribution of sentence vectors in theThree Commentaries

mixed and interlaced colors. But the Commentaryof Zuo still shows differences, converging into asingle color block around the point (-7.5, 2). Thisphenomenon is consistent with what the word vectors offer, and it reflects that the Commentary of
Zuo is significantly different from the other twobiographies in style.
Based on the mapping results of word vectors
and sentence vectors, the style of the Commentary of Gongyang and the Commentary of Guliang
are relatively similar, and the Commentary of Zuo
shows distinct style differences. This result aligns
with the views of ancient and modern scholars who
have carefully read Three Commentaries on the
Spring and Autumn Annals and can deepen the conclusion that the Commentary of Zuo is different inlanguage style.

4

Conclusion

ference between the Three Commentaries can beobserved from multiple perspectives, such as automatically mining different types of entities, orobserving the usage habits of words from the partof speech.
From the perspective of quantitative linguistics,
the language style differences between the Three
Commentaries will be observed through different
levels of language measurement indicators. At
the word level, the index selects the average word
length to measure the difference in word length,
selects the word density, standard type ratio, andsingle word ratio to measure the difference in therichness of the Three Commentaries vocabulary,and observes the information carrying capacity of
the richness of the Three Commentaries by calculating the entropy of text information. At thesentence level, the writing characteristics of theThree Commentaries were examined through average sentence length, sentence dispersion, sentence fragmentation, and other indicators. From
the above two perspectives, the linguistic characteristics and stylistic differences of the Three Commentaries on the Spring and Autumn Annals can be
examined from a new perspective, which provides
new verification ideas for the research related toThree Commentaries on the Spring and Autumn
Annals.

5

Acknowledgments

This work was supported by the National NatuFrom the perspective of Natural language process- ral Science Foundation of China [grant number
ing, uses a deep learning model with good versa- 72004095], and the National Social Science Fundtility to calculate the language style differences be- of China [grant number 21&ZD331].
tween different levels of the Three Commentaries
on the Spring and Autumn Annals. It concludes that
the Commentary of Zuo differs from the Commen- References
tary of Gongyang, and the Commentary of Guliang,
Wangheng Chen. 2021. Political Aesthetic Thoughts ofrealizing the mining research on the language charthe Three Commentaries on the Spring and Autumn
acteristics of ancient books.
Annals. Wuhan University Journal (Philosophy &Social Science), 74(6):80–94.
In the follow-up research, we will use other methods to examine the differences between the Three
Shuiqing Huang, Dongbo Wang, and Lin He. 2015. DisCommentaries. From the perspective of natural
cussion on Automatic Word Segmentation of Pre-Qin
language processing, this study has verified theClassics Using Sinological Index Series as the dofeasibility of the general language model in discovmain vocabulary. Library and Information Service,59(11):127–133.
ering the differences in the Three Commentaries,and subsequent language models suitable for ancient Chinese, such as GuwenBERT, SikuBERT, Liu Liu, Bin Li, Weiguang Qu, and Xiaohe Chen. 2013.
Automatic Acquisition of Age Characteristics of Preand other pre-training based on ancient Chinese
Qin Vocabulary and Automatic Judgment of Docudomain data enhancement model to further observe
ment Age. Journal of Chinese Information Processdifferences in language styles. In addition, the difing, 27(5):107–113.
120

Liu Liu and Dongbo Wang. 2018. A Review of Named
Entity Recognition. Journal of the China Society for
Scientific and Technical Information, 37(3):329–340.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.
Heran Qin and Dongbo Wang. 2020. Application of
Keyword Extraction in Pre-Qin Ancient Chinese under the Digital Humanities——Taking Chun Qiu Jing
Zhuan as an Example. Library Journal, 39(11):97–
105.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERTNetworks. ArXiv:1908.10084 [cs].
Lin Tong, Bing Liu, Jingpeng Deng, Dajun Liu, Yongsheng Yang, Xueyuan Hu, Zefeng Kang, and Ruili
Huo. 2021. Quantitative analysis and thinking on
bibliographic information of existing ancient Chinese ophthalmology books. Journal of Traditional
Chinese Ophthalmology, 31(6):449–452.
Runhua Xu, Dongbo Wang, Huan Liu, Yuan Liang, andKang Chen. 2022. Research on Automatic Summarization of History as a Mirror for the Digital Humanities of Ancient Books——Taking the SikuBERT
Pre-training Model as an Example. Library Tribune,
42(12):129–137.
Qi Zhang, Chuan Jiang, Youshu Ji, Minxuan Feng, Bin
Li, Chao Xu, and Liu Liu. 2021. Construction of an
automatic tagging model for part-of-speech integration of word segmentation for multi-domain pre-Qinclassics. Data Analysis and Knowledge Discovery,
5(3):2–11.
Lianzhen Zhao, Yiqin Zhang, Jiangfeng Liu, Dongbo
Wang, Minxuan Feng, and Bin Li. 2022. Researchon Automatic Punctuation of Pre-Qin and Han Classics for Digital Humanities—Taking SikuBERT Pretraining Model as an Example. Library Tribune,
42(12):120–128+137.
Hao Zhou, Dongbo Wang, and Shuiqing Huang. 2021.
Research on Automatic Recognition of Context ofCitations in Ancient Books——Taking Commentary
Documents as an Example. Information studies: theory & application, 44(9):169–175.

121

A Joint Model of Automatic Word Segmentation and Part-Of-Speech
Tagging for Ancient Classical Texts Based on Radicals
Bolin Chang1,2 , Yiguo Yuan2 , Bin Li1,2B , Zhixing Xu1,2 , Minxuan Feng1,2 , Dongbo Wang3,21School of Chinese Language and Literature, Nanjing Normal University, Nanjing, China
2Center of Language Big Data and Computational Humanities, Nanjing Normal University, Nanjing, China
3
College of Information Management, Nanjing Agricultural University, Nanjing, China
Blibin.njnu@gmail.com
Abstract
The technique of word segmentation and partof-speech tagging in ancient Chinese plays acrucial role in the field of information processing in ancient Chinese. The current state ofancient Chinese word segmentation and partof-speech tagging technology presents pressing
issues that require immediate attention, suchas low accuracy and efficiency. This study
employs a methodology that combines word
segmentation and part-of-speech tagging. It
establishes a correlation between fonts and radicals, trains the Radical2Vector radical vector
representation model, and integrates it with
the SikuRoBERTa word vector representation
model. Finally, it connects the BiLSTM-CRF
neural network.The study investigates the combination of word segmentation and part-ofspeech tagging through an experimental approach using a specific data set. In the evaluation dataset, the F1 score for word segmentation is 95.75%, indicating a high level of accuracy. Similarly, the F1 score for part-of-speechtagging is 91.65%, suggesting a satisfactory
performance in this task. This model enhances
the efficiency and precision of the processing ofancient books, thereby facilitating the advancement of digitization efforts for ancient books
and ensuring the preservation and advancementof ancient book heritage.

1

Introduction

notable enhancements in both labelling efficiency
and accuracy. The progress of this technology not
only facilitates the processing of ancient Chinese
texts, but also exerts a significant influence on interconnected disciplines, including literature, history,
philology, and digital humanities.
The maturation of ancient Chinese automaticword segmentation and part-of-speech tagging technologies is occurring with the ongoing advancement of computer technology. Nevertheless, thecurrent utilisation of these methodologies is confronted with two distinct obstacles as a result of thenumerous distinctive characteristics of ancient Chinese. Firstly, it is worth noting that there remains
potential for enhancing the precision of word segmentation and part-of-speech labelling in the context of ancient Chinese. While several technologies
currently available have the capacity to partiallysubstitute manual labelling, their level of accuracy
falls short of totally replacing manual labelling.
Consequently, a significant amount of proofreading labour is necessary during the later stages. Secondly, there is a need for additional enhancementin the effectiveness of word segmentation and partof-speech tagging in the context of ancient Chinese.
The prevailing approach involves conducting word
segmentation as the initial step, followed by partof-speech tagging. Nevertheless, this sequencing
will result in diminished processing efficiency andhas the potential to propagate errors in word segmentation to the subsequent part-of-speech tagging
phase, so exacerbating the influence of these errors
and subsequently diminishing overall accuracy.
This paper utilizes the Word2Vec model to incorporate the radical information of Chinese characters. It proceeds to train the Radical2Vector modeland combines it with SikuRoBERTa to form theEmbedding layer. Subsequently, the BiLSTM-CRF
neural network is connected to conduct an experiment on the integration of word segmentation andpart-of-speech tagging in ancient Chinese. The

The challenge of automatically segmenting words
and assigning part-of-speech tags to ancient Chinese text is a crucial area of study within the discipline of natural language processing. The primary objective of this project is to employ computer technology for the precise identification ofword boundaries in ancient Chinese writings, aswell as the exact assignment of appropriate partof-speech labels to these words, including nouns,
verbs, conjunctions, and others. By implementing
this procedure, the conventional task of manual
labelling can be efficiently alleviated, leading to122

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 122–132,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_015

utilization of ancient Chinese word segmentationand part-of-speech tagging facilitates the exploration of profound insights within ancient texts,
thereby advancing the digital advancement and utilization of these texts. Furthermore, it contributes
to the preservation and progression of ancient literary works.

2

Related Work

The co-examination of automatic word segmentation and part-of-speech tagging in the context of ancient Chinese is a common area of research. Huang(2002) conducted a study on part-of-speech tagging
in ancient Chinese using the hidden Markov model.
They applied this model to analyze "The Analectsof Confucius" and "Tao Te Ching". Although thestudy employed a set of 22 part-of-speech tags, itmade significant contributions to the field. In theirstudy, Fang (2009) developed a text segmentation
program called Yu Segmentation Program. The
researchers focused on ancient books such as "TheClassic of Tea" and employed a model algorithm
that utilized tree pruning to achieve efficient text
segmentation of these classical texts. The F1 score
for word segmentation has been reported to be approximately 86% by Min Shi (2010). A comparative experiment was conducted to evaluate the performance of the Conditional Random Fields (CRF)
model in the tasks of automatic word segmentation,
part-of-speech tagging, and integration of ancient
Chinese. Both features and integrated processing
contribute to the enhancement of the F1 value. Runhua Xu (2012) proposed a method that utilizes
structured annotations to enhance the word segmentation process. In their study, Shuiqing Huang(2015) employed the CRF model to analyze word
categories, phonetics, and probability features. Notably, their analysis yielded a remarkable F1 valueof 97.47%. According to the study conducted by
Xiaoyu Wang (2017), This paper examines the issue of automatic word segmentation in Middle Ancient Chinese by employing a combination of the
CRF model and a dictionary. It also investigates
the impact of inconsistent word segmentation on
the results of artificial word segmentation in Middle Ancient Chinese through experimental analysis. Additionally, the paper introduces characterclassification as part of the research methodology.
The dictionary information exhibits two notablefeatures. Firstly, the word segmentation F1 value
achieved a remarkable accuracy rate of over 99%

in the closed test. Secondly, in the open test, theword segmentation F1 value ranged between 89%
and 95%, further highlighting the effectiveness ofthe dictionary information. Ning Cheng (2020)
employed the Word2Vec-BiLSTM-CRF model toinvestigate the amalgamation of part-of-speech tagging for sentence segmentation and part-of-speech
analysis in ancient Chinese texts.
Numerous studies have been conducted on theutilization of vector representations of strokes,
parts, components, and radicals to facilitate Chinese information processing, both in contemporary
and ancient contexts. In their study, Tao (2019)
introduces a new model called Dual-channel Word
Embedding (DWE) that aims to effectively capture
both sequential and spatial information of characters. The author argues that this model demonstrates a logical and advantageous approach in representing the morphology of Chinese language. In
their study, Zhang (2021) presents a novel model
called the Feature Subsequence based Probability
Representation Model (FSPRM) for the purpose of
acquiring Chinese word embeddings. The model
incorporates both morphological and phonetic features, specifically stroke, structure, and pinyin, ofChinese characters. By designing a feature subsequence, the model captures a wide range of semantic information pertaining to Chinese words.
The efficacy of the proposed method is substantiated through a series of comprehensive experiments
conducted on various tasks including word analogy,
word similarity, text classification, and named entity recognition. The results of these experiments
consistently indicate that the proposed method surpasses the performance of the majority of existingstate-of-the-art approaches. In the study conductedby Shi (2015), a novel deep learning techniquereferred to as "radical embedding" is introduced.
The author provides a rationale for this approachby drawing upon principles derived from Chinese
linguistics. Furthermore, the feasibility and usefulness of this technique are assessed through a seriesof three experiments. In their study, Yu (2017)presents a method for simultaneously embedding
Chinese words, characters, and subcharacter components at a detailed level. The performance of ourmodel is shown to be superior through evaluation
on both word similarity and word analogy tasks.
In their study, Han (2018) utilized a shared radical
level embedding approach to address the task of
Simplified and Traditional Chinese Word Segmen-

123tation. Notably, their method does not require any
additional conversion from Traditional to Simplified Chinese. The integration of radical and character embeddings results in a reduction in parametercount, while facilitating the sharing and transfer ofsemantic knowledge between the two levels. This
integration significantly enhances performance. In
their recent publication, Tang (2021) introduces
a pioneering model named Moto, which aims toenhance embedding through the incorporation ofmultiple joint factors. The empirical findings indicate that our Moto model attains state-of-the-art
performance with an F1-score of 0.8316, representing a 2.11% improvement, when applied to Chinese
news titles. Furthermore, it achieves an accuracy of
96.38 (a 1.24% improvement) on the Fudan Corpus
dataset and 0.9633 (a 3.26% improvement) on the
THUCNews dataset. Among the various researchendeavors, the investigation into the utilization ofradical vectors stands out as the most prominent.
On one hand, this phenomenon can be attributed
to the relatively straightforward acquisition of thecorresponding relationship data between Chinesecharacters and their radicals. On the other hand,
the inclusion of radical vectors has been found toenhance the efficacy of Chinese information processing tasks.
It is evident that among the aforementioned studies, only one specifically addresses the topic ofancient Chinese classical Chinese, with a specificfocus on automating sentence segmentation tasks.
The absence of vector representations for strokes,
parts, components, and radicals in ancient Chinese
information processing has the potential to enhance
the morphology of ancient books. This article endeavors to analyze the impacts of research. This
study exclusively focuses on the radical vector representation and application of ancient Chinese characters, primarily due to limited resources.

3

Model Architecture

3.1

Embedding

The embedding layer, also known as the input layer,
is a fundamental component in neural network architectures. It is responsible for transforming input
data into enhancing the caliber of vectorized representation of historical Chinese text within the coding layer of the model constitutes a pivotal aspectin advancing the automated processes of sentence
segmentation and word segmentation in ancient
Chinese. In order to utilize natural language as in-put for the neural network model, it is necessary toconvert it into a vector representation. The BERT
model, constructed by Transformer’s bidirectional
encoder, is currently one of the most advanced technologies for language vector representation. Therefore, this research has opted to utilize the BERT
model. The SikuRoBERTa model serves as thefoundational approach for generating vector representations of Chinese characters. SikuRoBERTa isa vector representation model developed by Wang
Dongbo et al. that is specifically designed for ancient Chinese. This model is built upon the BERT
architecture. The training corpus utilized in thisstudy is the renowned Wenyuange "Siku Quanshu"
collection, which consists of approximately 500
million word instances. The word list encompasses
a total of 21,128 characters.
The exclusive reliance on Chinese character vectors is insufficient in fully capturing the interrelationships among Chinese characters. It is imperative to delve into a comprehensive characterizationof the intrinsic information embedded within Chinese characters. Chinese characters are a form of
semantic and phonetic characters, wherein the radicals, components, and even strokes of these characters possess a certain capacity to convey meaning. Hence, in the domain of character-based sequence labeling, the inclusion of semantic information from these entities is frequently employed toenhance the precision of lexical analysis. precision.
Firstly, it is imperative to differentiate between thefour concepts of strokes, parts, component, and radicals of Chinese characters. This article aligns withthe principles outlined in "A General Theory ofModern Chinese" edited by Jingmin Shao (2017).
(1) The stroke represents the fundamental building block of regular script glyphs.
(2) The part refers to a unit of character construction in the Chinese writing system. It is comprised of strokes, can be utilized autonomously, andserves the purpose of constructing Chinese characters. Components can also be considered as units
of word formation that are derived through one ormore segmentations of the complete word.
(3) The component refers to the structural component obtained through a single segmentationof the combined character using the dichotomy
method.
(4) The radical is component or subcomponent
that can combine to create characters in groups.
The characters that share a common component

124are grouped together in the "character set", withthis component being positioned at the forefront as
the leading unit. This arrangement serves as thefoundation for character retrieval.
Using the character "時" as a case study, Table 1
reveals the presence of a shared denotative symbol"日" in the parts, components, and radicals of "時".
This symbol serves as a pictograph, also known
as a meaning, for "時" characters. However, it isimportant to note that there is no direct and exclusive correspondence between the pictographs andradicals found in Chinese characters. In certain
instances, the complete representation of a Chinesecharacter necessitates the inclusion of all its constituent components or radicals. For instance, thepictograph for the character "闖" is denoted by the
combination of "門" and "馬", which collectively
convey the meaning of door. This character "膽",in turn, signifies "肉".
Word building unit
strokes
parts
components
radicalsComposition of "時"丨乛一一一丨一一亅丶
日士寸
日寺
日

Table 1: The strokes, parts, components and radicals of"時"

Figure 1: Transformation from traditional Chinese corpus to radical corpus

tion for Chinese characters, necessitating its utilization in conjunction with word vectors. There exist
two methods for integrating character vectors andradical vectors. The first approach involves concatenating the radical vectors and character vectors
to form extended vectors, which are subsequently
fed into a Bidirectional Long Short-Term Memory(BiLSTM) feature extractor. The second method
entails combining the radical vectors and character
vectors without further elaboration. The vectors
are inputted into two distinct BiLSTM feature extractors, with the exception of the hidden size, thehyperparameters of these two feature extractors remain consistent. In conclusion, it is imperative to
meticulously adjust the hyperparameters in orderto achieve the most optimal radical vector representation model and input methodology.
3.2

This paper establishes a mapping between fonts
and radicals based on a dataset comprising over
70,000 Chinese characters and their corresponding radicals. Subsequently, the ancient Chinese
traditional corpus known as "Siku Quanshu" is converted into radicals using the established mapping
relationship between fonts and radicals. Pleaserefer to Figure 1. This study utilizes the radical corpus and employs the Word2Vec training methodology to train the Radical2Vector model, which represents radical vectors. The Word2Vec algorithm
is widely recognized as a prominent method fortraining word vectors. It effectively maps words orradicals onto a continuous vector space, enabling
the identification and representation of semantic
and morphological similarities among them. By
utilizing the Radical2Vector model that has undergone rigorous training, it is possible to acquire thevectorized representation of individual radicals.
While the radical vector does contain internal
information pertaining to Chinese characters, its
informational capacity is restricted. Consequently,
it cannot serve as a standalone vector representa-

Neural Networks

The neural network layer is connected subsequent
to the Embedding layer. The neural network architecture comprises two distinct layers, BiLSTM
layer and CRF layer.
The BiLSTM is a type of neural network thatincorporates bidirectional long short-term memory
units. The recurrent neural network under consideration possesses the capability to effectively model
sequential data. The BiLSTM model encompasses
both forward and backward directions, enabling thesimultaneous consideration of contextual information. This characteristic renders it highly effective
for tasks involving sequence labeling. By utilizing
BiLSTM, the model is able to acquire a greater
amount of global semantic information.
The CRF model, also known as the conditionalrandom field, is a statistical model used in machinelearning and pattern recognition. The proposed approach is a statistical model designed for sequencelabeling tasks, with the capability to optimize thelabeling results on a global scale. Given that theoutput of the BiLSTM model is a probability matrix, it can be observed that the outcomes at each125

time step are mutually independent. Consequently,
the impact of the preceding label on the current
label cannot be taken into account. To address thisissue, the current innovation opts for CRF modeland integrates it following BiLSTM model. The
CRF is a graph model that can be used to represent
the joint probability distribution of a label sequence
given an observation sequence. It is commonly employed to enforce constraints on the labeling results
produced by BiLSTM model, ensuring that the output labels adhere to the rules of a valid sequence.
Furthermore, the CRF can also be utilized to compute the optimal solution of the BiLSTM output
sequence, thereby enhancing the effectiveness ofsequence labeling.
The Embedding layer incorporates both word
vectors and radical vectors, resulting in the formation of two distinct model structures when theneural network is spliced. These structures are illustrated in Figure 2 and Figure 3. Based on theanalysis of Figure 2 and Figure 3, it is evident
that the two input methods for radical vectors exhibit distinct characteristics. The former approach
involves the concatenation of word vectors and radical vectors within the embedding layer, requiring
the construction of a set of hidden layers using BiLSTM. Conversely, the latter method necessitates
the integration of radical vectors with other components. The word vector and radical vector are
separately fed into two distinct BiLSTM hidden
layers in order to generate two sets of BiLSTM
feature vectors. These LSTM feature vectors aresubsequently concatenated.

4Integrated Labeling Strategy

The tasks of Chinese automatic word segmentationand part-of-speech tagging are typically performed
independently, with the outcome of automatic word
segmentation serving as the foundation for part-ofspeech tagging. Hence, the general approach inChinese lexical analysis involves the sequential
implementation of automatic word segmentation
followed by part-of-speech tagging. The concept
of integrated tagging can be attributed to Shuanhu
Bai (1996), who proposed a combined approachfor word segmentation and part-of-speech tagging
to address the issue of ambiguous domains in contemporary Chinese automatic word segmentation.
However, Shuanhu Bai did not conduct a comprehensive assessment of the practicality of integrated
tagging. Ng (2004) provide a comprehensive anal-

ysis of the viability of integrated tagging in theirscholarly work. The authors conducted a comparative analysis of two strategies for Chinese word
segmentation, namely part-of-speech tagging andintegrated tagging, using the maximum entropy
model. The findings indicate that the integrated
method, which relies on word annotation, demonstrates superior performance. The initial utilizationof the integrated tagging method in the domain ofancient Chinese can be attributed to the research
conducted by Min Shi (2010). The CRF model
was employed to carry out experiments on word
segmentation and part-of-speech tagging for PreQin Chinese. The findings of the study indicatedthat the integrated strategy was effective. In comparison to the two-step strategy, it demonstrates anotable enhancement in the efficacy of word segmentation and part-of-speech tagging. Hence, thisstudy also employs an integrated labeling approach.
To achieve integrated labeling, the output label ofeach word is determined by combining the word’s
position and its corresponding part of speech. The
lexical tagging system for word position information consists of a set of four lexemes: B for begin,
I for middle, E for end, and S for a single word.
The "Basic Collection of Pre-Qin Chinese Parts of
Speech Tags" prescribes the use of part-of-speechtags. For instance, the tag "v" is employed to indicate verbs, while the tag "n" is used to indicate
nouns, among others. The hyphen (-) serves asa connector between the lexeme marker and thepart-of-speech marker. An illustration of this canbe seen in the token B-v, which represents a wordthat initiates a verb.

5

Experiment

5.1

DatasetThe selection of "Zuo Zhuan" as the experimental
corpus is based on the following rationales: The"Zuo Zhuan" holds the distinction of being the inaugural chronicle history book in our nation’s history,
encompassing a comprehensive narrative. Furthermore, it boasts the highest word count among all
pre-Qin literature publications. The extensive bodyof literature, consisting of over 200,000 characters, is well-suited for conducting automated word
segmentation and part-of-speech tagging experiments on ancient Chinese through the applicationof deep learning techniques. Furthermore, the reliability of the electronic corpus of "Zuo Zhuan"utilized in this study is reasonably assured. In ad-

126

Figure 2: The first input method of radical vector

dition to addressing punctuation and collation, theresearch group also examined the matter of variant
texts in relation to Yang Bojun’s (1990) work titled"Spring and Autumn Zuozhuan Zhuan". Furthermore, our research team has conducted artificial
segmentation and tagging of the electronic corpusof "Zuo Zhuan". The aforementioned tagged corpus exhibits a commendable level of quality andis deemed appropriate for utilization as an experimental corpus. In their respective studies, Min
Shi (2010), Chengming Li (2018), and Ning Cheng(2020) employed the "Zuo Zhuan" as the corpus
for conducting automated lexical analysis of ancient Chinese. In order to facilitate a meaningfulcomparison with their experimental findings, it isimperative for this study to employ the identical"Zuo Zhuan" annotated corpus during the experimentation process.
Hence, the partitioning of the "Zuo Zhuan"
dataset in this study aligns with the experimental design of the baseline model. Specifically, theinitial ten volumes of "Zuo Zhuan" serve as thetraining corpus, while the final two volumes are
utilized as the test corpus. Table 2 displays theprecise scale of the experimental set.
DatasetTraining set
Test set

Tokens
194,995
33,298Types
166,141
28,131

Table 2: "Zuo Zhuan" training set and test set size

Among the datasets, the ratio of word case occurrences in the training set to the test set is approximately 5.86, while the ratio of overall word caseoccurrences is approximately 5.91. In general, thetraining set is approximately six times larger than
the test set in terms of size ratio.
This study employs the conventional word tagging technique to accomplish the task of automatedlexical analysis. To do so, we must develop a tag
set that is suitable for both word segmentation andpart-of-speech tagging tasks.
5.2

Equipment and Environment

The model employed in this study was constructedusing the PyTorch 1.7.1 framework, with the programming language of choice being Python 3.8.
Regarding the system configuration, the central processing unit (CPU) employed is the Intel i7-13700F
operating at a clock speed of 2.90GHz. The memory capacity of the system amounts to 64GB, while
the graphics processing unit (GPU) utilized is the
NVIDIA GeForce RTX 4090. Furthermore, thememory size associated with the GPU is 24GB.
This particular system configuration has the capability to guarantee both the efficiency and speed ofmodel training.
5.3

Hyper-parameters

Radical2Vector can be described as a vector representation model that captures the essence of ancient127Figure 3: The second input method of radical vector

Chinese radicals. This model is constructed by ap- size of the vector representation for words or radiplying the Word2Vec training method to a radical
cals in the continuous space. The term "number ofcorpus sourced from "Siku Quanshu," which con- iterations" pertains to the frequency at which thetains over 700 million word examples. During the
corpus is traversed during the training process. In
training process of Word2Vec models, it is common
each iteration, the parameters of the model will beto encounter the need for adjusting four key hyper- updated in order to optimize the vector representaparameters. These hyperparameters include thetion of words or radicals. The term "window size"
choice of training algorithm, which encompasses
pertains to the maximum distance separating theboth Continuous Bag of Words (CBOW) and Skip- context and the present word (or words), thereby
Gram methods, as well as the feature vector dimen- determining the extent of the context. This papersion, the number of iterations, and the window size. combines various hyperparameter selections as outThe CBOW model is a technique that utilizes con- lined in Table 3 and conducts an initial experimenttextual information to predict the current word orfor parameter tuning.
words as part of a training task. On the other hand,This study initially selects the initial splicing
the Skip-Gram model is a method that employs thetechnique of word vector and radical vector, andcurrent word or words to predict the surrounding
proceeds to conduct a comparative experiment on
context as part of a training task. The training tasks. the training algorithm. Initially, the CBOW and
The dimension of the feature vector is a crucial
Skip-Gram models underwent training with vector
parameter in the Word2Vec model as it dictates thedimensions of 128, 256, and 512, respectively. This
128

Hyperparameters
training algorithm
vector dimension
iterations
window size

Value
CBOW/Skip-Gram
128/256/512/768
5/10/15/20/25
3/4/5/6/7/8

Table 3: Hyperparameters for Radical2Vector Model

Model name
CBOW-128d
CBOW-256d
CBOW-512d
Skip-128d
Skip-256d
Skip-512d

Word segmentation
95.73
95.56
95.75
95.58
95.73
95.63

POS tagging
91.54
91.45 Figure 5: The performance of CBOW-512d at different
91.65 iterations when the window size is 5
91.38
91.35
training does not significantly contribute to word
91.35
segmentation; however, it does enhance the efficacy of part-of-speech tagging. The Skip-Gram
Table 4: F1 value (%) of CBOW and Skip-Gram models
approach is not deemed appropriate for the trainwith different radical vector dimensions
ing of radical vectors. To ascertain the potential
enhancement of the model’s performance with a
training was conducted with iteration number 10
larger radical vector dimension, this study employs
and window size 5. The resulting models were la- the Continuous Bag-of-Words (CBOW) approach
beled as CBOW-128d, CBOW-256d, CBOW-512d, to train a model with a vector dimension of 768,
Skip-128d, Skip-256d, and Skip-512d. Next, em- referred to as CBOW-768d. In comparison to theploy the initial radical vector input approach to
CBOW-512d model, the integrated tagging of thiscarry out a comprehensive experiment involving
model exhibits a decrease of 0.1 in the F1 value forword segmentation and part-of-speech tagging onword segmentation and a decrease of 0.17 in the F1
the "Zuo Zhuan" dataset. The empirical findings
value for part-of-speech tagging.
are presented in Table 4.
In general, the selection of the CBOW training
Figures 4 and 5 display the performance of
method and the configuration of a vector dimension
CBOW-512d across various iterations with a win- of 512 are deemed more suitable. This study madedow size of 5, as well as the performance of CBOW- adjustments to the number of iterations and window
512d across different window sizes with a fixed
size of CBOW-512d, based on the given rationale.
number of iterations at 10. These figures aim to
Based on the findings presented in Figure 4 andinvestigate the impact of the number of iterations
Figure 5, this study ultimately determines that theand window size on the model’s influence.
optimal number of iterations for CBOW-512d is
10, while the most effective window size is 5. The
model is referred to as Radical2Vector.
5.4

Figure 4: The performance of CBOW-512d at different
iterations when the window size is 5

The chart illustrates that the CBOW-512d model
demonstrates the most favorable outcome. Additionally, the Word2Vec method’s radical vector

Vector Composition

The preceding data represents the performance ofRadical2Vector in the initial approach of radical
vector input, wherein the word vector and radical vector are combined and fed into a series of
BiLSTM hidden layers to produce LSTM feature
vectors. In this particular instance, there is a slightenhancement observed in the impact of part-ofspeech tagging. This study employs the Radical2Vector methodology to carry out experiments
pertaining to the second input modality. The experimental results of the two input methods arepresented in Table ??. The second input method of

129

the radical vector, Radical2Vector, exhibits minimal improvement efficacy.

6

Evaluation

In this study, the Radical2Vector model was selected as the representation model for ancient Chinese radicals. The radical vector was combinedwith the word vector and fed into the same set of
BiLSTM hidden layers. This approach, referred
to as the first radical vector input method, wasemployed. The incorporation of radical vectors
enhances the efficacy of part-of-speech tagging;
however, its impact remains somewhat constrained.
To assess the impact of the model proposedin this research paper, the evaluation metrics employed include accuracy rate (P), recall rate (R),and harmonic mean (F1). The model presented
in this study is then compared to the outcomesachieved by participating teams in the open test
TestA of the first international ancient chinese word
segmentation and pos tagging bakeoff(Li et al.,2022). The findings are juxtaposed, as illustrated
in Table 6. The training and test sets utilised inthis study align with the evaluation dataset. Furthermore, the training approach and outcome statistics presented in this article adhere to the criteria
outlined for open evaluation. Consequently, themodel’s computational findings can be comparedto the evaluation results to assess its impact onword segmentation and part-of-speech tagging.

7

Discussion

Innovation can improve ancient Chinese word segmentation and part-of-speech labeling. Reasons
include these. Ancient Chinese radicals are relatedwith form and meaning. The word segmentationand part-of-speech tagging model captures ancient
Chinese character structural similarities better byintegrating radical information. The resemblancehelps the model reliably identify and categorize
ancient Chinese characters, improving word segmentation precision. Radicals also relate to ancient
Chinese character semantics. Radical information
helps the model learn radical semantics and apply
them to part-of-speech labeling. Some radicals associate with nouns, whereas others with verbs or
adjectives. Semantic information can improve partof-speech labeling. Ancient Chinese word segmentation and part-of-speech tagging require knowledge of ancient literature and culture. Radicals area vital part of ancient Chinese characters. Innova-

tive information improves the word segmentationand part-of-speech tagging model’s understandingof historical manuscripts’ lexicon and expressions,improving ancient Chinese language processing
computational capabilities.
Lexical analysis is better with integrated tagging.
The integrated labeling technique reduces categorylabels during multi-classification tasks like lexical
analysis. This improves lexical analysis. This work
uses the four-lexeme tag set for automatic word
segmentation and 21 part-of-speech tags for tagging. Integrated tagging reduced the training set of"Zuo Zhuan" to 59 integrated tags. Strategy has 84
category labels. This is because ancient Chinese
auxiliary words (u), quantifiers (q), and concurrent
words (j) were single-character terms. These linguistic elements are only combined with the singleword marker (S), not with beginning (B), medial(I), or final (E) markers. The "Zuo Zhuan" dataset
contains terms without three-character words. Thisapplies to prepositions, adverbs, modal particles,
and onomatopoeia. This method reduces class labels further by adding in-word (I) tagging. Certain
characters vary and limit the part-of-speech scopeof their words on different lexemes. This limits
character consequences. Thus, the integrated tagging technique integrates external knowledge andautomated processing by leveraging the interrelated
and complimentary nature of word segmentationand part-of-speech information. Thus, this study
labels everything.

8

Conclusion

This study employs deep learning techniques toextract the radical information of Chinese characters, thereby achieving the integration of automaticword segmentation and part-of-speech tagging in
ancient texts. This study utilizes a dataset comprising over 70,000 Chinese characters and theircorresponding radicals to establish a correlation between fonts and radicals. Additionally, it employs
the Radical2Vector model to train a radical vector
representation. An experiment was conducted onthe "Zuo Zhuan" dataset to examine the integration of word segmentation and part-of-speech tagging, utilizing the SikuRoBERTa-Radical2VectorBiLSTM-CRF model in conjunction with the original SikuRoBERTa. The model’s automatic word
segmentation achieved an F1 value of 95.75% on
the test set, while the automatic part-of-speech tagging achieved an F1 value of 91.65%. The present

130Input Method
First
SecondTask
Word Segmentation
POS Tagging
Word Segmentation
POS Tagging

P
95.52
91.44
95.38
91.08

R
95.97
91.86
95.85
91.53

F1
95.75
91.65
95.61
91.31

Table 5: The integrated labeling effect of Radical2Vector on the two input methods (%)

Evaluation
FDU
ZNNU
HIT
BLCU
NJUPT
This article

Word Segmentation
PR
F1
95.81 96.88 96.34
95.73 96.84 96.28
92.78 90.18 91.46
91.2 93.49 92.33
91.09 93.41 92.24
90.91 92.4 91.65
90.56 92.29 91.41
78.14 86.31 82.02
95.52 95.97 95.75

POS Tagging
PR
F1
92.05 93.07 92.56
91.88 92.94 92.41
88.97 86.48 87.71
85.41 87.56 86.47
85.27 87.45 86.35
83.55 84.92 84.23
83.13 84.72 83.92
57.35 63.35 60.2
91.44 91.86 91.65

Table 6: Comparison between the model in this paper and the results of the evaluation teams (%)

study introduces an integrated model that utilizes
radicals for word segmentation and part-of-speechtagging in ancient Chinese. This model demonstrates a high level of performance, significantly
enhancing the efficiency and accuracy of tagging
ancient book corpora. Consequently, it facilitates
the digitization process of ancient books and actively contributes to the advancement of researchin this field. The topic of discussion pertains to theconcepts of inheritance and development.

Miao Fang, Yi Jiang, Qi Zhao, and Xin Jiang. 2009. Automatic word segmentation for chinese classics of tea
based on tree-pruning. In 2009 Second International
Symposium on Knowledge Acquisition and Modeling,
volume 1, pages 438–441. IEEE.

9

Liang Huang, Yinan Peng, Huan Wang, and Zhenyu Wu.
2002. Pcfg parsing for restricted classical chinese
texts. In COLING-02: The First SIGHAN Workshop
on Chinese Language Processing.

Acknowledgments

This research was supported by National Language
Commission Project (YB145-41), Key Project ofAncient Books Work (22GJK006) and National
Social Science Foundation of China major project(21&ZD331, 22&ZD262). We are grateful to thereviewers for comments which helped us to improve the paper.

References
Shuanhu Bai. 1996. An integrated method of chinese
word segmentation and part-of-speech automatic tagging. Chinese Information, 2:46–48.
Ning Cheng, Bin Li, Sijia Ge, Xingyue Hao, and Minxuan Feng. 2020. A joint model of automatic sentence
segmentation and lexical analysis for ancient chinese
based on bilstm-crf model. Journal of Chinese Information Processing, 34(4):1–9.

Han He, Lei Wu, Xiaokun Yang, Hua Yan, Zhimin
Gao, Yi Feng, and George Townsend. 2018. Duallong short-term memory networks for sub-character
representation learning. In Information TechnologyNew Generations: 15th International Conference on
Information Technology, pages 421–426. Springer.

Shuiqing Huang, Dongbo Wang, and Lin He. 2015. Exploring of word segmentation for fore-qin literature
based on the domain glossary of sinological index
series. Library and Information Service, 59(11):127.
Bin Li, Yiguo Yuan, Jingya Lu, Minxuan Feng, Chao
Xu, Weiguang Qu, and Dongbo Wang. 2022. The
first international ancient chinese word segmentationand pos tagging bakeoff: Overview of the evahan
2022 evaluation campaign. In Proceedings of theSecond Workshop on Language Technologies for Historical and Ancient Languages, pages 135–140.
Chengming Li. 2018. Research on Lexical Analysis ofAncient Books Based on Deep Learning. Ph.D. thesis,
Nanjing, China: Nanjing Normal University.
Hwee Tou Ng and Jin Kiat Low. 2004. Chinese part-ofspeech tagging: One-at-a-time or all-at-once? wordbased or character-based? In Proceedings of the1312004 Conference on Empirical Methods in Natural
Language Processing, pages 277–284.
Jingmin Shao. 2017. General Theory of Modern Chinese. Shanghai Educational Publishing House.
Min Shi, Bin Li, and Xiaohe Chen. 2010. Crf based
research on a unified approach to word segmentation and pos tagging for pre-qin chinese. Journal of
Chinese Information Processing, 24(2):39–45.
Xinlei Shi, Junjie Zhai, Xudong Yang, Zehua Xie, andChao Liu. 2015. Radical embedding: Delving deeper
to chinese radicals. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2:
Short Papers), pages 594–598.
Xunzhu Tang, Rujie Zhu, Tiezhu Sun, and Shi Wang.
2021. Moto: Enhancing embedding with multiple
joint factors for chinese text classification. In 2020
25th International Conference on Pattern Recognition (ICPR), pages 2882–2888. IEEE.
Hanqing Tao, Shiwei Tong, Tong Xu, Qi Liu, and Enhong Chen. 2019. Chinese embedding via stroke
and glyph information: A dual-channel view. arXiv
preprint arXiv:1906.04287.
Xiaoyu Wang and Bin Li. 2017. Automatically segmenting middle ancient chinese words with crfs. Data
Analysis and Knowledge Discovery, 1(5):62–70.
Runhua Xu and Xiaohe Chen. 2012. A methodof segmentation on zuo zhuan by using commentaries. Journal of Chinese Information Processing,26(2):13r17.
Bojun Yang. 1990. Annotations to the Spring and Autumn Annals. Zhonghua Book Company.
Jinxing Yu, Xun Jian, Hao Xin, and Yangqiu Song. 2017.
Joint embeddings of chinese words, characters, andfine-grained subcharacter components. In Proceedings of the 2017 conference on empirical methods innatural language processing, pages 286–291.
Yun Zhang, Yongguo Liu, Jiajing Zhu, and Xindong Wu.
2021. Fsprm: A feature subsequence based probability representation model for chinese word embedding.
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1702–1716.

132

Introducing an Open Source Library for Sumerian Text Analysis
Hansel Guzman-Soto and Yudong Liu
Computer Science Department
Western Washington University
Bellingham, Washington 98225{guzmanh,liuy2}@wwu.edu

Abstract
The study of Sumerian texts often requires domain experts to examine a vast number of tables. However, the absence of user-friendly
tools for this process poses challenges and consumes significant time. In addressing this issue, we introduce an open-source library thatempowers domain experts with minimal technical expertise to automate manual and repetitive
tasks using a no-code dashboard. Our library
includes an information extraction module thatenables the automatic extraction of names andrelations based on the user-defined lists of nametags and relation types. By utilizing the tool
to facilitate the creation of knowledge graphs,
which is a data representation method offering
insights into the relationships among entities inthe data, we demonstrate its practical application in the analysis of Sumerian texts.

1

Introduction

but it also is not scalable. Additionally, given that
Sumerian is a low-resource language, the availability of dedicated software tools is scarce, limiting
scholars’ access to user-friendly NLP (natural language processing) toolkits.
To address these issues, we introduce an opensource library that facilitates the seamless integration of processing and NLP models, thereby enabling more comprehensive and expedited analysis of Sumerian texts. The library consists of two
key components: a pipeline and a dashboard. Currently the pipeline provides functionalities for dataprocessing and information extraction, equipping
users with the necessary tools to build robust andefficient software solutions. The dashboard offers
a user-friendly interface which requires minimal
technical preparing for domain experts to automate
their workflow in analyzing Sumerian tablets, ultimately accelerating their research progress.

2

Related Work

The study of Sumerian texts offers a valuable opportunity to gain insights into the earliest written
There are existing tools that process or perform
languages and its associated historical context. As- NLP tasks tailored for specific tasks such as Masyriologists have conducted studies such as proso- chine Translation (Pagé-Perron et al., 2017; Punia
pography (Jacobs, 2007; Dahl, 2007; Liu, 2021) et al., 2020) and Sumerian text annotation (Tablan
and social network analyses (Kulikov et al., 2021; et al., 2006; Smith, 2010; Liu et al., 2015; Luo et al.,Pottorf, 2022) on these texts, enabling a deeper un- 2015; Chiarcos et al., 2018). Most notably, thederstanding of administrative and economic history
Cuneiform Digital Library Initiative (CDLI) hosts
as well as the involved families and individuals dur- several repositoriesthat process Sumerian in variing the Ur III period (ca. 2112-2004 BC). However, ous data formats such as CoNLL-U and RDF (Rethis type of studies often necessitates the identi- source Description Framework), and perform varification of named entities and their relationships
ous NLP tasks. Although these tools may provide
within a specific timeframe, demanding domain
versatility for different tasks, they require adequate
experts to meticulously examine a vast number oftechnical knowledge for modifying their utilization.
ancient Sumerian tablets. This process can be time- Without such expertise, modifying these resources
consuming and challenging.
to accommodate the diverse requirements of AsCurrently, non-technical users primarily depend
syriology can be daunting. Therefore, the needon SQL and Excel to perform repetitive tasks suchfor a more accessible platform becomes apparent,
as manually locating and recording instances of in- underscoring the importance of our work in this
dividuals and their relationships across tablets. Not
space. To the best of our knowledge, no existing
only does this result in a less intuitive interface, dashboard currently allows scholars to easily uti133
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 133–137,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_016

Figure 1: Architecture of the system.

lize tools or scripts specifically designed for the
analysis of Sumerian tablets.

3

System Description

3.1

System Architecture

Fig. 1 illustrates our system’s structure. It features
a dashboard interface for users to upload their own
data such as tablets or a customized list of namedentity tags used by the backend pipeline. The backend pipeline handles user requests, such as data
annotation, entity extraction or relation extraction,using specific library components which can be
configured by the users on the fly. Additionally,users can create knowledge graphs, stored in a
Neo4j database, leveraging the system’s entity andrelation extraction capabilities.
The library, accessible via this link1 is designed
to enable researchers to seamlessly integrate theirworkflow into our pipeline for their specific use
cases. While the implementations are still relatively
preliminary, the modular nature of the components
involved ensures their adaptability for a wide range
of applications. In the following sections, we will
provide detailed descriptions of each component
we have developed.
3.2

CDLI Extractor

The entry point to the pipeline is the CDLI Extractor, comprising three components: ATF (a1text markup format used by CDLI to describe
inscriptions on Cuneiform tablets and other artifacts) (Robson, 2014) Parser, Transliteration
Cleaner, and ATF Normalizer. This component
is built to load and process tablets from the CDLI
repository, which are written in ATF.
ATF Parser reads tablets in ATF format andstores it into an internal data structure that preserves all metadata, tablet content and positional
information. For now, we support data from CDLI
which has tablets in ATF format. Other formats exist such as Open Richly Annotated Cuneiform Corpus (ORACC) (Robson, 2014) and the Database of
Neo-Sumerian Texts (BDTNS) (Molina, 2002) for
which we plan to offer support.
Transliteration Cleaner then handles broken
tablets and normalizes transliterations to follow a
specific format. For example, for the transliterationof “1(disz)”, we may want this to map simply to“1” because the meaning is intact but it is easier for
us to process.
ATF Normalizer aims to establish a standardized format enabling the uniform processing of datafrom diverse sources, including CDLI, ORAAC,
and BTDNS. Currently, this component normalizes
CDLI data to a unified format, with plans to extend
its functionality to standardize data formats fromother sources.
Fig. 2 shows a working example of this module.

https://github.com/WWU-Sumerian-NLP

134

Figure 2: A working illustration of the CDLI Extractor. The ATF parser takes a CDLI tablet and parses and storesit into an internal data structure. The Translit Cleaner then performs cleaning on numeric symbols, damaged
annotations, and annotator’s correction or guesses markers. Finally, the Normalizer standardizes transliterations
from various tablet sources.

3.3

Information Extractor

As named entities and entity relationships are often the key information for Sumerian text analysis,
our Information Extractor module currently contains two modules: Entity Extractor and Relation
Extractor.
Entity Extraction We use a simple approachbased on string matching. In this process, eachword is examined, and target words are labelledwith entity types drawn from a list of known entities (Bansal et al., 2021). As aforementioned, thepipeline is designed to be flexible, allowing users
to input a customized list of named entity labels tobe processed. While simple, this approach effectively automates the manual annotation work andestablishes a centralized platform to leverage theannotated data for downstream tasks. Future iterations will port existing Named Entity Recognition
models to our library and provide them as option
to users.
Relation Extraction It involves finding connections between entities. The process involves the application of user pre-defined rules for relations using regular expressions. The pipeline allows users
to define and pass a list of regular expressions forthe system to search through.
3.4

Dashboard for Non-technical Users

application takes these patterns, iterate through alldata, and display the results to the user. 3) Configure different components within the pipeline. For
example, users could configure ATF parser to filterby tablet metadata such providence or by broken
tablets. 4) Search or filter for the results of eachcomponents output. 5) Download data or use therelationship tab to feed relations to a knowledge
graph stored in Neo4j.
Server Architecture We have a server in placethat acts as an intermediary between our dashboard
and NLP libraries. Our server’s backend imports
our NLP libraries to use for each task and stores
data in a relational database to maintain the stateof data across multiple services. For server implementation, we use the Mux library in Go. The
dashboard is designed for easy extension. To support a new tab for a service, users only need tocreate a new form in the frontend, add an entry in
our server’s database, and create a corresponding
endpoint in our backend that uses the service. We
are also developing features that will allow users toaccess their own Sumerian tablets for a variety ofdownstream tasks.

4

Evaluation and Implementation
Considerations

We aim to create reproducible, replicable tools that
A No-Code Dashboard To facilitate the use of ourcan be easily customized and interchanged withinlibrary, we have developed a user-friendly dash- the Assyriologist community. This is reflected inboard that enables users to view, modify, and up- our pipeline architecture which will allow for theload their data (see Fig. 3). It currently supports theseamless integration of various components, enfollowing features: 1) Upload, add or delete entity
abling users to modify and adapt the tools accordnames with their corresponding entity tag. 2) De- ing to their specific needs. This modularity not only
fine relation types with specific pattern rules. Our
promotes the reuse and repurposing of individual135Figure 3: Dashboard interface designed for the streamlined data upload and knowledge graph generation throughinteractive widgets: “Entities”, “Relation_Types”, and “Relationships”. With the “Entities” widget, users can input
entity lists with tags, triggering entity extraction across their dataset. Extracted entities are then cataloged in a
database and displayed in a corresponding table.

components but also encourages collaboration andknowledge sharing within the Assyriologist community.
Our decision to utilize the Go programming language, is primarily motivated by its speed. It offers
up to a 30-fold speed increase, resulting a highly responsive user dashboard. For example, tasks suchas entity extraction, which could require 5-20 minutes in Python, now demand only 1-5 seconds in
Go. As we continue to introduce more customization options, algorithms and features, maintaining
this speed becomes essentials for a good user experience. Furthermore, this efficiency extends toserver interactions, ensuring swift communication
between the frontend and backend.

imals, and locations. Connections between nodes
depict relationships or actions, and each connection
is enriched with tablet metadata, including tablet
number, year, and region. For instance, the node
‘ARAD2-mu’ (a person) is connected to the node‘sila4’ (lambs) with an edge labeled ‘delivers’. The
graph not only illustrates the volume of deliveries,
recipients, and geographic routes but also provides
a comprehensive view of individual interactions
over time and space. It gives insights into the networks of individuals and the broader prosopological landscape, shedding light on societal structures,
relationships, and economic dynamics.

6

Conclusion and Future Work

This paper introduces an open-source library designed to empower domain experts in processing
and analyzing Sumerian cuneiform tablets throughKnowledge graphs serve as a powerful tool for rep- an no-code dashboard. The application of knowlresenting data as a network of interrelated entities, edge graphs enhances the analysis via large-scale
enabling us to answer queries such as “who did
entities and relation visualization. The current imwhat to whom”. For illustrative purposes, we draw
plementation shows an initial but promising step inupon the work (Liu, 2021) to demonstrate the use
accommodating configurable components that areof knowledge graphs in studying prosopographyagnostic to various NLP tasks. As the pipeline’sof a family engaging in an animal delivery busi- capabilities expand, we invite collaborations toness during the Ur III period. In this knowledge
broaden its applications, potentially encompassing
graph, nodes represent entities such as people, an- a wider range of ancient Mesopotamian languages.
136

5

Use Case: Creating Knowledge Graphswith Our Tools

References

Proceedings of the 28th International Conference on
Computational Linguistics, pages 3454–3460.

Rachit Bansal, Himanshu Choudhary, Ravneet Punia,
Niko Schenk, Jacob L Dahl, and Émilie Pagé-Perron.
2021. How low is too low? a computational
perspective on extremely low-resource languages.
arXiv:2105.14515.

Steve Tinney Eleanor Robson. 2014. Oracc: The open
richly annotated cuneiform corpus.

Christian Chiarcos, Ilya Khait, Émilie Pagé-Perron,
Niko Schenk, Jayanth, Christian Fäth, Julius Steuer,
William Mcgrath, and Jinyan Wang. 2018. Annotating a low-resource language with llod technology: Sumerian morphology and syntax. Information,
9(11):290.

Valentin Tablan, Wim Peters, Diana Maynard, Hamish
Cunningham, and K Bontcheva. 2006. Creating tools
for morphological analysis of sumerian. In LREC,
pages 1762–1765.

Eric JM Smith. 2010. Query-Based Annotation and theSumerian Verbal Prefixes. University of Toronto.

J.L. Dahl. 2007. The Ruling Family of Ur III Umma:
A Prosopographical Analysis of an Elite Family inSouthern Iraq 4000 Years Ago. Peeters Publishers
Booksellers.
Dennis Jacobs. 2007. The secret life of judges. 75
Fordham L. Rev. 2855.
Anya Kulikov, Adam Anderson, and Niek Veldhuis. 2021.
Sumerian Networks: Classifying
Text Groups in the Drehem Archives. IDEAH.
Https://ideah.pubpub.org/pub/q22859lx.
Changyu Liu. 2021. Prosopography of individuals delivering animals to Puzriš-Dagan in Ur III
Mesopotamia,” Akkadica 142/2, 2021, pp. 113-142.
Akkadica, 2021(24.0):112–142.
Yudong Liu, Clinton Burkhart, James Hearne, and Liang
Luo. 2015. Enhancing sumerian lemmatization by
unsupervised named-entity recognition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
1446–1451.
Liang Luo, Yudong Liu, James Hearne, and Clinton
Burkhart. 2015. Unsupervised sumerian personalname recognition. In The Twenty-Eighth International Flairs Conference.
Manuel Molina. 2002.
sumerian texts.

Bdtns: Database of neo-

Émilie Pagé-Perron, Maria Sukhareva, Ilya Khait, andChristian Chiarcos. 2017. Machine translation andautomated analysis of the Sumerian language. In
Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social
Sciences, Humanities and Literature, pages 10–16,
Vancouver, Canada. Association for Computational
Linguistics.
Andrew Pottorf. 2022. Social Stratification in Southern
Mesopotamia during the Third Dynasty of Ur (ca.
2100–2000 BCE). Ph.D. thesis, Harvard University
Graduate School of Arts and Sciences.
Ravneet Punia, Niko Schenk, Christian Chiarcos, andÉmilie Pagé-Perron. 2020. Towards the first machinetranslation system for sumerian transliterations. In

137

Coding Design of Oracle Bone Inscriptions Input Method
Based on “ZhongHuaZiKu” Database
Dongxin Hu
College of Liberal Arts , Capital Normal University , 10089 Beijing , China
602201042@qq.com

Abstract
Based on the oracle bone glyph data in the“ZhongHuaZiKu”database, this paper designs a
new input method coding scheme which is easy tosearch in the database, and provides a feasible
scheme for the design of oracle bone glyph input
method software in the future. The coding scheme
in this paper is based on the experience of the pastoracle bone inscriptions input method design. In
view of the particularity of oracle bone inscriptions,the difference factors such as component
combination, phonetic code and shape code ( letter )are added, and the coding format is designed asfollows : The single component characters in theidentified characters are arranged according to theformat of “ structural code + pronunciation fullspelling code + tone code ” ; the multi-component
characters in the identified characters are arranged
according to the format of “ structure code + split
component pronunciation full spelling code +overall glyph pronunciation full spelling code”;
unidentified characters are arranged according to
the format of “ y + identified component
pronunciation full spelling + unidentified
component shape code ( letter ) ”.Among them,
the identified component code and the unidentified
component shape code are input in turn according
to the specific glyph from left to right, from top tobottom, and from outside to inside. Encoding
through these coding formats, the heavy code rate
is low, and the input habits of most people are also
taken into account.

1

Previous design and inspiration oforacle bone inscriptions input method

In the past, some scholars designed the input
method of oracle bone inscriptions from theperspective of shape code in coding. For example,
Mr.Xu Song of Central China Normal University
developed a method in 1995, which applied 26
English letters and 9 Arabic numerals tocorrespond to more than 500 characters in oraclebone inscriptions, and realized the input of oraclebone inscriptions by keyboard input characters. By
2012, researchers such as Li Qingsheng of Anyang
Normal University jointly developed an input
method of oracle bone inscriptions based on the
dynamic description library of oracle bone
inscriptions. On the basis of the coding and writing
specifications of modern Chinese characters, theinput side uses the dynamic description method todescribe the oracle bone inscriptions with directed
strokes and strokes, and combines the extended
coding area with the external description character
library. It is more effective to solve the input
problem of variant characters and unliteracy inoracle bones.
Some scholars have developed image method,visual input method and handwritten input methodfrom the perspective of non-coding to solve theinput problem of oracle bone inscriptions. In 1990,
Zhou Demin et al.of Henan University firstdeveloped the Calculator Oracle Information
Processing System ( CJPS ), which laid an
important foundation for the subsequent researchand development of related input methods. In 2004,
Mr. Liu Yongge and Li Qingsheng of Anyang
Normal University developed a visual oracle bone
inscriptions input method. The principle of the
input method is to provide the input person with atable of oracle bone inscriptions. The input person
selects the corresponding radicals contained
according to the oracle bone inscriptions that hewants to input. The program presents the results
containing these radicals to the input person in theform of candidates. The input person clicks on theglyph he wants to input to complete the input. After
that, in 2020, Mr.Liu Yongge, Mr.Li Qiang andothers from the Key Laboratory of Oracle Bone
Inscription Information Processing of Anyang
Normal University jointly developed a new oraclebone script handwriting input method. Based on
the latest research results of artificial intelligence

138
1
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 138–147,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_017

deep learning and convolutional neural network,
the oracle bone script recognition network andrecognition module were developed. The methodof using this input method is to operate the mouseto write the oracle bone script that you want to
input to the virtual handwriting board to complete
the recognition of oracle bone script, and thengenerate the glyph candidate, and then click the
candidate glyph to complete the input of oraclebone script.
From the above content, there are two main
problems in the design of oracle bone inscriptions
input method in the past. On the one hand,
computer professionals only design the input
method of oracle bone inscriptions from the two
directions of shape code and non-coding, and do
not use phonetic code to participate in coding. The
reason is that phonetic code can not encode theunidentified characters in oracle bone inscriptions.1
Whether it is from the perspective of speech or
keyboard input, most of us are used to associatingphonetic symbols with text . Considering thatpeople whose mother tongue is Chinese or people
whose mother tongue is not Chinese, the first thing
they learn when learning Chinese is the Chinese
pinyin scheme, we think that setting phonetic codesin the input method coding is very
convenient. .From the perspective of ancient
Chinese characters ， since oracle bone
inscriptions are already identified , they must havea clear pronunciation. The commonly usedcharacters in oracle bone inscriptions are basically
identified glyphs, it is precisely these identified
glyphs that we often use when inputting characters .
According to these , we should not abandon thephonetic code when designing input method
encoding .
On the other hand, the shortcomings of using
only shape code to encode are generally notconvenient for input learners to learn and use.
Some are used to input the roots to retrieve
alternative characters. This method is similar to thefive-stroke input method to split today’s regular
script characters, but its drawbacks are reflected in
the fact that the keyboard is as inconvenient forusers to master as the five-stroke input method. It
is not in line with the character theory for some1

Although “Yin Qi Wen Yuan” Data Platform(http://jgw.aynu.edu.cn) has provided this input method, it
does not provide input method software that can be used awayfrom the website, so its coding principle is not clear.

oracle bone inscriptions, and it is not convenient todistinguish the large number of variant charactersin oracle bone inscriptions. Some search foralternative glyphs according to the method of
stroke input （ Nie Yanzhao and Liu Yongge .
2010.）, but most of the modern so-called strokes
are applicable to the glyph decomposition of Li andKai characters, while many of the more pictorial
characters in oracle bone inscriptions cannot be
described by the concept of strokes.For example,
the relatively representative glyphs of oracle bone
inscriptions 2 , 3 , etc., strokes cannot truthfully
describe the shape at the top of the glyph ; the 车characters of oracle bone inscriptions are
5

4

and

. This special glyph of the record segment
and the nuances between the glyphs cannot be
combined and split simply by strokes. Some
combine the similar four-corner number retrieval
method with the configuration codes such as closed
curve stroke and its extension line structure , crossstroke structure , discrete stroke structure, etc（LiuYongge and Li Qiang . 2020.）. The “Oracle Bone
Inscription Six-digit Code Search Font Library”
is based on these three aspects as the basis forcoding, but this search font library does not containas many glyphs as ours.The most difficult thing forusers is to learn this coding rule, which does notmeet our requirements in simplicity and efficiency.
Some use the method of dynamic description,
based on the coding and writing norms of modern
Chinese characters, using concepts and techniquessuch as directed strokes and pen elements todescribe oracle bone inscriptions.（Li Qingsheng,
Wu Qinxia, Wang Lei . 2012.）. The premise ofthis method is that the input must have a deep
understanding of oracle bone glyphs, and thewriting norms of modern Chinese characters are akind of rules with strong regularity and serious
symbolization, which is not very suitable for oraclebone glyphs with strong realism.
Since oracle bone inscriptions have a highdegree of pictography, from the professional pointof ancient Chinese characters’ view, we hope toprovide the academia with a coding scheme that2

Jia Gu Wen He Ji 6816
Jia Gu Wen He Ji 27888
4
Jia Gu Wen He Ji 584 front side5
Jia Gu Wen He Ji 10405 front side
3

139
2

conforms to the professional cognition of ancient
Chinese characters. We urgently need a set of input
method coding scheme that can be easily acceptedby professionals to the greatest extent, faithful tothe correct description of oracle bone inscriptions
as much as possible, and convenient for users tolearn and use.

2

A new design scheme of oracle bone
inscriptions input method
2.1 Technical route of oracle bone
inscriptions input method design
The Oracle Bone Inscriptions Input method isdesigned using Microsoft 's Text Service
Framework ( TSF : Text Service Framework ). It isa COM-based input method framework that does
not depend on specific input devices and cansupport multiple languages. It provides a simple
and scalable technology for implementing text
input and natural language processing technology.
The text service framework includes three main
components : application, TSF manager and text
service. The architecture is shown in the following
figure.

Picture 1 ：TSF architecture“Application” refers to the application softwarethat supports and has adopted TSF, such as
Microsoft 's MS Office, Notepad and other word
processing programs. The application accesses
text by implementing a COM server that supports
a specific interface, and communicates with TSF
by using an interface exposed by the TSF manager.
Applications that support TSF do not need toconsider the specific details of the input method,and can receive text input from the “text service”
to achieve a series of operations such as displaying,editing, and storing text.

“Text service” refers to the text input processor,
which can be keyboard input, handwritten
recognition input or speech recognition input andother input programs. After registering with TSF,
users can use language bar or keyboard shortcuts tointeract with the text service. The text service canobtain text from the application or write text to theapplication. Text services can also associate data
and attributes with text blocks. The oracle bone
inscriptions input method implemented in thispaper is a text service that inputs oracle bone
inscriptions characters through the keyboard.
“TSF Manager” is an intermediary between an
application and one or more text services,
implemented by the operating system to enableapplications and text services to share text. The text
service does not interact directly with theapplication, and all communications are performed
through the TSF manager.
Oracle Bone Inscriptions Input method
implements the basic elements of TSF, such as
Thread Manager, Client Identifiers, Document
Manager, Edit Context, Ranges, Compartment,
Properties and Composition.
The “Thread Manager” is responsible forcompleting the task of connecting the applicationand the text service. These tasks include activatingor suspending the TSF text service, creating the
document manager, and maintaining the correct
association between the document and the input
focus.
The “client identifier ” is an identifier assignedby the thread manager that is received and must be
maintained by clients such as applications and text
services. The client needs to provide its own
identifier when calling various TSF methods.
The continuous text stream created by the “editcontext” through the interface can be created by
the application and provided to the text service. In
some cases, the text service can also create an edit
context as needed.
The “document manager” is responsible formaintaining the last-in-first-out buffer, and thecontent stack stores the list of edited content
managed by the document manager.
An “input combination” is a temporary input
state that enables the text service to keep theapplication and user input text in a state of constantchange. The application can obtain the display
attribute information of the input combination anduse this information to display the input
combination state to the user. The application
140
3

determines how to display the text and what kindof operation to the text according to whether there
is an input combination.

2.2

Coding scheme designed by oraclebone inscriptions input method

Professor Huang Tianshu of Tsinghua University
presided over the press and publication of major
scientific and technological projects of the Chinesecharacters project “中 华 字 库- Oracle bone
inscriptions collection and collation” ( 06101041BJNF2328-03 ), its network platform has
been sorted out 12685 Oracle bone inscriptions, thenumber of this glyph is the past Oracle Bone
Inscriptions Input method can not be compared. We
divide the 12685 oracle glyphs into two categories :
literate and unliterate, and encode them separately.
Among them, literate is divided into two
categories : single component and multicomponent. Unliteracy is divided into three
categories : components that are identified but not
identified as a whole, some components can be
identified but the rest are not identified, andcomponents are not identified at all.

2.2.1 Coding scheme of identified glyphs
2.2.1.1 Coding scheme of single component
characterOn the basis of “ natural classification ” ,Mr.Huang Tianshu summarized and sorted out theradicals of oracle bone inscriptions into four
categories :“象物” “象人” “象工” and “other”（Huang Tianshu. 2020.）. The “象物” refers to
all non-living and living things in nature, “象人”
refers to the shape of people and their five senses
and limbs, “象工” refers to the products of human
wisdom，“other” refers to parts that cannot beclassified .For the input method itself, a constraint
condition is added to the limited coding position,
which can greatly reduce the repetition rate.
Therefore, we roughly divide the structure of
single-component characters into four categories :象物, 象人, 象工, and others. According to the fullspelling of the first letter of the character “物”, andat the same time, in order to distinguish it from “合文” in the following text, “v” is used to refer to “象物” . According to this setting method, the otherthree types of codes can be set as follows: “象人”corresponds to“r”,“象工”corresponds to“g”,
and “other” corresponds to“t” （ “other” inChinese is “其他”， based on the full spelling of
the first letter of the character “他” is “t”）. It may

be the first time in the history of input method
development to classify and encode single
component characters by natural classification.
We set this type of coding in the first place of the
input order. Because the identified characters inoracle bone inscriptions generally havecorresponding interpretation opinions, there will becorresponding pronunciations of regular script
characters in later generations. This pronunciation
also belongs to the important coding attribute ofsingle component characters in oracle bone
inscriptions, so we set the corresponding Chinese
pinyin spelling in the second place of the input
order. From the perspective of reducing therepetition rate, under the constraints of the first twocodes, sometimes there may be situations where
the accuracy of the alternative characters is notenough. For example, under the premise that the
expected input phonetic code“you”is added, theinput situation is divided into vyou , ryou , gyou ,tyou. The corresponding vyou codes are“柚”“ሇ”etc., and the corresponding ryou codes are“又”“尤”etc. Corresponding to gyou codes,
there are“酉”“卣”etc., corresponding to tyou
codes, there are“㞢”“由”“穬”etc., and it
can be seen that the repetition rate is already very
low, but there are many variant characters of thesame character in oracle bone inscriptions. After
entering the coding, the number of options
becomes the number of variant characters of one
character plus the number of all variant charactersof another or two characters. The number is still alot. In the input method software or patents thathave appeared, there is no design for encoding
tones. The tone symbols and corresponding codes
we designed are listed below :Intonation
CodingHigh-level toney(first tone)
rising tonep(second tone)
falling-rising tones(third tone)
falling toneq(fourth tone)
Table 1 ：Tones table

If we add the attribute difference of tone on thisbasis. Coding becomes vyouy, vyoup, cvyous,
cvyouq, ryouy, cryoup, ryous, ryouq, gyouy, gyoup,
gyous, gyouq, tyouy, tyoup, tyous, tyouq and so on.
In this way, in the previous coding, except that thecharacters“酉”and“卣”under the“gyou”
141
4

code are not distinguished, the characters under theother three codes can be distinguished. Therefore,
the coding format of “ structure code +pronunciation full spelling code + tone code”iscompletely feasible.
In the following, we show some practicalexamples of single component character coding.
Intermediate code
OracleInputof “中华字库”
glyphscoding
0E9C7Bgzheny╴trenp⩃
0E9C86
vyueq⩄
0E9C88tdingy0E9C79⨾gzus
0E9C75⡂
ଭ
0E79E0
vyangpଣ
0E79D1rhuangp
0E86DAჄvzhiq
჊
0E86E9vlaip
Ⴠ
0E86D6rruoq
Table 2 ：Single component characters coding table

2.2.1.2 Coding scheme of multi-component
characters
The classification of multi-component characters isthe most detailed. We divide the structure of multicomponent characters in oracle bone glyphs into 14
categories，and according to the general shape ofthe structure, it is coded with English letters withsimilar shapes : the left and right structure
correspond to the position of “ ⿰ ” , and thecorresponding code is “h” ; the corresponding
position of the upper and lower structure is “⿱”,and the corresponding code is “ z ” ; thecorresponding position of the full inclusion
structure is “⿴” , and the corresponding code is“ o ” ; the corresponding position of the upper
three-inclusion structure is “ ⿵ ” , and thecorresponding code is “ n ”; the corresponding
position of the lower three-inclusion structure is“ ⿶ ”, and the corresponding code is “ u ” ; thecorresponding position of the left three-inclusion
structure is “⿷” , and the corresponding code is“c ” ; the right three-inclusion structure
corresponds to the position of “ ” , and thecorresponding code is “ b ” ; the corresponding
position of the upper left contains structure is “⿸”,and the corresponding code is “ p ” ; thecorresponding position of the upper right contains
structure is “ ⿹ ”, and the corresponding code is“ q ” ; the corresponding position of the lower left
inclusion structureis “⿺”, and the corresponding code is “ l ” ; thecorresponding position of the lower right inclusion
structure is “ ” , and the corresponding code is“ j ” ; the corresponding position of the covering
structure is “ ⿻ ”, and the corresponding code is“ f ” ; The corresponding position of the upper,
middle and lower structure is “ ⿳ ”, and thecorresponding code is “ e ” ; the corresponding
position of the left, middle and right structure is“⿲”, and the corresponding code is “ m ”. The
above-mentioned glyph structure and thecorresponding coded letters are set up on the basisof the principle that the structural form of the firstlevel component is as close as possible to theletters .
We believe that the full-spelling syllable coding,which conforms to the typing and recognition
habits of modern Chinese people, is very suitablefor encoding multi-component characters, but it isalso a problem to match the full-spelling codingwith what coding. Tone coding is very suitable for
distinguishing single-component characters. If
multi-component character coding is designed touse tone coding on the basis of full pinyin syllables,
the distinguishing ability of this coding will begreatly reduced. For example, in oracle bone
inscriptions, the multi-component characters withpronunciation of fú are supported, 扶， and 穡
etc., the three-character components arecompletely different but cannot be distinguished.
In addition, some of the components in the multicomponent characters have been deformed andvoiced during the evolution of the glyph. Thechanges in this component can not be distinguishedby the pronunciation of the characters. For example,
兴 characters are generally from the same ⋯deformed into ߱ ; ≻left component is deformed
into 㐁（簟）ዅ, and the overall evolution is ᅿ.
The above two aspects of coding problems,“fullspelling + tone”method is not able to solve.
Therefore, we add two coding items, the wholeglyph pronunciation and the disassemblycomponent pronunciation, to the structural coding,that is, the coding format of“structural code +disassembly component pronunciation fullspelling + whole glyph pronunciation fullspelling”. This three-stage coding design scheme
for multi-component characters of Chinesecharacters may be original.
142
5

In the following, we show some practicalexamples of multi-component character coding.
Intermediate
Oraclecode of
Input coding
glyphs“中华字库”
zrourouduo⩁
0E9C83zzhiwangwang
ଥ
0E79D3zshengmuxing
0E79DC‫ܩ‬
nmianwanbin‫ܦ‬
0E79D9hyiliyi჈
0E86E4
0E9C42⨬
nmianshizongዩ
0E860E
mchiwujieyu
Ÿ
0E7342
ezhiweizhiwei
0E95E6ᮐozhubeigu⤹
0E9965uzhikanchu
ᚱ0E8765pyanziyou
ᬪ
0E94C5fmeigemie
0E86AFႭ
fjiannvyan‫ޚ‬
0E7BD2
broushitun
৻
0E75F0oweichuangzang
Table 3 ：Multi-component characters coding table

However, there are some of the multicomponent characters that can be analyzed forstructure and components, but they cannot identify
the overall pronunciation of the characters. For theconvenience of coding, we classify all thesecharacters into the category of “ unidentified
characters”during coding, such as ჆ can beanalyzed as 犬 and 大,ᇢ can be analyzed as 殳 and米,ު can be analyzed as 目 and 口 , but these are
not the exact overall pronunciation, we for theconvenience of coding, this kind of multicomponent characters into the category of literacy.
There are a large number of“合文”in oraclebone inscriptions. This kind of glyph refers to thephenomenon of combining several original
independent glyphs into one glyph. For“合文”,
although the identity of each part of the new glyph
is a character rather than a component, thecombination of “ 合 文 ” is actually similar to“ multi-component character ” in terms ofstructure. In order to take into account theindependent and common characteristics of“合文”, we regard “合文” as an input method
structure category that can be independently
classified, and the subsequent coding writes thepronunciation of each part according to the reading
order of “ 合 文 ” , and the coding format isroughly“w + split component full spelling”,

The coding order of each character in“合文”isarranged according to the order of reading.
When encoding the “合文”, we need to pay
attention to the following aspects : somecombinations of “合文” are connected or evenhave overlapping parts, such as 大 and 丁 isᱞ, 柚and 京 is⤆, 上 and 甲 is⦹, 三 and 牛 isᑄ, and soon . Some combination methods are similar to the“借笔” in the combination characters, such as 大and 甲 isཫ, 五 and 璧 is⦆, 五 and 牢 is༽, 妣 and丙 isᓍand so on. Although some two characters
are separated, the “character spacing” is slightly
closer than the normal character spacing, such as
大 and 庚 is␶, 母 and 癸 is␸, 祖 and 己 is➝ andso on , this is also the most common “合文” . In
some combination forms, one of the components is
separated, and even the separated components arequite close to the other character. This kind of
combination is not easy to identify, such as 武 and乙 is⼣, 三 and 牡 isѷ, 龐 and 母 is⎴, 武 and 丁is⚷and so on. Although some of the combined
texts are separated from each other, one of thecharacters is simple, such as 多 and 子 is ⸡. In
addition, in order to facilitate the explanation of therules of the input method, we also incorporate thesituation of “ 重 文 ” into the category ofcompound characters. At present, only one
phenomenon of“重文”is found in oracle bone
inscriptions, that is, 有 and 佑 duplicate charactersisṗ.
In the following, we show some practicalexamples of“合文”coding.
Intermediate code
OracleInputof “中华字库”
glyphscodingwzudingᅓ
0E8848wshangjia
0E8B2E῍wshiyiyue
0E7D0Eఉ
wbaoyi⦋
0E9A2Ewwubi
⦆
0E9A20wxiaogao
0E7D90Ꮊ
wyoujing
パ
0E8F1F
wfuding⹵
0E9A82
wxiaolao
0E977E⣠
wyouyou᷼
0E94E3wsanniu
ᑄ
0E8128wliuyueᵐ
0E91FB
wwushi
0E79C1Ѵ
wduoziᇊ
0E7DCD
Table 4 ：“合文”characters coding table1436

2.2.2 Unidentified glyph coding scheme
2.2.2.1 All components are identified but
the whole character does not
identified

0E75E6ycaijie
0E9D74yhzhui
0E8858yyux

0E79CByyangmumin
0E9FF4/0E9118
yiikou
yyykou0E7353/0E73A7

0E8109yrioda
0E735D
yzuoyyou
0E7E94yodao

0E8FB0yochu
0E967Aypanren
0E9683yshuio
0E82FAyxjie

0E73BFymjiewang
0E7D2Dyygan

0E8675ywixing
0E7F13yfuy
0E933C
ymuyx!
0E95B2
ykoukoux"
0E74F1ywda
Table 6 ：Some components can be identified
characters coding table

This part is relatively simple, although we do not
identify the pronunciation of the glyphs, do not
identify which glyphs they correspond to later
generations, but each component in the glyphs is
identified, so this part of the code can be coded
according to the “y + component” format, and thepinyin of the component can be written in the orderfrom left to right, from top to bottom, and fromoutside to inside.
In the following, we show some practicalexamples of coding.
Intermediate codeof “中华字库”
0E86180E7786
0E7966
0E7D7E
0E79A2

Oracleglyphs
嚑
禚
 曣
 簑
 

Input coding
yyanripu
yyanripu
yriyanyrishiyriyuan

0E8F54
yyuhuo 碙
0E906Cyyuji 碂
0E7F03ybaohuo 碔
0E93D9yzhuilihuo
筶
0E9157yzhebuhuoᨎ
Table 5 ：All components are identified characterscoding table

2.2.2.2 Some components can be identified
but the rest do not.
In addition to the glyphs that are clearly fit
structures, we forcibly separate the uncharacterized
glyphs that may be part of the single body into
several parts for the convenience of the input
method design. The coding order of this part iswritten in accordance with “ y + identified
components + unidentified components”. The
order of identified components and unidentified
components is also arranged in the order from left
to right, from top to bottom, and from outside to
inside. The unidentified components are
represented by the selection of 26 letters similar totheir shapes according to the specific glyphs.
In the following, we show some practicalexamples of coding.
Intermediate codeof“中华字库”
0E7D86
0E734D

Oracleglyphs

Input coding
ytianooymunn

The middle part of the font of 0E7D86 is like“田”, and the closed semicircle on both sides isreplaced by two “o”.
The lower part of the glyph of 0E734D is “目”,
and the upper part looks like two upward raised
curved pens, so it is replaced by two “n”.
The left side of the font of 0E75E6 is “才”, andthe right side is both an undetermined andinseparable component. The component on theright side is composed of the head of “廌” and“ 卩 ”. In this case, if the coding is designed
according to this splitting, the coding will become
very long, so we choose the “卩” which is easier to
identify to replace the component on the right side.
The lower part of the glyph of 0E9D74 is “隹”,
and the upper part looks like “H”.
The upper part of the glyph of 0E8858 is “雨”,
and the lower part does not know what animal it
refers to. Because there are cross strokes in thelower part of this component, and the repetition
rate of coding “yyux” is very low, we use “x” toreplace the following components.
The glyph periphery of 0E79CB may be“皿”.Although the inner component does not knowwhat it is, it can be forcibly disassembled into twoparts : “羊” and “木”.
The left side of the glyph of 0E967A is “爿”,
and the right side of the component is not “人” butstill like the shape of human.
144
7

>
0E850Ayiiixx?
0E7687yuu
ys
0E8326
ⱘys
0E8327
ⱙ@
0E97F8yk
A
0E8E6Fyk
B
0E9172yx
C
0E9096yyD
0E78F8yl
E
0E78FDyoo
F
0E8C8Ayl
G
0E9F9Byy
H
0E9FC3yh
I
0E98B2yuiJ
0E7814yeo
K
0E8680yto
L
0E7E64yox
M
0E7EAAyi
N
0E7E3Dym
O
0E8100yuu
P
0E761Fyl
Q
0E80F6yj
R
0E7F29yiS
0E818Eyf
T
0E89F5ycj
U
0E8A23ym
V
0E8E21yomx
W
0E8415yoy
Table 7 ：Components completely unidentified
characters coding table

The glyph of 0E73BF has half part of the “王”,
and the left half is like the kneeling figure. The
upper side of the left component that cannot be split
is like “m”, so it can be forced to split into “m+卩”.
Part of the glyph of 0E8675 is an obvious “行”.
the upper side of the non-separable part that almost
encloses the “行” is like “w”, the lower side is like“I”.
0E933C may be a single font. The top is “目”
and “屮”, and the lower part can no longer be split.
But the part like “屮” is not “屮”, so we use theapproximate trident “Y” instead. Because therepetition rate of encoding “ymuyx” is also very
low, the lower part is replaced by “x”.
2.2.2.3 Components completely unidentified
In this case, we can only split these glyphs into
several parts, and arrange the letters similar to eachpart according to their order in the glyph. In orderto make the input of this part of the glyph easier,we try to arrange the parts with roughly similar
shapes in the same letter as much as possible.
In the following, we show some practicalexamples of coding.
Intermediate code of“中华字库”
0E774A
0E94F7
0E8AB9/0E97C5/0E72FC
0E7E8E
0E8482
0E999E
0E7BB30E71800E87B90E8B290E8B2D
0E7EDC
0E9A2D/0E8275/0E96CF
0E778B
0E832D
0E97B30E717C
0E82830E813C
0E899D
0E9740
0E9822
0E7E23

Oracleglyphs
#
$
% & '
(
)
*
+
,
.
/
0
123
4
5
6
7
8
9
:
;
<
=

Input
coding
yooox
youx
youuy
ym
yy
yi
yii
yam
yyooa
yooy
yuooy
yox
yox
ymmy
yco
yhh
yh
yi
yo
yww
yuo
yyooo
yoooy

In order to facilitate the reader to understand ourideas, we split the description of complex
characters . Due to space constraints, we list someof the more special examples to illustrate.
According to the strokes, we can see that theupper two curved pens of 0E774A 6 form threerings, and only “o” is a ring in the 26 letters.
Therefore, we have compiled three “o”, and there
are two crossed strokes. The image of “x” is moreconsistent, so this character is coded as “yooox”.
From top to bottom, 0E94F7 is an image of aring, a “凵” shape, two eyes, and a combination ofa person’s upper limb and a frog’s lower limb. Thisglyph has many and obvious distinguishingfeatures. For the convenience of input, the part ofthe eye shape that can not be encoded. In otheraspects, the “凵” shape can be encoded by “u” ,and the rest of the glyph is similar to “☆”, but the
26 letters are not similar to it, so the “x” with cross6

For the convenience of writing, we use the intermediate code
to replace the original character, and the corresponding
character can refer to the above table.

145
8

stroke features is used to encode. So this characteris coded as “youx”.
From top to bottom, 0E8AB9 is a ring, two “凵”
shapes, and the rest can be seen as an inverted “Y”
shape, so the character is encoded as “youuy” . The
glyphs 0E97C5 and 0E72FC look similar to
0E8AB9, so we think these three characters can use
the same code.
The shape of 0E8482 is similar to that of oraclebone inscriptions “歺” , but it should not be thesame character. The whole shape of this characteris three-line intersection, so it is encoded by theletter “y” .
The shape of 0E7180 is originally a single body,
but in order to facilitate input and avoid coding
repetition, we divide this font into two parts. The
top tip shape is like “A”, so we use “a” to encode
it. The lower part is like a bird spreading its wings,
which is similar to “M”, so we use “m” to encodeit.
The shape of 0E8B2D from top to bottom isapproximately “凵” , two circles, inverted triangle,
and other strokes. A vertical stroke is connectedunder the triangle below, and they are combined
together to be similar to the “Y” shape, so the word
is encoded as “yuooy” . Similarly, the coding of
0E8B29 is “yooy” .
The glyph of 0E7EDC can be divided into twoparts. The periphery is a circle, and the inside isthree lines that intersect at the same point. Theintersecting lines can still be encoded by “x” .
0E9A2D, 0E8275 and 0E96CF all have similar
characteristics, like the larger version of “田” . The
periphery of the three is basically closed and can becoded with “o”.There are many dry cross lines atthe center of the font, which can be coded with only
one “x”. We use the same coding on the two typesof glyphs, which may lead to high repetition rate,
but there are few cases similar to 0E7EDC glyphs,
and there are not many uncharacterized glyphs
similar to 0E9A2D,0E8275 and 0E96CF.Therefore,
these two types of glyphs are easy to distinguish in
the input process and will not affect the efficiency
of input.
Both 0E8326 and 0E8327 are on the 合集 22507,
and it remains to be further investigated whether
they are glyphs or characterization symbols.
However, the shape and composition of the two arevery strange. Like today’s one-stroke, it is notcommon in oracle bone inscriptions. Because weuse “s” to encode the curved linear components of

the rope shapes in other glyphs, we also use “s” toencode here.

3

Conclusion

According to our internal test program, the abovecoding design is indeed feasible, and the coding
repetition rate is very low, which is conducive toaccurate search. The number of glyphs involved in
the input method coding scheme we designed isunprecedented, so our coding design will be closer
to the real situation of oracle bone inscriptions thanprevious coding designs. We try to provide a
coding scheme for the input method in line with theprofessional cognition of ancient Chinesecharacters. Therefore, we are different from theprevious design : the coding design is carried out
for different types of Oracle glyphs, and theconcept of “natural classification” is added to thecoding of single component characters. For theunidentified glyphs, we also imitate the multicomponent characters as much as possible to carryout the separation in line with the cognition ofancient Chinese characters to encode, and use theEnglish letters to refer to the unidentified parts withsimilar shapes. Not only that, we have also
implemented the coding form of “shape code +phontic code” that has not been tried in the past.
The remaining number of variants that are notoften used is not a lot of unidentified glyphs.
Although some of these are not encoded according
to the knowledge of ancient Chinese philology,
there are still general rules to follow. For example,
we use “o” to refer to the closed form component,“x” to refer to the cross part of the two lines, “y”
to refer to the trident part, “x” to refer to theunidentified component that cannot be split
without increasing the repetition rate, and so on.
The design of o, x and y is similar to Oracle Bone
Inscriptions Six-digit Code Retrieval Font, but werefer to it from the perspective of “component” ofancient Chinese characters, not using the conceptof “stroke” of subsequent of Chinese characters.
Nevertheless, it still takes a lot of effort to form aregular coding design for the unidentified glyphs.
At present, the learning manual matching the
input method formed on the basis of this coding
design is still in preparation, and the preparation of
the learning manual is the subject we will study
next. In the following research, we will improve thepart of the above coding design that is notconvenient for fast input, and try to fit the coding
rules with strong regularity as much as possible forthe unidentified glyph part.

146
9

Acknowledgments
Thanks to Mo Bofeng, Liu Ying, and Li Aihui for
their guidance on this research. Thanks to Mr. Deng
Jian for his guidance in computer technology.
Thanks to the anonymous reviewers for theirvaluable comments on this paper.

References
Huang Tianshu. 2020.A Preliminary Study on the Sideand Head of the Oracle Bone . Collection of Huang
Tianshu 's Oracle Bone Science. .Zhonghua Book
Company. page172-181.(in Chinese)
Li Qingsheng, Wu Qinxia, Wang Lei . 2012 Oracle
Bone Inscription Input Method Based on Oracle Bone
Inscription Character Dynamic Description
Library .Chinese Journal of Information Issue 4. (inChinese)Liu Yongge and Li Qingsheng . 2004. Design and
Implementation of Visual Oracle Bone Inscription
Input Method , Computer Engineering andApplication Issue 17. (in Chinese)
Liu Yongge and Li Qiang . 2020.Summary of Oracle
Bone Inscription Input Method . Yindu Journal . Issue
3.（in Chinese）
Liu Zhixiang and Liu Xiaorong . 2019. Oracle Bone
Inscriptions Six-digit Code Retrieval Font. Sichuan
Dictionary Publishing House.（in Chinese）
Nie Yanzhao and Liu Yongge . 2010. Free Stroke Input
Method of Oracle Bone Inscriptions , Chinese Journalof Information. Issue 6.(in Chinese)
Xu Song and Hu Jinzhu . 1995. Realization of Oracle
Image Code Input Method , Journal of Central China
Normal University ( Natural Science Edition ). Issue
3. (in Chinese)
Zhou Demin, Wang Guoan, Zheng Tongbin, Su Yue, Li
Feng . 1990. Design and Implementation of Computer
Oracle Bone Inscriptions Information Processing
System ( CJPS ).Henan Science and Technology.
Issue 1. (in Chinese)147
10Word Sense Disambiguation for Ancient Greek:
Sourcing a training corpus through translation alignment

Alek Keersmaekers, Wouter Mercelis, Toon Van Hal
University of Leuven, Belgium{alek.keersmaekers,wouter.mercelis,toon.vanhal}@kuleuven.be

Abstract
This paper seeks to leverage translations ofAncient Greek texts to enhance theperformance of automatic word sense
disambiguation (WSD). Satisfactory WSD
in Ancient Greek is achievable, provided
that the system can rely on annotated data.
This study, acknowledging the challenges
of manually assigning meanings to every
Greek lemma, explores strategies to derive
WSD data from parallel texts using
sentence and word alignment. Our results
suggest that, assuming the condition of highword frequency is met, this techniquepermits us to automatically produce asignificant volume of annotated data,
although there are still significant obstacles
when trying to automate this process.

1

Aims

This contribution aims at making active use oftranslations of Ancient Greek texts in order toimprove results in automatic word sense
disambiguation (WSD). Section 2 outlines thegeneral research context, showing that decent
WSD in Ancient Greek is, in the current stage,
feasible if the system can be trained on annotateddata. Given the impracticality of manuallyannotating word meanings to all Greek lemmas,
this paper explores the possibility of generating asignificant volume of annotations automatically.
Section 3 surveys related work both at the level ofour general aim – word-sense disambiguation ofAncient Greek – and at the level of themethodology we adopt for attaining automaticallyannotated data for word-sense disambiguation, viz.
sourcing from parallel texts via sentence and wordalignment. After detailing the methodology
adopted (Section 4), we subsequently discuss theresults obtained, possible avenues for improvement
and perspectives for applications (Sections 5-7).

2

Research
context:
onomasiological searches

towards

It is generally known that in natural languages there
is not a one-to-one mapping between form andmeaning: one form or term can express various
meanings or concepts (e.g. ‘bright’ can refer tolight or intelligence) and vice versa (e.g. there arevarious ways to express that a person is intelligent,
including ‘bright’, ‘clever’, ‘smart’ etc.). In
semantic theory, studying the various meanings
that a specific form expresses is called the‘semasiological’ perspective, while studying thevarious forms that can be used to express a certain
meaning is called the ‘onomasiological’perspective (see Geeraerts, 2010).
This has important practical consequences:while it is straightforward to query most annotatedcorpora for specific terms, querying it for specificconcepts is usually far less straightforward (see, forinstance, Goossens, 2013). Most corpora have notbeen annotated semantically, given that the
annotation is labor-intensive and often subjective,
and semantics is multifaceted. However, to avoid
manual annotation, one could make use of socalled ‘vector-based models of meaning’ or ‘word
embeddings’, which retrieve computational
representations of meaning in a bottom-up mannerfrom a large, unannotated dataset (Lenci, 2018).
In the context of Ancient Greek, exploratory
studies of vector-based models for detecting
onomasiology have begun to emerge, starting fromthe premise that these models can be harnessed to
identify words bearing a similar or related meaning
to a given target word (Keersmaekers and Van Hal,
2021 & 2022). In this case, if the researcher alreadyknows some terms that can express a particularconcept (say γλῶσσα and φωνή for the concept‘language’), they can use these models to look forterms that are similar to these target words and by
148

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 148–159,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_018

doing so fully map the onomasiology of thisconcept.
However, one complication is polysemy. Whenusing a vector-based model1 to find the ten nearest
neighbors of the term γλῶσσα, for example, theresults are all body parts, such as οὖς ‘ear’, ὀδούς‘tooth’, ὀφθαλμός ‘eye’, χεῖλος ‘lip’ and φάρυγξ
‘throat’. The explanation for this is predominantly
linked to the polysemy of γλῶσσα, which candenote both ‘language’ and ‘tongue’. The lattermeaning is particularly prominent due to the
corpus’s extensive inclusion of medical data, whichconstitutes 14% of all training data, in which‘tongue’ is more frequently referred to.
One possible solution is WSD: if we could
separate all tokens of γλῶσσα that mean ‘tongue’
from those meaning ‘language’, we could look forthe nearest neighbors of γλῶσσα when only thetokens meaning ‘language’ are taken into account.
Again, vector-base models can be employed for
this: indeed, several transformer-based embedding
models such as BERT (Devlin et al., 2019) do nolonger model the ‘general’ meaning of a word but
the meaning of a word in context. Such an approachfor Ancient Greek is discussed in Mercelis et al.
(Forthc.), using ELECTRA (Clark et al., 2020) as alanguage model. When this model was used in an
unsupervised way, the results were disappointing,possibly due to data sparsity. However, when used
in a supervised way, by finetuning the transformer
network, decent results could be achieved with
only about 150 training examples (for binary
meaning distinctions) or 300 (for ternary meaning
distinctions).
To overcome the problems related to theacquisition bottleneck in obtaining annotated data
(Lefever et al., 2011: 320; Pasini, 2020), this paper
will discuss an automated way of creating datasets
for WSD, by exploiting parallel texts (Greek
original texts and English translations). Thisapproach initially involves aligning sentences.
Subsequently, within the aligned sentences,individual words are aligned. This two-step process
will enable us to annotate polysemous words in
Greek with English labels, thus trying to get a holdof their polysemy.

3

1 This example is retrieved from the vector models

based on word vectors created using singular value
decomposition incorporating syntactic dependency features.

described in Keersmaekers & Van Hal 2021, which are

3.1

Related work
WSD for Ancient Greek

While the problem of automatic WSD has beentackled for decades already for English, interest incomputational semantics has only raised recently
for Ancient Greek, and the literature on this topic istherefore very limited. The only studies that we areaware of are Mercelis et al. (Forthc.), as discussed
in Section 2, and McGillivray et al. (2019). While
Mercelis et al. (Forthc.) directly exploredsupervised and unsupervised WSD using large
language models, the angle of McGillivray et al.
(2019) is somewhat different in that they explore
how computational methods can be used for lexical
semantic change detection. Focusing on threepolysemous words (viz. μῦς, ἁρμονία and κόσμος),
they explore their polysemy over time and genre
using a Bayesian topic model, and match the results
to manually annotated datasets of these words.
3.2

Word and sentence alignment

Word alignment used to be one of the key steps inthe process of statistical machine translation.
Statistical word alignment, represented by
GIZA++ (Och and Ney, 2003) formed a strong
baseline, which was only surpassed recently bylarge language models such as BERT (Devlin et al.,2019) and RoBERTa (Liu et al., 2019), based ontransformer techniques. Nowadays, attention
mechanisms in these large language models havemade the word alignment task obsolete in machinetranslation pipelines. Nonetheless, in recent years
word alignment made a comeback, albeit not solely
in function of machine translation (Li, 2022). Our
paper can be situated in this newfound interest inword alignment, as we focus on aligning words tocreate datasets for WSD.
Li (2022) provides a comprehensive summary of
the history of word alignment, along with an
overview of potential strategies for executing thistask. Given that the word alignment task isinherently multilingual, most approaches employ amultilingual language model such as mBERT(Devlin et al., 2019) or XLM-RoBERTa (Conneau
et al., 2020), which is then fine-tuned for thealignment task. In our case, this is more complex,
given that Ancient Greek is in general notincorporated in such multilingual models. Hence,149

we used the recently released PhilBERTa model(Riemenschneider and Frank, 2023), a trilingual
model trained on English, Ancient Greek, and Latin
texts.
Yousef et al. (2022a) recently investigatedtranslation alignment at the word level, with aparticular focus on Ancient Greek. They utilized
multilingual embeddings from which they selected
the most similar pairs, signifying aligned words.
They employed two alignment techniques: theapproach of Jalili Sabet et al. (2016) and that of
Dou and Neubig (2021). While according to Li(2022) both techniques handle the word alignment
task proficiently, the highest-performing techniquein Li’s (2022) dataset was a span-extraction modelby Nagata et al. (2020). This approach is widely
recognized for its application in Question
Answering, as the Stanford Question Answering
Dataset (SQuAD) (Rajpurkar et al., 2016) was
designed with this technique in mind.
Chousa et al. (2020) released a similar model –also based on span-extraction – for sentencealignment. This model achieved state-of-the-art
results on various modern language combinations(German – English, French – English, Japanese –English), beating previous approaches such as
VecAlign (Thompson and Koehn, 2019).
3.3

Translation alignment for WSD

Parallel texts have a long-standing tradition in
WSD, with its roots traced back to the work of Ng
et al. (2003) (cf. Pasini, 2020: 4939 for moredetails). Our approach in this contribution isbilingual, viz. Ancient Greek – English. Over the
past decade or so, there has been an emphasis on amultilingual rather than bilingual approach toparallel corpora for WSD (see e.g. Lefever et al.,2011). Most of these approaches rely on themassive European parliament corpus (see, e.g.
Delli Bovi et al., 2017). Rather than concentrating
solely on direct annotation transfer on the token
level, certain researchers propose a more holistic
approach. This involves taking into account thewider context provided by the entire parallel
corpus, rather than merely focusing on parallel
sentences (van der Plas and Apidianaki, 2014).
More recently, scholars have proposed multilingual
approaches in which translation parallels are
replaced with propagation methods. Starting fromcontextualized word embeddings in English andrelying on multilingual data from knowledge bases(such as WordNet and Wikipedia), such approaches
can automatically generate training data for
languages without labeled data for WSD (Barba etal., 2020). Recent research has also pointed out thatthe generation of translations can improve thequality of WSD (see e.g. Luan, 2020).

4
4.1

Methodology
Data

In our undertaking Ancient Greek is the source
language and English the target language, given theabundance of English translations and manuallyaligned data. For our source language, we started
from the GLAUx corpus (Keersmaekers, 2021),
which encompasses approximately 32M Greek
tokens, spanning roughly from the 8th century BC
to the 4th century AD. As for the target language,
the majority of our English data was drawn fromthe Perseus project (Smith et al., 2000).2 However,
we also incorporated openly accessible online
editions for certain lengthy texts not available in
Perseus, such as Dionysius of Halicarnassus’s
Roman Antiquities. Both the GLAUx data and theEnglish translation data incorporate information
about the texts’ structure (e.g., division into books,
chapters, sections, verses, etc.). This facilitated thealignment of ‘paragraphs’ in both languages. We
use the term ‘paragraph’ loosely here, referring to
the shortest shared structural unit between the
Greek text and its translation, which can be, forinstance, a section, chapter (if no sections areprovided), or, in the case of poetic texts, a group ofverses. In total, we were able to link around 7.2
million Greek tokens (approximately a quarter of
the GLAUx corpus) to an English translation.
We trained word alignment models using datafrom the Alpheios project3 and from the UGARIT
project (Yousef et al., 2022b) as training data
(66929 tokens). For the sentence alignment task,
we used the same data sources, supplemented with
Pedalion data (Keersmaekers et al., 2019), as well
as a parallel New Testament corpus and data fromthe Greek Learner Texts Project.4 In addition to this,we also annotated data ourselves. In total, thisamounted to 15178 training sentences.

2 Data taken from https://github.com/PerseusDL/canonical-

4 See https://greek-learner-texts.org andgreekLit.
3 https://alpheios.net/pages/tools/

https://github.com/jtauber/plato-texts.

150During the development stage, we assessed theword alignment task using the same gold standard
data (5076 tokens) employed by Yousef et al.
(2022a). This facilitated a direct comparison of ourresults with their work. We evaluated the sentencealignment model using our own held-out data (879sentences). This dataset was the most appropriate
for evaluating performance as it consisted ofparallel paragraphs, whereas for other datasets, wewere forced to artificially combine sentences into
existing or sometimes even entirely newparagraphs (fixed at a length of 10 sentences), since
they did not provide paragraph data. Given thelength of some of these paragraphs in ourevaluation dataset, this dataset posed a significant
challenge for the model in accurately predicting
sentence alignments.
4.2

Sentence alignment

Our target corpus, GLAUx, is paragraph-aligned,
requiring us to first conduct sentence alignment toenable word alignment within these sentences.
Segmenting Ancient Greek paragraphs into
sentences is a straightforward process, given theexistence of meticulous editions of the available
texts and the general lack of abbreviations thatmight complicate splitting at full stops. Thus, ouraim is to extract the English sentences that
correspond to a particular Ancient Greek sentencefrom an entire English paragraph.
To achieve this, we employ a span-extraction
approach, based on the work of Chousa et al.
(2020), as discussed in Section 3.2. This method
represents the state-of-the-art approach and is
methodologically quite similar to the wordalignment model. The key distinction lies in thefocus of extraction: tokens from sentences in thecase of word alignment, and sentences fromparagraphs for sentence alignment.
4.3

For each pair of parallel sentences, we used theEnglish sentence as the context. Then, for every
token in the Ancient Greek sentence, we treatedthis sentence as a ‘question’, similar to theterminology used in SQuAD. In this sentence, thecurrent token was demarcated with a special
separation token. Both the context and the‘question’ were processed by a PhilBerta model(Section 3.2), fine-tuned for the span-extractiontask. The model then predicted the start and end
indices of the corresponding English token in thecontext, or the English sentence, thereby aligning
the Ancient Greek and English tokens.
Upon completing this process, we secured a
corpus that was aligned at the word level.

Word alignment

As noted earlier, there are several strategies forword alignment. For this task, we selected the spanextraction approach as well. This method was the
top performer in the study by Li (2022), 5 and it
utilizes annotated data, to which we had access.
Additionally, choosing a different approach toword alignment than Yousef et al. (2022a) allowed
us to compare the outcomes.

Lemma’sFrequency bandγλῶσσα; λόγος; φωνή
ῥῆμα

1
2

Table 1: Linguistic terms.
Lemma

Frequency bandαἴσθησις; καταλύω;ἀλλότριος
βίοτος; ἀπαντάω; μιαρός
ἱστός; ἀνύω; ξηρός

1
2
3

Table 2: Randomly selected terms.

4.4

From translation alignment to WSD

To investigate how useful the word-aligned results
are for WSD, we created two test sets of (a) words
referring to metalinguistic concepts and (b)randomly selected polysemous words, as shown in
Tables 1 and 2 respectively. The first set of words
was handpicked by our team, as this work was
initiated within the framework of a project focusedon the onomasiology of linguistic concepts. The
second set was chosen to extend the validation of
our approach beyond the confines of this specificproject. To be precise, we utilized the word list by
Van Hal (2013), which provides information on thefrequency (in four frequency bands) and polysemyof various Greek words, excluding those that are
extremely common. From the first three frequency
bands of Van Hal (2013), we randomly selected one
noun, one adjective, and one verb.

5 Note, however, that the target languages of these studies

are all modern languages that are less inflectional thanGreek and can utilize larger language models.

151

Next, for each of the target words listed in Table
1-2, we extracted the word alignments retrieved
with our automatic models. The results were quite
messy, containing many one-to-many alignments(likely due to our training data): an example(γλῶσσα) is shown in Table 3. Additionally, they
contain inflected forms (‘tongues’) as well asfunction words such as articles and prepositions,due to linguistic differences between the two
languages (i.e. Greek uses case marking, does nothave an indefinite article and uses definite articles
differently from English etc.). We therefore further
cleaned the data by (a) tokenizing the results, (b)removing punctuation, (c) removing stop words
and (d) lemmatizing each word in the results, using
the
NLTK
packages
stopwords
andWordNetLemmatizer (Bird et al. 2009). After doing
so, we further removed noise by calculating thefrequency of each remaining lemma and removing
all the lemmas that occur less than 1% in the total
results. An example of the final output for γλῶσσα
is given in Table 4. Although the table still contains
some noise (e.g. the adjectives ‘rare’, ‘good’ and‘ordinary’), most of the results are clear
translations of the word γλῶσσα.
Nevertheless, the results contained several
synonyms or very closely related words (e.g. ‘lip’
and ‘mouth’ in Table 4). To use these results for
WSD, they therefore need to be clustered in someway. In order to obtain a first idea which criteria theclustering should use, we performed the clustering
manually, although automatic clustering is
obviously necessary if one wants to scale up thisapproach to the full Greek corpus. Concretely, weused both frequency and meaning relatedness as
criteria: in all cases, we clustered very closely
related meanings (i.e. near-synonyms) together, butalso clustered meanings when they were only
somewhat closely related but were infrequent. In
other words, we used a pragmatic criterion: if there
were too little examples of a specific meaning, it
would be problematic to learn this meaning
through WSD, so it would be worth it to combine
them with examples of another related meaning,
even if some meaning granularity was lost by doing
so. We did not assign irrelevant words to a cluster(e.g. ‘rare’, ‘good’, ‘ordinary’ in Table 4), but
simply discarded them from the dataset. The resultsof the manual meaning clustering can be seen in
Tables 5-8 in Appendix. To create a final dataset for
WSD, for each cleaned up word alignment wechecked if it contained any of the words assigned

to one of the clusters, and if not, the example wasdiscarded. Next, one could use these results to trainmodels for WSD, take the tokens from the Greek
corpus that were assigned to one of the meanings
that they are interested in (e.g. the linguistic
meaning of γλῶσσα in our case) and calculate thenearest neighbors based on these tokens, as detailedin Section 2. However, we did not perform this stepin the scope of this paper.
Alignment#

Alignment#

tonguethe tongue
tongueswith tongues
aof the tongue.
his tongue

57
18
11
9
7
6
5
5

my tongue
in a tongueofthe tongues
speecha tongue
lips
…

5
5
5
5
4
4
3
…

Table 3: Example of word alignment results:
γλῶσσα.
Alignment#

Alignment#

tongueword
speech
rare
lip

168
15
12
8
6

language
good
voice
mouth
ordinary

4
4
3
3
3

Table 4: Cleaned results of γλῶσσα55.1Results
Sentence alignment

Firstly, we evaluated the model on the held-out data
described in Section 4.1. This resulted in an
accuracy (exact matches) of 73% (644/879). The
F1-score, which also takes into account partial
matches, was 86%.
Since such a quantitative evaluation can bemisleading (since the test data might not entirely
match our target corpus), we also manuallyconducted an evaluation of sentence alignment
performance using 133 sentences from the target
corpus chosen at random. The accuracy was 65%(86/133), somewhat lower than the 73% of theautomatic evaluation, indicating that these results152

might be too rosy.6 Out of the remaining 35% ofsentence alignments that were not correct, half ofthem (25/47) were partially correct, i.e. the Greek
sentence contained the English translation but
included more text, or vice versa.
The size of the training corpus at the sentencealignment task appears to be of great importance. It
was our hypothesis that non-problematic
corresponding sentences (in a 1-to-1 ratio, i.e.
without Greek sentences that are mapped tomultiple English sentences, or vice-versa) thatwere combined into artificial paragraphs (cf.
Section 4.1) would contribute little. This turned out
not to be the case. A model trained on data without
these artificial paragraphs performed significantly
worse, with an accuracy of 43% and an F1-score of
41% on the held-out data.
5.2

Word alignment

In contrast with the results shown in Li (2022), thespan-extraction approach implemented in ourmodel performed worse than the approach of Jalili
Sabet et al. (2016) and Dou and Neubig (2021), as
used by Yousef et al. (2022a). The comparison isdifficult however, as they not only used anotheralignment approach, but also utilized anothertraining dataset. Their best-performing model
achieved an F1-score of 81.5, and an Alignment
Error Rate (AER) of 18.7. It is, however, notexactly clear how the metrics are computed, viz.
how punctuation and source words that do not havean alignment (e.g. untranslatable particles) areexactly handled. In the gold dataset, tokens withoutalignment are not annotated. Thus, it is not clear
whether they are included in the evaluation or not.
The scores including these source tokens andpunctuation, are an F1 of 47.7 and an AER of 43.5
(5076 tokens in total). If we leave these out, the F1score rises to 59.6, and the AER is 35.9. For thescope of this project, the former evaluation is themost important, as the WSD task is mainlyinterested in content words such as verbs, nouns
and adjectives. In contrast with these part-ofspeech classes, the left-out tokens are mainly
punctuation marks and untranslatable particles,
which are of less importance for the WSD task.

5.3

Manual clustering of the results

The results derived from applying word alignmentand subsequently manually clustering them, as
outlined in Section 4.4, are presented in Tables 5-8(found in the Appendix). A notable observation isthat a considerable proportion of the data,accounting for 49% of all aligned tokens on
average across all target words, included manytranslations that could not be neatly clustered(labelled as ‘other’ in these tables). This percentage
varied from 24% (for μιαρός) to as high as 84% (forἱστός). These typically fall into two categories: (a)words that were excluded by the frequency filter(see Section 4.4) or (b) incorrect word alignments.
Concerning category (a), there are instances where
the frequency filter eliminates relevant terms. A
case in point is ‘Latin’ for γλῶσσα, which wasfiltered out despite clearly referring to the linguistic
sense of γλῶσσα (contextually appearing in ‘λέγεινἱκανῶς ἑκατέραν γλῶτταν’ which was roughly
translated to ‘to speak both Latin and Greek
fluently’). Conversely, when the frequency filter is
not used, the data evidently becomes cluttered withirrelevant results. For instance, some of the singleoccurrence results for γλῶσσα include ‘she-bear’,‘of frigidity’, and ‘power of lubricating’, which areunquestionably incorrect translations for γλῶσσα.
Given that translation alignment at both thesentence and word levels only reaches a respective
F1-score of 86 and 60 percent, it is inevitable thatthe data will contain numerous errors, resulting
from either inaccurate sentence or word alignment.
Since the frequency threshold was relatively
low, for less frequent words (viz. βίοτος, μιαρός,ἱστός, and ἀνύω) no words were filtered out,
allowing us to assess how many alignment pairs
were relevant for the task described in this paper.
As can be deduced from Tables 5-8, for βίοτος 40%of all alignments were irrelevant, for μιαρός 24%,for ἱστός 84% and for ἀνύω 67%. This averages out
to 54%, meaning that only half of the alignments
were relevant for compiling a WSD dataset.
This has serious consequences for the
possibilities of automating this approach. On the
one hand, the frequency filter was absolutely
necessary, given the amount of noise present in thedata, which would make automatic clustering
problematic. On the other hand, if an absolute
frequency filter would have been used (e.g.

6 Although the differences are barely statisticallysignificant, with p=0.05 with Fisher’s exact test.

153filtering out translations that occur less than 3times), this would lead to data sparsity for lessfrequent words. Therefore an obvious solution
would be expanding the data, either by improving
the alignment results or by adding more parallel
English translations to the data.
On a brighter note, this method is clearly capableof retrieving a sufficient number of relevant
examples for more frequent terms, thus creating a
useful dataset for WSD. Nevertheless, there areseveral important considerations. Firstly, it is worth
noting that the manual clustering was highly
subjective: another researcher may well havegrouped the words differently than we did. In such
instances, an automatic clustering method mightoffer greater objectivity, even though automatic
methods carry their own inherent biases. Generally
speaking, the use of parallel translations is more
effective when meanings can be more clearly
differentiated (e.g., in the case of ἱστός, where there
is a stark difference between ‘mast’ and ‘loom’),rather than when the differences are somewhat
vague (for instance, for λόγος, the distinctions
between ‘word’, ‘statement’, and ‘report’ are notalways easily discernible).
Secondly, the level of granularity that is possible
to distinguish is dependent on the number ofexamples for a specific sense, especially when
taking into account that some senses are morepresent in the data that we are using than othersenses. While for λόγος many fine-grained
distinctions can be made, for γλῶσσα only ageneral ‘linguistic’ sense can be distinguished,conflating the translations ‘voice’, ‘speech’,‘language’ and ‘word’. Meanwhile, for some WSD
is not possible at all: for ξηρός all translations
pointed to ‘dry’ (while the word also has othermeanings in Greek, such as ‘slim’ and ‘harsh’).
Finally, one obvious issue is that this method
assumes that the English translation equivalents do
not have the exact same sense ambiguity as the
Greek words. This does not always hold true. In theγλῶσσα-case, for instance, the English term
‘tongue’ can occasionally signify ‘language’, as
exemplified in phrases like ‘mother tongue’. This
interpretation is also found in some of the more
antiquated translations within our corpus. Anotherexample is αἴσθησις, where ‘sense’ in English issimilarly ambiguous between the meaning‘sensation, perception’ and ‘faculty forexperiencing the outside world’. This issue could
be solved in multiple ways, e.g. by using paralleltranslations from other languages that do not havethis sense ambiguity. Alternatively, WSD could be
conducted on the English data. However, this adds
another automated step, which may potentiallycompromise the quality of the final results.

6
6.1

Avenues for better results
Improving alignment

Clearly, as the previous section demonstrates,
inaccurate alignment results significantly curtailthe volume of data that can be employed for WSD.
Therefore, enhancing automatic alignments is avital step towards further improvements.
On a foundational level, our work relied on an
existing multilingual RoBERTa model, namely
PhilBerta. However, given potential mismatches
between the data format of PhilBerta and GLAUxdata (for instance, in terms of Unicode encoding ofaccents or tokenization), it might prove beneficial
to adopt an English-Greek model that is moreclosely attuned to the GLAUx data.
Regarding sentence alignment, potential
improvements could be realized by augmenting thetraining data. Considering our current training set
is rather limited (comprising 15,178 sentences),
expanding it is one possible avenue for enhancing
results (a step we are presently exploring; cf.
Section 6.4). However, this inevitably entails asignificant amount of manual work. An alternative
strategy is to refine the alignment method itself.
Our current method relies solely on wordembedding information. While this might function
effectively for language models with extensive data,
Greek embedding models could be too sparse to beeffectively deployed in isolation. Supplemental
information might thus bolster the results, such as
sentence position within a paragraph (naturally,
Greek and English sentences tend to occupy similar
positions within identical paragraphs) and thefrequency of matches between the English
translation of a Greek word, using a bilingual
dictionary, and the English sentence. Moreover,
the word alignment task could inform sentencealignment: very low probabilities in wordalignment might signal that sentence alignment has
misidentified a sentence. Lastly, an entirely
different approach than the one employed in thisstudy could also be considered. Adopting an
unsupervised approach like VecAlign (Thompson
and Koehn, 2019) could address the problem of154

having to depend excessively on annotatedexamples.
Given that our method for word alignment isbased on the same technique as sentence alignment,
all the above considerations hold true for theformer task as well. However, manually annotating
word alignment proves to be even more laborintensive than sentence alignment. Hence,
unsupervised models may prove particularly
advantageous for this task.
6.2

Improving the clustering

While the alignment results could be improved
further, the task is inherently challenging and it istherefore likely that a significant amount of noise
will always persist in the data. Thus, it is vital to
implement effective techniques for filtering thisnoise. The simple frequency filter used in this study
could potentially be too restrictive in someinstances, such as with the Greek word μιαρός,which has several one-time translations for theconcept ‘miserable’. To address this, we might
consider semantic similarity (operationalized
through language models) as an additional criterion,
specificallybyincluding
low-frequency
translations if they show substantial semantic
similarity to a higher-frequency translation.
For this study, we manually performed theclustering, but naturally, automatic clustering isnecessary if we aim to extend this approach to theentire Greek corpus. A feasible method might
involve clustering words with similar static
embeddings in English.
6.3

Alternative methods

The applicability of new techniques for WSD andtranslation alignment, as discussed in Section 3.3,
to Ancient Greek remains uncertain. When it
comes to multilingual approaches, there is a
scarcity of multilingual parallel corpora featuring
Ancient Greek, with the exception of Biblical texts.
However, repositories like <remacle.org> andhodoi elektronikai <hodoi.fltr.ucl.ac.be> could
facilitate the creation of a trilingual Greek, English,
and French corpus. The potential of propagation
methods (which necessitate knowledge bases) andautomatic translations in enhancing WSD in
Ancient Greek is unclear.

7 For sentence alignment, the accuracy rose from 73% to85%, while the F1 score increased from 86% to 92%. For

One reviewer commented that instead of themethod proposed in this paper, one could collecttraining data from dictionaries, as was done by
Bamman and Burns (2020). Indeed, this was thestrategy we initially pursued, using a digital versionof the Liddell-Scott-Jones (LSJ) dictionary that we
automatically linked with the GLAUx corpus.
However, it soon became clear that relying
exclusively on this dataset was only possible for a
few highly frequent words with many examples inthe dictionary: even when excluding the irrelevant
word alignments (classified as ‘other’ in Table 58), the amount of data we could retrieve from wordalignment was ten times larger than from the LSJ
dictionary, and there was only one word (λόγος) for
which we could retrieve more than 100 examples
from LSJ (243 in total, which is still much smaller
than the 3128 examples from word alignment).
However, these dictionaries might still provide
supplementary data, or provide a solid base forclustering the word alignments (i.e. by showing
which English translations ‘group together’ for one
specific meaning).
6.4

Progress made after peer review

While the results discussed in this paper might notseem too promising initially, we found that wewere able to substantially improve the results by
expanding and cleaning up the training data forboth sentence and word alignment, and expanding
our parallel Greek-English corpus with some otheropenly available translations.7 For example, for theword γλῶσσα we now obtained 829 relevant results,
after removing 187 results by applying thefrequency filter, while previously we only had 210
relevant results after removing 152 results (seeTable 8).

7

Conclusions and perspectives

In view of the extensive research conducted on
WSD for modern languages, the comparative
neglect of classical languages is striking. However,significant progress can be made in the near future
to rectify this disparity, thanks in part to thecomprehensive philological studies conducted in
the past. With a robust lexicographical tradition
replete with translated example sentences, and aprolific translation history, classical language
resources, once available in a digital shape, haveword alignment, the F1 score improved from 59.6 to 70.6,
while the AER dropped from 35.9 to 24.0.

155the potential to unlock promising possibilities for
WSD applications.
The methodology presented in this paper
appears to be a promising means to achieve ourgoals – coming to an onomasiological disclosure of
the Ancient Greek corpus. A critical prerequisite,
however, is the availability of a substantial volumeof data, suggesting that the approach is effective
predominantly for frequently used words.
Apart from this, we believe that this approachholds intrinsic value. For texts that have digital
English translations available, we can makeeducated predictions regarding the meanings of theindividual tokens. Additionally, this approachprovides insights into the distribution of word
senses as distinguished by lexicographers inAncient Greek.

Acknowledgments
We would like to thank the reviewers for theirvaluable comments and suggestions. The researchin this article was made possible by grant numbers
HBC.2021.0210/KRIS:0508000001361 (VLAIO)
and G052021N (FWO, Research Council –Flanders).

References
Bamman, David, and Patrick J. Burns. 2020. “Latin
BERT: A Contextual Language Model for ClassicalPhilology.” ArXiv:2009.10053 [Cs], September.
http://arxiv.org/abs/2009.10053.
Barba, Edoardo, Luigi Procopio, Niccolò Cam
polungo, Tommaso Pasini, and Roberto Navigli.
2020. “MuLaN: Multilingual Label Propagation for
Word Sense Disambiguation.” In Proceedings of
IJCAI, 3837–44.
Bird, Steven, Edward Loper, and Ewan Klein. 2009.
Natural Language Processing with Python. O’Reilly
Media Inc.
Chousa, Katsuki, Masaaki Nagata, and Masaaki
Nishino. 2020. “SpanAlign: Sentence Alignment
Method Based on Cross-Language Span Prediction
and ILP.” In Proceedings of the 28th International
Conference on Computational Linguistics, 4750–
61. Barcelona (Online): International Committee on
Computational
Linguistics.
https://doi.org/10.18653/v1/2020.coling-main.418.
Clark, Kevin, Minh-Thang Luong, Quoc V. Le, andChristopher D. Manning. 2020. “ELECTRA: PreTraining Text Encoders as Discriminators Rather
Than Generators.” CoRR abs/2003.10555.
https://arxiv.org/abs/2003.10555.

Conneau, Alexis, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke
Zettlemoyer, and Veselin Stoyanov. 2020.
“Unsupervised
Cross-Lingual
Representation
Learningat
Scale.”
arXiv.
https://doi.org/10.48550/arXiv.1911.02116.
Delli Bovi, Claudio, Jose Camacho-Collados,
Alessandro Raganato, and Roberto Navigli. 2017.
“EuroSense: Automatic Harvesting of Multilingual
Sense Annotations from Parallel Text.” In
Proceedings of the 55th Annual Meeting of the
Association for Computational Linguistics (Volume
2: Short Papers), 594–600. Vancouver: Associationfor
Computational
Linguistics.
https://doi.org/10.18653/v1/P17-2094.
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. “BERT: Pre-Training of
Deep Bidirectional Transformers for Language
Understanding.” In Proceedings of the 2019
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short
Papers), 4171–86. Minneapolis).
Dou, Zi-Yi, and Graham Neubig. 2021. “Word
Alignment by Fine-Tuning Embeddings on Parallel
Corpora.” arXiv. http://arxiv.org/abs/2101.08231.
Geeraerts, Dirk. 2010. Theories of Lexical Semantics.
Oxford & New York: Oxford University Press.
Goossens, Diane. 2013. “Assessing Corpus Search
Methods in Onomasiological Investigations.”
Corpus Perspectives on Patterns of Lexis 57: 271.
Jalili Sabet, Masoud, Philipp Dufter, François Yvon,and Hinrich Schütze. 2021. “SimAlign: High
Quality Word Alignments without Parallel Training
Data
Using
StaticandContextualized
Embeddings.”
arXiv.
http://arxiv.org/abs/2004.08728.
Keersmaekers, Alek. 2021. “The GLAUx Corpus:
Methodological Issues in Designing a Long-Term,
Diverse, Multi-Layered Corpus of Ancient Greek.”
In Proceedings of the 2nd International Workshop
on Computational Approaches to Historical
Language Change 2021, 39–50. Online: Associationfor
Computational
Linguistics.
https://doi.org/10.18653/v1/2021.lchange-1.6.
Keersmaekers, Alek, Wouter Mercelis, Colin
Swaelens, and Toon Van Hal. 2019. “Creating,
Enriching and Valorizing Treebanks of Ancient
Greek.” In Proceedings of the 18th International
Workshop on Treebanks and Linguistic Theories(TLT, SyntaxFest 2019), 109–17. Paris: Associationfor
Computational
Linguistics.
https://doi.org/10.18653/v1/W19-7812.

156

Keersmaekers, Alek, and Toon Van Hal. 2021. “A
Corpus-Based Approach to Conceptual History ofAncient Greek.” In Cognitive Sociolinguistics
Revisited, edited by Gitte Kristiansen, Karlien
Franco, Stefano De Pascale, Laura Rosseel, andWeiwei Zhang, 213–25. Berlin & Boston: Walter deGruyter.
———. 2022. “In Search of the Flocks: How to
Perform Onomasiological Queries in an Ancient
Greek Corpus?” In Proceedings of the LREC 2022
Second Workshop on Language Technologies for
Historical and Ancient Languages (LT4HALA2022), 73–83. Marseille.
Lefever, Els, Véronique Hoste, and Martine De Cock.
2011. “ParaSense or How to Use Parallel Corpora
for Word Sense Disambiguation.” In Proceedings ofthe 49th Annual Meeting of the Association for
Computational Linguistics: Human Language
Technologies, 317–22. Portland: Association for
Computational
Linguistics.
https://aclanthology.org/P11-2055.
Lenci, Alessandro. 2018. “Distributional Models of
Word Meaning.” Annual Review of Linguistics 4(1):
151–71.
https://doi.org/10.1146/annurevlinguistics-030514-125254.

Using Multilingual BERT.” In Proceedings of the2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), 555–65. Online:
Association for Computational Linguistics.
https://doi.org/10.18653/v1/2020.emnlp-main.41.
Ng, Hwee Tou, Bin Wang, and Yee Seng Chan. 2003.
“Exploiting Parallel Texts for Word Sense
Disambiguation: An Empirical Study.” In
Proceedings of the 41st Annual Meeting of the
Association for Computational Linguistics, 455–62.
Sapporo:Associationfor
Computational
Linguistics.
https://doi.org/10.3115/1075096.1075154.
Och, Franz Josef, and Hermann Ney. 2003. “A
Systematic Comparison of Various Statistical
Alignment Models.” Computational Linguistics 29(1): 19–51.
Pasini, Tommaso. 2020. “The Knowledge Acquisition
Bottleneck Problem in Multilingual Word Sense
Disambiguation.” In Proceedings of the TwentyNinth International Joint Conference on Artificial
Intelligence, 4936–42. Yokohama: International
Joint Conferences on Artificial Intelligence
Organization.
https://doi.org/10.24963/ijcai.2020/687.

Li, Bryan. 2022. “Word Alignment in the Era of Deep
Learning:
A
Tutorial.”
arXiv.
https://doi.org/10.48550/arXiv.2212.00138.
Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du,
Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
2019. “RoBERTa: A Robustly Optimized BERT
Pretraining
Approach.”
arXiv.
https://doi.org/10.48550/arXiv.1907.11692.
Luan, Yixing, Bradley Hauer, Lili Mou, and Grzegorz
Kondrak. 2020. “Improving Word Sense
Disambiguation with Translations.” In Proceedingsof the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), 4055–65.
Online: Association for Computational Linguistics.
https://doi.org/10.18653/v1/2020.emnlp-main.332.
McGillivray, Barbara, Simon Hengchen, Viivi
Lähteenoja, Marco Palma, and Alessandro Vatri.
2019. “A Computational Approach to Lexical
Polysemy in Ancient Greek.” Digital Scholarship inthe
Humanities34(4):
893–907.
https://doi.org/10.1093/llc/fqz036.
Mercelis, Wouter, Toon Van Hal, and Alek
Keersmaekers. Forthcoming. “Tongue, language ornoise? Word Sense Disambiguation in Ancient
Greek with corpus-based methods.” In International
Colloquium of Ancient Greek Linguistics.
Nagata, Masaaki, Katsuki Chousa, and Masaaki
Nishino. 2020. “A Supervised Word Alignment
Method Based on Cross-Language Span Prediction

van der Plas, Lonneke, and Marianna Apidianaki.
2014. “Cross-Lingual Word Sense Disambiguation
for Predicate Labelling of French.” In Proceedingsof TALN 2014 (Volume 1: Long Papers), 46–55.
Marseille: Association pour le Traitement
AutomatiquedesLangues.
https://aclanthology.org/F14-1005.
Rajpurkar, Pranav, Jian Zhang, Konstantin Lopyrev,
and Percy Liang. 2016. “SQuAD: 100,000+
Questions for Machine Comprehension of Text.”
arXiv. https://doi.org/10.48550/arXiv.1606.05250.
Riemenschneider, Frederick, and Anette Frank. 2023.
“Exploring Large Language Models for ClassicalPhilology.”
arXiv.
https://doi.org/10.48550/arXiv.2305.13698.
Smith, David A., Jeffrey A. Rydberg-Cox, and Gregory
R. Crane. 2000. “The Perseus Project: A Digital
Library for the Humanities.” Literary and Linguistic
Computing 15 (1): 15–25.
Thompson, Brian, and Philipp Koehn. 2019.
“Vecalign: Improved Sentence Alignment in Linear
Time and Space.” In Proceedings of the 2019
Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint
Conference on Natural Language Processing(EMNLP-IJCNLP), 1342–48. Hong Kong:
Association for Computational Linguistics.
https://doi.org/10.18653/v1/D19-1136.
Van Hal, Toon. 2013. Ankura. Basiswoordenlijst
Oudgrieks. Antwerpen & Apeldoorn: Garant.

157

Yousef, Tariq, Chiara Palladino, Farnoosh Shamsian,Anise d’Orange Ferreira, and Michel Ferreira dos
Reis. 2022. “An Automatic Model and Gold
Standard for Translation Alignment of Ancient
Greek.” In Proceedings of the Thirteenth Language
Resources and Evaluation Conference, 5894–5905.
Marseille:
European
Language
Resources
Association.
https://aclanthology.org/2022.lrec1.634.
Yousef, Tariq, Chiara Palladino, Farnoosh Shamsian,and Maryam Foradi. 2022. “Translation Alignment
with Ugarit.” Information 13 (2): 65.
https://doi.org/10.3390/info13020065.

158

A Appendices
Lemma

Translations

N

Lemma

TranslationsNαἴσθησις

perception, sensation
sense, faculty
memory,knowledge,
consciousness, opinion
other (unclassified)subvert,overthrow,undermine,
suppress,
destroy, abolish, depose,dissolution
end, break, cease, stoplodge
other (unclassified)another, property, others,
belongs, belonging, else,
possessions
alien, foreign, strange,
strangers
other (unclassified)

118
43
20

ἱστός

raft, keel, ship, mast
weaving, loom, tambour
other (unclassified)attain, prove, accomplish,
gain,
achieve,finish,obtain, complete, reach,stop, fulfil
haste, proceed, renew
other (unclassified)dry, withered, arid, wet,dried, barren, liquid, moist,
dessicant, wateryother (unclassified)

7
6
68
22

καταλύω

ἀλλότριος

ἀνύω

173
109

33
7
234
148

ξηρός

77
210

Lemma

Translations

Nγλῶσσαtongue, mouth, lip
voice, speech, language,
word
other (unclassified)voice, cry, vocal
speech, language, utterance,
word, tongue
soundother (unclassified)say, talk, speech, statement,
said, saying, speak
word
argument, reason
story, report, discourse
formuladiscussion
account
eloquenceother (unclassified)
name,saying,word,
expression, term
verb
sentence, phrase
speechother (unclassified)

177
33

Lemma

Translations

Nβίοτος

life, agemean, victual, property,
substance, gold, wealth,
livelihood
estate, housefoodother (unclassified)meet, encounter
come, goconfront
other (unclassified)infamous, wretch, bad,cruel,
abominably,
wretched,
abominable,rogue, foul, trouble, …
polluted, pestilential, stain,
blood, defiled, unclean,pestilent, filthy
other (unclassified)40
16μιαρός

78

Table 7: Results for randomly chosen terms(frequency band 3).

Table 5: Results for randomly chosen terms(frequency band 1).

ἀπαντάω

4
53
126

φωνή

λόγος

6
4
44
96
42
5
238
54

ῥῆμα
14

21

Table 6: Results for randomly chosen terms(frequency band 2).

152
355
112
60
294
1032
850
520
433
92
82
66
53
3616
108
17
16
4
92

Table 8: Results for linguistic terms.

159

Enhancing State-of-the-Art NLP Models for Classical Arabic
Tariq Yousef

Lisa Mischer

Hamid Reza Hakimi

Maxim Romanov“The Evolution of Islamic Societies (c. 600-1600): From Algorithmic Analysis into Social History”
Emmy Noether Junior Research Group, Hamburg University{firstname.lastname}@uni-hamburg.de

Abstract
Like many other historical languages, ClassicalArabic is hindered by the absence of adequate
training datasets and accurate "off-the-shelf"models that can be readily used in processing
pipelines. In this paper, we discuss our ongoingwork to develop and train deep learning models specially designed to manage various tasks
related to classical Arabic texts. We specifically concentrate on Named Entity Recognition, classification of person relationships, toponym classification, detection of onomastic
section boundaries, onomastic element classification, as well as date recognition and classification. Our efforts aim to confront the difficulties tied to these tasks and to deliver effective
solutions for analyzing classical Arabic texts.
Though this work is still under development,
the preliminary results presented in the paper
suggest excellent to satisfactory performanceof the fine-tuned models, successfully achieving the intended objectives for which they weretrained.

1

Introduction

the creation of the network of knowledge. This
network will serve as the main research frameworkof the project for the study of the social history of
the Islamic world.
Classical Arabic, like all other historical and ancient languages, lacks adequate training datasets
and accurate “off-the-shelf” models that can be directly employed in the processing pipelines. In
light of this, our objective is to make a valuable
contribution to the field by creating comprehensive training datasets for various tasks related toclassical Arabic text processing and analysis. Furthermore, we aim to develop, train, and fine-tune
models that can be easily integrated and sharedwith fellow researchers in the community, facilitating their work and promoting further advancementsin the field of classical Arabic language processing.
In the following sections, we present our inprogress work in developing and training deeplearning models tailored for handling diverse tasks
relevant to classical Arabic texts. Specifically, we
focus on NER, person relationships classification,
toponyms classification, onomastic section boundaries detection, onomastic entities classification, as
well as date recognition and classification.

Arabic chronicles and biographical collections preserve a plethora of information on long-term environmental and societal processes that shaped and2 Related Work
molded Islamic societies. Numerous and extensive,these written texts are the richest “mine” of infor- Recent advancements in deep learning and language modeling have significantly propelled themation and are particularly valuable for the period
development of Natural Language Processing
before the 15th century, for which exceptionally(NLP) models for the Arabic language. Several
few archival documents are available.
Our work focuses on constructing the social his- transformer-based language models are currently
tory of the Islamic world from historical and bio- available and provide state-of-the-art performancein various downstream tasks.
graphical texts that constitute a significant part of
the Arabic written tradition. The project studies
A RA BERT (Antoun et al., 2020) is the firsta vast corpus of digitized texts and relies on a se- transformer-based language model for the Araries of computational methods for identifying andbic language. The CAM E L L AB (Obeid et al.,linking relevant information from the corpus. Cur- 2020) introduced a collection of pre-trained modrently, the project is at the stage of fine-tuning rele- els for several Arabic NLP tasks such as Part-ofvant NLP-based approaches. The work describedspeech (POS) tagging, named entity recognitionin this paper is the methodological foundation for(NER), sentiment analysis, and text classification.
160
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 160–169,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_019

Figure 1: An example illustrating a typical biography.

Figure 2: Translation of the example in Figure 1.

These models have been trained using different
corpora, namely, classical Arabic CA, dialectal
Arabic DA, modern standard Arabic MSA, and the
MIXED corpus which comprises all available corpora. FARASA1 (Abdelali et al., 2016) offers diverse solutions and models for Arabic text processing. It also provides a RESTful API, allowing users
to access its functionalities and leverage languageindependent solutions.
Further, several pre-trained models have been
trained for different downstream tasks such as
A RAT5 (Nagoudi et al., 2021) and A RAGPT2 (Antoun et al., 2021b) for Arabic language generation and understanding; A RA ELECTRA (Antoun
et al., 2021a), ARBERT, and MARBERT (AbdulMageed et al., 2021) for language representation.
The majority of the models mentioned in this context have been trained primarily on modern Arabic texts. However, their applicability to classicalArabic texts varies in terms of performance. No-

tably, the CAM E LBERT-CA model2 is the only
model that is specifically trained on classical Arabic texts. It offers the highest initial performance,
if compared to all the other models. Serving asa cornerstone for our research, this model formed
the basis for our initial annotations and subsequentfine-tuning, allowing us to adapt it to our specifictasks and requirements.

3

CorpusTexts utilized in our project are a sub-corpus of the
OpenITI corpus (Nigst et al., 2023).3 At the moment, our sub-corpus includes 101 multi-volume
texts (c. 71 million tokens), which include approximately 495 thousand biographical records. Mostof these texts—about 70 of them—come from theperiod of 1000–1600 CE and from all the major regions of the Islamic world, spanning from Spain (alAndalus) and North Africa (al-Maġrib), to Egypt
2

CAMeL-Lab/bert-base-arabic-camelbert-ca
https://github.com/openiti/, Open Islamicate Texts Initiative.
3

1

https://farasa.qcri.org/.

161

(Mis.r) and Syria (al-Šām), to Iraq (al-‘Irāq), Iranian provinces (Fārs, Khurasān, etc.) and Central
Asia (Mā-warā’-l-nahr).
Figure 1 illustrates a typical structure of biographies in our corpus (Figure 2 offers a translation for
additional clarity). The onomastic section, which
provides details about the biographee’s name, genealogy, origins, as well as some social and religious background, is typically located at the beginning of the biography. The onomastic section
may also mention members of the immediate andextended family. This section is usually followed
by information about the biographee’s education:with whom they studied, in which cities, and, sometimes, what specifically they studied. The section
on teachers is often followed by a section on biographee’s students, who are listed in a structurally
similar manner. In some biographies, descriptionsof the biographee’s characteristics are given as well,either as the opinion of the main text’s author or as
opinions of other earlier biographers. In the middle, biographies often include other important facts
from the life of biographees. Usually, concluding
sections of biographies provide information on thedate and place of biographee’s death, and, occasionally, the location of biographee’s burial.

After preparing our sub-corpus, we began working with a random subset of biographies. Initially,
we utilized CAM E L L AB models for lemmatization, Named Entity Recognition (NER), and Partof-Speech (POS) tagging to perform the initial annotation steps. Also, in the initial stage, we employed our own rule-based models for date recognition and classification. Following this, we proceeded with the first round of manual refinement
and correction conducted by domain experts, resulting in the successful correction of 1,100 biographies. To ensure consistency and accuracy,
annotators performed cross-validation to ensure
the correctness and consistency of the annotations.
Using this refined dataset, we fine-tuned the NERmodel and trained a model specifically designed todetect boundaries of the onomastic section. Subsequently, we employed these models to annotate
a subset of biographies and repeated the cycle ofmanual correction and fine-tuning. This iterative
process will continue until we achieve a stable performance that meets our expectations, enabling themodels to accurately perform the intended tasks.

5

NER for Classical Arabic

Named Entities Recognition aims to identify entities within the biographies and classify them into
4 Methodology
three main categories, namely, T OPONYM, P ER Manually created data plays a crucial role in the
SON , and M ISC . This task can be viewed as a
training process of machine learning models. Largetoken classification task, where each token in theand accurate training datasets are particularly im- text is assigned a specific label. There is a notableportant to the development of more precise models
distinction between modern Arabic names and trawith improved performance. Historical and classi- ditional Arabic (Islamic) names. Modern names
cal languages pose a unique challenge as there is of- often follow a similar structure to Western names,
ten a lack of readily available training datasets. Cre- including a given name and a surname or family
ating such datasets requires domain experts withname. Traditional Arabic names typically consist
specialized knowledge to perform accurate dataof up to six different elements, though not all ofannotation. Given the limited resources available, these elements have to be present in each case andwe have decided to employ the active learning pro- they may appear in any order. Traditional Arabic
cess (Wang and Hua, 2011) as a solution to generatenames and their elements are explained in moretraining data for the various tasks we aim to tackle. detail in the section 6.2.
Figure 3 illustrates the active learning paradigm we
In the initial phase, we employed CAMeLadapted as an efficient strategy to produce accurate
BERT-CA-NER model to create the first roundtraining data to be used for model training. Figureof annotations. Then, annotators refined the au4 illustrates a lightweight annotation scheme thattomatic annotations and added the missing annowe developed to increase the easiness, speed, andtations according to our annotation scheme. The
accuracy of manual annotation.4
first manual correction round resulted in a train4
ing dataset comprising 1,100 sentences, including
Inspired by markdown, our scheme relies on short opening tags, where the end of the tagged entity is determined by
3,244 persons, 692 toponyms, and 198 miscellaa number indicating the number of tokens. For example, aneous entities. Next, we used this dataset to finetag P3T can be placed at the beginning of a 3-token entity
indicating a person, who was a teacher of the biographee.
tune CAMeLBERT-CA-NER model and used the162Figure 3: Development Process.

Precision
RecallF1TOPONYM
zero-shot fine-tuned
83.45%
92.95%
91.10%
95.94%
87.11%
94.42%PERSONzero-shot fine-tuned
56.56%
94.29%
64.25%
95.35%
60.16%
94.82%MISC
zero-shot fine-tuned
3.28%
71.21%
2.50%
74.21%
2.84%
72.68%

Table 1: NER Model Performance.

trained model to automatically annotate a set of
biographies. Subsequently, the next step resulted
in a bigger training dataset 3,826 sentences containing 10,333 persons, 1,906 toponyms, and 612
miscellaneous entities. Table 1 provides a comparative analysis of the performance between the
CAMeLBERT-CA-NER model (zero-shot) and themost recent fine-tuned model. Notably, there isa significant improvement in the identification of
persons’ names, with an increase of approximately
34.6% in the F1 score. This improvement can be
attributed to the fact that the initial model is trainedon modern Arabic person names which are structurally different if compared to traditional Arabic
names.
Furthermore, our M ISC entities did not overlap much with the M ISC entities of the original
CAMeLBERT-CA-NER model. Our fine-tuned
model started to learn from our annotations, resulting in a promising performance with an F1 scoreof 72.68%.
5.1

Person Relationships Classification

we defined six main classes (as shown in table 2).
Detailed classification of the roles of individuals
mentioned in biographies will allow us to modeland generate complex networks, which will help
the project to study the social organization of communities of Muslims across different periods andregions of the growing Islamic world. Table 2 illustrates the dataset utilized for training. The roleclassification information was manually added toperson tags, which were generated automaticallyusing the fine-tuned NER model.
Based on our reading of annotated biographies,
we have singled out several consistently present
classes of persons. Some of these classes can be described as unimportant, especially those that do notindicate any kind of direct contact to biographees.5
We ended grouping them into the U NDEFINED
class. Aiming for better performance and reliableclassification, we trained two models with a reduced number of classes. The first model classifies
the detected persons in the biography into threemain classes T EACHER, S TUDENT, and U NDE FINED . The second model classifies persons into
four classes FAMILY, O PINION, C ONTACT, andU NDEFINED. Then we merge the classes from both

The NER model achieved great performance in detecting persons in the biographies. Further, we
5Lots of persons appear in so-called “chains of transmiswanted to determine the relationships betweension” (Ar. isnād); most of these individuals never met thethese detected persons and the biographee, a person
biographee and therefore are not part of that biographee’s
in whose biography they appear. For this purpose, immediate social network.
163

Classes
T EACHER (T)S TUDENT (S)O PINION (O)FAMILY (F)C ONTACT (C)U NDEFINED (X)
Total# Entities
2,891
2,117
905
603
562
3,372
10,325

Table 2: Training Dataset for the Persons Classification
Task.

models allowing persons to be associated with twoclasses simultaneously. For instance, a person may
be classified as both S TUDENT and FAMILY, incases, for example, when a biographee is a studentof his father.
Table 3 shows the classification results revealingthat the T EACHER and S TUDENT classes achieved
good performance compared to other classes because the dataset contains a substantial number ofentities belonging to these specific types. Currently,
we are working to expand the training dataset bycorrecting and refining the automatic annotations
created by the models. We believe that having
a bigger training dataset would enhance the performance, especially for the labels of the second
model.
5.2

Toponym Classification

ond dataset that has been pre-annotated with thepreliminary fine-tuned CAM E LBERT-CA-basedmodel and then manually corrected. We used therule-based model as our baseline to compare theresults of the fine-tuned CAM E LBERT-CA-basedmodel to. Table 5 shows the evaluation of the classification task results. For now, the achieved performance of both models—the rule-based one aswell as the fine-tuned CAM E LBERT-CA-based
one—is not satisfying. The low results for thefine-tuned CAM E LBERT-CA model are due toa lack of sufficient training data as our manually labeled dataset only contains 437 classified
toponyms. Since we achieved high-accuracy results for all other token classification tasks with our
fine-tuned CAM E LBERT-CA models with bigger
training datasets, we are currently expanding ourtraining datasets.

6

Onomastic Entity Recognition forClassical Arabic

Each biography starts with a robust onomastic section on the biographee. The onomastic section isparticularly valuable as it gives information on various backgrounds of the biographee. To identify
the respective descriptors in the text, two different
models are required. The first model identifies theboundaries of the onomastic section in the text ofa biography. Applied to the identified onomastic
section, the second model recognizes and classifies
different onomastic elements.

In addition to the recognition of toponyms in the
biographies, we are also interested in the relation
between the biographee and the place mentioned. 6.1 Onomastic Section Boundaries Detection
We defined six main classes, namely, places of: We formulated this problem as a token classifiB IRTH, D EATH, B URIAL, K NOWLEDGE transfer, cation task. In this approach, the model assigns
R ESIDENCE, and U NDEFINED places. For the train- labels to individual tokens within the given text.
ing of the preliminary model, we used a dataset ofSpecifically, we utilized three labels. 1) B-ONOM
1,047 biographies; for the subsequent fine-tuning
represents the beginning of the onomastic section,of the preliminary model, we used 824 biographies. indicating that the token marks the start of the relTable 4 illustrates the datasets utilized for the train- evant section; 2) I-ONOM indicates that the token
ing.
is inside the onomastic section, and 3) O, which isThis model was trained with two datasets. The
assigned to tokens that are outside the onomastic
first dataset was initially annotated with a rule- section.
based model which classified toponyms based on
The model was trained with 3,848 biographies
certain keywords preceding them. This data, with- labeled using with the active learning cycle overout any manual revisions, was then used to train
two rounds. The recent fine-tuned model achieved
our preliminary model. We then used this prelimi- a precision 87.39%, a recall of 88.24%, and annary model to annotate the second dataset. We thenF1 score of 87.81%. However, it is important tomanually corrected this second dataset, creating amention certain challenges and factors that influrevised set of training data.
enced the numerical evaluation results. Largely,
The evaluation of the model is based on this sec- this is due to minor inconsistencies in the manu164

Precision
RecallF1

T EACHER
92.60%
94.44%
93.51%S TUDENT
94.16%
94.65%
94.40%O PINION
95.28%
84.87%
89.78%C ONTACT60.26%
56.88%
58.52%FAMILY80.88%
75.86%
78.29%U NDEFINED
90.19%
93.54%
91.83%

Table 3: Person Relationships Classification Results.

ClassesB IRTH (B)D EATH (D)B URIAL (G)K NOWLEDGE (K)R ESIDENCE (R)U NDEFINED (X)
Total# Entities
23
60
16
77
183
78
437

Table 4: Training Datasets for the Toponym Classification Task.

ally labeled training data, e.g. whether or not toinclude punctuation marks at the end of the onomastic section. Despite these challenges, however,
it is important to stress that for the purposes ofour project, the achieved level of accuracy is perfectly sufficient. The minor discrepancies, which
are mainly due to closing tags being placed afteran extra punctuation mark or an extra token, haveno effect on the accuracy of the final outcome of
the main research task.
6.2

Onomastic Entity Classification

Traditional Arabic (Islamic) names, as they appear
in biographical collections, are quite different fromtheir modern counterparts and are more akin to
the short social profiles of individuals. With up to
six different onomastic elements that may occur
in any order and not all of them are always available, they give us the biographee’s:6 1) “personal
name” (ISM, Ar. ism); 2) the list of mainly maleancestors, which has the structure of “the son of... the son of ... etc.” (NAS, Ar. nasab); 3) “descriptive names”, which describe tribal, religious,
professional, geographical, and other affiliations(NSB, Ar. nisba); 4) a “patronymic” name that has
the form of “The Father of ... / Abū Fulān” or “The
Mother of ... / Umm Fulān” (KUN, Ar. kunya);5) “honorific titles” (LQB, Ar. laqab); and 6) the6
The main source of methodological guidance for thiswork is Malti-Douglas and Fourcade 1976, which summarizes
the main research method of the O NOMASTICON A RABICUM
Project.

“name of renown” (SHR, Ar. šuhra). The “descriptive names” (NSB, Ar. nisba) are the most valuable
onomastic element for the goals of our project asthey allow us to model different social and historical processes in the context of the development of
the Islamic world.
Once the onomastic section within the biographical text has been identified, our next objective is torecognize and classify discrete onomastic elements
present within it. For this purpose, we trained a token classification model that distinguished among
the six main classes, described above (ISM, NSB,
NAS, SHR, KUN, and LQB).
First, we pre-annotated the initial data set with
our rule-based model (Table 6), which was built on
data from the O NOMASTICON A RABICUM (Institute de Recherche et d’Histoire des Texts).7 Morespecifically, we developed an onomastic gazetteer
from data elements which were classified as ism,kunya, laqab, nisba and šuhra in the descriptionsof persons collected in the O NOMASTICON A RA BICUM . Further, the gazetteer included technical terms, used in texts to explain the spelling ofrare names. We used our rule-based model to assign classes to all tokens inside onomastic sections;these assignments were then manually corrected.
The model was trained with 2,011 biographies. Table 7 shows the training results. Notably, ISM
achieved the best performance since it is unique ineach onomastic section and it comes almost always
as the first entity in the section. The training process adhered to the project’s active learning cycle.
The initial training data was generated by manually correcting labels derived from the rule-basedmodel. Subsequently, the onomastic entity recognition model was trained using this corrected sample.
The subsequent samples were then annotated using
the onomastic entity recognition model.

7

Date Recognition, Classification, andParsing

The aim of this model is threefold: 1) recognitionof dates; 2) classification of dates; 3) parsing dates
7

165

See, https://onomasticon.irht.cnrs.fr.

B IRTHD EATHB URIALK NOWLEDGE
R ESIDENCEU NDEFINED

Precision
78.57%
85.45%
77.78%
68.25%
88.59%
36.99%

rule-based
Recall100%
78.33%
77.78%
59.72%
52.38%
75%

F1
88%81.74%
77.78%
63.70%
65.84%
49.54%

Precision
45.45%
50%
0%96.55%
57.14%
41.67%

fine-tuned
Recall100%58.33%
0%73.68%
77.61%
22.73%

F1
62.5%
53.85%
0%83.58%
65.82%
29.41%

Table 5: Toponym Classification Results.

Classes
ISM
NSB
NAS
KUN
LQB
SHR
Total

# Entities
1,888
3,260
3,856
910
285
179
10,378

Table 6: Training Dataset for the Onomastic Entities
Classification Task.

to numerical values. The research focus of the
project is on the period of c. 600-1600 C.E. andinformation from the texts is assigned to a certain
point in time somewhere within this period. We areparticularly interested in what kind of information—
historical events—can be associated with specificpoints in time.
First, the model searches the Arabic text witha regular expression (regex) which will matchphrases reporting on dates. The regex matches
days, days of the week, months, and years. Additionally, it captures ten preceding tokens, which
are considered the context of a date. At the moment, we are primarily interested in years and their
thematic contexts.8 When the regex finds an occurrence of a date phrase, it usually returns two
main groups of elements. One of the groups is thecontext; another one is a series of spelled-out numerals of the date statement, including ones, tens
(decades), hundreds (centuries), and, in late texts,
a thousand (for the first millennium). The model
then uses a dictionary that returns numerical values of date statement element. Summing up thesenumerical values gives us the actual value of thedate. Additional regex is then applied to the datecontext to check if it has any of the most common
contextual vocabulary. For example, tokens likewulida (he was born) or wulidat (she was born)
are used to classify dates as dates of birth. If thecontext contains more than one term from the dateclassification dictionary, we use the one closest to
the date statement. We defined six main classesof dates, which are B IRTH, D EATH, K NOWLEDGE
transfer, appointment or termination of an O FFICE,P ILGRIMAGE, and U NDEFINED dates.
Table 9 shows the classes and their number ofoccurrences in the test dataset. The model was
evaluated with 1,047 biographies. The numericalvalue extraction by this model achieved a mean
average percentage error (MAPE) of 1.55% and amean average error (MAE) of 4.76 years if the datephrase was correctly recognized as such.
Table 8 shows the results for this rule-basedmodel. One of the reasons for a low precision
for the class U NDEFINED is that this class is assigned whenever no other indicator was in the ten
preceding tokens. Those ten tokens are not enoughfor every case, so sometimes the indicator was the
11th preceding token, and therefore the date was
mistakenly classified as U NDEFINED. Overall, theresults by this rule-based model show promising
performance, especially since the parsing is alreadyworking very well for recognized dates. Still, daterecognition presents a lot of obstacles.

So far, the model only returns information about
a date if it’s explicitly stated in the recognizedphrase. However, not all information is always
presented explicitly. A common instance is theomission of the century, as authors often expect
readers to infer the exact century from the context.
Consequently, we need to enhance our model toderive any missing data from other dates provided
in the biography, headers of chapters containing
8
Year statements are the most frequent type of date statethe biography (especially when biographies arements; more precise indications of time are significantly lessfrequent and, structurally, are more diverse and less consistent.
grouped into periods—a common practice in our
166Precision
RecallF1

ISM
99.07%
97.72%
98.39%

KUN
99.42%
97.73%
98.57%

NSB
97.15%
97.99%
97.57%

NAS
96.55%
98.07%
97.30%

LQB
75.32%
80.56%
77.85%

SHR
79.07%
70.83%
74.73%

Table 7: Onomastic Elements Classification Results.

Precision
RecallF1

B IRTH
96.03%
90.98%
93.44%

D EATH
94.64%
84.57%
89.33%K NOWLEDGE
89.66%
59.09%
71.23%O FFICE66.67%
16.67%
26.67%P ILGRIMAGE
80.00%
50.00%
61.54%U NDEFINED
52.41%
81.46%
62.20%

Table 8: Date Recognition and Classification Results.

ClassesB IRTH (B)D EATH (D)K NOWLEDGE (K)O FFICE (O)P ILGRIMAGE (P)U NDEFINED (X)
Total# Entities
133
376
44
12
8
85
658

fine-tuned an NER model, specifically tailored forhistorical and biographical texts in classical Arabic, which exhibits excellent performance in detecting persons and toponyms. Moreover, we trainedmodels to further classify detected persons, basedon how they are related to the biographee. Thismodel achieved good performance, particularly inthe classification of teachers and students within
the biographical context. The model for toponym
Table 9: Evaluation Dataset for the Date Classification
classification is still under development as we are
Task.
lacking sufficient training data. We are currently
working on increasing our training dataset for thistask.
sources), or the scope of the historical source where
Further, we trained a boundaries detection model
the biography was found. We are still assessing the
to locate the onomastic section inside biographies,
most efficient approach to implement this disamand yet another model that identifies onomastic
biguation.
elements within that section. The model for date
Another challenge involves handling date staterecognition, classification, and parsing achieved
ments that refer to periods or approximate years.
promising results for the main goals of the project.
For example, instead of exact numbers, we may
Still, date recognition is not a trivial task and wefind words like ba‘d. and nayyif, which refer to an
are researching ways to overcome limitations such
unspecified year within a specified decade. Whileas the missing centuries.
these date statements cannot be translated into
Although this work is still in progress, the preprecise numerical values, this limitation does notliminary results reported in the paper indicate theseverely impact our project. Given the extensive
excellent-to-satisfactory performance of the fineperiod we are studying, we typically operate on thetuned models, effectively meeting the intended goal
granularity of decades, rounding exact years to thefor which they were trained. However, our ongonearest decade when necessary.
ing efforts involve expanding the training datasets
Additionally, authors sometimes report alternaand further fine-tuning the models with the aim oftive dates for the same event, either by detailingachieving even better results.
both dates fully or by abbreviating the second date.
Finally, our contribution extends beyond theIn such cases, we make an effort to collect andtrained models themselves. We have also develprocess both dates.
oped and curated valuable training datasets thatcan serve as a resource for other researchers and
8 Conclusions and Future Work
contribute to the advancement of work in the field
In the preceding sections, we have outlined ourof classical Arabic. These datasets provide a founefforts in adapting existing state-of-the-art Ara- dation for further exploration and improvements in
bic NLP models to specific research tasks. We
the current models.
167

Figure 4: Example of automatically annotated biography from al-Sahāwı̄’s al-D
. aw’ al-lāmi‘.
¯

Our future work can be summarized as follows.
In the short term, our primary focus lies in generating additional training data to facilitate further
fine-tuning of the models. This iterative process is
aimed at continuously improving the performanceof the models until reaching a stage where theycan effectively annotate the entire corpus. Figure
4 illustrates an exemplar output of our annotation
pipeline, wherein the annotated text has undergone
processing by all the discussed models. After annotating all biographies in our corpus and extracting
all relevant metadata (onomastic elements, persons,
toponyms, dates, etc.), our subsequent objective
is to organize this information into networks comprised of overlapping thematic clusters. These thematic clusters will serve as an analytical framework, enabling us to explore and derive insightsfrom the interconnections and relationships among
various social, professional, and religious groups,
within extensive historical and geographical contexts as they are recorded in our vast corpus. Additionally, the project explores the development ofthese networks through spatial and temporal analysis, which is grounded in the recognition of dates
and toponyms. Overall, this network will serveas the main research framework of the project forthe study of the social history of the Islamic world.
Further, this project will help to identify weakly
researched topics in the field of Arabic studies andat the same time provide a new research tool for
fellow researchers to start working on these topics.

Limitations

present additional challenges. To tackle these obstacles, we have adopted an active learning development cycle, which allows us to efficiently andrapidly generate training data. Furthermore, in certain cases, we decided to reduce the number oflabels or split the labels into two sets and train two
separate models instead of one model in order toget better performance.

Acknowledgement
This research is a part of the work within the Emmy
Noether Research Group (№445975300), “TheEvolution of Islamic Societies (c.600-1600 CE): Algorithmic Analysis into Social History” (EIS1600),
funded by the German Research Foundation (DFG)
and hosted at Universität Hamburg.

References
Ahmed Abdelali, Kareem Darwish, Nadir Durrani, andHamdy Mubarak. 2016. Farasa: A fast and furious
segmenter for Arabic. In 15th Annual Conferenceof the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies, pages 11–16. Association for Computational Linguistics.
Muhammad Abdul-Mageed, AbdelRahim Elmadany,
et al. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105.

Ancient and classical languages, including classi- Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.
Arabert: Transformer-based model for arabic lancal Arabic, face significant challenges in terms ofguage understanding. In LREC 2020 Workshop Lanthe availability of adequate training datasets andguage Resources and Evaluation Conference 11–16
pre-trained models. Creating such datasets is a
May 2020, page 9.
non-trivial task, demanding considerable time andresources. It necessitates the involvement of do- Wissam Antoun, Fady Baly, and Hazem Hajj. 2021a.
AraELECTRA: Pre-training text discriminators formain experts possessing the requisite knowledge to
Arabic language understanding. In Proceedings ofperform annotations in accordance with prescribed
the Sixth Arabic Natural Language Processing Workguidelines or annotation schemes. The scale of theshop, pages 191–195, Kyiv, Ukraine (Virtual). Assodataset and the complexity of classification tasks
ciation for Computational Linguistics.
168

Wissam Antoun, Fady Baly, and Hazem Hajj. 2021b.
AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages
196–207, Kyiv, Ukraine (Virtual). Association for
Computational Linguistics.
Institute de Recherche et d’Histoire des Texts. Onomasticon Arabicum.
Fedwa Malti-Douglas and Geneviève Fourcade. 1976.
The Treatment by Computer of Medieval Arabic Biographical Data: An Introduction and Guide to the
Onomasticum [i.e., Onomasticon] Arabicum. Number 6 in Série Onomasticon Arabicum. Editions duCentre national de la recherche scientifique.
E Moatez Billah Nagoudi, A Elmadany, and M AbdulMageed. 2021. Arat5: Text-to-text transformers
for Arabic language understanding and generation.
arXiv preprint arXiv:2109.12068.
Lorenz Nigst, Maxim Romanov, Sarah Bowen Savant, Masoumeh Seydi, and Peter Verkinderen. 2023.
OpenITI: a Machine-Readable Corpus of Islamicate
Texts.
Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima
Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl
Eryani, Alexander Erdmann, and Nizar Habash. 2020.
CAMeL tools: An open source python toolkit for
Arabic natural language processing. In Proceedingsof the Twelfth Language Resources and Evaluation
Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
Meng Wang and Xian-Sheng Hua. 2011. Active learning in multimedia annotation and retrieval: A survey.
ACM Transactions on Intelligent Systems and Technology (TIST), 2(2):1–21.

169

Logion: Machine-Learning Based Detection and Correction of TextualErrors in Greek Philology
1Charlie Cowen-Breen1 , Creston Brooks2 , Johannes Haubold3 , Barbara Graziosi3
Department of Applied Mathematics and Theoretical Physics, University of Cambridge
ccbreen@mit.edu
2
Department of Computer Science, Princeton University
3
Department of Classics, Princeton University{cabrooks, jhaubold, barbara.graziosi}@princeton.edu
Abstract

We present statistical and machine-learning
based techniques for detecting and correcting errors in text and apply them to the challenge of textual corruption in Greek philology.
Most ancient Greek texts reach us through along process of copying, in relay, from earlier
manuscripts (now lost). In this process of textual transmission, copying errors tend to accrue.
After training a BERT model on the largest premodern Greek dataset used for this purpose todate, we identify and correct previously undetected errors made by scribes in the process of
textual transmission, in what is, to our knowledge, the first successful identification of such
errors via machine learning. The premodern
Greek BERT model we train is available foruse at https://huggingface.co/cabrooks/
LOGION-base.

1

Introduction

or function, etc. Discovery of such errors typically requires a sophisticated understanding of an
author’s writing tendencies and the context of aparticular text.
This motivates the use of contextual language
models for the detection of scribal errors. In thispaper, we propose Logion, a framework for detecting scribal errors based on contextual language
models.1 Logion consists of three stages: in thefirst stage, a contextual language model learns conditional word distributions for a selected corpus;
in the second stage, potential errors are identified
according to statistics derived from the learned distribution; lastly, in the third stage, corrections areproposed for the words identified as potentiallyerroneous. While not all words flagged by the algorithm will be genuine scribal errors, a “shortlist”
of potential scribal errors can point philologists
to previously undetected errors which, after being
corrected, restore the original meaning.
To summarize, our main contributions are as
follows:

Ancient texts have been passed down by scribes
over hundreds of years, in a process known as textual transmission. Scribes occasionally make mis• We present a premodern Greek BERT trained
takes, some of which lie undiscovered to this day.
with what we believe to be the largest dataset
As unchecked errors have the potential to change
used for this purpose to date.
the meaning of a text, finding and correcting scribal• We propose Logion, a framework for scribal
errors is a central aim in Greek philology.
error detection and emendation based on conIn a proof-of-concept paper, we presented the
textual language models.2
first scribal mistakes detected by contextual lan• We study the effectiveness of Logion at detectguage models (Graziosi et al., 2023). In this paper,ing artificially generated scribal errors, andwe describe and study the approaches used to arshowcase real errors which it has already disrive at those results and evaluate the algorithm’s
covered.
effectiveness on artificially generated errors.
In this paper, we concentrate on the discovery
Prior to Graziosi et al. (2023), to the best ofof scribal errors in the works of the Byzantine auour knowledge, scribal errors were found only byhand—that is, with domain experts carefully read- thor Michael Psellus, who is a convenient choice at
1
ing the text until they find potential errors, and then
The name “Logion” derives from an ancient Greek word
meaning “oracle” to emphasize that model-generated results
using database searches to assess textual problems
from human interpretation.
and propose solutions. These errors include sim- benefit
2
Our code and shareable data is available atplifying difficult expressions, omissions, replacing
https://github.com/charliecb/Logion/tree/main/
one word for another with a similar sound, shape, error_detection_and_correction.
170
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 170–178,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_020

a proof-of-concept stage for philological reasons.
However, we remark that these methods may be applied to any premodern text passed down by scribes,
provided sufficient data is available.

2

specific works of interest, as outlined in subsection 3.1.
The second and third stages harness the learned
distributions of the fine-tuned BERT to detect andemend errors, respectively. Before describing thelater stages in full, we briefly recount the conditional distributions which BERT learns. Given asequence of tokens w1 , . . . , wn , consider a single
token wi and denote the surrounding (bidirectional)
context by w−i = (w1 , . . . , wi−1 , wi+1 , . . . , wn ).
From the masked-language model (MLM) trainingtask, the model learns the distribution

Related Work

In a study also related to the restoration of premodern Greek, Assael et al. (2022) train a multi-task
transformer-based model to date, place, and fillgaps in ancient Greek inscriptions. Inscriptions
display the original text on stone, pottery, or othermedia, whereas most of what survives from antiquity reaches us via a long tradition of hand-copying
from earlier exemplars. For this reason, Assael
et al. (2022) focus on gaps in inscriptions causedby physical damage but not on copying errors intexts.
In English and other modern languages, previous work on textual error detection has typicallyfocused on spelling and grammar checking (Etoori
et al., 2018) (Ganiz et al., 2020) (Naber, 2003),
while textual errors introduced by scribes are often more complex (e.g. Figure 4). For this reason, identifying scribal errors more closely aligns
with out-of-distribution detection, in which thetask is to discern whether samples—in our case,words—are likely to have been generated by agiven distribution—in our case, the author’s bodyof work—or instead are out of distribution—i.e.,
the result of an error in transmission. Ren et al.
(2019) propose the use of likelihood ratios to determine out-of-distribution samples for images andgenomic sequences, a metric which we slightly
modify. Sometimes error detection is validated byphilological experts; at other times it is confirmed
directly by manuscripts that were either sidelined
or misread by previous scholars in the course of
preparing the first or subsequent printed editions.
To our knowledge, this paper is the first to identify
and correct scribal errors via machine learning.

3p(w|w−i )(1)over tokens w which occur in the ith position of asentence when surrounded by context w−i (Devlin
et al., 2019). For inference on words comprised ofmultiple tokens, we extend p to a distribution oversequences of tokens via beam search. Therefore,
in what follows, when (w1 , . . . , wk ) is a sequenceof words, rather than tokens, we will let p(w|w−i )denote the corresponding distribution over words
w which is derived from Expression 1 via beam
search.
In the second and third stages, described in subsection 3.2 and subsection 3.3, respectively, existing statistical theory is applied to the learned
distribution p to determine the tokens which aremost likely to contain errors, and subsequently topropose emendations. The stages are illustrated
together in Figure 1.
3.1

BERT Training

We initially trained a BERT model on a dataset of
6.4 million words of premodern Greek, which we
gratefully received from Pranaydeep Singh. This isthe base model used in Graziosi et al. (2023). Singhet al. (2021) assembled this data from open-source
databases, such as the Perseus Digital Library andthe First1KGreek corpus, in the course of training
a BERT model for ancient and medieval Greek. We
subsequently assembled a much larger dataset ofroughly 70 million words.4 We divided this data
into a 90-10 train-test split and trained the BERT
model using two NVIDIA A100 GPUs for 200
epochs until validation loss stabilized. To prepare
the tokenized input, we maximized the amount ofpunctuation-separated text in each input, up to alimit of 512 input tokens. We used a batch size of

Methodology

Logion is a three-stage framework for the discovery
and emendation of textual errors in a given corpus.3
The initial stage involves training a BERT model,
which undergoes broad pre-training on premodern
Greek text followed by subsequent fine-tuning on
3

It also has other philological functionalities we do notdescribe here, but which we explore in Cowen-Breen et al.
(2023) and Graziosi et al. (2023).

4

171
2

See Appendix A.

16 and a mask ratio of 0.15.5
To evaluate the impact of Singh et al.’s pretraining on modern Greek, we trained two models,
one with random initialization and one initialized
from Singh et al. (2021)’s Ancient Greek BERT.
Both times, we used Singh et al. (2021)’s tokenizer
which had been created for Modern Greek subwords, since they themselves fine-tuned a Modern
Greek BERT. We find that both trainings converge
to the same validation loss after a small numberof epochs, indicating no discernible benefit frompre-training on Modern Greek. A future model
may be more effective with a tokenization optimized for Ancient Greek: see section 6. The resulting premodern Greek BERT model is available
for use at https://huggingface.co/cabrooks/
LOGION-base.
To learn more accurately the distribution of particular works in which we would like to identify
errors, we then perform a fine-tuning of the broadly
trained premodern Greek BERT. We partition selected works into a 90-10 train-test split and continue training using the MLM objective until validation loss stabilizes.

theoretical guarantees, as shown by Proposition 1
in subsubsection 3.2.3.

3.2

3.2.2 Rare Words
While low chance may seem to be the most intuitive indicator of errors, we find that the othertwo metrics are helpful for avoiding false positives.
If chance were the only metric considered, genuine but rare words would be incorrectly flagged
as errors.7 Moreover, scribal errors are sometimes
graphically or phonetically similar to the correct
text. Thus, we choose to flag low-chance portionsof text which are close in sound or shape to highconfidence model suggestions. As experimentallydemonstrated in subsection 4.3, accuracy at detecting artificially generated errors is greatly improvedby considering both chance and confidence, in comparison to using either metric alone.

1. Given a word wi with (bidirectional) context
w−i , the chance of word i is defined asp(wi |w−i )that is, the probability that the word exists in
its given context, as determined by the model.
2. The model’s confidence at word i is definedasmax p(w|w−i )word wthat is, the probability of the top suggested
replacement in the given context around position i, as determined by the model.
3. The scribal distance at word i is defined as
d wi , argmax p(w|w−i )word wwhere d(x, y) denotes the Levenshtein distance between strings x and y.

Error Detection

In this section, we show how certain metrics derived from the distribution p learned by BERT function as indicators of the likelihood that a given word
contains an error.
Given a corpus, the goal is to flag words which
are most likely to be erroneous, in order to provide
domain experts with a shortlist of potential errors
and emendations. A word is flagged if it satisfies
certain conditions based on the metrics we define
below.
3.2.1 Metrics
We propose three metrics for flagging potential errors. Additional metrics may achieve higher accuracy at error detection in the future.6 That said, themetrics presented here have the benefit of certain

3.2.3 Combining Metrics
Depending on the application of interest, one can
combine metrics in various ways to generate error
flags. In what follows, we present two ranking
schemes that appear to be effective at finding either
real scribal errors or artificial errors introduced inorder to test the effectiveness of our approach.

5

At the task of 1-token prediction on our test set, the model
achieves 84.4% top-1 accuracy and 95.2% top-5 accuracy, andobtains a low pseudo-perplexity of 2.162 (Wang and Cho,2019). We note that these metrics are dependent on specifictokenizations and should not be compared to models withdifferent tokenizations.
6
These metrics are certainly not the only ones that would
lead a human philologist to consider a word suspicious, butthey serve for now as a useful tool, as evidenced by Graziosi
et al. (2023). In the future, we expect that more end-to-end
methods—such as training for detecting errors directly—andregressions accounting for more metrics will outperform whatis shown here.

7

This is because chance considers only the absolute probability of a word wi in context w−i , instead of the relative
probability when compared to plausible alternatives. Such
relative probabilities are achieved by the chance-confidence
ratio, which we present in the next section.

172
3

Figure 1: Logion pipeline. Here, the given text has been corrupted with a change from “jumps” to “bumps.” In thefirst stage (left), a BERT model is fine-tuned to learn p(·|w−i ) for a given corpus. In the second stage (middle), to
identify the error, Logion computes the CCR of each word in the sentence (this is depicted as the brightness of eachword) and identifies “bumps” as having the lowest CCR. In the third stage (right), to correct the error, a change from“bumps” to “jumps” is proposed based on the learned distribution, when restricted to words which are one-charactermodifications of “bumps” (here, k = 1). Error and emendation proposals are then vetted by domain experts.

in Appendix B.8

Chance-confidence ratio rankings

Proposition 1. (Correspondence between CCR
and relative probabilities of sentences) Let p(s)be a joint distribution on sentences s. Given asentence s, suppose that

Suppose that we are given a sequence of wordss = (w1 , . . . , wn ). As a measure of likelihood forthe ith word to be an error, we propose the quantity
ρi (s) :=p(wi |w−i )maxw∈Wk (wi ) p(w|w−i )(2)s∗ = argmax p(s′ )s′ ∈W1 (s)Then s∗ = s if and only if ρi (s) > 1 for all i.
Moreover, if s∗ ̸= s and i∗ is the word index atwhich s∗ differs from s, thenwhere
Wk (x) = {y : d(x, y) ≤ k}i∗ = argmin ρi (s)and k is a fixed positive integer. To motivate ρi (s)ias a measure of the likelihood that the ith word is∗erroneous, we note that, intuitively, ρi (s) is small Furthermore, s is obtained by replacing wi∗
with the model top suggestion at i∗ restricted towhen its numerator is small and its denominator is
W1 (wi∗ ).
large, i.e. when word i has low chance and is closein Levenshtein to a high-confidence model suggesIn other words, the proposition states that, assumtion. By the discussion in subsubsection 3.2.2, then, ing a joint probability distribution exists,9 the CCR
we expect erroneous words to correlate with words
indicates the one-character alteration of s which
for which ρi (s) is small.
the model determines most likely to have been writWe will refer to ρi (s) as the chance-confidence
ten by the author.10 This motivates ranking wordsthratio (CCR) of the i word of s. This name de- by their CCR (i.e., by the values ρi (s)) in order torives from the fact that if the distributions p(·|w−i ) detect plausible errors. In section 4, we artificiallyused to compute chance and confidence are further
generate errors and find that the word with index
conditioned on the event that d(·, wi ) ≤ k, then theargmin ρi (s)ratio of the (conditioned) chance and confidence isiequal to ρi (s).
8
For alterations of k > 1 characters, the proposition genOne natural motivation for the CCR is the fol- eralizes to the corresponding statement with the assumption
instead that s′ lies in the set of all sentences which differ fromlowing: suppose we are allowed to change only one
s in a single word by at most k characters.
character of a sentence and want to do so in such
9
For methods of constructing joint distributions froma way that it most resembles what a given authormasked language model conditionals, see Torroba Hennigen
and Kim (2023).
has written. Then, the character which we should
10
That said, care must be taken in concluding that s∗ waschange is exactly the one which would result in thethe original formulation of the author. Scribal errors may skew
smallest CCR of the affected word. This is formal- toward easier readings of the text and may thus increase p.
ized in the following proposition, which we proveThis is an effect we consider further in section 6.
173
4

indeed contains an error 90% of the time, showing that such rankings are effective at detecting
artificially generated errors (see Table 1 and Figure 3). Moreover, in 98% of such instances, thetop model suggestion at the erroneous word wi ,argmaxw∈W1 (wi ) p(w|w−i ), recovers the correct
ground-truth word.
Another interpretation of ρi is that it is thelikelihood-ratio statistic, assuming the prior on
w which is uniform on Wk and vanishes elsewhere. In this sense, the CCR builds on Ren et al.
(2019) and Gangal et al. (2020), which achieved
success at detecting out-of-distribution samples
with the likelihood-ratio statistic. This interpretation amounts to treating the ground truth word at
position i as an unknown parameter w, the valueof which determines the conditional distributionp(w−i |w) of the surrounding words. In this case—again assuming that scribes only make errors which
do not exceed a Levenshtein distance of k from theoriginal text—we can formulate error detection as
the hypothesis testing problem
H0:

The word wi is correct as written.

H1:

The original word has been alteredwhich pass the threshold, the model’s top suggestion is the same before and after thresholding for
scribal distance.
Thresholds determine the precision and recallof the model when it is used to identify erroneouswords. For applications where one wishes to finda list of strong candidates for erroneous words (i.e.
high precision is desirable), one can set the confidence threshold to be high (e.g. 90%) and thechance and scribal distance thresholds to be low(e.g. 10−6 and 2, respectively). For applications inwhich one wishes to find more corrupted words andcan tolerate sifting through weaker candidates (i.e.
high recall is desirable), one can set the confidence
threshold to be low (e.g. 50%) and the chance andscribal distance thresholds to be high (e.g. 10−4
and 4, respectively).
3.3

Emendation

Once a subset of the corpus has been flagged as
potentially erroneous, we can easily propose emendations via Proposition 1. In the case k = 1, forexample, Proposition 1 suggests that the highest
probability one-character alteration of the input text
is found by replacing the flagged word (say, wi )with the model top suggestion at position i whenrestricted to only one-character alterations:and lies in Wk \ {wi }.
The corresponding likelihood-ratio statistic for thishypothesis test is given byargmax p(w|w−i )w∈W1 (wi )This is the scheme which is employed for the experiments in the following section. Since producing
more than one suggested emendation can be helpful for domain experts, in practice, we report any
number of the most likely words w ∈ Wk (wi ) according to the distributionp(w−i |wi )maxw∈Wk p(w−i |w)
In a Bayesian framework with uniform prior on
Wk , one can see that this is equivalent to the CCR.
In Figure 2 (i), we plot the distribution of thelikelihood-ratio statistic under the hypotheses H0
and H1 . The distributions under each hypothesis
are distinct, allowing for formal hypothesis testing
via the likelihood-ratio test.

p(w|w−i )for any k ≥ 1.

4

Thresholding

Experiments

In some applications, thresholding for each met- In this section, we study the effectiveness of theric individually can provide more flexibility for
proposed approach at finding artificially generated
generating a shortlist of errors. In Graziosi et al. errors, while noting that the proposed approach has(2023), the results were generated by thresholding
already resulted in the discovery of real errors, as
for confidence of at least 50%, scribal distance atoutlined in Graziosi et al. (2023). A sample of thatmost 3, and ranking the remaining words in orderwork is shown in section 5.
of increasing chance. A selection of flags resulting
4.1 Artificially Generated Errors
from this scheme is shown in section 5. The choiceof a 50% threshold for confidence is convenient
Artificially generating scribal errors is made difbecause it respects the property that, among words
ficult by the fact that the data-generating mecha174
5

(i)

1.0

(ii)

0.8

0.6

0.6

0.4

0.4

0.2

0.2
0.0

(iii)

0.8

0

10

20

30

40

0.0

0

10

20

30

40

7
6
5
4
3
2
1
0

H0
H1

0.0

0.5

1.0

1.5

2.0

2.5

Figure 2: Distribution of metrics under hypotheses H0 and H1 . The metrics shown are (i) CCR as in Expression 2,(ii) chance, (iii) confidence. Each horizontal scale is − log T , where T is the metric associated with the plot. Here,
H1 is modeled by the scheme to generate artificial errors described in section 4, where we restrict to only single
token replacements in order to produce samples efficiently. Each plot contains roughly 500,000 samples from H0
and 1,000 samples from H1 .

nism is inherently complex and difficult to reproduce. Such errors are often dependent on individual
scribes, the context in which they were working,and their interest in what they were copying: scribal
errors can be quite varied and complex.
That said, some errors are fairly banal, such aschanges in pronunciation that can result in spelling
errors due to phenomena such as itacism.11 For
the purpose of this simulation, we generate scribal
errors in the following manner: within every paragraph, we replace a randomly chosen characterwith another random character such that the modified word is in the dictionary of words used by theauthor at least ten times. If the modified word does
not meet this criterion, we continue substituting
characters until it does. This process ensures that asimple dictionary check could not catch the errors
we generate.
4.2

do not involve all three metrics: (i) ranking by confidence when restricted to scribal distance 1, and(ii) ranking by chance alone.
1.0

CDF

0.9
0.8
0.7
chance-confidence ratio
chanceconfidence

0.6
0.5
0%

2%

4%
6%
8%
Percentile of erroneous word

10%

Figure 3: Artificial errors are inserted into one word
per paragraph (average 230 words). The metrics ofevery word in the paragraph are computed (CCR, chancealone, confidence restricted to scribal distance 1), andthe percentile of the erroneous word is measured whenranked by metric within the paragraph. This plot shows
the cumulative distribution function of these percentiles.
90.5% of words rank first (i.e. 0th percentile) in theirparagraph by CCR, and 59.7% of words rank first by
chance, in agreement with Table 1. Error bands arecomputed via the DKW inequality with 99% coverage
probability.

Results

Within every paragraph, we rank words by CCR(Equation 2), as described in subsubsection 3.2.3
with k = 1. Out of 615 randomly generated
instances, the erroneous word ranked first 556
times, yielding a 90.5% top-1 accuracy. Among
instances in which the erroneous word ranked first,
the ground-truth word was the top suggested replacement for the erroneous word 98.1% of thetime. The results are summarized in Table 1.

Figure 3 shows the distribution of artificially corrupted words when ranked by the ranking schemes
proposed. The distribution of each ranking scheme
is heavily left-skewed: more than 85% of erroneous
4.3 Ablation Studywords lie in the bottom 10% of words when ranked
To demonstrate that consideration of all three met- by any metric. This suggests that each of the rankings proposed correlates with artificial errors.
rics introduced in Section 2.3 improves accuracy at
detecting artificial errors, here we compare ranking
However, when ranking by either scheme (i) orby CCR to two alternative ranking schemes whichscheme (ii), the corrupted word is ranked first in
less than 60% of cases; in comparison, the CCR
11
The term itacism describes a confusion between different
metric ranks the corrupted word first in 90.5% ofvowels and diphthongs, all of which came to be pronounced/i/.
cases (see Table 1). Therefore, we conclude that
175
6

Accuracy
Top-1
Top-5
Top-10

CCR
90.5%
95.9%
97.6%

Chance alone
59.7%
88.2%
93.1%

Confidence alone
54.2%
81.1%
83.5%

Table 1: Accuracy at detecting a single artificial error out of 230 words according to different schemes of combining
metrics. Best performance is achieved by using CCR, although chance is also a viable metric. Since the task is togenerate shortlists of potential errors for review (and domain experts can often verify quickly whether a flaggedword is a true error) top-10 accuracy is a significant metric here.

ranking words by either scheme (i) or scheme (ii)is less effective in identifying artificial errors thanranking by the CCR.

5

led to the realization that the manuscript actuallyreports "ἡδὺ", not "ἦ δὲ". The sentence can now
be translated as follows: "This man, having been
king twice, prayed for a third and fourth term. For,he said, ‘sun after clouds is sweet’." In this case,then, the error turned out to be not in the earlymanuscript but in subsequent readings of it.

Philologically Significant Results

The metrics presented here have successfully identified errors that were previously undetected, ranging
from scribal errors in the manuscripts, typographical errors in printed editions, and errors caused bydigitization. These findings underwent philological peer review and have been published in TAPA,
the research journal of the Society for ClassicalStudies.
In Graziosi et al. (2023), we show at proofof-concept stage how the approaches introduced
here improve on previous knowledge of premodern
Greek texts by identifying and sometimes solving several different philological problems. For
detailed examples and further discussion, please
see Graziosi et al. (2023). Below, we offer a single example to illustrate one type of error which
may be detected (in this case a misreading of themanuscript on the part of modern scholars rather
than an actual error in the manuscript itself).
In Psellus’s Hist. brev. at lines 81.89–90, Aerts’
edition reads:

6

Future Work

One major line of future work concerns developing an application which is adopted by domain experts and used to assist their work. Given any text,
such an application would be capable of generating
shortlists of suspected errors and proposed emendations for review. Future research directions inthis area include developing efficient and linguistically motivated sub-word tokenization schemes andthe capability to include or exclude sections of thedataset from consideration at inference time: this isrelevant when one is interested in performing error
detection on a section of text which was includedin the training set without retraining the model
entirely. In working towards the latter goal, one
promising architecture is DEMix, which enables
dynamic expert mixtures at inference time (Gururangan et al., 2021). Another idea for future work,and one which sets scribal error detection apartfrom traditional error detection, concerns treating
scribal modifications as diffusion processes. As
scribal errors are often contextually driven, text
altered by scribes may paradoxically evaluate tohaving higher probability than the original text.12
On this view, then, the text evolves over time as adiffusion process with a transition kernel derived
from p (for example, one option is to model the
trajectory of the text by Gibbs sampling accordingοὗτος δὶς βασιλεύσας ηὔχετο καὶ τρὶςκαὶ τετράκις· ἦ δὲ γάρ, φησι, μετὰ νέφος ὁ ἥλιος.
Houtos dis basileusas ēucheto kai tris kai
tetrakis: ē de gar, fēsi, meta nephos, hohēlios.
‘This man, having been king twice,prayed for a third and fourth term. For
indeed, he said, there is sun after clouds.’

When thresholding for confidence and scribal
12
In philology, this is the principle known as lectio difficildistance, the token δε was one of the lowest chance
ior potior. Because “the normal tendency is to simplify, totokens in the test set. The algorithm output de- trivialize, to eliminate the unfamiliar word or construction,”
the more difficult reading (i.e., lectio difficilior) should in
picted in Figure 4 and the subsequent examinationsome circumstances be taken to be the authentic one (West,of the manuscript on which this edition is based 1973).
176
7

Figure 4: Algorithm output that led to the discovery of a scribal error in the words η δε. Top line: words are
given a grayscale color according to their CCR, as in Equation 2; the word δε was flagged because it obtained thesmallest CCR of all words in its given context (the surrounding 512 tokens, not all of which are pictured here).
Below top line: algorithm-generated suggestions, given a grayscale color according to their likelihood. In each case,the algorithm suggests merging two words by deleting the space before δε. The third suggestion, ηδυ, is, in fact,transmitted in the relevant manuscript and must be what was originally written by the author (Graziosi et al., 2023).
The small probability awarded here reflects the complexity of scribal errors. Some are trivial, including the ones wegenerate artificially; others, including this one, are harder to emend.

References

to the conditionals p(wi |w−i )). Diffusion models
are designed to recover original data from diffused
data, so it may be fruitful to apply such models forrecovery of original text from scribally-modified
text (Sohl-Dickstein et al., 2015). While not itself
a diffusion model, ELECTRA is a promising architecture for such future work (Clark et al., 2020).

7

Y. Assael, T. Sommerschield, B. Shillingford, et al.
2022. Restoring and attributing ancient texts using
deep neural networks. Nature, 603:280–283.
K. Clark et al. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In
Proceedings of the 8th International Conference on
Learning Representations, Addis Ababa, Ethiopia. International Conference on Learning Representations.

Conclusion

C. Cowen-Breen, C. Brooks, J. Haubold, andB. Graziosi. 2023. Logion: Machine learning forgreek philology.

In this study, we have trained a BERT model tosupport philological work on premodern Greek
texts: in particular, we use statistical and machinelearning-based approaches to identify scribal errorsthat accrue in the process of textual transmission
and to propose emendations. In a broader sense,
this research aims to contribute to the future ofphilology, understood as a discipline concernedwith preserving, elucidating, and making publicly
accessible the global archive of premodern texts.
Some of what we have presented here is of relevance also for authors and languages we have notconsidered, as well as for modern text editing in
general.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
2019. Pre-training of deep bidirectional transformers for language understanding. In Proceedings ofthe 2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, volume 1 (Longand Short Papers), pages 4171–4186. Association
for Computational Linguistics.
P. Etoori, C. Manoj, and M. Radhika. 2018. Automatic
spelling correction for resource-scarce languages using deep learning. In Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics. Student Research Workshop, pages 146–152.
Association for Computational Linguistics.
V. Gangal, A. Arora, A. Einolghozati, and S. Gupta.
2020. Likelihood ratios and generative classifiers
for unsupervised out-of-domain detection in task oriented dialog. In Proceedings of the Thirty-Fourth
AAAI Conference on Artificial Intelligence (AAAI-20),
pages 7764–7771. Association for the Advancementof Artificial Intelligence.

Acknowledgements
We are grateful to the three anonymous peer reviewers, Kasia Kobalczyk, Simon Babb, Suma
Bhat, David Cox, Justin Curl, Bernhard Haubold,
Max Haubold, Peter Heslin, Mika Hyman, Mirjam Kotwick, Karthik Narasimhan, Maria Pantelia,
Stratis Papaioannou, Pranaydeep Singh, and David
Smith for helpful feedback and advice for future
steps.

M. Can Ganiz et al. 2020. Grammar and spell checking for turkish language. Cse4197 – analysis anddesign document, T.C. Marmara University, Facultyof Engineering, Computer Engineering Department.
B. Graziosi, J. Haubold, C. Cowen-Breen, and
C. Brooks. 2023. Machine learning and the future ofphilology: A case study. TAPA, 153(1):253–284.

177
8

all training data that can be disseminated; we further include instructions and code for reproducing the error detection methods we present here athttps://github.com/charliecb/Logion.

S. Gururangan et al. 2021. Demix layers: Disentangling
domains for modular language modeling.
D. Naber. 2003. A rule-based style and grammar
checker. Diplomarbeit Technische Fakultät, Universität Bielefeld.

Appendix B

J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lakshminarayanan. 2019.
Likelihood ratios for out-of-distribution detection. In
Advances in Neural Information Processing Systems
32 (NeurIPS 2019), volume 19, pages 14640–14651,
Red Hook, NY. Curran Associates Inc.

Proof of Proposition 1.
max p(s′ ) = max

max

p(w1 , . . . , w, . . . wn )

= max

max

p(w|w−i )p(w−i )s′ ∈W1 (s)1≤i≤n w∈W1 (wi )1≤i≤n w∈W1 (wi )P. Singh, G. Rutten, and E. Lefever. 2021. A pilot study
for bert language modelling and morphological analysis for ancient and medieval greek. In Proceedings of
the 5th Joint SIGHUM Workshop on Computational
Linguistics for Cultural Heritage, Social Sciences,
Humanities and Literature, pages 128–137. Association for Computational Linguistics.

p(w|w−i )p(w−i )p(s)1≤i≤n w∈W1 (wi ) p(wi |w−i )p(w−i )p(w|w−i )= p(s) max max
1≤i≤n w∈W1 (wi ) p(wi |w−i )1= p(s) max
1≤i≤nρi (s)= max

J. Sohl-Dickstein et al. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on
Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265. PMLR.

maxwhich establishes that, for s∗ ∈ W1 (s),p(s∗ ) = max p(s′ )s′ ∈W1 (s)L. Torroba Hennigen and Y. Kim. 2023. Deriving language models from masked language models. In
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2:
Short Papers), pages 1149–1159, Toronto, Canada.
Association for Computational Linguistics.

if and only if s∗ differs from s in word i∗ andρi∗ (s) = min ρi (s).
iOn the other hand, we have

A. Wang and K. Cho. 2019. BERT has a mouth, andit must speak: BERT as a Markov Random Field
Language Model. In Proceedings of the Workshop
on Methods for Optimizing and Evaluating Neural
Language Generation, pages 30–36, Minneapolis.
Association for Computational Linguistics.

ρi (s) ==M. L. West. 1973. Textual Criticism and Editorial Technique Applicable to Greek and Latin Texts. Teubner,
Stuttgart.

p(wi |w−i )maxw∈W1 (wi ) p(w|w−i )
min

p(s)

s′ ∈W1 (s):s′ differs from s at word i p(s′ )>1

if and only if ∀s′ such that s′ differs from s only
in word i, and only by one character, we havep(s) > p(s′ ). If this holds for all i, then s = s∗by definition. If not, then for some i, it holds thatp(s) ≤ p(s′ ). In this case, by uniqueness of the
maximum, for some i for which this holds, we musthave p(s) < p(s′ ). Thus s ̸= s∗ .

Appendix A

Data for premodern Greek faces a specific problem which needs to be addressed. The best online
database is the Thesaurus Linguae Graecae (TLG).
It is not open access (unlike the best databases forother ancient languages, e.g. Latin). We are grateful to the TLG Director for providing us with someof the data we used for our models; we were instructed, however, that it cannot be disseminatedfurther, because of the license currently restricting access to the TLG. The global archive of premodern texts is an important reservoir of linguistic
and cultural diversity which should be accurately
digitized and made freely available. For now, we
make available the models we trained along with
178
9

Classical Philology in the Time of AI: Exploring the Potential of Parallel
Corpora in Ancient Languages
Tariq Yousef
University of Southern Denmark

Chiara Palladino
Furman University

Farnoosh ShamasianLeipzig University

yousef@imada.sdu.dk

chiara.palladino@furman.edu

farnoosh.shamsian@uni-leipzig.de

Abstract1This contribution presents an overview of Parallel Text Processing, particularly Translation
Alignment, and illustrates the current statusof this task in ancient languages. In the firstpart, we provide the fundamental principles ofParallel Texts and give an overview of theirapplications for the study of ancient texts. In
the second part, we indicate how Parallel Texts
can be leveraged to perform other NLP tasks,
including automatic alignment, dynamic lexica
induction, and Named Entity Recognition. In
the conclusion, we emphasize current limitations and future work.

such as morphology and word order, contribute toadditional difficulties.
The goal of this paper is to offer a programmaticsurvey of the current status of TA research in thespecific domain of ancient languages, particularly
Ancient Greek. As such, we will cover many different applications, both in Philology and Computer
Science, with the intent of demonstrating the potential of this method in the study of ancient languages.
Our aim is to illustrate how TA and parallel corpora can be used for a wide range of research, tocontribute to existing debates and to inspire newquestions.

Introduction

2

Design and Concept of Translation
Alignment Tools

Parallel Text Processing refers to various computational tasks based on parallel corpora (Véronis, Since TA was established, several tools have been
2000). Parallel corpora are collections of texts thatdesigned to collect TPs, with or without integrated
show some level of equivalence between them: forreading environments for visualizing the alignexample, a text and its translations, or different
ments (overviews are provided in our previous
versions of the same text.
works Yousef and Janicke 2020; Yousef 2023). In
The most important task in Parallel Text Process- the context of ancient languages, a limited numing is Text Alignment, that is, the automatic estab- ber of tools have been developed. These include
lishment of equivalences across various types of
Alpheios (Almas and Beaulieu, 2013), DUCAT
units: document, chunk, sentence, and word (Kay
Citation Alignment Tool (Blackwell et al., 2020),
and Röscheisen, 1993). The task of aligning a text
Benner’s tool for aligning the Bible (Benner, 2014),
against its translation(s) is called Text-Translationand U GARIT1 , designed to enable word-level alignAlignment (from now on, TA). The output of TA
ments in low-resourced languages (Yousef et al.,is defined as Translation Pairs (TPs), which corre- 2022c). Currently, U GARIT counts about 50
spond to pairs of the various units aligned (chunks, aligned languages, 700 users, and more than a milsentences, words, etc.).
lion TPs2 , establishing itself as the most popular
TA can be considered a subfield of Text Align- tool in this area.
ment: however, it has very unique challenges atU GARIT was designed as a crowd-sourcing
tributed to the complex dynamics underlying the
project to collect training data for automatic alignrelationship between texts and their translations. ment methods for ancient languages, but it exIn particular, word-level TA poses considerable
panded into a range of diverse applications, mostly
complexity due to the inherent uncertainty in estab- thanks to its global community of scholars andlishing individual equivalences: translations are not1
https://ugarit.ialigner.com/
perfect transpositions of the originals, and tend to2
Of this number, about 240,000 TPs are automaticallyalter, normalize, expand or simplify parts of the text. generated through traditional statistical automatic alignment
Moreover, structural differences across languages, tool (Giza++).
179
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 179–192,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_021

students. The alignment workflow, which allows
bilingual and trilingual alignments, is simple andintuitive.U GARIT allows different types of TPs:
word-to-word (1-1), word-to-phrase (1-N), phraseto-word (N-1) and phrase-to-phrase (N-N).
Alignments are immediately published online.
The visualization of published alignments allows
the user to compare aligned texts token by token,
providing a transliteration service for non-Latinalphabets, statistical information about the percentage of aligned and not-aligned tokens, types oflinks, a downloadable list of TPs, and an embedding option (Figure A.1).
The tool integrates a dynamic lexicon, which canbe triggered through the search function or simply
by clicking on a word in an aligned text. The results are visualized as a radial cluster dendrogram,a tree view, and as a list of words with frequency(Figure A.2). The lexicon extracts all the translation equivalents of a given word across the whole
database, providing a list of all languages in whichthat word has been translated.

3.1

Qualitative Studies: Pedagogy andTranslation Studies

Manual or supervised TA is essential for the creation of high-quality Gold Standards and training
datasets. However, it can also be configured asa close reading task for translation study and language learning. In recent years, efforts have been
undertaken in the realm of Digital Philology, inthe context of a major emphasis on the development of open resources for innovative approaches
to learning Classical languages (Crane et al., 2023).
3.1.1

Pedagogy and User Behavior

Parallel corpora on U GARIT are currently being
used to teach Ancient Greek, Latin, and Persianin several universities, including Leipzig, Furman,
Sâo Paulo, Tufts, University of Zagreb, Göttingen,
Cattolica University, but also in schools across Europe, such as the Liceo G. Peano Tortona in Italy.
The active engagement with the text throughthe effort of establishing fine-grained equivalences
stimulates a reflective approach to the text and creates an opportunity to design exercises that invite
3 Applications of Translation Alignment
language learners to reflect upon the cultural andlinguistic specificities of ancient texts through the
In many modern languages, TA is successfully em- contrastive comparison with modern translations:ployed in a wide range of NLP tasks. For example, through specific exercises tailored to the level, stuit is essential in word- and phrase-based Statisti- dents are stimulated to reflect on the depths ofcal Machine Translation (SMT) pipelines (Brown
semantic and linguistic differences, and their imet al., 1993; Koehn et al., 2003); it can be used topact on the very operation of translating (Palladino,
analyze the output of Neural Machine Translation2020). Moreover, this process encourages a critimodels (NMT) and assess their performance qual- cal approach to translations as interpretations, reity (Neubig et al., 2019); to filter and clean noisy
thinking their role in understanding ancient texts,
parallel corpora (Kurfalı and Östling, 2019; Zarin, aand enabling the students to be part of a broader
et al., 2015); to transfer linguistic annotation fromconversation about the reception and significanceone text to its translation, such as Semantic Roleof a text over time. Palladino et al. 2021 provide
labels, POS tags, Named Entity tags (Yousef, 2015; a series of use cases showing how TA can be used
Ni et al., 2017; Huck et al., 2019). Parallel Corpora
for learning Ancient Greek or Latin at various levaligned at word-level can support the work of pro- els, through a series of reflective and project-based
fessional translators (Liu, 2020), bilingual lexicon
exercises. Most importantly, the comparison of
induction (Marchisio et al., 2021), and word sense
different translations of the same texts provides a
disambiguation (Procopio et al., 2021). Moreover, tangible sense of the different strategies employed
they provide extremely useful information for vo- by professional translators, and gives a strong pragcabulary assimilation and language teaching (Vy- matic understanding of the fluidity of translations
atkina and Boulton, 2017), and for the study of theand their (in)ability to transmit the original in its
history of transmission of a corpus (Laviosa, 2021). full meaning. Shamsian and Crane 2022 showed
In the following sections, we will survey the cur- how TA can be integrated with grammar explanarent state of TA research for ancient languages, tions and other types of annotations to create bornillustrating how the parallel corpora created on
digital pipelines for learning ancient languages,
U GARIT are used for qualitative and quantitativeeven at beginner level. Through TA, students areresearch.
able to critique existing scholarly translations and180

reflect on how to create more accurate representations of the original. This process is particularly
useful in linguistic contexts where available translations are mostly derivative from translations inother languages, like in the case of Persian.
3.1.2

Empirical study of translations andintertextual phenomenation, creating very distinctive results for specificlanguage pairs, but also that cultural context, text
genre and modalities of transmission have an impact in determining structural differences in translations.
3.2

Quantitative Studies: AI and Parallel
CorporaWhile translations constitute a crucial aspect in thehistory of the transmission of ancient texts, very The various applications described above show theimportance of parallel corpora for the study of texts
few studies have used computational approaches
to investigate them. In this area, manual and au- from different perspectives. For this reason, it is imtomatic TA provides an extremely promising re- portant to develop workflows for automated alignsource. Bizzoni et al. 2017 used an automatic align- ment tasks, which support the scalability of both
qualitative and quantitative research. While thisment workflow based on the Needleman-Wunscharea is very well developed for modern languages,
algorithm, using proper names as anchors to alignselected passages of the Odyssey against a large cor- it is still in its infancy for ancient ones. In the following section, we will show current efforts in thepus of French translations, to identify large-scale
improvement of automatic alignment models, andtrends in translation practices across the 16th andindicate how automatic TA can be used to enhance
17th century. Shukhoshvili 2017 used U GARIT
the performance of important NLP tasks.
to support the creation of a complete translation
Until the advent of transformer-based models,of Plato’s Theaetetus into Georgian and used thethestate of the art of automatic TA was statisresulting corpus to investigate cross-linguistic dytical methods, such as Giza++ (Och and Ney,
namics between the two languages. Somewhat in2003), fast_align (Dyer et al., 2013) and EfLomAl
the opposite direction, Xie 2023 used U GARIT to(Östling
and Tiedemann, 2016). However, the perexamine the Ancient Greek translation of the Latin
text of the Res Gestae: the method applied com- formance of statistical alignment models relies on
the presence and size of training datasets in thebined close reading to inspect specific semantic
phenomena, and distant reading through the con- form of parallel sentences.
Recently, however, Neural Machine Translationsultation of the alignment statistics provided by thetool. Interestingly, while Xie 2023 found a remark- (NMT) and multilingual transformer models haveintroduced the possibility of creating accurate alignable degree of accuracy in the corpus, the trilingual
ments even with no training datasets (Jalili Sabetalignment of the Rosetta Stone performed by Amin
et al., 2020). Most notably, transformer models faet al. 2023 on U GARIT demonstrated that the threecilitate the creation of contextualized word embedversions of the text bear considerable differences
dings, which encode information about a meaning
and they cannot be considered one and the sametext. Finally, Palladino et al. 2022a propose a work- of a word based on its context. Pre-trained multilingual transformer models, such as Multilingual
flow that combines close reading and quantitative
Bert (mBERT) and XLM-RoBERTa, achieved sigindicators to support alignment-based evaluation oftranslations of Ancient Greek texts: the set of crite- nificant performance improvements for numerous
cross-lingual tasks (Conneau et al., 2019b; Devlin
ria includes frequency of link types, percentage ofet al., 2018a).
aligned and not-aligned words, intersection acrosstranslators, POS intersection, in combination with
Language models are now increasingly used for
close reading of selected passages.
various NLP tasks in ancient languages (SommerThe ever-increasing amount of corpora inschield et al. 2023 provide a comprehensive survey
U GARIT also allows for big-data exploration sce- in the field). Most current applications are develnarios. Palladino and Yousef 2023 used theoped with a strong interest in POS tagging andU GARIT database to investigate cross-linguistic
morphological analysis. To the best of our knowldynamics, studying how language and culture af- edge, we are pioneers in employing transformer
fect the establishment of word equivalents between
models to automate TA tasks in ancient corpora,text and translation. Their data show how different
and to leverage on the resulting parallel texts tolanguage systems influence the process of transla- explore new possibilities in other NLP tasks. We
181

Figure 1: The alignment workflow.

use Ancient Greek as a case study, but the model
we developed is multilingual and can be fine-tuned
for other ancient languages.
3.2.1

Alignment Guidelines and Gold
Standards

In order to evaluate the performance of automaticalignment models, it is essential to have highquality gold standard datasets. Gold Standards are
typically created by two or more annotators, whose
Inter-Annotator Agreement (IAA) is measured toensure consistency in the dataset. Guidelines are
created and used to ensure that annotators are following a similar strategy.
While there is no lack of guidelines and standards for modern languages,3 we developed thefirst ones specifically aimed at ancient languages:
using Ancient Greek as case study, we consideredtranslations into English, Portuguese, Persian,4 andscholarly Latin. These guidelines can be used forthe evaluation of automatic alignment tasks, butalso as a general reference for students and scholars who wish to create their own parallel corpus
for other purposes (Ferreira et al., 2022; Palladinoet al., 2022b; Palladino and Shamsian, 2022).
The resulting Gold Standards are based on a
corpus of 5,500 words from Ancient Greek epic
poetry and prose (Homer, Xenophon, and Plato)and on 100 fragments of Ancient Greek translated
into Latin from the Digital Fragmenta Historicorum Graecorum (DFHG)5 . Two annotators aligned
each corpus separately, after having drafted the
Guidelines. The resulting IAA was measured at
86.17% for GRC-ENG and 83.31% for GRC-POR,
and GRC-LAT 90.50%.
Our guidelines considered the same general principles established for modern languages (Lambert
et al., 2005), but working within the specificities ofan ancient language: for example, we had to care-fully address the impact of high inflection and theinconsistency shown in the translation of linguistic
and rhetorical structures. As a result, while mostguidelines cover 7-10 classes of phenomena, ours
covered 14 main classes with several subclasses.
Therefore, it is easy to understand how the alignment of an ancient text may result in higher ambiguities than modern corpora traditionally usedin TA: moreover, modern corpora are usually technical texts, which leave little space for variation,
but that is not the case for ancient texts, which arenecessarily literary or even poetical. Although ourguidelines reach and exceed the 80% threshold of
optimal consistency, it is important to reflect on
the origins of disagreements across annotators inorder to individuate areas of improvement for boththe Gold Standards and automatic TA models: factors such as the native language of the annotators,
their proficiency with the language/s, their familiarity with the text and specific dialect, and the timeat their disposal may all have an impact on theirperformance. This qualitative study is part of our
future work.
3.2.2

The U GARIT Ancient Greek Alignment
Model

In our previous works (Yousef et al., 2022b,d), we
have trained an automatic TA model that employs
the recent advances in language modelling and isable to generate accurate word-level alignmentseven with small amounts of training data. In thiscontext, we adapted the pipeline illustrated in Figure 1 proposed by (Jalili Sabet et al., 2020; Dou
and Neubig, 2021).
The core concept is to leverage pre-trained
multilingual contextualized language models suchas M BERT (Devlin et al., 2018b) and XLMRO BERTA (Conneau et al., 2019a) or fine-tuned
versions of them. A similarity matrix can be derived based on distance/similarity metrics that cal3
An overview of available resources can be found on theculate the similarity for every two tokens based
U GARIT website: https://ugarit.ialigner.com/
guidelines.php.
on their embeddings. Then, the word-level align4
This set of guidelines has not yet been used for the crements can be predicted by employing an extraction
ation of Gold Standards, therefore we did not employ it foralgorithm over the similarity matrix.
evaluation purposes.
5
https://www.dfhg-project.org/
The initial experiments we conducted on the pre182

Experiment

Languages

Data Size

Phase 1GRC Monolingual

12 Millions Tokens

Phase 2GRC-ENG, GRC-LAT
GRC-KAT

45.000 sentences

Phase 3Mixed dataset

5000 sentences
190k TPs

SourcePerseus DL, TreeBanking,First1kGreek
Perseus DL, DFHG,UGARITUGARIT

Table 1: The proposed fine-tuning strategy.

ENG

LAT

POR

Softmax
Match
Argmax
Itermax
Softmax
Match
Argmax
Itermax
Softmax
Match
Argmax
Itermax

Precision
80.80%
65.42%
84.95%
78.43%
85.67%
62.18%
88.46%
81.27%
63.84%
50.00%
66.01%
59.67%mBERT
RecallF1
56.91% 66.78%
72.76% 68.90%
52.47% 64.87%
64.08% 70.53%
84.64% 85.15%
87.97% 72.86%
80.80% 84.46%
84.78% 82.99%
61.27% 62.53%
72.61% 59.22%
54.92% 59.96%
64.06% 61.79%

AER
32.72%
31.31%
34.57%
29.14%
14.83%
27.55%
15.09%
17.06%
37.40%
41.50%
39.76%
38.35%

Precision
92.62%
79.22%
94.44%
91.05%
94.64%
80.61%
95.52%
92.21%
76.11%
58.79%
77.25%
72.22%XLM-RoBERTa
RecallF166.85% 77.65%
87.26% 83.05%
63.32% 75.81%
71.65% 80.19%
92.39% 93.50%
96.30% 87.76%
91.38% 93.40%
93.33% 92.77%
75.61% 75.86%
86.17% 69.89%
71.10% 74.05%
81.02% 76.37%

AER
21.88%
17.17%
23.70%
19.42%
6.47%
12.50%
6.55%
7.25%
24.13%
31.01%
25.81%
23.91%

Table 2: Evaluation results of the automatic alignment model on three gold standard datasets.

trained M BERT and XLM-RO BERTA (Zero-Shot)
showed significantly poor performance on Ancient
Greek-English, Ancient Greek- Latin, and Ancient
Greek-Portuguese datasets. Therefore, fine-tuning
those models was necessary to achieve better performance. Due to the availability of parallel sentences and in order to obtain the best outcome fromthe training process, we conducted several experiments employing multiple training objectives (Douand Neubig, 2021) aiming to find the best training
strategy. Each experiment tested various combinations of unsupervised and supervised training.
Table 1 illustrates our proposed training strategy
which consists of three phases. The initial stage involved training pre-existing models using monolingual Ancient Greek corpora, which encompassed a
total of 12 million tokens. Subsequently, the model
underwent an unsupervised fine-tuning process utilizing a collection of 45,000 parallel sentences.
This fine-tuning phase encompassed sentences in
Greek-English, Greek-Latin, and Greek-Georgian.
Ultimately, the model underwent supervised finetuning, where it was refined using precise manualalignments extracted from the U GARIT database.

against the gold standard datasets using Precision,
Recall, F1 and Alignment Error Rate AER.

Table 2 presents the performance evaluation of
our model during phase 3, utilizing three gold standard datasets: Greek-English, Greek-Latin, andGreek-Portuguese. We evaluated the model’s performance using four alignment extraction heuristics
and two fine-tuned models: mBert-based modeland XLM-RoBERTa-based model. Notably, thefine-tuned XLM-RoBERTa models consistently
outperformed the mBERT-fine-tuned models acrossall cases, demonstrating their superior performancein alignment extraction. The Match heuristic significantly outperformed other models regarding Recall. However, it achieved always the lowest Precision. On the other hand, the Argmax heuristic
consistently achieved the highest precision but thelowest recall. Both the Softmax and Itermax heuristics demonstrated balanced performance, with arelatively equal consideration given to recall andprecision. Itermax showcased superior recall compared to Softmax, while Softmax displayed better
precision than Itermax. Overall, the performanceof these heuristics varies in terms of recall and
The performance of the model was evaluatedprecision, with each exhibiting strengths and weak183

nesses. The choice of the appropriate heuristic will
depend on the specific requirements and prioritiesof the task at hand, balancing the trade-off between
recall and precision based on the desired outcomes.
Our alignment model is available on H UGGINGFACE 6 and can be downloaded and used locally.
In order to make it more accessible, however, we
implemented an online tool7 that integrates the pretrained model and allows users to simply paste theirtexts and align them automatically, with an option
to visualize and download the results (Figure A.1).
The pre-trained alignment model can be used toscale all the qualitative operations described above,
but also for a variety of downstream tasks. In thefollowing sections, we will describe our preliminary results in the areas of Bilingual Lexica Induction and Named Entity Recognition.
3.2.3

Bilingual Lexica Induction

The significance of aligned word-level parallel corpora as a data source for terminology banks andbilingual dictionaries is emphasized by Véronis
2000. These resources are highly valuable to improve the performance of professional translators,
to enrich and train translation memory software,to retrieve terminology lists for technical texts,
or in lexicographic studies. However, it is worth
noting that not all language pairs can be easilyaligned, especially when dealing with ancient andlow-resourced languages. In this proof of concept,
we applied automatic dictionary induction to produce high-quality translation pairs for languages
that do not share parallel texts. Additionally, we
represented the acquired translation pairs within agraph-based data structure. This approach allows
us to integrate manual alignments and dictionary
entries and facilitate performing clustering or pivoting to generate translation pairs of languages withno direct connections.
Corpora: we used 400,000 parallel sentences in
6 languages (Ancient Greek, Arabic, English, Hebrew, Latin, and Persian). Our corpus derives fromthe Bible8 , the Perseus Digital Library9 , and the
DFHG corpus.
Alignment: we used our fine-tuned alignment model for Ancient Greek to perform theword/phrase alignments. We employed Itermax
heuristic to extract the most accurate translation
pairs from the similarity matrix since it achieved
the highest Phrase Alignment Accuracy (Yousef
et al., 2023; Yousef, 2023).
Graph Generation: Figure 2 illustrates the proposed graph structure. We model every translation
pair as two nodes connected with an edge. Additional relations can be added to indicate different
linguistic features if they are available. For example, connecting a phrase with its constituent words
or linking a word with its lemma. These relations
can be beneficial for running sophisticated queries.
Shi et al. (2021) proposed a matching ratio that considers the alignment frequency and how frequently
the two words co-occurred in the corpus. However,this ratio works only with one-to-one alignments.
Therefore we proposed an alignment score that considers phrases as well:score(s, t) =2 ∗ A(s, t)
A(s|Lt ) + A(t|Ls )(1)

Where A(s, t) indicates how many times the twowords/phrases are aligned together, A(s|Lt ) indicates how many times s is aligned in total towords/phrases in the same language as t, andA(t|Ls ) indicates how many times t is aligned in
total to words/phrases in the same language as s.

Figure 2: The graph structure of the induced TPs.

The resulting graph contains over 614k nodes
and 1,620k edges from Automatic Alignment, andan additional 193k TPs collected from U GARIT as
Manual Alignment. Moreover, graph clustering
algorithms such as C HINESE W HISPER (Biemann,2006), a hard partitioning and flat clustering al6
gorithm, can be applied to cluster graph entries
https://huggingface.co/UGARIT/
grc-alignment
into sets containing words/phrases that are seman7
http://ugarit-aligner.com
tically related or share the same meaning. Figure8
https://github.com/christos-c/
3 shows a cluster of aligned words/phrases in varibible-corpus.
9
ous languages. This cluster is one of 7300 clusters
http://www.perseus.tufts.edu/hopper/.
184

Figure 3: An example from graph clustering results.

obtained after filtering out the relations with frequency less than 5, alignment score less than 0.25,
and running C HINESE W HISPER clustering algorithm for 20 iterations. Figure A.4 shows anothercluster with an extended visualization, in which
manual alignments and PART_OF relations are displayed.
The results of this work are available on our
GitHub10 : The resulting dictionaries can provide
an invaluable resource to establish equivalences
across languages that are not normally translated
into each other: for example, a Persian speakerstudying Ancient Greek can use this resource toextract Persian equivalents of Greek words, insteadof relying on English or French translations. Moreover, the dictionaries provide insights into realworld use of the words, as they derive directly fromcontextual usages in texts. Therefore, the development of this application will considerably improvethe use of parallel corpora for teaching, translating,and language learning.
Our future work in this regard includes expanding the corpus of accurate manual alignments forother low-resourced languages, and expanding themonolingual and bilingual datasets to improve theaccuracy of the model in other languages. We will
also develop a user interface with various searchand visualization functions.
3.2.4 NER for Ancient Greek
An additional application of our alignment model
pertains to enhancing the efficacy of Named Entity
Recognition (NER) in the context of ancient languages through the employment of annotation projection. This workflow leverages on cross-lingualtransfer: the basic principle is that, if NER mod10

https://github.com/UgaritAlignment/els reach accurate results in one language, we canuse an automatic alignment workflow to align anannotated text with another one, for which NERmodels do not achieve such a high performance.
This principle, called annotation projection, consists in projecting NER annotations performed on
English translations on an aligned text in an ancientlanguage, so that Named Entities can be extracted
and classified through the alignment process.
NER is in great demand among scholars of ancient languages. However, it comes with significant challenges including OCR-generated errors
and noisy data, complexity of the sources, lack ofgold standards and guidelines. The only survey on
the topic for historical languages is provided by
Ehrmann et al. 2021, with some recent updates inSommerschield et al. 2023. New pipelines basedon transformers have shown considerable improvement in this area, although NER remains a particularly challenging task (Palladino et al., 2020;
Yousef et al., 2022a; Burns, 2023; Yoo et al., 2022).
While most of these experiments use a direct training approach with annotated datasets ofNamed Entities in the target language, we propose
a novel workflow that integrates annotation projection and leverages on our automatic alignment
model. Figure 4 illustrates our pipeline: we collect a parallel corpus of Ancient Greek and English
translations; automatically annotate the text of the
English translations using AllenNLP, an accurate
off-the-shelf NER system11 ; then, we employ automatic word alignment to retrieve translation pairs,
11

We benchmarked three high-quality English NER models,
namely, spaCy, AllenNLP and flairNLP to select the modelwith the highest accuracy on our corpus. The comparison revealed that AllenNLP and flairNLP significantly outperformed
spaCy, and their performance was very close.

185

Figure 4: Named-Entity annotation projection pipeline.

and project the annotations from the English translations onto the corresponding tokens in the Ancient Greek text using a direct mapping heuristic.

4 Conclusions and Future Work

Parallel Corpora are today’s Rosetta Stones (Véronis, 2000). They can be used for a variety of philoWhile AllenNLP provides four entity classes(PERS, LOC, ORG, MISC), we only used PERS, logical and computational tasks, as they provide
a medium between languages and cultures. This
LOC, and MISC, as the ORG entity label does notstudy shows the importance of parallel text proapply intuitively to ancient naming systems (seefurther on this issue Ehrmann et al. 2021; an alter- cessing, specifically in the context of Translation
native strategy for labeling is proposed for Latin by Alignment, for various activities in the study of
low-resource languages. The value of TA emerges
Burns 2023.).
in its various applications, which include language
We tested the workflow on the Bible corpus, uslearning, NLP development, dictionary extraction,
ing English for annotation and selecting versions
and research on translations and cross-linguistic
in Ancient Greek, Latin, and Arabic12 . We decided
interactions.
to expand the range of languages beyond Ancient
Most importantly, the development of accurate
Greek, which is still the most present in training
TA models can significantly contribute to improvedatasets, to show the potential of the multilingual
the performance of important NLP tasks in ancientmodel.
languages through the mdeium of annotation proTwo domain experts performed qualitative evaljection. For this reason, we plan a significant exuation on 100 random verses (about 550 entities
pansion of monolingual and bilingual corpora forper dataset) and assigned a score as shown in tasupervised and unsupervised training, in order toble 3. The evaluation results show an accuracy
improve performance on other ancient languages.
of 86.63% in Ancient Greek, 82.34% in Latin,
Moreover, we will test analogous workflows based
and 75.54% in Arabic: understandably, Arabic
on annotation projection for other NLP tasks, suchshowed the worst performance because we had
as POS tagging and lemmatization. In this sense,much less corpora available for training. The mostthe development of accurate sentence alignment
common errors were found in the misclassification
workflows is fundamental, as it can significantlyof entities, sometimes as a consequence of the factenhance the performance of word-alignment modthat English translations adopted a different entity
els.
type. Most notably, many ethnonyms (MISC in our
Despite the great success of transformers and landataset) were translated with place-names in Enguage models, we want to emphasize that manually
glish, and therefore classified as LOC in the ancientannotated corpora and guidelines are still essenlanguage. Moreover, incomplete or partial aligntial to ensure accurate performance and to detectments were frequent in multi-token entities, such
patterns of error. Gold Standards and output evaluaas "Jesus Christ", "Simon Zelotes", and "Pontius
tion require strong disciplinary expertise, especially
Pilate".
in scenarios where the research questions are complex. For this reason, as already emphasized by
12
All versions were taken from the Bible Corpus on GitHub,
Sommerschield et al. 2023, the best efforts in thewhile the Ancient Greek version was retrieved from the
Perseus Digital Library.
domain of automatic text processing are achieved
186

ScoreCorrect alignment / Correct NER
Incorrect alignment / Correct NER
Correct alignment / Incorrect NER
Incorrect alignment / Incorrect NER

Ancient Greek
86.63%
7.26%
5.28%
0.83%

Latin
82.34%
12.87%
3.96%
0.83%

Arabic75.54%
21.16%
2.98%
0.33%

Table 3: Manual evaluation of 100 randomly selected verses.

by multidisciplinary teams, where the contributionof scholars of the language and philologists can provide better information about the idiosyncrasies ofthe material, and crucially contribute to the evaluation of the results. High-quality philological work
is essential for progress in this field, and the only
way we can produce reliable tools that will be usedby Digital Humanists and Humanists as well.

Acknowledgements
We are most grateful to all the people who contributed to the development of U GARIT, the Alignment Models, and who provided Gold Standards
and annotated datasets: Gregory Crane, Monica Berti, Anise d’Orange Ferreira, Michel Ferreira dos Reis, Josh Kemp, David Wright, Maia
Shukhoshvili, Sisi Xie, Brian Clark, and all thescholars and students who worked on Translation
Alignment on Ugarit.

References
Bridget Almas and Marie-Claire Beaulieu. 2013. Developing a New Integrated Editing Platform for Source
Documents in Classics. Literary and Linguistic Computing, 28(4):493–503.
Miriam Amin, Angelos Barmpoutis, Monica Berti,
Eleni Bozia, Josephine Hensel, and Franziska
Naether. 2023. The Digital Rosetta Stone Project.
In Rita Lucarelli, Joshua A. Robertson, and Steve
Vinson, editors, Ancient Egypt, New Technology., volume 17 of Harvard Egyptological Studies, pages 58–
84. Brill, Leiden - Boston.
Drayton Benner. 2014. A Tool for a High-Carat GoldStandard Word Alignment. In Proceedings of the
8th Workshop on Language Technology for Cultural
Heritage, Social Sciences, and Humanities (LaTeCH),
pages 80–85.
Chris Biemann. 2006. Chinese whispers - an efficientgraph clustering algorithm and its application to natural language processing problems. In Proceedingsof TextGraphs: the First Workshop on Graph Based
Methods for Natural Language Processing, pages 73–
80, New York City. Association for Computational
Linguistics.

Yuri Bizzoni, Marianne Reboul, and Angelo Del Grosso.
2017. Diachronic Trends in Homeric Translations.
Digital Humanities Quarterly, (2).
Christopher W. Blackwell, Chiara Palladino, Mackense Greico, and Allie Bolton. 2020. DUCAT: Passage/Translation Alignment with the CITE Architecture. In Proceedings of the DH2020 Digital Humanities Conference 2020, Ottawa.
Peter F. Brown, Stephen A. Della Pietra, Vincent J.
Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter
Estimation. Computational Linguistics, 19(2):263–
311. Place: Cambridge, MA Publisher: MIT Press.
Patrick J. Burns. 2023. LatinCy: Synthetic Trained
Pipelines for Latin NLP. ArXiv:2305.04365 [cs].
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Unsupervised
cross-lingual representation learning at scale. CoRR,
abs/1911.02116.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Unsupervised Cross-lingual Representation Learning at Scale.
CoRR, abs/1911.02116. ArXiv: 1911.02116.
Gregory Crane, Alison Babeu, Lisa M. Cerrato, Amelia
Parrish, Carolina Penagos, Farnoosh Shamsian,James Tauber, and Jake Wagner. 2023. Beyond translation: engaging with foreign languages in a digital
library. International Journal on Digital Libraries.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018a. BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. : 1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018b. BERT: pre-training ofdeep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Zi-Yi Dou and Graham Neubig. 2021. Word Alignment
by Fine-tuning Embeddings on Parallel Corpora. In
Proceedings of the 16th Conference of the European
Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online.
Association for Computational Linguistics.

187

Chris Dyer, Victor Chahuneau, and Noah A. Smith.
2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, pages 644–648, Atlanta,
Georgia. Association for Computational Linguistics.

Kelly Marchisio, Philipp Koehn, and Conghao Xiong.
2021. An Alignment-Based Approach to SemiSupervised Bilingual Lexicon Induction with Small
Parallel Corpora. In Proceedings of Machine Translation Summit XVIII: Research Track, pages 293–304,
Virtual. Association for Machine Translation in theAmericas.

Maud Ehrmann, Ahmed Hamdi, Elvys Linhares Pontes,
Matteo Romanello, and Antoine Doucet. 2021.
Named Entity Recognition and Classification on Historical Documents: A Survey. ArXiv:2109.11406[cs].

Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel,
Danish Pruthi, and Xinyi Wang. 2019. compare-mt:
A Tool for Holistic Comparison of Language Generation Systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations),
pages 35–41, Minneapolis, Minnesota. Association
for Computational Linguistics.

Anise d’Orange Ferreira, Michel Ferreira dos Reis, andTariq Yousef. 2022. Critérios ou Convenções de Alinhamento do Grego às Traduções em Português. Publisher: Zenodo. Version Number: 1.0.
Matthias Huck, Diana Dutka, and Alexander Fraser.
2019. Cross-lingual Annotation Projection Is Effective for Neural Part-of-Speech Tagging. In Proceedings of the Sixth Workshop on NLP for Similar
Languages, Varieties and Dialects, pages 223–233,
Ann Arbor, Michigan. Association for Computational
Linguistics.
Masoud Jalili Sabet, Philipp Dufter, François Yvon,and Hinrich Schütze. 2020. SimAlign: High Quality Word Alignments Without Parallel Training Data
Using Static and Contextualized Embeddings. In
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online.
Association for Computational Linguistics.

Jian Ni, Georgiana Dinu, and Radu Florian. 2017.
Weakly Supervised Cross-Lingual Named Entity
Recognition via Effective Annotation and Representation Projection. In Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1470–1480,
Vancouver, Canada. Association for Computational
Linguistics.
Franz Josef Och and Hermann Ney. 2003. A systematiccomparison of various statistical alignment models.
Computational Linguistics, 29(1):19–51.
Chiara Palladino. 2020. Reading Texts in Digital Environments: Applications of Translation Alignment
for Classical Language Learning. The Journal ofInteractive Technology and Pedagogy, (18).

Martin Kay and Martin Röscheisen. 1993. Texttranslation Alignment. Computational Linguistics,
19(1):121–142.

Chiara Palladino, Maryam Foradi, and Tariq Yousef.
2021. Translation Alignment for Historical Language Learning: a Case Study. Digital Humanities
Quarterly, 015(3).

Philipp Koehn, Franz J Och, and Daniel Marcu. 2003.
Statistical phrase-based translation. Technical report,
University of Southern California Marina Del Rey
Information Sciences Inst.

Chiara Palladino, Farimah Karimi, and Brigitte Mathiak. 2020. NER on Ancient Greek with minimal
annotation. DH2020 Ottawa. Book of Abstracts.

Murathan Kurfalı and Robert Östling. 2019. Noisy parallel corpus filtering through projected word embeddings. In Proceedings of the Fourth Conference on
Machine Translation (Volume 3: Shared Task Papers,
Day 2), pages 277–281.
Patrik Lambert, Adrià De Gispert, Rafael Banchs, andJosé B. Mariño. 2005. Guidelines for Word Alignment Evaluation and Manual Alignment. Language
Resources and Evaluation, 39(4):267–285. Publisher: Springer.
Sara Laviosa. 2021. Corpus-based Translation Studies: Theory, Findings, Applications, volume 17 ofApproaches to Translation Studies. Brill, Leiden Boston.
Kanglong Liu. 2020. Corpus-Assisted Translation
Teaching: Issues and Challenges, volume 7 of Corpora and Intercultural Studies. Springer.

Chiara Palladino and Farnoosh Shamsian. 2022. Translation Alignment: Ancient Greek to English. Annotation Style Guide and Gold Standard. Publisher:
Zenodo Version Number: 1.0.
Chiara Palladino, Farnoosh Shamsian, and Tariq Yousef.
2022a. Using Parallel Corpora to Evaluate Translations of Ancient Greek Literary Texts. An Application of Text Alignment for Digital Philology Research. Journal of Computational Literary Studies,(1).
Chiara Palladino, David J. Wright, and Tariq Yousef.
2022b. Translation Alignment: Ancient Greek toLatin. Annotation Style Guide and Gold Standard.
Publisher: Zenodo Version Number: 1.0.
Chiara Palladino and Tariq Yousef. 2023. To say almost
the same thing? A study on cross-linguistic variation in ancient texts and their translations. Digital
Scholarship in the Humanities.

188

Luigi Procopio, Edoardo Barba, Federico Martelli, andRoberto Navigli. 2021. MultiMirror: Neural Crosslingual Word Alignment for Multilingual Word Sense
Disambiguation. volume 4, pages 3915–3921. ISSN:
1045-0823.
Farnoosh Shamsian and Gregory R. Crane. 2022. OpenResources for Corpus-Based Learning of Ancient
Greek in Persian. Journal of Interactive Technology
and Pedagogy, (21).
Haoyue Shi, Luke Zettlemoyer, and Sida I Wang. 2021.
Bilingual lexicon induction via unsupervised bitext
construction and word alignment. arXiv preprint
arXiv:2101.00148.
Maia Shukhoshvili. 2017. Methodology of Translation
Alignment of Georgian Text of Plato’s “Theaetetus”.
International Journal of Language and Linguistics,
4(4).
Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer,
John Bodel, Jonathan Prag, Ion Androutsopoulos,
and Nando de Freitas. 2023. Machine Learning forAncient Languages: A Survey. Computational Linguistics.
Nina Vyatkina and Alex Boulton. 2017. Corpora in Language Teaching and Learning. Language Learning
and Technology, (3).
Jean Véronis, editor. 2000. Parallel Text Processing:
Alignment and Use of Translation Corpora. Text,
Speech and Language Technology. Springer Netherlands, Dordrecht-Boston-London.
Sisi Xie. 2023. Textual Alignment of Res Gestae: Translation in Historical Languages. The Stoa: a Review
for Digital Classics.

Tariq Yousef, Chiara Palladino, and Stefan Jänicke.
2022a. Transformer-based named entity recognition
for ancient greek.
Tariq Yousef, Chiara Palladino, Farnoosh Shamsian,Anise d’Orange Ferreira, and Michel Ferreira dos
Reis. 2022b. An automatic model and Gold Standard
for translation alignment of Ancient Greek. In Proceedings of the Thirteenth Language Resources and
Evaluation Conference, pages 5894–5905, Marseille,
France. European Language Resources Association.
Tariq Yousef, Chiara Palladino, Farnoosh Shamsian, andMaryam Foradi. 2022c. Translation Alignment with
Ugarit. Information, 13(2):65. Number: 2 Publisher:
Multidisciplinary Digital Publishing Institute.
Tariq Yousef, Chiara Palladino, David J. Wright, andMonica Berti. 2022d. Automatic Translation Alignment for Ancient Greek and Latin. In Proceedings of
the Second Workshop on Language Technologies for
Historical and Ancient Languages, pages 101–107,
Marseille, France. European Language Resources
Association.
Ieva Zarin, a, Pēteris ikiforovs, and Raivis Skadin, š. 2015.
Word alignment based parallel corpora evaluationand cleaning using machine learning techniques. In
Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages
185–192.
Robert Östling and Jörg Tiedemann. 2016. Efficient
Word Alignment with Markov Chain Monte Carlo.
The Prague Bulletin of Mathematical Linguistics,
106.

A

Haneul Yoo, Jiho Jin, Juhee Son, JinYeong Bak,
Kyunghyun Cho, and Alice Oh. 2022. HUE: Pretrained Model and Dataset for Understanding Hanja
Documents of Ancient Korea. In Findings of the
Association for Computational Linguistics: NAACL
2022, pages 1832–1844, Seattle, United States. Association for Computational Linguistics.
Tariq Yousef. 2015. Word alignment and named-entity
recognition applied to greek text reuse. MSc’s Thesis. Alexander von Humboldt Lehrstuhl für Digital
Humanities, Universität Leipzig.
Tariq Yousef. 2023. Translation Alignment Applied toHistorical Languages. Ph.D. thesis.
Tariq Yousef, Gerhard Heyer, and Stefan Jänicke. 2023.
Evalign: Visual evaluation of translation alignment
models. In Proceedings of the 17th Conference of
the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages
277–297.
Tariq Yousef and Stefan Janicke. 2020. A Survey of
Text Alignment Visualization. IEEE Transactions on
Visualization and Computer Graphics, PP:1–1.

189

AppendixFigure A.1: Ugarit manual alignment tool, Side-by-side visualization of bilingual aligned texts.

Figure A.2: Visualization of translation pairs search results.

190

Figure A.3: Ugarit automatic alignment tool.

Figure A.4: An example from graph clustering results with extended relations.

191

Figure A.5: An example of the annotation projection using the proposed pipeline.

192

Using Word Embeddings for Identifying Emotions Relating to the Body ina Neo-Assyrian Corpus
Ellie Bennett and Aleksi Sahala
University of Helsinki
eleanor.bennett@helsinki.fi
aleksi.sahala@helsinki.fi

Abstract

Neo-Assyria is the best represented time period
currently available on the Open Richly Annotated
Cuneiform Corpus (Oracc). We can therefore useword embeddings to gather quantitative data ofwhether the body words identified by Steinert toexpress emotions were used similarly to emotion
words during the Neo-Assyrian period. In addition, word embeddings will allow us to determine
whether any of these body words can be consideredas part of a semantic field of emotions.

Research into emotions is a developing fieldwithin Assyriology, and NLP tools for Akkadian texts offers new perspectives on the data.
We use PMI-based word embeddings to explore
the relationship between parts of the body andemotions. Using data downloaded from Oracc,
we ask which parts of the body were semantically linked to emotions. We do this through examining which of the top 10 results for a bodypart could be used to express emotions. After
identifying two words for the body that have themost emotion words in their results list (libbu
and kabattu), we then examine whether thoseemotion words were indeed used in this mannerin the Neo-Assyrian textual corpus. The results
indicate that of the two body parts, kabattu wassemantically linked to happiness and joy, andhad a secondary emotional field of anger.

1

2

The data for this project consists of two sections:the corpus of Neo-Assyrian texts, and the list of
Akkadian words that relate to parts of the body.
2.1

Introduction

The study of emotions in ancient Near Easterncultures has grown in the past five years, withtwo edited volumes covering the topic (Hsu and
Llop Raduà, 2021; Sonik and Steinert, 2022). A
key question has been how ancient languages used
the body to express emotions - or the ‘embodimentof emotions’. For Akkadian, this question has beenaddressed by Ulrike Steinert (Steinert, 2022, 2021).
Her work involved the traditional approach of close
reading Akkadian texts and close analysis of themost important Akkadian dictionaries to identify
words relating to emotions, and how the body wasused to express them. She identified 22 Akkadianwords that refer to parts of the body that were used
to express emotions (Steinert, 2022). As Steinert’s
results were reflective of Akkadian language as awhole, we were interested in whether her results
could be replicated for a textual corpus from a morenarrow timespan - namely the Neo-Assyrian period(c. 934-612 BCE).1
1
There is growing research into emotions in Neo-Assyrian
material (Valk, 2016; Battini, 2022; Bach, 2022; Nadali, 2022;

Data

Neo-Assyrian corpus

The textual corpus was downloaded from Oracc.2
The texts were selected according to metadata tags
found in the Oracc data. We chose a ‘fuzzy’ approach to the data, and cast a wide net in order
to include as many Neo-Assyrian texts written in
Akkadian as possible. We therefore included texts
with the following tags for language and time period:• ‘Akkadian’; ‘akkadian’; ‘Akkadian with Sumerian incipits’; ‘Akkadian, Aramaic?’; ‘Akkadian, with Aramaic epigraph’; ‘Akkadian?’;‘Assyrian’; ‘Akkadian, Aramaic’• ‘Neo-Assyrian’; ‘9th/8th century’; ‘8th/7th
century’; ‘9th century’; ‘7th century’; ‘8th
century’
The data is in a word per line format, where every
word is represented in the following lemmatised
Schaudig, 2022; Bonatz, 2022; Morello, 2022), but scholarship is still limited with regards to embodied emotions.
2This was done through a remix of the following script,
which resulted in a lemmatised version of the Oracc dataset
in line with Oracc standards: https://github.com/
niekveldhuis/compass.

193
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 193–202,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_022

form:3

be computed from the vectors by using cosine similarity (Equation 1).

lemma[guideword]EPOS4
Stop words are given as ‘<stop>’,5 unlemmatised words as ‘_’,6 and ‘#’ to indicate the end of atext.
The data underwent a process of minimal cleaning to remove duplicates that appeared due tospelling errors, resulting in a corpus of 7,969 texts,
1,014,890 tokens, and 19,436 unique word forms.7
2.2

Word selection

We referred to Ulrike Steinert’s work on embodied
emotions in Akkadian to select the words that weremost likely to be similar to emotion words in the
Neo-Assyrian dataset (Steinert, 2021, 2022). We
selected 22 words relating to the head (5 words),
torso (or the whole body) (3 words), organs (6words), and limbs (8 words).8 They can be viewed
in Table 1.

3

Word Embeddings

Word embeddings represent words as real-valued
vectors in a multi-dimensional vector space. In simple terms, this vector space can be understood as amatrix, where the rows represent each word in the
corpus and the columns represent abstractions of
their co-occurrences with other words. This property makes word embeddings useful for measuring
lexical similarity: if two words A and B are similar
to each other in meaning, they likely occur in similar contexts, and thus their vectors should show
higher similarity to each other. The most closely
related words (often called nearest neighbors) can3

As Akkadian is a highly inflected language, and the research was not focused on syntax or morphology, we worked
from the lemmatised version of the dataset. This follows previous Assyriological research projects based on Oracc data
(Svärd et al., 2020; Sahala and Svärd, 2021; Bennett, 2023)
4This is how the Akkadian words will appear in the tablesof this contribution for the ease of non-Assyriological readers,
but in the main body we will be using Assyriological standardsof italicising the lemma.
5
A full list of these can be found in the accompanying Zenodo repository, the url of which can be found in Appendix A.
6
An unlemmatised word in Oracc can be due to many factors, such as a broken text (indicated by ‘x’ in transliterations),
or a word with uncertain meanings.
7
The dataset is part of the supplementary material, which
is described in Appendix A. The corpus can be broken down
according to genre in the following manner: letters (33%),
royal inscriptions (17%), transactions (15%), scholarly texts(14%), and the remaining 21% was a mix of administrative,
religious, legal, political, literaturary, and untagged texts.
8
We did not include bodily emissions in this research,
but could be an interesting future field of emotions research(Sonik, 2022a).

cos(A, B) =A·B
∥A∥∥B∥(1)

In word embedding calculations, this metric hasa theoretical upper bound of 1 in case the words
are perfect synonyms, and a lower bound of 0 incase the semantic relationship between the words
is independent.9
Some of the better known tools for building wordembeddings are Word2vec (Mikolov et al., 2013)
and fastText (Bojanowski et al., 2017), which bothuse shallow neural networks to learn the word embedding space from the corpus. While the neural network based methods are powerful in large
datasets, these methods are known to have issues insmaller datasets comprising only a million words(Jungmaier et al., 2020). For low-resource languages and small corpora, word embeddings canbe successfully built by using count-based methods, such as word association measures combinedwith matrix factorization (Levy et al., 2015; Jungmaier et al., 2020). The count-based method usedin this paper produces vectors only for words thatare documented in the data set. Therefore the similarity between out-of-vocabulary words cannot bemeasured like in fastText.
PMI-embeddings is a Python script for building
count-based word embeddings for low-resource
languages.10 It combines several features fromvarious research papers on count-based word embeddings that utilize Pointwise Mutual Information (PMI) (Church and Hanks, 1990) and Singular Value Decomposition (SVD). Briefly described,
PMI-embeddings builds a sparse matrix of the vocabulary that encodes statistical significance of cooccurrences in terms of PMI within a symmetriccontext window of arbitrary size. This sparse matrix is then truncated into desired dimensionality(typically between 60 and 300) using SVD, yielding the final vector space that can be saved into thestandard Word2vec format to be used in various
NLP applications.
The script allows tuning several hyperparameters
to find optimal settings for the given dataset. These
include, but are not limited to, Dirichlet Smoothing
9

Negative scores up to -1 are also possible depending on
how the word embeddings are created, but in this paper discussion on negative values is not necessary.
10
https://github.com/asahala/pmi-embeddings/

194

Head
ı̄nu[eye]Npānu[front]Npû[mouth]N
qaqqadu[head]Nrēšu[head]NBody
pagru[body]Nšı̄ru[flesh]N
zumru[body]NOrganskabattu[liver]Nkalı̄tu[kidney]N
karšu[stomach]N
libbu[interior]N
qerbu[centre]Ns.urru[interior]NLimbs
ahu[arm]Ndurā’u[arm]N
idu[arm]Nizı̄ru[arm]Nkirimmu[(crook-of)-arm]Nkittabru[arm]Npurı̄du[leg]Nzāqu[arm]N

Table 1: The 22 Akkadian words relating to the body selected for this study organised according to whether theyrefer to the head, the whole body or torso, the organs, or limbs.

(Turney and Pantel, 2010; Jungmaier et al., 2020),
several variants of PMI such as shifted and context distribution smoothed PMI (Levy et al., 2015)
with various different shifting mechanisms, Dynamic Context Window that gives less significancefor co-occurrences that are further apart (Sahlgren,2006), and Context Similarity Weighting (Sahala
and Lindén, 2020) that downsamples noise causedby duplication and repetitiveness in the dataset,
which is a problem especially in formulaic Mespototamian royal inscriptions.
If a human-evaluated gold standard for semantic
similarity is available, well performing parameters
can be searched by using the hypertune.py script
distributed with PMI-embeddings. Currently thedefault settings of PMI-embeddings have been defined for the first millennium Akkadian texts using
a work-in-progress gold standard based on independent word similarity rankings done by five Assyriologists. This gold standard is distributed as apart of PMI-embeddings.11
3.1• Symmetric dynamic context window of threewords, meaning that the co-occurrences arecalculated within a span of three preceding
and following words to the center word, andthat the co-occurrence frequencies are reciprocals of their distance to the center word giving
less importance to words that co-occur farther
away from each other.
• Context Similarity Weighting with a k-valueof 3, meaning that co-occurrences in repetitive or partially repetitive contexts are downsampled with a weight risen to the powerof 3. This gives significant penalty to cooccurrences also in partially repetitive contexts.

4

Results

Parameters

The default parameters use a vector dimensionalityof 300 and the following features for calculating
the sparse PMI matrix:• Shifted PMI with a shift value of 7 using
the formula implemented in Jungmaier et al.
2020. This allows some co-occurrences to exist in the sparse matrix even if they are notstatistically significant. Typically all statistically independent co-occurrences would bedisregarded, which may be harmful in sparse
datasets.
11
https://github.com/asahala/pmi-embeddings/tree/main/eval built in cooperation with the University of Helsinki, the LMU Munich
and the University of California, Berkeley.

Figure 1: Ranges of the differences between the firstand last cosine score in the top ten results for Akkadianbody words. Libbu is highlighted in black.

We queried the resulting .vec file for each of the
22 Akkadian words relating to the body for the
10 words whose vectors were the most similar to195the Akkadian body word, as well as their cosine
similarity score.
Libbu had the largest range of cosine similarity
scores between the first and tenth result out of all
the results lists. Kalı̄tu (“kidney”) had the smallest
difference in cosine score in its top ten results. The
mean of the differences in cosine similarity scoreof the top 10 results was 0.16, and the median was
0.15, indicating most of the body words queried
had a similar range of usage in the Neo-Assyrian
corpus.
The difference between the body words with thehighest (libbu) and second highest (ahu) range incosine scores was 0.085 - the largest difference
between the cosine ranges (Fig. 1).12 The large
range in cosine score in comparison to the otherresults lists suggests that libbu was used in manydifferent contexts, whereas the smaller difference
between kalı̄tu’s first and tenth results suggest it
had a more specific usage in Neo-Assyrian texts.
The words in the results lists for each body word
were then compared to a master Excel file that classified emotions according to 18 different emotion
categories: sadness, distress, suffering, anger, happiness, schadenfreude, pleasure, fear, hate, love,
desire, disgust, sympathy, envy, pride, surprise,
shame, sexual arousal.13
If a word in the results lists occurred in this Excel
file, the principle emotional field was imported to
the results list for analysis. This highlighted thewords in the results list that could be used to denoteemotions. This was our first indicator of whether abody word was part of an emotional semantic field.
4.1

libbu
Word
kindu[earth]Nhı̄pu[break]N
gilittu[terror]Nqı̄lu[burning]N
haluppu[(a-tree)]Nhūs.u[pain]N
hibs.u[swelling]N
terku[blow]Nzenûtu[anger]N
papānu[(a-kind-ofrush)]N

EmotionNone
Sadness
Fear
None
None
Sadness
None
Fear
Anger
None

Table 2: The ten words most similar to libbu in the NeoAssyrian corpus.

kabattuWordtes.û[defecate]V
s.abru[blinker]AJ
alālu[sing-a-joyfulsong]Velēs.u[swell]Vmussahhiru[turned-tosomeone)]AJ
hadû[be(come)joyful]Vmušnēšu[keepingalive]AJ
aggu[furious]AJ
qurruru[(meaningunknown)]AJ
epēru[feed]VWords relating to the organs

The group of words referring to organs (column 3
in Table 1) was most likely to have a word in its results list that could express emotions. Of the word
relating to organs, libbu (“interior”) and kabattu(“liver”) had the highest number of emotion words
in their results list (5 and 4 respectively, as seen in
Tables 2 and 3).14
12

Removing libbu from the results does not change themean and median of the ranges of cosine scores for the top 10
results of each body word.
13
The full list of emotional fields can be found in the accompanying Zenodo repository (Appendix A).
14
Although the guideword translates this as “interior”, libbu
is actually a difficult word to translate. The precise definition
is a general sense of “inside”, including “inside” a house or
the body. The lack of precision for precisely which organ
inside the human body libbu referred to has led to scholars
translating the term variously as “heart”, “liver”, and “mind”(CAD L: 164 libbu). We decided to keep the original sense of

Score
0.786
0.525
0.496
0.462
0.458
0.457
0.453
0.443
0.441
0.432

Score
0.793
0.788
0.770

EmotionNone
None
Happiness

0.715
0.701

Happiness
None

0.697

Happiness

0.683

None

0.661
0.661

Anger
None

0.660

None

Table 3: The ten words most similar to kabattu in the
Neo-Assyrian corpus.

196

The results for libbu have a surprising feature.
The scores between the first and second results(kindu and hı̄pu, respectively) have the biggest difference of any two scores in the results lists. Kindu,a Sumerian term found in bilingual texts twice, hasa score of 0.786, and hı̄pu has a score of 0.525 - a
difference of 0.261. As kindu only occurs twicein the corpus, and libbu occurs 5,710, this result
is either a quirk of the PMI embeddings script, or- more likely - is representative of the wide usage
range of libbu in Neo-Assyrian texts.
The emotion words in the results list of libbu
are overwhelmingly negative: hı̄pu (“break”) andhūs.u (“pain”) were used in the expression of sadness (Wende, 2022).15 Gilittu (“terror”) and terku(“blow”) were used to express fear, and zenûtu
(“anger”) was used for anger (Wende, 2022; Bach,2022; Svärd et al., 2020).16
In comparison, the emotion words similar to kabattu were mostly positive (Table 3). Alālu (“singa joyful song”), elēs.u (“swell”), and hadû (“be joyful”), were all used to denote happiness (Wende,2022; Bach, 2022).17 The only outlier is aggu (“furious”), which was used to express anger (Wende,2022; Bach, 2022).18
Of the other words relating to organs, qerbu(“centre”) had the third highest number of words relating to emotions in its results list, with 2 results.19
Zā’eru (“hostile”) was the 4th highest result, witha cosine similarity score of 0.508.20 It was used in
the expression of despisement or hate. The ninth
word in the results was maqtu (“fallen one”), witha cosine similarity score of 0.471.21 The principal
emotional field of this word is surprise.
Finally, s.urru’s (“interior”) results list also featured a word relating to emotions: rūbu (“anger”),
ranked ninth, with a cosine similarity score of
0.669.22
the word, and have simply left it as the guideword “interior”.
CAD K: 11 kabattu.
15
CAD H: 195 hı̄pu; CAD H: 260 hūs.u.
16
CAD G: 71 gilittu.
17
CAD A1: 331 alālu; CAD E: 88 elēs.u; CAD H: 25 hadû.
18
CAD A1: 150 aggu.
19
CAD Q: 216 qerbu.
20
CAD Z: 14 zā’iru.
21
CAD M1: 254 maqtu.
22
As with libbu, s.urru was also used to denote a general“inside” of the human body, and cannot be connected to any
individual (or group of) organs as we understand them today.
CAD S.:259 s.urru; CAD R: 400 rūbu.

4.2

Words relating to the limbs

Of the eight words relating to the limbs, four had
words that could be used to express emotions intheir results lists.
Purı̄du (“leg”) had the highest number of emotion words in its list, with muštarhu (“presumptuous one”) ranked third (0.543 cosine similarity) andpetû (“open”) ranked fourth (0.519 similarity).23
Muštarhu expresses pride whereas petû expressedhappiness.
Two of the other words for parts of the arm
had positive emotions appear in their results list.
Zāqu (“arm”) saw .tābu (“good”) as the most similar word, with a cosine similarity score of 0.751,
and was used to express happiness.24 The ninth result for kirimmu (“crook of the arm”) was narāmtu(“beloved”, cosine value 0.508), used to express
love.25
Lastly, the seventh result for ahu (“arm”) was
adirtu (“gloominess”) (cosine similarity of 0.546),
and was used to express sadness.26 This was the
only result connected to a negative emotion for thewords relating to limbs.
4.3

Words relating to the head, torso, andwhole bodyNone of the Akkadian words relating to the head,torso, or the whole body had a word denoting emotions in their results lists.

5

Analysis

The only body words that had more than three results which could be used to express emotions were
libbu and kabattu. Previous scholarship regarding
Neo-Assyrian embodied emotions has identified
these were often used in phrases to express emotions (Bach, 2022; Luukko, 2021; Morello, 2022;
Sonik, 2022b). The results appeared to thereforealign with current Assyriological research.
Thus far, we have identified words in results lists
that could be used to express emotions. Many ofthese words can be translated in different ways andhave alternative usages. This section will assess
whether the words in the results lists for libbu andkabattu were indeed used to convey emotions in the
Neo-Assyrian corpus. This will indicate whether
23
CAD P: 517 purı̄du; CAD M2: 286 muštarhu; CAD P:
338 petû.
24
CAD Z: 64 zāqu; CAD T.: 19 .tābu.
25
CAD K: 406 kirimmu; CAD N1: 342 narāmtu.
26
CAD A1: 205 ahu; CAD A1: 127 adirtu.

197

emotions in general were part of the semantic field
for either libbu or kabattu. It will also examine
whether the emotions conveyed with these wordsalign with current Assyriological research into embodied emotions.
5.1

Emotion words similar to libbu

Libbu is consistently identified in Assyriologicalscholarship as the main seat of emotion, and is described in Akkadian phrases as where anger, fear,
and joy were felt (Bach, 2022; Luukko, 2021; Steinert, 2022; Schaudig, 2022).
The first word relating to emotions in libbu’s
results list was hı̄pu (“break”), which was the second overall result. In the Neo-Assyrian corpus,
hı̄pu was attested 72 times, and was used in three
ways: to describe a break in a text the scribe wascopying;27 to describe quarrying lapis lazuli;28 thebreakage of physical objects;29 and to describe thebreaking of the libbu as an expression of sadness.
Only 3 attestations were to express emotion andwere only found in the royal inscriptions of Assurbanipal in the construction hı̄p libbu (“broken
libbu”, often translated as “broken heart”) (Wende,2022).30 Therefore, the primary usage of hı̄pu in
the Neo-Assyrian corpus was not to express emotion, but when it was, it was done so only withlibbu.
The second word on the results list that could
express emotion was gilittu (“terror”). Gilittu occurs most frequently in the Neo-Assyrian corpus indisregard formulae of oracular queries (22 out of 27
attestations) (Svärd et al., 2020; Wende, 2022).31
In all of the attestations, gilittu was used to ex27
In order to aid non-Assyriologists, we refer the reader
to the electronic versions of the texts hosted on Oracc.
When possible, the urls also link to the exact location of
the word under discussion in the text. For examples ofhı̄pu used to decribe a break in an original tablet the scribe
copied, this can be seen in the first three lines of http://oracc.org/blms/P394721, lines rev. ii 8-19 in http://oracc.org/cams/gkab/P338598, and four lines in
http://oracc.org/dcclt/nineveh/P386432.
28
For
examplehttp://oracc.org/rinap/
rinap1/Q003421.4.5;
http://oracc.org/
rinap/rinap4/Q003232.125.4;
and
http:
//oracc.org/rinap/rinap4/Q003230.308.5.
29
For example in a recipe for making glass in line rev. 37
in http://oracc.org/glass/P394484.
30
http://oracc.org/rinap/rinap5/Q003819.
5.2;
http://oracc.org/rinap/rinap5/
Q007602.249.2;
http://oracc.org/rinap/
rinap5/Q003710.875.3.
31
For example, in http://oracc.org/saao/
saa04/P238980.29.12,
http://oracc.
org/saao/saa04/P239001.16.1,
and
http:
//oracc.org/saao/saa04/P238965.22.1.

press fear, and more specifically acute terror. As
gilittu has a clear semantic field of fear in the NeoAssyrian corpus, its appearance as third on the results list for libbu suggests that of the emotions,libbu was connected strongly to fear.
The third emotion word on libbu’s results list
was hūs.u (“pain”). It is ranked sixth in the similarity rankings, and it is unsurprising to see hūs.ualongside hı̄pu in the results list for libbu, as thesewords were part of the compound expression hūs.
hipi libbi (“an emotional or physical pain within the
libbu”).32 However, in the Neo-Assyrian corpus
this phrase was not used to express an emotion, but
in scholarly commentaries to describe an abdominal pain.33 Thus, in the Neo-Assyrian corpus hūs.uconnected libbu with a medical semantic field.
The fourth word relating to emotions was terku(“blow”). Terku could be used to describe athrobbing emotional response, but of the 14 NeoAssyrian attestations this was only found in 1 text.
The text was a letter to the Assyrian king and includes a description of someone dying from a throbbing libbu due to hearing the speech of the king.34
The rest of the attestations were in two royal inscriptions and the omen series Šumma tirku. In allof these cases, terku was used to describe a physicaldark spot on either human skin or lambs’ wool.35
Overall, even though terku could be used to express a strong emotion or physical response, in the
Neo-Assyrian corpus it was used for anatomical
descriptions.
The final word used to express emotions in the
results list for libbu is zenûtu (“anger”). It had aclear connection with the emotional field of anger,
but was only used in two royal inscriptions fromthe reign of Esarhaddon in passages explaining
the anger of the god Marduk prior to destroying
Babylon.36 It followed the phrase ēziz libbašu (“hislibbu was furious”), which explains it’s appearance
in the results list, and indicates libbu’s connection
32

CAD H: 260 hūs.u.
For
example,http://oracc.org/ccpo/
P296515.8.1,
http://oracc.org/ccpo/
and http://oracc.org/ccpo/
Q005179.6.1,
P461217.7.2.
34
http://oracc.org/saao/saa10/P313436.
19.2
35For example, http://oracc.org/cams/gkab/
P363488.12.2,
http://oracc.org/rinap/
rinap4/Q003388.5.5,
http://oracc.org/
rinap/rinap4/Q003387.7.7.
36
http://oracc.org/rinap/rinap4/Q003335.
18.1,
http://oracc.org/rinap/rinap4/
Q003342.9.7.

198

33

to the semantic field of anger.
Overall, only two of the words that appearedon the results list for libbu were used exclusively
to express emotions in the Neo-Assyrian corpus:
gilittu and zenûtu. They expressed fear and anger,
respectively.
Traditional Assyriological research states libbu
was the seat of joy based on phrases that locate
joy within the libbu (Steinert, 2022; Luukko, 2021).
Our results diverge from this perspective, and offer
an alternative perspective of the emotions associated with libbu. Neither gilittu nor zenûtu wereused to express joy, and none of the words in libbu’s
results list that could be used to express emotions
were used to express joy. Interestingly, 27 attestations pair elēs.u with libbu. Elēs.u’s absence on
the results list for libbu further strengthens the argument that whilst joy might have been connectedwith libbu, libbu was not primarily associated withjoy or happiness.
Moreover, whilst libbu was used in a similarmanner to gilittu and zenûtu, the varied usage ofother words in the result list suggests that emotions
were not the principle semantic field for libbu. Thiswas made even more clear once the results list was
expanded to 20 results, as the results pointed to asemantic field more akin to illness and agriculture.
These can be found in the supplementary material(Appendix A). These results from close reading
also corroborate the suggestion from the ranges incosine similarity scores (Fig. 1), which suggested
libbu would have a broader semantic field thanother Akkadian body words in this study.
Emotions were therefore not the principle semantic field for libbu, and the results list speak to it’s
diverse usage in the Neo-Assyrian corpus. However, when it was used to express emotions, it wasused to express fear and anger.
5.2

Emotion words similar to kabattu

In Assyriological scholarship, kabattu is consistently identified as an important part of the bodywhere emotions were felt, especially for feelinghappiness and joy (Bach, 2022; Sonik, 2022b).
Of the emotion words in the results list for kabattu, alālu (“to sing a joyful song”) was the mostsimilar, and was ranked third in the results list. Of
the seven attestations in the Neo-Assyrian corpus,
it was used to express the gods’ joy five times,37
37

http://oracc.org/saao/saa03/P334929.
85.1,
http://oracc.org/rinap/rinap2/
Q006494.149.2,
http://oracc.museum.

and the joy of the king twice.38 Alālu was found inliterary texts, royal inscriptions, and a hymn. All ofthese genres drew upon literary topoi, which suggests alālu was imbued with metaphorical meaning(Wende, 2022).
Elēs.u (“to swell”)was the second word related
to emotions in the results list for kabattu, and was
ranked fourth. It is mostly attested in Neo-Assyrian
royal inscriptions (35 attestations out of 37 in the
corpus), and was largely used to describe the happiness and good moods of the gods and the kings(Bach, 2022).39 In six texts it was paired with kabattu (such as ētelis. kabattı̄), desribing how joy
made the liver swell.40
Hadû (“to become joyful”) was the third word
related to emotions in the results list for kabattu,
and was ranked sixth. Elēs.u and hadû were twoof four words identified by Mikko Luukko that
expressed joy or happiness in Assyrian archival
material (Luukko, 2021). Hadû specifically related
to divine joy and the joy of the king, which aligns
with Luukko’s view that the happiness and joy ofsuperiors was a concern for subordinates (Luukko,2021).41 There is also an interesting usage wherehadû was used to describe the happiness of fathersin literary texts.42 This points to a particular typeof joy that hadû expressed, and kabattu could besemantically connected to not just joy, but a specifickind of joy.
The final word relating to emotions that was on
the results list for kabattu was aggu (“furious”),
org/cams/gkab/P338326.37.2,
http://oracc.org/cams/gkab/P363581.33.4,
andhttp://oracc.org/cams/gkab/P338328.30.4.
38
http://oracc.org/rinap/rinap2/Q006488.
194.7,
http://oracc.org/rinap/rinap2/
Q006483.336.1.
39For
examplehttp://oracc.org/rinap/
http://oracc.org/
rinap2/Q006488.168.9,
rinap/rinap4/Q003230.496.4,
and
http:
//oracc.org/rinap/rinap5/Q007625.37.4.
40
http://oracc.org/cams/anzu/Q002771.
60.5,
http://oracc.org/ribo/babylon6/
http://oracc.org/rinap/
Q006325.77.2,
rinap2/Q006596.62.4,
http://oracc.org/
rinap/rinap2/Q006482.325.2, http://oracc.
org/rinap/rinap2/Q006483.189.8, and http://oracc.org/rinap/rinap2/Q006605.77.1.
41
For
example,http://oracc.org/riao/
Q004661.10.1, http://oracc.org/saao/saa09/
P337163.23.7, http://oracc.org/saao/saa01/
P224485.22.1, and http://oracc.org/saao/
saa04/P237053.13.8.
42
http://oracc.org/atae/huzirina/
P338675.56.2, http://oracc.org/cams/gkab/
P338321.83.3, and http://oracc.org/cams/
gkab/P338675.56.2.

199

which was ranked eighth. It was mostly used inroyal inscriptions to describe how the wrath of thegods was not appeased.43 Even though this wasa different emotional field, it ties in to the themeof emotions of gods seen in the usage of the otheremotion words similar to kabattu.
The findings align with the usage of kabattu insimilar types of texts. Of the 176 attestations, 120
were in royal inscriptions, which were heavily inspired by literary topoi in order to express emotions
as agreed upon by the king and the most seniorcircle of scribes (Bach, 2022).44 Most of the attestations of alālu, elēs.u, hadû, and aggu werein similarly literary texts. In addition, the results
demonstrate an overwhelming concern regarding
the happiness of gods and kings, aligning with previous scholarship suggesting happiness was a key
aim of Mesopotamian kingship (Morello, 2022;
Luukko, 2021).
Overall, the results for kabattu included words
that were used in texts like royal inscriptions which
made extensive use of literary motifs. Of the words
that were similar to kabattu, and could be used toexpress emotions, all were indeed used as such in
the dataset. Three out of four were used to express
happiness or joy. Therefore, kabattu was principally part of the semantic field of happiness andjoy. The fourth result (aggu) suggests a secondary
semantic field for kabattu was anger, based on thesewords’ usage in royal inscriptions and literary texts.

6

Conclusions

PMI-embeddings has therefore proven to be an important tool to not only identify which areas of the
body were most associated with emotions, but toidentify which body parts were semantically connected to specific emotional fields in Neo-Assyrian
texts.
We highlighted two words that were most similarin usage to words that could be used to express emotions: libbu and kabattu. Our much more limited
list of Akkadian words for body parts that were semantically linked with emotions than Steinert’s list

suggests the Neo-Assyrian corpus has temporallyspecific methods of embodying emotions. Future
research could compare our results to datasets built
from texts of a different period, such as Middle Assyrian. Libbu and kabattu were the two Akkadianwords relating to the body that were most likely
to be part of the semantic field of emotions, aligning with the findings of previous Assyriologicalresearch (Sonik, 2022b; Steinert, 2021; Luukko,2021; Wende, 2022).
We have been able to corroborate results withclose readings of texts for kabattu, and solidify thatin the Neo-Assyrian texts a primary semantic fieldof the liver was not just emotions generally, butmore specifically the semantic fields of happinessand anger.
We have also demonstrated how the results from
PMI-embeddings can complement and add to close
reading approaches in order to provide a more nuanced reading of Neo-Assyrian embodied emotions.
We demonstrated that the emotion words most similar to the usage of libbu were in the emotional fieldof fear and anger, which does not align with traditional close reading approaches. Furthermore, ourresults demonstrate that emotions were only one
facet of the many usages of libbu in Neo-Assyrian
texts.
PMI-embeddings is therefore a tool that can both
corroborate results from close-readings, but more
importantly has provided new perspectives on how
Neo-Assyrian texts embodied emotions. On thebasis of this single case study, PMI-embeddings
will become vital in the suite of tools for digital
Assyriology.

Acknowledgements

This research was conducted as part of the project
Embodied Emotions: Ancient Mesopotamia andToday, funded by the Finnish Cultural Foundation.
The PMI-embeddings script was developed withthe assistance of the Centre of Excellence Ancient
Near Eastern Empires, funded by the Academy ofFinland (decision number 352747).
We would like to thank Dr. Ulrike Steinert, who
43This was in eight out of 12 attestations. For example,
graciously allowed us access to the research of the
http://oracc.org/rinap/rinap5/Q003705.
538.3,
http://oracc.org/rinap/rinap5/
project “Akkadische und Hethitische EmotionsbeQ003703.316.3, and http://oracc.org/rinap/
griffe im Kontext” (AHEC).
rinap5/Q003778.23.3.
44This research could not have been carried out
For example, http://oracc.museum.upenn.
edu/rinap/rinap2/Q006596.62.4,
http:without the richly annotated dataset on Oracc, al//oracc.museum.upenn.edu/rinap/rinap4/
Q003286.275.1,
and http://oracc.museum. most all of which was manually inputted. We
upenn.edu/rinap/rinap5/Q007615.16.6.
would like to thank the Steering Committee of200

Oracc, Jamie Novotny, Eleanor Robson, Steve Tinney, and Niek Veldhuis. We would also like tothank the PIs of the many Oracc sub-projects, andthe scores of researchers who were involved in thelemmatisation and digitisation process.

Abbreviations
CAD: The Assyrian Dictionary of the Universityof Chicago

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013. Efficient estimation of word representations in vector space.
Nathan Morello. 2022.
Joy and Happiness in
Mesopotamian Royal Inscriptions. In Karen Sonik
and Ulrike Steinert, editors, The Routledge Handbook of Emotions in the Ancient Near East, 1 edition,
pages 455–471. Routledge, London.
Davide Nadali. 2022. Shaming the Enemy in Assyrian Palace Reliefs and Royal Inscriptions. In Karen
Sonik and Ulrike Steinert, editors, The Routledge
Handbook of Emotions in the Ancient Near East, 1
edition, pages 614–627. Routledge, London.

References
Johannes Bach. 2022. Emotions and Assyrian Kingship. In Karen Sonik and Ulrike Steinert, editors,
The Routledge Handbook of Emotions in the Ancient
Near East, 1 edition, pages 329–358. Routledge, London.
Laura Battini. 2022. Sennacherib’s Sieges and Deportations Reliefs: How to Increase Emotions. Avar: An
Interdisciplinary Journal of Life and Society in theAncient Near East, 1(2):313–359.
Ellie Bennett. 2023. Age and Masculinities during the
Neo-Assyrian Period. Journal of Cuneiform Studies,
75:123–154. Publisher: University of Chicago press.
Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information.
Dominik Bonatz. 2022. Fear and Terror in Assyrian
Palace Reliefs. In Karen Sonik and Ulrike Steinert,
editors, The Routledge Handbook of Emotions in
the Ancient Near East, 1 edition, pages 525–543.
Routledge, London.
Kenneth Ward Church and Patrick Hanks. 1990. Word
association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29.

Aleksi Sahala and Krister Lindén. 2020. Improving word association measures in repetitive corpora with context similarity weighting. In Proceedings of the 12th International Joint Conferenceon Knowledge Discovery, Knowledge Engineeringand Knowledge Management, IC3K 2020, Volume
1: KDIR, Budapest, Hungary, November 2-4, 2020.
SCITEPRESS Science And Technology Publications.
Aleksi Sahala and Saana Svärd. 2021. Language technology approach to “seeing” in Akkadian. In The
Routledge Handbook of the Senses in the Ancient
Near East, 1 edition, pages 561–575. Routledge, London.
Magnus Sahlgren. 2006. The word-space model: using distributional analysis to represent syntagmaticand paradigmatic relations between words in highdimensional vector spaces.
Hanspeter Schaudig. 2022. Anger and Hatred in NeoAssyrian and Neo-Babylonian Royal Inscriptions. In
Karen Sonik and Ulrike Steinert, editors, The Routledge Handbook of Emotions in the Ancient Near
East, 1 edition, pages 631–647. Routledge, London.

Shih-Wei Hsu and Jaume Llop Raduà, editors. 2021.
The Expression of Emotions in Ancient Egypt andMesopotamia. BRILL.

Karen Sonik. 2022a. Emotions and Body Language.
The Expression of Emotions in Visual Art. In Karen
Sonik and Ulrike Steinert, editors, The Routledge
Handbook of Emotions in the Ancient Near East, 1
edition, pages 269–325. Routledge, London.

Jakob Jungmaier, Nora Kassner, and Benjamin Roth.
2020. Dirichlet-smoothed word embeddings for lowresource settings. In Proceedings of the 12th Language Resources and Evaluation Conference, pages
3560–3565, Marseille, France. European Language
Resources Association.

Karen Sonik. 2022b. Emotions and Body Language.
The Expression of Emotions in Visual Art. In Karen
Sonik and Ulrike Steinert, editors, The Routledge
Handbook of Emotions in the Ancient Near East, 1
edition, pages 269–325. Routledge, London.

Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned
from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225.
Mikko Luukko. 2021. Expressions of Joy and Happiness in Neo-Assyrian. In Shih-Wei Hsu and Jaume
Llop Raduà, editors, The Expression of Emotions
in Ancient Egypt and Mesopotamia, pages 255–282.
BRILL.

Karen Sonik and Ulrike Steinert, editors. 2022. The
Routledge Handbook of Emotions in the Ancient Near
East, 1 edition. Routledge, London.
Ulrike Steinert. 2021. Pounding Hearts and Burning
Livers: The Sentimental Body in Mesopotamian
Medicine and Literature. In Shih-Wei Hsu and Jaume
Llop Raduà, editors, The Expression of Emotions
in Ancient Egypt and Mesopotamia, pages 410–469.
BRILL, Leiden.

201

Ulrike Steinert. 2022. Emotion and the Body: Embodiment, Conceptual Metaphor, and Linguistic Encoding of Emotions in Akkadian. In The Routledge
Handbook of Emotions in the Ancient Near East, 1
edition, pages 51–87. Routledge, London.
Saana Svärd, Tero Alstola, Heidi Jauhiainen, Aleksi
Sahala, and Krister Lindén. 2020. Fear in Akkadian Texts: New Digital Perspectives on Lexical Semantics. In S-W Hsu and J. Llop-Raduà, editors,
The Expression of Emotions in Ancient Egypt andMesopotamia, number 116 in Culture and History of
the Ancient Near East, pages 470–502. Brill, Leiden.
Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research,
37:141–188.
Jonathan Valk. 2016. “They Enjoy Syrup and Ghee atTables of Silver and Gold”: Infant Loss in Ancient
Mesopotamia. Journal of the Economic and Social
History of the Orient, 59(5):695–749.
Janine Wende. 2022. Akkadian Emotion Terms. In
Karen Sonik and Ulrike Steinert, editors, The Routledge Handbook of Emotions in the Ancient Near
East, 1 edition, pages 150–170. Routledge, London.

A

Supplementary Data

Supplementary data is available to view for thisresearch. Accompanying files can be found via thefollowing url:
https://doi.org/10.5281/zenodo.
8272738
The data includes:• A ReadMe file.
• The Neo-Assyrian corpus downloaded from
Oracc.
• A .txt file listing the terms for the body.
• The Excel spreadsheet with the emotion words
and their emotional categories.
• The vector files generated by the method outlined in the main text.
• An Excel file with the top 10 results for each
word, their cosine similarity score, the frequencies for the words, and their principal
emotional fields.
• All images used in this publication.

202

BabyLemmatizer 2.0 – A Neural Pipeline for POS-tagging andLemmatizing Cuneiform Languages
A. J. Aleksi Sahala
University of Helsinki, Finland
aleksi.sahala@helsinki.fi

Krister Lindén
University of Helsinki, Finland
krister.linden@helsinki.fi

Abstractin the Open Richly Annotated Cuneiform Corpus(Oracc) (Tinney et al., 2006). The simplest finite
We present BabyLemmatizer 2.0, a linguissurface form, the third person singular G-present
tic annotation pipeline for POS-tagging andinaddin "he/she gives" is spelled in seven different
lemmatizing cuneiform languages, as well as
ways
in Oracc: logographically IN.SUM, SUM,
pretrained models for a variety of ancient
SUM{+in},logo-syllabically SUM-in and syllabiMesopotamian languages and dialects. We evalcally i-na-din, i-na-di3 -in and ina-ad-din. Similarly
uate the system on two dialects of Akkadian:
Assyrian and Babylonian, as well as on two
the third person singular G-preterite and G-perfectgenealogically unrelated cuneiform languages:
forms iddin and ittadin are spelled in eight and five
Sumerian and Urartian. We also test our systemdifferent ways in Oracc, respectively.
on Ancient Greek and Latin to experiment with
Part-of-speech (POS) tagging is another imporits performance on non-cuneiform languages.
tantconcept in the NLP of morphologically comOur system achieves a POS-tagging accuracy
plex
languages. Besides its obvious use, that is,
between 95-98% and a lemmatization accuracyof 94-96% depending on the language or disearching for words that belong to a certain POSalect. The system can predict correct POS-tags
class, POS-tags can be used to some extent to disfor 83-91%, and lemmata for 68-84% of outambiguate lemmatization. For instance, in Akkaof-vocabulary word forms depending on thedian the logogram IGI can denote various concepts
language or dialect.
depending on its context. If preceded by a preposition, it often denotes being in front of something
1 Introduction(e.g. ina IGI = ina pāni), but in other contexts it
Lemmatization is a linguistic annotation task that
can also mean šı̄bu "witness", nāmuru "be(come)labels words with their dictionary forms. This isvisible" or ı̄nu "eye", among many other readings
essential for morphologically complex highly in- and meanings.
flectional and agglutinative languages, where theTraditionally Akkadian and other cuneiform
relationship between surface forms and their dictio- language lemmatization and POS-tagging hasnary forms are opaque. In historical languages with
been done with rule-based systems, including aless standardized spelling, lemmatization becomes
dictionary-based and morphology-based methods.
even more crucial because also the relationshipThe disadvantage of dictionary-based lemmatizers
between the surface forms and their graphemic rep- is that they are unable to provide POS-tagging or
resentations may be obscure, and make searchinglemmatization for previously unseen word forms.
attestations of words belonging to highly inflec- Although morphology-based tools can produce antional part-of-speech classes difficult, or close tonotations for unseen word forms as long as theirimpossible, without cumbersome regular expres- lemmata and morphology have been defined, theysion based search queries.
struggle to deal with spelling variation, which isThis issue can be demonstrated with the Akka- difficult to describe using rules without producing
dian verb nadānu "to give", which occurs in 367 dif- excessive over-generation.
ferent surface forms and in 477 different spellings
In this paper we present an OpenNMT-based
0neural lemmatizer and POS-tagger for AkkadianAleksi Sahala was responsible for developing the tool,training and evaluating the models and writing the paper. Krisand other cuneiform languages. The presented neuter Lindén was the PI of the project and provided feedback for
ral network based approach aims to solve both ofthe manuscript. BabyLemmatizer 2.0 is accessible at https://github.com/asahala/BabyLemmatizer
the previously mentioned issues. It can learn many203
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 203–212,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_023

to-many relations between all spellings of wordforms and their possible lemmata in context, anduse these learned mappings to predict annotation
for previously unseen word forms, also in previously unseen spellings.
Lemmatization of cuneiform texts opens them toa variety of computational methods, including, but
not limited to semantic and network analysis, andenables harmonization of existing resources, whichis essential for the digitalization of Assyriologicalresearch.

2

Cuneiform and the Cuneiform
Languages

ically to give hints on how a logogram next to themshould be read by repeating some of its sounds
syllabically (Jagersma, 2010).
2.1

Transliteration of Cuneiform

Transliteration of cuneiform aims to represent theoriginal text in the Latin alphabet sign by sign. Conventionally, logograms are written in capital letters(except in Sumerian), and the syllabic signs arealways written in lowercase. The marking of determinatives and phonetic complements vary. In paper
publications they are written in superscript, but in
the Oracc notation they are wrapped in curly brackets. Phonetic complements are distinguished fromdeterminatives using a plus sign, as in APIN{+ru}
for the Akkadian word ikkaru "farmer" (Tinney andRobson, 2019).
Another detail relevant to this paper in cuneiform
transliteration is indexing that aims to separate
cuneiform signs with similar readings from eachother. The index of the sign (or its reading) is expressed in subscript numbers. For instance, there
are two common cuneiform signs that indicate thesyllable /šu/. To keep these two signs separate intransliteration, they are transliterated as šu and šu2 ,allowing the reader to know which sign was used in
the original source. This makes the transliterationof cuneiform reversible and more transparent.

The cuneiform writing system was used in ancient
Mesopotamia from the middle of the fourth millennium BCE until the first or the second century CE.
According to modern understanding, it was firstdeveloped by the Sumerians and later adapted byspeakers of several other languages such as Akkadian, Elamite, Hittite, Hurrian and Urartian.
Originally cuneiform was a logographic writing system, where all the signs denoted various
concepts, such as numbers and commodities thatwere relevant to trade, taxing and ownership in theearly Mesopotamian society. Around 2800 BCE
the writing system took its first clear steps toward
a more phonetic expression of human language by
2.2 Languages
allowing certain logograms to be used for marking
syllabic values (Michalowski, 2008). For example, For this paper, relevant languages are Akkadian,
the sign KA that originally denoted the Sumerian
Sumerian and Urartian.
word for mouth /kag/ began also to mark a phoAkkadian is best known as the language of the
netic syllable /ka/, which allowed ancient scribes toBabylonians and Assyrians. It belongs to the Eastexpress more abstract ideas such as combinations
Semitic languages and is documented in writingof grammatical affixes. After the cuneiform writ- from the Old Akkadian period ca. 2400 BCE toing system had been adopted to the East-Semitic
the first or the second century CE. The Assyrian diEblaic and Akkadian languages around the 25th
alect, once spoken in the northern Mesopotamia, iscentury BCE, the use of syllabic signs became
divided into three chronological variants: Old Aswidespread, as logograms alone were too ambigu- syrian (1950-1500 BCE), Middle Assyrian (1500ous for expressing the Semitic stem-internal mor- 1000 BCE), and Neo-Assyrian (1000-612 BCE).
phology (Michalowski, 2008).
The Babylonian dialect is divided into Old BabyloCuneiform signs can be used for four distinct
nian (2000-1500 BCE), Middle Babylonian (1500purposes. The two basic uses are logograms that 1000 BCE), Neo-Babylonian (1000-626 BCE) and
express ideas such as "king", "wife", "temple" orLate Babylonian (626 BCE-100 CE). An artificial"to build", and syllabograms that express syllable- language known as Standard Babylonian was also
like sounds like /ma, mu, mi, me/. The remaining
used in literary contexts by Akkadian speaking
uses are determinatives and phonetic complements. scholars for over a millennium. Although this lanThe former were used to classify words into various
guage was based on Old Babylonian, the texts writcategories, such as divine names, trees or woodenten in Standard Babylonian often contain residue
objects, and places among many others. Phonetic
from the contemporary spoken Babylonian and Ascomplements, on the other hand, were used sporad- syrian dialects.
204

Akkadian features a complex morphology thatcombines linear (prefixation and suffixation) andnonlinear (root-pattern morphology and infixation)
processes (Huehnergard and Woods, 2008). Akkadian is written mostly using syllabic signs, but aselection of logograms is also used. The extent
of logogram use varies depending on the time period and genre. Typically, everyday texts, such as
letters, do not contain many logograms, but they
are abundantly used until the later time periods inscholarly texts.
Sumerian was an isolate language first attested
in writing in the middle of the fourth millennium
BCE. Sumerian died as an everyday vernacular in
the 18th century BCE and transformed into a literary language used by the Babylonian and Assyrian
scholars in various contexts until the end of thecuneiform tradition circa the first or the second
century CE (Jagersma, 2010). As an agglutinating
language with 10 grammatical cases, possessive
suffixes and heavy verbal prefixation its morphology is quite rich, not as opaque as that of Akkadian.
Although Sumerian is generally written in a logosyllabic manner, the earliest texts were purely logographic, and some texts written after the second
millennium BCE used only syllabic signs. Counterintuitively, these syllabic, so-called unortographic,
texts are often the most difficult ones to understanddue to their high ambiguity (Michalowski, 2011).
Urartian was a language spoken in Asia Minor and the northern reaches of Mesopotamia. It
belonged to the Hurro-Urartian language family
and is attested between the 9th and the 7th century
BCE on inscriptions written in the Neo-Assyrian
cuneiform script (Wilhelm, 2008). Similarly toSumerian, Urartian is a heavily agglutinating language with a complex morphology, including ninegrammatical cases, Suffixaufnahme (stacking of
nominal suffixes in genitive constructions) and richverb affixation. Due to the relatively low numberof surviving inscriptions and their repetitive nature,
the Urartian language is far less understood than
Akkadian or Sumerian.

3

Digital Resources

important digital resources include the Cuneiform
Digital Library Initiative (Englund et al., 1998)(ca. 350,000 entries, including texts and metadata), Database of Neo-Sumerian Texts (Molina,2002) (ca. 105,000 Neo-Sumerian administrative
documents), The Electronic Babylonian Library
Fragmentarium (Jiménez et al., 2018) (16,000 fragments), Archibab (Charpin, 2009) (10,000 texts),
Ebla Digital Archives (Milano and Maiocchi, 2016)(3,000 texts) and Achemenet (Briant and Henkelman, 2009) (4,000 texts). For a more detailed survey on cuneiform language resources, see Charpin(2014).
Oracc contains ca. 2.22 million words of Akkadian as of 2023. As the total number of words inknown Akkadian tablets and inscriptions has been
estimated to be around 10 million (Streck, 2010), a
majority of Akkadian texts remain unannotated todate.1For Sumerian, Oracc hosts texts comprising 4.45
million words and they include the vast majority of
the important Sumerian texts and archives. Mostof the Sumerian data has already been lemmatized,
and therefore the need for Sumerian annotation
tools is not as urgent as it is for Akkadian. Nonetheless, many witnesses of Sumerian composite texts
still lack lemmatization.
For Urartian, Oracc contains texts comprising
26,000 words, 24,000 of which have already beenlemmatized. The total number of non-digitized
texts existing outside Oracc is not clear to us.

4Previous Work

Akkadian lemmatization and POS-tagging havebeen approached with finite-state morphology on
several occasions since the late 1980s. The firstattempt to morphologically analyze, lemmatize
and POS-tag Akkadian with finite-state transducers was taken by Kataja and Koskenniemi (1988).
Barthélemy (1998) and Macks (2002) used Prolog
Definite Clause Grammars for parsing Akkadianverbal morphology, and later a procedural approachto Akkadian verb morphology was taken by Sahala(2014). Bamman (2012) built a finite-state model
for lemmatizing Old Assyrian letters, and Sahala
et al. (2020) published the BabyFST, a finite-statemodel for Babylonian.

The most relevant digital resource to the work presented in this paper is the Open Richly Annotated
Cuneiform Corpus, better known as Oracc (Tinney
1
There is no reliable estimate of the total number of Akkaet al., 2006). It contains ca. 112,000 cuneiform
dian words in various digital resources, but alongside Oracc, attexts in various languages, including but not lim- least 30,000 texts exist in other digital resources with varying
ited to Sumerian, Akkadian and Urartian. Other
accessibility (Charpin, 2014).
205

For Sumerian, morphological analysis, POStagging and lemmatization have been done with
the GATE Java Suite (Tablan et al., 2006) and more
recently with a dictionary-based approach by Chiarcos et al. (2018).
To date, the most comprehensive lemmatizer for
cuneiform languages is L2 (Tinney, 2019), a dictionary and rule-based tool that has been used toannotate Oracc. L2 is also capable of providing
morphological analysis for Sumerian.
For a more comprehensive survey on Computational Assyriology see Sahala (2021), and on theuse of Machine Learning in ancient language processing Sommerschield et al. (2023).

5

were not removed from the Sumerian data
as homophones with different indices can belong to different POS-classes and denote completely different lemmata (see section on tokenization). We chose to separate literary texts
from administrative texts due to their differing vocabulary and grammar. This data set
comprised 268,000 words.
• The Sumerian (administrative) data set consisted of all Sumerian Early Dynastic, Old
Babylonian, Old Akkadian, Ebla and Lagaš
II administrative texts in Oracc’s ePSD2 corpus. The Ur III corpus was excluded because
it would have completely overwhelmed thisdata set with its 81,000 texts. As in the data set
above, the subscript indices were preserved.
This data set consisted of 570,000 words.

Data

All our cuneiform language data comes from Oracc
JSON dumps downloaded in January 2023. For theexperiments done in this paper, we extracted all the
texts written in Sumerian, Akkadian and Urartian.
We selected the data from Oracc as follows:• The Urartian data set comprised all texts
from Oracc labeled as Urartian, the majority of the data coming from the eCUT (Christiansen et al., 2016). In total, this set consistedof 24,000 words.
• The Neo-Assyrian data set comprised all
texts from Oracc labeled as Neo-Assyrian dialect. In total, this set consisted of 331,000
words. This corpus consists mostly of royal
inscriptions and letters that primarily comefrom SAAo (Radner et al., 2005) and ATAE
(Novotny et al., 2017).
• The First Millennium Babylonian data set
consisted of all Oracc texts labeled as any
variant of Babylonian or Akkadian, excluding
Neo-Assyrian, in the first millennium BCE,
thus containing Standard Babylonian, NeoBabylonian and Late Babylonian texts. In
total, this consisted of 1.33 million words belonging to a wide range of genres. The largest
portions of data came from RINAP (Frameet al., 2007), ADSD (Pirngruber et al., 2018),
SAAo, RIBO (Frame et al., 2015) and HBTIN(Pearce et al., 2011).
• The Sumerian (literary) data set consisted of
all Sumerian texts in Oracc’s ePSD2/Literary,eSD2/earlylit and ePSD/Praxis* (Tinney et al.,2017). In this data set, the subscript indices

To test our system on historical non-cuneiform
languages, we used the Latin and Ancient Greek
PROIEL treebanks (Haug and Jøhndal, 2008), comprising 205,000 and 210,000 words respectively.
5.1

Training Data Cleanup

All our models are trained by using the Oracc data,
but we run it through heuristic cleanup rules toprovide more consistent learning results for themodels and to minimize the amount of unwanted
and meaningless errors in the evaluation, such as{d}x-x being tagged as a divine name in one place
but as an unknown POS-class somewhere else.
For the Akkadian data, we merge some inconsistent lemmatizations with their most common representations in the data (e.g. aganutillû vs. agannutillû) and correct obvious lemmatization errors
such as bēlēšu, which is de facto the phonologicaltranscription of the word instead of lemma. We
also apply the Helsinki normalizations (Jauhiainen
et al., 2019) to all divine names in the corpus tomake their lemmatization consistent. Therefore
variation such as Anunnaki, Anunnaku and Anunak
is consistently mapped into Anunnaki. Unfortunately the normalizations are available only fornames that occur in the first millennium Akkadiantexts.
For all cuneiform languages, we do the following
normalizations:
1. Remove all lacuna indicators such as various brackets, exclamation marks and question
marks.

206

2. Remove all entries that have been transliterated as asterisks. This convention is used insome composite texts where that only exist inphonological transcription, such as iddinū forvarying spellings in the witnesses like id-di-nu
and id-di-nu-u2 .
3. Remove all entries without lemmatization unless they are supposed to be unlemmatized, as
is the case of lacunae (breakages in tablets)
and numbers.
4. Remove all lemmatizations from numbers andforce the POS-tag n to them.
5. Force POS-tags for broken personal names,
place names and divine names if the determinative is visible but the words have not been
POS-tagged in Oracc. This can be done inhigh confidence for divine names and personalnames.
6. Force the POS-tag u for broken words in casethey do not have a POS-tag.
We did not do any modifications to the Ancient
Greek and Latin data. These data sets were used asthey are distributed in the Universal Dependencies
GitHub repository.

6

Description of the System

Our system first pre-annotates the input text usinga encoder-decoder model, and then aims to correct possible errors by using simple post-correction
rules. The system is based on the Open Neural Machine Translation Toolkit (OpenNMT) (Klein et al.,2017) and handles the POS-tagging and lemmatization almost completely as a machine translation
task. Relying purely on OpenNMT makes the tooleasy to setup and allows more flexibility and easiercustomization. The whole pipeline is written in
Python and it comes with an easy-to-use commandline interface and extensive documentation.
As our tool is purely based on Oracc notation,
it aims to harmonize various digital resources andto encourage various projects to publish their data
openly in Oracc.

where the encoder is a two layer BiLSTM that reads
the sequence of logo-syllabically tokenized input.
The decoder for generating the output character sequences is a two layer unidirectional LSTM with input feeding attention. However, we train the model
for a lesser amount of steps relative to the training
data size, as it improves the training speed but does
not seem to affect the model’s performance. We
use a batch size of 64 and start the learning rate
decay halfway through the training process.
6.2

Tokenization

We tokenize the input sequences in a special way
that is particularly suitable for the logo-syllabic
cuneiform writing system. From here on, we refer to this as logo-syllabic tokenization. In logosyllabic tokenization, syllabic signs and phonetic
complements that represent phonetic sequences are
encoded as space-separated character sequences,whereas logograms and determinatives are encoded
as indivisible tokens. We retain indices for logograms, because homophonic logograms can referto different parts-of-speech and lemmata (e.g. in
Akkadian DUG3 = .tābu "good" and DUG4 = qabû"speak"), but for syllabic signs indexation is removed to bring homophonic readings such as šuand šu2 closer to each other. Based on our observations, splitting compound logograms such as
MA.NA into MA and NA yields better results thanhandling them as monolithic units.
The tagger is trained with 5-grams of logosyllabically tokenized word forms and it aims topredict the POS-label for the center word wrapped
inside double angle brackets (see Table 1 for tokenization examples). The lemmatizer is trainedwith tokenized word forms followed by its POStag, as well as the word’s previous and following
POS-tags to provide shallow information about
the word’s context. This input string sequence ismapped to its lemma on the character level, enabling the system to infer unseen lemmata.
6.3

Lemmatization and POS-tagging Process

The input text is first tokenized logo-syllabicallyand fed into the tagger. The tagger output is thenused as the context information for the lemmatizer,
which produces the fully annotated output consist6.1 Network Architecture
ing of the lemma and the POS-tag.
Our neural network architecture for both, the POSThe post-correction comprises two steps. First,
tagger and the lemmatizer, follows the architecture
we calculate the distribution of lemmata assignedof the Universal Lemmatizer (Kanerva et al., 2021). for each word form + POS-tag pair in the training
We use a deep attentional encoder-decoder network, data and in case any single lemma constitutes more
207

Original
SourceTargetSourceTargetCombined{m}KU6 -li-i-di ina ŠA3 -bi x MA.NA{m} KU6 - l i - i - d i | i n a « ŠA3 - b i » x | MA . NANŠA3 - b i P0=PRP P1=N P2=ul i b b u
libbu + N

Table 1: Example of logo-syllabic tokenization. The upper part shows the tokenization fed into the tagger, the center
word wrapped in double angle brackets, and the wanted output N (noun). The lower table shows the tokenization
fed into the lemmatizer, including the center word and its POS-tag along with the preceding and the following
POS-tags, and the wanted output as a sequence of characters.

than 70% of the lemmatizations of the given pair,
we replace the predictions made by the neural network with this lemmatization. Next, we repeat thesame step but instead of using word forms and their
POS-tags, we also use the POS-tags assigned to thepreceding and the following words. These steps
aim to ensure, that close to unambiguous lemmata
are always lemmatized consistently. However, due
to the fact that the context information is taken into
account already in the neural lemmatization, thepost-correction no longer improves the lemmatization results significantly as in the previous version
of BabyLemmatizer (Sahala et al., 2022). Therefore the post-correction is now mostly used for
assigning lemmatizations with confidence scoring.
Confidence scoring aims to assist humans tomanually verify and correct the lemmatization results. This system is mainly designed for detecting out-of-vocabulary (OOV) words, that is, wordforms that were not present in the training data,and categorizing these words into different classes
based on their spellings. The lowest confidence
score of 0 is given to OOV words written logographically as the logogram and its lemma has asuppletive relation. The score of 1 is given forlogo-syllabic spellings, which may have partially
suppletive relationship to their lemmata. Syllabic
OOV spellings are given a confidence score of 2.
Confidence scores between 3 and 5 are assigned
for in-vocabulary words. The score of 3 is given
to highly ambiguous words, such as polyvalent
logograms that exist in contexts that have not been
observed in the training data. The score of 4 isgiven to words that show low or unlikely ambiguity,and the highest score of 5 is given to words thathave low ambiguity and exist in a POS-context thathas been witnessed in the training data.

are to be lemmatized, this data set should be broken
into smaller subsets, for example in four batches of
2,500 texts each.
After each lemmatization batch, the tool generates OOV lexicons for all low confidence scoreclasses. These lists are sorted by frequency, allowing maximal number of corrections per each corrected entry. The lemmatizations can be corrected
simply entering the corrected lemma and POS-tag
for any word form in the OOV list, or accepting
the already given lemmatization by removing thesymbol # from the beginning of the line. Whenthe lemmatizer is run again, the system appends
these changes to the model’s lexicon, allowing it
to lemmatize them correctly in the future. Afterthe lemmatization results of the current batch areconsidered to be clean enough, the model can beretrained by using the the data from the current
batch appended to the model’s existing training
data, yielding an updated model augmented withnew manual corrections. This approach should
significantly reduce the time needed for manual
corrections after each batch.
6.4

CoNLL-U+ for Cuneiform

Our tool uses an extended CoNLL-U format for
input and output.2 The first ten columns follow
the standard notation, reserving the XPOS field for
the Oracc POS-label. In addition to the conventional fields (ID , FORM , LEMMA , UPOS , XPOS ,FEATS , HEAD , DEPREL , DEPS , MISC), our CoNLLU+ format has the following fields: ENG for the
English translation, NORM for phonological transcription, LANG for word’s language, FORMCTX
and XPOSCTX for storing temporary context information for the system, SCORE for the confidence
scoring and LOCK for write-protecting the field in

2
The lemmatization process is designed to be iterSee https://universaldependencies.org/
ative. For example, if a batch of 10,000 new texts
format.html
208case the file has manual corrections that the systemshould not overwrite.
The pipeline handles the conversion of CoNLLU+ into OpenNMT-compatible target and source
files and conversion of simple rawtext transliteration into CoNNL-U+.

7

Evaluation

For evaluation, we train ten models for each data
set. We use a 80/10/10 train/dev/test split and estimate the model’s accuracy, that is, the percentage
of correct analyses over all analyses, using 10-foldcross-validation. We measure the accuracy in two
categories: first, for all the word forms in the test
set, and second for the OOV word forms only to
examine the models’ ability to predict labels forwords that were not present in the training data.
The results are summarized in Table 2 and Table
3 respectively. Confidence intervals of the crossvalidation are shown in parentheses.
We ignored all fully broken words in the evaluation, as assigning empty labels for completely
destroyed words is trivial.
7.1

Results for Cuneiform Languages

With the current data sets, the minimum accuracy scores for POS-tagging and lemmatizationof known word forms are 95% and 92%, respectively. For OOV words, the system achieves minimum accuracy of 81% for POS-tagging and 68%
for lemmatization. OOV lemmatization accuracy
Category
NN POS-tagger
NN Lemmatizer
NN Combined
PC POS-tagger
PC Lemmatizer
PC Combined
OOV-rate

Urartian
96.97 (±0.39)
93.45 (±0.66)
92.49 (±0.72)
96.97 (±0.39)
94.12 (±0.57)
93.18 (±0.63)
8.26

seems to vary greatly depending on the language
and the diversity of the data set, the most striking difference being between the first millennium
Babylonian (68%) and the Sumerian literary texts(84%). This difference can be explained partly by
the diversity of the Babylonian data, and partly bySumerian morphology, which is significantly moretransparent than that of Akkadian. The low performance in the OOV lemmatization in administrative Sumerian can be explained by inconsistencies
in the data especially in proper nouns. At times,
Oracc renders their lemmata as sequences of signs
separated by dots, whereas at times the dots are
not used, for instance, {d}nin-mar{ki} is lemmatized as Ninmar, Nin.mar, Nin.MAR, which makes
it difficult for the model to learn how to generalize.
About 18% of OOV lemmatization errors and 13%of all lemmatization errors in this particular data
set are caused by such inconsistencies.
7.2

Comparison Against Version 1.0

Compared with the earlier version of BabyLemmatizer, the current tool clearly outperforms its neuralnetwork performance, which significantly reduces
the improvement gained from post-correction. For
comparison, we used the same 500,000 word Babylonian evaluation data set as we used in our earlier
report (Sahala et al., 2022). Better performance ofthe neural network translates directly into a better
performance in OOV word lemmatization, improving the prediction accuracy of Lemma+POS labels

Neo-Assyrian
97.67 (±0.17)
95.35 (±0.20)
94.48 (±0.26)
97.72 (±0.17)
95.47 (±0.21)
94.59 (±0.28)9.20Babylonian
96.80 (±0.18)
95.02 (±0.31)
93.82 (±0.36)
96.80 (±0.18)
95.14 (±0.30)
93.94 (±0.34)
6.06

Sum. (lit.)
94.79 (±0.28)
94.67 (±0.25)
92.28 (±0.37)
94.77 (±0.28)
94.66 (±0.27)
92.27 (±0.37)
16.80

Sum. (adm.)
96.32 (±0.08)
95.50 (±0.10)
94.57 (±0.09)
96.32 (±0.08)
95.53 (±0.08)
94.60 (±0.07)
5.21

Table 2: Results of the 10-fold cross-validation for the neural net (NN) and the post-corrected (PC) results. Combined
represents word forms where both, lemma and POS-tag were predicted correctly. OOV-rate shows the average
percentage of OOV words in the test set.

Category
POS-tagger
Lemmatizer
Combined

Urartian
83.00 (±1.96)
70.10 (±2.16)
65.73 (±2.06)

Neo-Assyrian
90.87 (±0.74)
71.16 (±1.14)
70.15 (±1.09)

Babylonian
85.78 (±0.81)
67.82 (±1.36)
65.52 (±1.31)

Sum. (lit.)
84.39 (±0.84)
83.93 (±1.00)
76.49 (±1.15)

Sum. (adm.)
81.17 (±0.77)
70.51 (±1.24)
67.20 (±1.28)

Table 3: Results of the 10-fold cross-validation for OOV words only. This table does not contain separate results for
NN and PC, because post-correction does not affect OOV words.

209

for OOV words on average by 10 percentage points.
The performance increase is summarized in Table
4.
Category
NN POS-tagger
NN Lemmatizer
NN Combined
PC POS-tagger
PC Lemmatizer
PC Combined

All
+0.14
+8.87
+8.91
+0.14
+0.35
+0.45

OOV
+4.06
+8.84
+10.48+4.06+8.37+10.01

Table 4: Average improvement in accuracy-% from v1.0
to v2.0, overall and for OOV words (NN for neural net
and PC for post-corrected).

7.3

Experiment on Latin and Ancient Greek

To test our system on non-cuneiform languages,
we tagged and lemmatized the PROIEL treebanks
for Latin and Ancient Greek (Table 5). For theselanguages, we used character sequences as input
format for both the tagger and the lemmatizer withthe same context information as in the logo-syllabic
tokenization (5-grams for tagger and adjacent POScontext for lemmatizer). We used the training, development and test data provided at the Universal
Dependencies GitHub.
Category
POS-tagger
Lemmatizer
Combined
OOV POS-tagger
OOV Lemmatizer
OOV Combined
OOV-rate

Greek
96.70
96.70
94.96
87.54
74.47
73.18
11.02

Latin
95.31
95.81
94.03
84.64
75.92
74.92
10.58

Table 5: Results for the Ancient Greek and Latin data.
The upper table shows the overall results and the lower
table the results for OOV words only.

8

Conclusions and Future Workthe current version can predict correct POS-tags
for 83-91%, and lemmata for 68-84% of the input word forms from transliteration. Comparedwith the earlier version, the current one has about
10% higher accuracy in OOV lemmatization and
POS-tagging due to better neural network performance. We also tested the system for lemmatizing
and POS-tagging the PROIEL Ancient Greek and
Latin treebanks, achieving results similar to thosewith the cuneiform languages.
In the future, we plan to add prediction for UD
POS-tags, phonological transcription and morphological labels for Akkadian, Sumerian and Urartian.
We also plan on adding full Oracc lemma prediction that includes the English translation of theword following Oracc’s lemma[translation]POS
format, but prior to this more data cleanup is required.

Acknowledgements
We wish to thank the Academy of Finland forfunding the Origins of Emesal Project (PI Krister
Lindén) and the Centre of Excellence in Ancient
Near Eastern Empires (PI Saana Svärd). We also
wish to thank Tero Alstola and Jonathan Valk (bothat the University of Helsinki) for their valuable
feedback on the Akkadian lemmatization results,
Niek Veldhuis (UC Berkeley) for the Oracc dataacquisition scripts,3 and Steve Tinney (Universityof Pennsylvania) for correcting certain broken data
sets in Oracc on our request.

References
David Bamman. 2012. 11-712 NLP Lab Report:
Akkadian-morph-analyzer.
François Barthélemy. 1998. A morphological Analyzer
for Akkadian Verbal Forms with a Model of Phonetic
Transformations. In Computational Approaches to
Semitic Languages.
Pierre Briant
Achemenet.

andWouterHenkelman.

2009.

We presented an updated version of BabyLemmatizer, a pipeline for POS-tagging and lemmatiz- Dominique Charpin. 2009. Archives Babyloniennes
ing cuneiform languages and evaluated its perfor(Archibab).
mance on Sumerian, first millennium Babylonian,
Dominique Charpin. 2014. Ressources Assyriologiques
Neo-Assyrian and Urartian texts extracted fromsur Internet. In Bibliotheca Orientalis LXXI no. 3-4.
Oracc to observe its performance for the first time
Christian Chiarcos, Ilya Khait, Émilie Pagé-Perron,outside Babylonian texts. The system achieves
Niko Schenk, Jayanth, Christian Fäth, Julius Steuer,a POS-tagging accuracy between 95-98% and a3lemmatization accuracy of 94-96% depending on
https://github.com/niekveldhuis/
the language or dialect. For OOV words only, compass
210

William Mcgrath, and Jinyan Wang. 2018. Annotating a low-resource language with LLOD technology: Sumerian morphology and syntax. Information,
9(11):290.

Piotr Michalowski. 2008. Sumerian. In R. D. Woodard,
editor, The Ancient Languages of Mesopotamia,
Egypt, and Aksum, pages 19–46. Cambridge University Press.

Birgit Christiansen, Mirjo Salvini, Dan Roberto, andStephan Kroll. 2016. Oracc: Electronic Corpus ofUrartian Texts (eCUT) Project.

Piotr Michalowski. 2011. The Correspondence of theKings of Ur: An Epistolary History of an Ancient
Mesopotamian Kingdom. Eisenbrauns.

Robert Englund, Jürgen Renn, Jacob L. Dahl, Bertrand
Lafont, and Steve Tinney. 1998. Cuneiform Digital
Library Initiative (CDLI).

Lucio Milano and Massimo Maiocchi. 2016. Ebla Digital Archives.

Grant Frame, Jamie Novotny, and Karen Radner. 2015.
Oracc: Royal Inscriptions of Babylonia Online.
Grant Frame, Karen Radner, Barry L. Eichler, Erle Leichty, and Steve Tinney. 2007. Oracc: The Royal
Inscriptions of the Neo-Assyrian Period.
Dag T. T. Haug and Marius Jøhndal. 2008. Creatinga parallel treebank of the old Indo-European Bible
translations. In Proceedings of the second workshop
on language technology for cultural heritage data
(LaTeCH 2008), pages 27–34.

Manuel Molina. 2002. The Database of Neo-Sumerian
Texts (BDTNS).
Jamie Novotny, Karen Radner, and Poppy Tushingham.
2017. Oracc: Archival Texts of the Assyrian Empire(ATAE).
Laurie Pearce, Stephanie Langin-Hooper, Chris Bravo,
Talia Prussin, and Jay Crisostomo. 2011. Oracc: Hellenistic Babylonia: Texts, Images and Names.
Reinhard Pirngruber, Maya Rinderer, and Craig A. Harris. 2018. Oracc: Astronomical Diaries Digital.

John Huehnergard and Christopher Woods. 2008. Akkadian and eblaite. In R. D. Woodard, editor, The Ancient Languages of Mesopotamia, Egypt, and Aksum,
pages 83–152. Cambridge University Press.
Bram Jagersma. 2010. A Descriptive Grammar of Sumerian. University of Leiden.
Heidi Jauhiainen, Aleksi Sahala, and Tero Alstola. 2019.
Oracc in Korp.
Enrique Jiménez, Jussi Laasonen, Aino Hätinen, Zsombor Földi, Adrian Heinrich, Tonio Mitto, Geraldina
Rozzi, Ilya Khait, and Fabioan Simonjetz. 2018. The
Electronic Babylonian Library (eBL).
Jenna Kanerva, Filip Ginter, and Tapio Salakoski.
2021.
Universal lemmatizer: A sequence-tosequence model for lemmatizing universal dependencies treebanks. Natural Language Engineering,27(5):545–574.
Laura Kataja and Kimmo Koskenniemi. 1988. Finitestate Description of Semitic Morphology: A Case
Study of Ancient Accadian. In Coling Budapest
1988 Volume 1: International Conference on Computational Linguistics.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Opensource toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages
67–72, Vancouver, Canada. Association for Computational Linguistics.
Aaron Macks. 2002. Parsing akkadian verbs with prolog. In Proceedings of the ACL-02 workshop on
Computational approaches to semitic languages.

Karen Radner, Eleanor Robson, Steve Tinney, and Jamie
Novotny. 2005. Oracc: The State Archives of Assyria
online.
Aleksi Sahala. 2014. Babylonian verbimorfologian automaattinen jäsentäminen. University of Helsinki.
Aleksi Sahala. 2021. Contributions to Computational
Assyriology (PhD Thesis). University of Helsinki.
Aleksi Sahala, Tero Alstola, Jonathan Valk, and Krister Linden. 2022. BabyLemmatizer: A Lemmatizer
and POS-tagger for Akkadian. In CLARIN Annual
Conference Proceedings, 2022. CLARIN ERIC.
Aleksi Sahala, Miikka Silfverberg, Antti Arppe, andKrister Lindén. 2020. BabyFST-towards a finite-statebased computational model of ancient babylonian. In
Proceedings of the Twelfth Language Resources and
Evaluation Conference, pages 3886–3894.
Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer,
John Bodel, Jonathan Prag, Ion Androutsopoulos,
and Nando de Freitas. 2023. Machine Learning forAncient Languages: A Survey. Computational Linguistics, pages 1–44.
Michael Streck. 2010. Großes fach altorientalistik: Der
umfang des keilschriftlichen textkorpus. In Mitteilungen der Deutschen Orient-Gesellschaft zu Berlin,
142.
Valentin Tablan, Wim Peters, Diana Maynard, Hamish
Cunningham, and K Bontcheva. 2006. Creating
Tools for Morphological Analysis of Sumerian. In
LREC, pages 1762–1765.
Steve Tinney. 2019. L2: How it Works.

211

Steve Tinney, Phillip Jones, and Niek Veldhuis. 2017.
Oracc: The electronic Pennsylvania Sumerian Dictionary 2.7.
Steve Tinney, Jamie Novotny, Eleanor Robson, andNiek Veldhuis. 2006. The Open Richly Annotated
Cuneiform Corpus.
Steve Tinney and Eleanor Robson. 2019. Oracc
ATF Primer. Oracc: The Open Richly Annotated
Cuneiform Corpus.
Gernot Wilhelm. 2008. Urartian. In R. D. Woodard,
editor, The Ancient Languages of Asia Minor, pages
105–123. Cambridge University Press.

212

Tibetan Dependency Parsing with
Graph Convolutional Neural Networks
Bo An
Institute of Ethnology and Anthropology,Chinese Academy of Social Sciences
Building 6, Zhongguancun Nandajie 27, Beijing, China
anbo@cass.org.cn

Abstract
Dependency parsing is a syntactic analysis method to analyze the dependency relationships between words in a sentence. Theinterconnection between words through dependency relationships is typical graphdata. Traditional Tibetan dependency
parsing methods typically model dependency analysis as a transition-based or
sequence-labeling task, ignoring the graphinformation between words. We proposea graph neural network (GNN)-based Tibetan dependency parsing method to address this issue.
This method treats
Tibetan words as nodes and the dependency relationships between words asedges, thereby constructing the graph dataof Tibetan sentences. Specifically, we use
BiLSTM to learn the word representationsof Tibetan, utilize GNN to model the relationships between words, and employ MLP
to predict the types of relationships between words. We conduct experiments on a
Tibetan dependency database, and the results show that the proposed method canachieve high-quality Tibetan dependency
parsing results.

1

Introduction

In recent years, the explosive growth of Tibetan text data, fueled by the popularizationof information technology in Tibetan areas,
has made the processing and deeper understanding of Tibetan information a hot research
topic in Tibetan natural language processing(NLP) (Faggionato and Meelen, 2019). Dependency analysis is an essential task for thesemantic modeling of texts, as it provides abasis for deep semantic analysis and has significant research and practical value. The results of dependency analysis can be directly
applied to numerous basic natural language
processing tasks, such as question answering(Cao et al., 2019), sentiment analysis (Xiaomei
et al., 2018), and named entity recognition (Jieet al., 2017).
Traditional Tibetan dependency analysis
methods can be mainly classified into two categories: (1) statistical learning-based methods(Hua et al., 2013), which usually require experts to design corresponding rules and features and then use statistical learning models
to model and predict dependency syntax. This
type of method heavily relies on Tibetan linguistic experts. (2) deep learning-based methods(An and Long, 2021) , which have been
widely applied in Tibetan information processing, such as word segmentation, text classification, and dependency analysis, with the rapid
development of deep learning. The major advantage of deep learning-based methods is thatthey do not require expert features. Tibetan
dependency analysis can be achieved througha fixed network structure and annotated data.
However, the methods above model Tibetan
dependency analysis as a classification or transition problem, ignoring the features of graphdata in dependency analysis. Graph data
features can better model the relationships
between different words and ignore the distance between words in the text, i.e., they
can model the dependency information between words that are far apart. They can alsomodel higher-order relationships through indirect relationships between words, which significantly impacts modeling word relationships,
such as AMR (Abstract Meaning Representation) (Wang et al., 2020).
This paper presents a method for Tibetan
dependency analysis based on graph convolutional neural networks. Tibetan word representations are modeled using Bert (Devlin
et al., 2018) and BiLSTM, followed by graphneural networks(GNN) (Zhou et al., 2020) for

213
Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 213–221,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_024

modeling dependency relationships between
words. MLP is then employed for relationshipclassification and determining the dependency
relationship types. The results on Tibetan dependency analysis data indicate that GNN cansignificantly enhance the performance of Tibetan dependency analysis, thus aﬀirming thevalue of graph information.
The main contributions of our work are
as follows: (1) We propose using graph convolutional neural networks (GCN) to modelthe dependency relationships in Tibetan sentences. (2) Experimental results show that the
proposed method outperforms other methods,
such as R-GCN (Schlichtkrull et al., 2018), in
Tibetan dependency analysis, which may be
due to insuﬀicient training data.
The main contributions are twofold:• We propose using GCN to model the dependency relationships in Tibetan sentences.
• Experimental results show that the
GCN+MLP method outperforms othermethods, such as R-GCN, in Tibetan dependency analysis, which may be due to
insuﬀicient training data.
The rest of the paper is organized as follows:
Section 2 introduces some of the most related
work, including Tibetan dependency parsing
models and graph neural networks. Our proposed model is detailed described in Section
3. Section 4 shows our experimental resultson the introduced Tibetan dependency analysis dataset and presents the effects of different
modules. We conclude our work in Section 5.

2

Related Work

This section briefly reviews related work, including Tibetan dependency parsing methods and neural-based methods for dependency
parsing.

first. The current Tibetan dependency analysis dataset includes the following: For instance,Hua et al. 2013 construct a Tibetan dependency tree semi-automatically. It includes
a word-pairs dependency classification model,
and dependency edges annotation model basedon Tibetan language grammar. Tashi and Duo2015 built a Tibetan dependency treebank
of multidimensional windows based on theirgrammar. Toudan et al. 2018 annotated dependency trees for sentences from Tibetan primary school textbooks. Wu et al. 2019 introduced a Tibetan dependency analysis datasetwith 1500 sentences annotated based on complex dependency grammar with 62 types ofdependency arcs. (An and Long, 2021) constructs a Tibetan dependency parsing datasetwith more than 5000 Tibetan sentences basedon an interlinearized annotation dataset.
Currently, most of the Tibetan dependency
parsing models are composed of two components: feature extraction and dependency prediction. A discriminant model is proposed to
conduct Tibetan dependency parsing based onfeature engineering by Tibetan experts (quecai rang and Zhao, 2013). And their model
was further utilized for the parsing of Tibetan
compound sentences. Xia et al. 2019 extracted unigram, bigram, trigram, and someTibetan-specific features for each word in thesentence and employed a perceptron classifierto perform Tibetan dependency parsing. Allof the above works were based on features designed by Tibetan language experts. Compared with these methods, the main advantage
of our method is that our model can extract
useful feature vectors automatically.
2.2 Neural-based method for
dependency analysis

In recent years, neural-based models haveachieved competitive performances in many
natural language processing tasks, such as
word segmentation, part-of-speech, and se2.1 Tibetan dependency parsing
mantic parsing. Furthermore, this line of
methodworks have two advantages: avoiding complex
feature engineering and better generalization.
The Tibetan dependency analysis dataset isDue to the above advantages, neural-based
the foundation for researching dependency
analysis methods. Therefore, the existing Ti- models are introduced for dependency analysis.
There are two main research directions for debetan dependency analysis data is introduced
pendency analysis: translation-based parsers
and graph-based parsers.
214

Figure 1: An example of Tibetan dependency tree.

Chen and Manning introduced the firstneural translation-based dependency parser(Chen and Manning, 2014), which utilizes afeedforward network to assign a probability toeach action the parser. Andor et al. (Andoret al., 2016) augments the above model witha beam search and a conditional random field
loss objective for correcting false predictions.
The Long-Short-Term Memory (LSTM) model
was employed to achieve the state-of-the-art
performance (Dyer et al., 2015; Kuncoro et al.,2016).
The first neural graph-based parser 2016
utilizes the attention mechanism from machine translation and LSTM to conduct dependency parsing. Hashimoto et al. 2016 extend
the graph-based parser as a multi-task neuralmodel and employ a bilinear MLP label classifier. Furthermore, Cheng et al. 2016 further
resolve the limitation of being unable to condition the scores of each possible arc on previous
parsing decisions of other graph-based parsers.
Dozat et al. 2016 propose bi-aﬀine classifiers
to predict arcs and labels for dependency analysis tasks and achieve state-of-the-art performances. Recently, with the wide use of deep
contextual embeddings (Peters et al., 2018),
Schuster et al. 2019 introduced a multilingualtransfer framework that utilizes deep contextual embeddings in an unsupervised fashion.
(An and Long, 2021) proposes a deeplearning-based Tibetan dependency parsing
method using BiLSTM and multi-layer perceptron. (Duo et al., 2021) models the Tibetan
dependency parsing task using deep learningbased transition. Recently, scholars have introduced deep learning methods to the taskof Tibetan dependency parsing task, resulting

in an improvement in the performance of Tibetan dependency parsing. Despite the potential of graph neural networks, they have notbeen utilized in Tibetan dependency parsing
tasks. Hence, this paper proposes a graph neural network-based Tibetan dependency parsing method to better model the relationships
between words.

3 The GCN-based Tibetan
Dependency Parsing Method
3.1 Task Definition

Dependency parsing is a natural language processing task that involves analyzing the grammatical structure of a sentence by identifying the relationships between the words in it.
Moreover, Figure 1 presents an example of a
Tibetan dependency tree.
Specifically, given a sentence S consistingof n words w1 , w2 , ..., wn , dependency parsing
aims to construct a directed acyclic graph G =(V, E), where V = v1 , v2 , ..., vn is the set ofvertices representing the words in S, and E ⊆V × V is the set of directed edges representing
the syntactic dependencies between words.
Each edge ei,j = (vi , vj ) in E is labeledwith a dependency type ri,j ∈ R, where R is
the set of all possible dependency types. The
dependency tree’s root is the vertex with noincoming edges. Thus, the dependency tree
T = (V, E ′ ) is a tree if it contains n − 1 edges
and satisfies the constraints mentioned above.
The output of a dependency parser is thedependency tree T that represents the sentence’s grammatical structure. This tree canbe used for various downstream applications,such as machine translation, information retrieval, and text summarization.
215

This work employs the dataset introducedby (An and Long, 2021). There are 34 typesof dependency arcs in our Tibetan dependency
grammar. We present them in Table (1).
3.2

GNN for Tibetan Dependency
Parsing

Tibetan dependency parsing includes word segmentation, word relation, and arc label prediction. The framework of Tibetan dependency
parsing is presented in Figure 2.
We employ SegT (Huidan Liu and Yeping,2012) for Tibetan word segmentation in thiswork. We utilize the Tibetan-Roberta-base to
implement the embedding layer, which generates the embedding for each Tibetan syllable.
Moreover, we employ BiLSTM (Kiperwasser
and Goldberg, 2016) to compose the syllable
embeddings into the word embedding hi .
We employ Graph Convolutional Networks(GCN) to model Tibetan dependency parsing.
GCNs can be used to model dependency parsing by constructing a graph representation of
the sentence, where the vertices represent thewords in the sentence, and the edges represent the syntactic dependencies between them.
Each vertex is associated with a word embedding hi , which captures the semantic information of the word. Formally, let G = (V, E)
be the graph representation of the sentence,
where V = v1 , v2 , ..., vn is the set of vertices
representing the words in the sentence, andE ⊆ V × V is the set of directed edges representing the syntactic dependencies between
words. Each vertex vi is associated with aword embedding hi ∈ Rd , where d is the dimension of the embedding. To capture theinteractions between the vertices in the graph,
GCNs perform graph convolution operations
on the vertex embeddings. Specifically, theembedding of each vertex is updated by aggregating the embeddings of its neighboring
vertices, weighted by an adjacency matrix A
that encodes the edge information. The graphconvolution operation can be expressed as:


∑
1
(l+1)
(l)
W (l) hj 
hi
= σ
ci,j
j∈N (i)l, N (i) is the set of neighboring vertices of vertex i, W (l) is the weight matrix at layer l, andci,j is a normalization constant that ensuresthat the sum of the weights of the neighborsof vertex i is 1. The activation function σ istypically a non-linear function, such as the rectified linear unit (ReLU).
After several graph convolution operations,the final vertex embeddings can be fed into aclassifier to predict the syntactic dependency
labels between the words.
3.2.1 Arc Prediction Layer
This layer comprises two classifiers; the firstpredicts the dependency head for each word,while the second classifies the type of dependency arc between the word and its head word.
Head Classifier. The input to this classifier is the feature vectors of each word, and it
outputs the index of the word’s head. Since
the number of words in a sentence is variable,
this is a variable-class classification task, making it impossible to utilize a multi-layer perceptron (MLP) typically used for category tasks.
We draw inspiration from the biaﬀine attention model (Dozat and Manning, 2016) to address this challenge and employ two MLP models to build the head classifier. Specifically, we
use Equation (1) and (2) to convert the feature
vector of each word into two vectors f⃗head ∈ dh
and f⃗dep ∈ dh , respectively, to represent thehead and dependent nodes of the dependencyarc.
f⃗ihead = M LP head f⃗i(1)

fidep = M LP dep f⃗i(2)

Next, the position of the head node for eachword i is calculated using a bilinear attention mechanism as per Equation (3), whereharc∈ dh represents the score for the j-thiword as the head node of the i-th word. Here,
Fdep ∈ dnh is a matrix obtained by concatenating the head representation f⃗ihead of all words
in the sentence, where n is the number ofwords in the sentence. Additionally, U ∈ dhh
is the parameter matrix, and ⃗u ∈ dh is theparameter vector.

(l)where hi is the embedding of vertex i at layer
216

harc
= Fdep Ufdep+ Fhead uji(3)Table 1: The types of Tibetan dependency arcs.

Item

Dependency Relationship

Label

Item

Dependency Relationship

Label

1

Subject predicate relationship

SBV

2

Direct object relationship

DOB

3

Indirect object relationship

IOB

4

Subject verb relationship

SBC

5

Predicative verb relationship

CPS

6

Modifier relationshipMOD

7

Apposition relationship

APP

8Quantitative relationship

QUN

9Constellation relationship

COO

10

Connection relationship

CON

11

Referential relationship

REF

12

Qualified relationship

DET

13

Negative relationship

NEG

14

Interrogative relationship

ITG

15

Location relationship

LOC

16

Time relationship

TMP

17

Expression relationship

EXP

18

Genitive relationship

GEN

19

Ergative relationship

ERG

20

Dative relationship

DAT

21

Comitative relationship

COG

22

Plural relationship

PLU

23

Honorific relationship

HON

24

Nominalized relationship

NML

25

ROOT relationship

ROOT

26

Tense and aspect relationship

TAM

27

Punctuation relationship

PUN

28

Description relationship

DES

29

Particle relationship

PAR

30

Target relationship

TAR

31

Auxiliary relationship

AUX

32

Manner relationship

MAN

33

Source relationship

SOU

34

Non-predicative verb relationship

PER

Figure 2: The framework of Tibetan dependency parsing.

217

Table 2: The statistics of the dataset.

Datasettrain
validation
test#Instances
4970
200
400

Average Length
6.8
6.7
7.1

root node. Notably, this metric does not consider the type of dependency arcs. UAS isheadcalculated as per Equation (7), where Nword
is the number of words labeled with the correct head word, and Nword represents the total
number of words in the dataset.
U AS =

Finally, the head node with the highest scoreis selected as per Equation (4) to correspondto the word.
headi = max hx
0<x<n

(4)

Type Classifier. The number of dependencies between head and dependent words isfixed, making this a fixed-class classification
problem. To better model the dependency relationship between the head and dependent
words, we use both the representations of thehead and dependent words to predict the typeof dependencies, as shown in Equation (5).
Where j is the head word index of word i;
W1 ∈ df∗m∗f and W2 ∈ d2f∗m a parameter matrix; m is the number of types of dependency
arcs; ⃗b ∈ dm is a parameter vector.
headlabel= f⃗jT W1~ri + (~rj ⊕ r~i )W2 + ~
b
i

(5)

The type of the dependency arc is predictedusing Equation (6).
typei = max

0<x<m

4

headlabelx

4.1

DatasetThe dataset is divided into three parts: thetraining set, validation set, and test set. Table(2) displays the statistics of the dataset.
4.2

LAS =

arc
Nword
Nword

(8)

4.2.3 UEM
The unlabeled exact match score (UEM) is defined as the percentage of sentences in which
all the words have the correct head words.
UEM is calculated using Equation (9), wherehead
Nsentenceis the number of sentences withcorrect head words for all their words, andNsentence is the total number of sentences inthe dataset.

(6)

In this section, we conduct experiments of Tibetan dependency analysis task on the dataset
from (An and Long, 2021) .

(7)

4.2.2 LAS
The labeled attachment score (LAS) is definedas the percentage of all words that have thecorrect head word and the correct type of dependency arc, including the root node. LAS
arc
is calculated as per Equation (8), where Nword
is the number of words with the correct head
word and type of dependency arc.

U EM =

Experiments

head
Nword
Nword

head
Nsentence
Nsentence(9)

4.2.4 LEM
The labeled exact match score (LEM) is defined as the percentage of sentences in which
all the words have the correct head words andtype of dependency arcs. LEM is calculated usarc
ing Equation (10), where Nsentenceis the number of sentences with correct head words andtype of dependency arcs for all their words.
U EM =

Evaluation Metrics

arc
Nsentence
Nsentence(10)

4.3 Experimental Settings

Four evaluation metrics are employed in thispaper, as follows.
4.2.1 UAS
The unlabeled attachment score (UAS) is defined as the percentage of all words that havefound their correct head word, including the

The validation set is utilized to determine thebest hyperparameters for the model. The hyperparameters of the model are then set as follows: Tibetan-Roberta-base1 is employed to1
https://huggingface.co/sangjeedondrub/tibetanroberta-base �218generate the embedding of Tibetan syllables,
and the vectors are composed into word embeddings based on BiLSTM. The dimensionof the word embedding is set at dw = 768,
whereas the dimension of the semantic role label is set at dl = 100.
The dimension of the multi-layer perceptron
matrix M LP head is set at 768 ∗ 100, and thedimension of M LP dep is also set at 768 ∗ 100.
The model’s dropout is set at 0.3, and it employs the adadelta optimizer, with the learning
rate set at 0.001. All parameters are randomly
initialized using a uniform distribution among[−0.2, 0.2].
Moreover, we compare our model withdeep learning-based Tibetan dependency parsing models, including word2vec + BiLSTM+ MLP (DL-BiLSTM), word2vec + RNN +MLP (DL-RNN), word2vec + GRU + MLP(DL-RNN) and word2vec + Stacked LSTM +MLP (DL-Stacked) from (An and Long, 2021).
And the word embedding is trained by fastText (Thavareesan and Mahesan, 2020). In
addition, we compare our model with R-GCN(Schlichtkrull et al., 2018) with similar settings.
All experiments were conducted on a GPU
server with a CPU configuration of 2* AMDSkyline 7742, 512G DDR4 RAM, and 4*
Nvidia A100 40G GPU cards.
4.4

The Overall Experimental Results

The Tibetan dependency parser is designed to
predict the head word and type of dependency
arc for each word in a sentence. Our framework is implemented using Pytorch, and theoverall results are presented in Table (4).
The experimental results demonstrate thatthe method proposed in this paper achieved
the best performance on all four metrics, highlighting the value of graph neural networks in
Tibetan dependency analysis. And our proposed method achieves better performances
than R-GCN, we speculated that R-GCN requires more data to train the relation representation matrix, whereas our training data
is suﬀicient to train the relation matrix effectively.
4.5

Ablation Study

sults, an ablation study was conducted to analyze the value of pre-trained language models
and graph neural networks in Tibetan dependency analysis. The experimental results arepresented in Table 3, where ”- GCN + LSTM ”
represents our proposed model replacing GCN
with LSTM, ”-Bert + GCN” represents our
proposed model replacing Bert with word2vec.
From these results, two conclusions can be
drawn: (1) Graph neural networks significantly impact the performance of Tibetan dependency analysis, and using word embeddings as the lexical representation method canstill improve the performance. (2) Tibetan
pre-trained language models also hold value
in Tibetan dependency analysis, and the performance of Tibetan dependency analysis declines to some extent when using word embeddings to replace the BERT model.

5 Conclusion
To address the issue of inadequate modeling of dependency graph information in current Tibetan dependency analysis methods,
this paper proposes a graph neural networkbased approach for Tibetan dependency analysis. Furthermore, a Tibetan pre-trained language model is employed to improve the performance further. The experimental results
demonstrate the effectiveness of the graph neural network and the Tibetan pre-trained language model for enhancing Tibetan dependency analysis. Large models such as ChatGPT have recently achieved significant resultsin natural language processing tasks such as
dialogue and knowledge extraction. In the future, we aim to explore large models for lowresource languages and their potential applications in low-resource scenarios.

6 Acknowledgments
This work is supported by the Natural Science Foundation of China (22BTQ010), theNational Natural Science Foundation of China(62076233) and the Innovation Project major
research of Chinese Academy of Social Sciences(2022MZSQN001).

References

To better understand the impact of different
Bo An and Congjun Long. 2021. Neural depenparts of the model on the experimental redency parser for tibetan sentences. Transactions
219

Table 3: Overall results of various models on Tibetan dependency parsing datasets.

Model
DL-RNNDL-GRUDL-LSTM
DL-Stacked
R-GCN
Our model

UAS
0.905
0.913
0.918
0.912
0.907
0.924

LAS
0.851
0.865
0.867
0.864
0.852
0.879

UEM
0.675
0.677
0.687
0.680
0.630
0.696

LEM
0.54
0.565
0.560
0.570
0.542
0.577

Table 4: The result of ablation study.

Model
Our model- GCN + LSTM
-Bert + GCN

UAS
0.924
0.913
0.918

LAS
0.879
0.865
0.867

UEM
0.696
0.677
0.687

LEM
0.577
0.565
0.560

on Asian and Low-Resource Language Information Processing, 20(2):1–16.

term memory. arXiv: Computation and Language.

Daniel Andor, Chris Alberti, David J Weiss,
Aliaksei Severyn, Alessandro Presta, Kuzman
Ganchev, Slav Petrov, and Michael Collins. 2016.
Globally normalized transition-based neural networks. arXiv: Computation and Language.

Christian Faggionato and Marieke Meelen. 2019.
Developing the old Tibetan treebank. In Proceedings of the International Conference on Recent Advances in Natural Language Processing(RANLP 2019), pages 304–312, Varna, Bulgaria.
INCOMA Ltd.

Qingxing Cao, Xiaodan Liang, Bailin Li, and Liang
Lin. 2019. Interpretable visual question answering by reasoning on dependency trees. IEEE
Transactions on Pattern Analysis and MachineIntelligence.
Danqi Chen and Christopher D Manning. 2014. A
fast and accurate dependency parser using neural networks. pages 740–750.
Hao Cheng, Hao Fang, Xiaodong He, Jianfeng Gao,and Li Deng. 2016. Bi-directional attention withagreement for dependency parsing. arXiv: Computation and Language.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. Bert: Pre-training ofdeep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Timothy Dozat and Christopher D Manning. 2016.
Deep biaﬀine attention for neural dependency
parsing. arXiv preprint arXiv:1611.01734.
Jiecairang Duo, Quecairang Hua, Keyou Huan,
and Rangdangzhi Cai. 2021. Transition basedneural network dependency parsing of tibetan.
In MATEC Web of Conferences, volume 336,
page 06018. EDP Sciences.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin
Matthews, and Noah A Smith. 2015. Transitionbased dependency parsing with stack long short-

Kazuma Hashimoto, Caiming Xiong, Yoshimasa
Tsuruoka, and Richard Socher. 2016. A joint
many-task model: Growing a neural network
for multiple nlp tasks. arXiv: Computation andLanguage.
Quecairang Hua, Wenbing Jiang, Haixing Zhao,
and Qun Liu. 2013. Semi-automatic building tibetan treebank based on word-pair dependencyclassification. Journal of Chinese Information
Processing.
Weina Zhao Jian Wu Huidan Liu, Minghua Nuo
and He Yeping. 2012. Segt:a practical tibetan
word segmentation system. Journal of Chinese
information processing.
Zhanming Jie, Aldrian Obaja Muis, and Wei Lu.
2017. Eﬀicient dependency-guided named entity
recognition. In Thirty-First AAAI Conferenceon Artificial Intelligence.
Eliyahu Kiperwasser and Yoav Goldberg. 2016.
Simple and accurate dependency parsing using bidirectional lstm feature representations.
arXiv: Computation and Language.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng
Kong, Chris Dyer, Graham Neubig, and Noah A
Smith. 2016. What do recurrent neural network
grammars learn about syntax. arXiv: Computation and Language.

220

Matthew E Peters, Mark Neumann, Mohit Iyyer,
Matt Gardner, Christopher Clark, Kenton Lee,and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv: Computationand Language.
Hua que-cai rang and Hai Xing Zhao. 2013.
Tibetan text dependency syntactic analysis
based on discriminant. Computer Engineering,39(4):300–304.
Michael Schlichtkrull, Thomas N Kipf, Peter
Bloem, Rianne Van Den Berg, Ivan Titov,and Max Welling. 2018. Modeling relational
data with graph convolutional networks. In
The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece,
June 3–7, 2018, Proceedings 15, pages 593–607.
Springer.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir
Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications tozero-shot dependency parsing. arXiv: Computation and Language.
Tashi-Gyal and Duo-La. 2015. Theory and methodof tibetan dependency treebank construction.
Journal of Tibet University.
Sajeetha Thavareesan and Sinnathamby Mahesan. 2020. Sentiment lexicon expansion using
word2vec and fasttext for sentiment prediction
in tamil texts. In 2020 Moratuwa engineeringresearch conference (MERCon), pages 272–276.
IEEE.
Nima-Zhaxi Toudan Cairang and Wanme Zhaxi.
2018. Study on the technique of tibetan dependence treebank building. Plateau Science Research, 2(03):103–109.
Tianming Wang, Xiaojun Wan, and Hanqi Jin.
2020. Amr-to-text generation with graph transformer. Transactions of the Association for
Computational Linguistics, 8:19–33.
XIA Wuji and HUAQUE Cairang. 2019. Dependency tree based tibetan semantic dependency
analysis. Journal of Tsinghua University (Science and Technology), 59(9):750–756.
Zou Xiaomei, Yang Jing, Zhang Jianpei, and Han
Hongyu. 2018. Microblog sentiment analysis
with weak dependency connections. KnowledgeBased Systems, 142:170–180.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan
Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang,
Changcheng Li, and Maosong Sun. 2020. Graphneural networks: A review of methods and applications. AI open, 1:57–81.

221

On the Development of Interlinearized Ancient Literature ofEthnic Minorities: A Case Study of the Interlinearization ofAncient Written Tibetan Literature
Congjun Long and Bo An
Institute of Ethnology and Anthropology,Chinese Academy of Social Sciences
Building 6, Zhongguancun Nandajie 27, Beijing, China{lcj,anbo}@cass.org.cn
Abstract
Ancient ethnic documents are essential to
China’s ancient literature and an indispensable civilizational achievement of Chinese culture. However, few research teams
are involved due to language and script
literacy limitations. To address these issues, this paper proposes an interlinearized
annotation strategy for ancient ethnic literature. This strategy aims to alleviate
text literacy diﬀiculties, encourage interdisciplinary researchers to participate instudying ancient ethnic literature and improve the eﬀiciency of ancient ethnic literature development. The interlinearized
annotation consists of original, word segmentation, Latin, annotated, and translation lines. In this paper, we take ancient
Tibetan literature as an example to explore the interlinearized annotation strategy. However, manually building a largescale corpus is challenging. To build alarge-scale interlinearized dataset, we propose a multi-task learning-based interlinearized annotation method, which can generate interlinearized annotation lines basedon the original line. Experimental results show that after training on about
10,000 sentences (lines) of data, our model
achieves 70.9% and 63.2% F1 values on thesegmentation lines and annotated lines, respectively, and 18.7% BLEU on the translation lines. It dramatically enhances theeﬀiciency of data annotation, effectively
speeds up interlinearized annotation, andreduces the workload of manual annotation.

1

Introduction2015), some of which record the process ofcreating the great history of the Chinese nation together and the vivid facts of exchanges
and interactions among various ethnic groups.
They contain rich national unity and progressideas and are necessary historical resources forwitnessing the Community of the Chinese Nation (Meng et al., 2023). Therefore, the indepth excavation of ancient ethnic literature
is conducive to promoting traditional Chinese
culture and showing the historical events ofthe formation of Sense of Community for the
Chinese Nation (Long et al., 2023).
China is rich in ancient ethnic literature, but
studying ancient ethnic literature faces many
diﬀiculties. First, the degree of digitizationis relatively low due to the lack of public resources; second, limited by language and script
literacy, the group of ancient ethnic literature
research and utilization is small. In exploring the formation of Chinese civilization andpromoting Chinese culture, how more disciplines and researchers pay attention to, study,
develop, utilize, and popularize the excellent
traditional culture contained in ancient ethnic
books is an issue worth exploring. The General Oﬀice of the CPC Central Committee and
the General Oﬀice of the State Council issued
the Opinions on Promoting the Work of Ancient Books in the New Era, emphasizing theneed to encourage interdisciplinary researchmethods. The ‘text structuring‘, ‘knowledge
systemization‘ and ‘intelligent utilization‘ ofancient books are actively carried out (Leiet al., 2022).
The documentary properties and unique cultural attributes of minority antiquarian literature have made it a focus of interdisciplinary
experts and a laboratory for interdisciplinary
research (Long et al., 2023). However, constructing most ancient ethnic literature re-

The excellent traditional culture of ethnicity isan essential part of Chinese culture, an important cultural heritage of the Chinese nation,
and a valuable asset to human civilization.
Many excellent traditional cultures have beenrecorded in ancient ethnic literature (Bender,
222

Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, pages 222–231,
held in Varna Bulgaria, Sept 8, 2023.
https://doi.org/10.26615/978-954-452-087-8.2023_025

sources is still diﬀicult to meet the needs ofmultiple disciplines. For example, experts incomputational linguistics focus on the information processing of ancient ethnic documents
and need a cooked corpus with information onword segmentation, annotation, entity recognition, and translation, and then carry out deep
text mining. Experts in the field of library intelligence explore the collection, collation, cataloging, and citation of multi-language ethnic
ancient texts from the perspective of knowledge organization and knowledge managementof ancient texts, and build catalog search libraries and full-text search libraries to serve
readers better. Linguistics researchers areconcerned about the phonology, vocabulary,and grammar of the national languages in themulti-language ancient ethnic document to assist in the construction of the ancient Chinese phonetic system, analysis and comparison of the Chinese and the people’s language
relations, to explore the language homology
differentiation clues, summarize the phonetic,
lexical and grammatical type characteristics
and the evolutionary path of the language.
Researchers in history focus on historical elements such as time, place, people, and eventsin the ancient texts of multi-language ethnic
groups and explore the political systems, economic systems, social histories, and foreign exchanges of different ethnic groups. Scholars
in ethnic culture explore the traditional culture, folk customs, cultural heritage, and traditional handicrafts recorded in the ancient texts
of multilingual ethnic groups. Experts in religion, philosophy, art, and traditional medicine
also hope to obtain the knowledge they needfrom multilingual ethnic literature.
To better meet the needs of multidisciplinary utilization of ancient ethnic texts, thispaper proposes a strategy of interlinearized annotation of ethnic ancient texts, converting thecontent of ethnic ancient texts into five lines ofdata, namely, the original line (original line),
the line of folk language sub-word (segmentation line), the line of Latin alphabet transcription (transcription line), the line of grammar
annotation (annotation line) and the line ofChinese meaning translation (translation line).
Researchers from different disciplines can use
these annotations to analyze and study the lit-

erature content. In conducting interlinearized
annotation research, manual annotation was
mainly used in the early stage. With theaccumulation of annotation data, this paper
proposes a multi-tasking framework based on
deep learning to automatically generate interlinearized annotation data to assist manual
annotation and finally build a large-scale textual structured database of ancient ethnic literature to lay the data foundation for further
development and utilization of ancient ethnic
documents in multiple disciplines.

2 Related Work
China is a multi-ethnic country, and in thelong history of the formation and developmentof the Chinese nation, people of all ethnic
groups have shared honor and disgrace andare closely related to each other, creating Chinese civilization and culture together. The
multilingual chapter-aligned, sentence-aligned,
and word-aligned historical documents handed
down or excavated archaeologically are thebest proof of the exchange and interminglingof people from all ethnic groups.
Among the Chinese and Tibetan bilingualaligned historical documents, chapter-aligned
documents are the most numerous, followed
by sentence-aligned and less word-aligned documents. Chapter-aligned documents are bothtranslated from Chinese into ethnic texts, suchas the four ancient Tibetan translations of
the Shang Shu Zhou Shu (Wong, 2016) in thefirst collection of the 1978 Paris photocopy of
the Selected Tibetan Documents of the Bibliothèque Nationale de French; there are alsotranslations from ethnic texts into Chinese,such as the oath on the west side of the TangFan Alliance monument, which is a ChineseTibetan aligned sentence pair (Li F G, 2007).
Most of the materials in the form of word control are found in the dictionary category andword list categories, such as the Great Collection of Translation Nominalities (Z, 2013),
the Dunhuang Tibetan texts P.T. 1257 and
P.T. 1261 (X, 2014), and the Imperial FiveStyle Qing Wenjian (Q, 2000). However, there
are not many materials on the full-text wordalignment of ancient literature texts. Scholarsof linguistics who study ancient ethnic literature often have to translate the documents.

223

In general, the source and translated texts
are still aligned in terms of chapter alignment,such as the translation of Baxie (Supplementary Text) (Ba S N, 1990), the Chinese translation of the History of Buddhism in Buton(Bu D, 2007), and the Tibetan King’s Tale
(Suo N J Z, 2002), among others. However,some scholars have also adopted the wordalignment model, such as the Study of Tibetan
Fatwas in the 8th-9th centuries (Z, 2007), etc.
Linguistic researchers have been more rigorous in organizing documentary materials, especially in dialectal and ethnolinguistic materials, and have mostly adopted the word alignment model. This paradigm is used for language text materials in the Chinese Ethnolinguistic Compendium Series, the Newly Discovered Languages of China Series, and the Endangered Languages of China Series. For example, an example of the annotated text for
the language of the security language on page
1918 of Languages of China.
The German publishing house Lincom
GmbH has been funding the publication ofinterlinearly annotated corpora and scholarly
works in minor languages worldwide for many
years. Tikaram Poudel published Rajbanshi
Grammar and Interlinearized Text (an IndoAryan language of Nepal and Bengal) in 2006
(Poudel, 2006); Karnakhar Khatiwada published A Reference Grammar in 2017 of Dhimal (King, 2008) describing writings and text
annotation Interlinearized texts in Dhimalwith Grammar notes (Khatiwada, 2017) (interlinearized annotated texts in Dhimal). To
date, the publisher has published more than
500 works in small languages. Sino-Tibetan
linguists Randy J. LaPolla & Dory Poa also
published Rawang Texts grammatically annotated texts at Lincom Europa (LaPolla andPoa, 2001).
Computer experts have developed interlinearized annotation tools to assist linguists inadvancing interlinearized annotation successfully. The American Standard Interchange
Language (SIL) organization has developed
Toolbox 1 annotation tool; British scholars
have developed Eudico Linguistic Annotator (ELAN) annotation tool 2 , and French scholars
12

https://software.sil.org/software-products/
http://sites.bu.edu/elsa/elan-coding/

adopted the Interlinear Text Editor software(ITE) technology. Chinese scholars have used
Toolbox tool to organize and publish the series ‘Grammatical Annotated Texts of Chinese
Ethnic Languages‘ (D, 2016), which is a totalof 20 books covering 20 languages or dialectsof five principal language families or groups
in China, namely Tibetan-Burmese, Miao-Yao,
Dong-Tai, South Asian and Altaic, with a
total word count of about 10 million words.
These software tools are widely used in the linguistic community, making it easier and fasterfor linguists to annotate the corpus and enhancing the standardization of corpus annotation. However, the common drawback is thatthey mainly rely on manual operations and failto introduce natural language processing techniques for low-resource languages, especiallynatural language information techniques.
Interlinearized annotation is similar to thegoal of word alignment in machine translation,
where the word alignment technique is to obtain word boundaries in sentence pairs andachieve translation alignment based on bilingual pairs, which is a core task in machinetranslation (Bahdanau et al., 2014). However,
the research results devoted to word alignment
methods belong to the early stage of statisticalmachine translation. With the development of
ethnolinguistic information processing and thepromotion of the ‘One Belt, One Road‘ strategy, machine translation of low-resource languages has become a popular research topic(Ranathunga et al., 2023), and some research
results discussing the word alignment between
Chinese and Mandarin have appeared. For example, Zhao Yang and Zhou Long discussed
the Min-Chinese scarce resource neural machine translation technique (Zhao Yang, 2019);(Su L Y, 2018) discussed the word alignment
method in Mongolian-Chinese machine translation; (Liu J M, 2011). Studied the Han-Vi
word alignment. However, the current machine translation commonly adopts deep neural network technology, which does not needto discuss word alignment methods separately.
In recent years, the concepts of ‘exploring
the origin of Chinese civilization‘ and ‘forging
a sense of Chinese national community‘ havebeen proposed, and interdisciplinary fields
have jointly focused on transcribed texts of224

ethnic minority oral discourse and ancient ethnic literature. The increasing demand for interlinearized annotation of ethnic texts has
made interlinearized annotation a researchparadigm. However, it is challenging to meetthe needs of multidisciplinary and multilingual interlinearized annotated corpus by manual annotation. This paper combines the linguistic fine-grained annotation paradigm andmulti-tasking techniques to conduct automaticinterlinearized annotation.

3
3.1

Ancient Ethnic Literature
Interlinearized Annotation
The Format of Ancient Literature
Interlinearized Annotation

The target of interlinearized annotation is thefull text of ancient ethnic literature without
explicit markers between words, which need
to be divided into ‘words‘ for the full text,
and then convert the traditional ethnic script
into Latin transcription or international phonetic symbols by word. The words are translated into the other language, and the function words are marked with their grammatical
function. The grammatical functions are labeled with English abbreviation tags common
to the linguistic community. The final output consists of a line of the original language,a Latin transcription or an International Phonetic Alphabet line corresponding to the lineof the minor language, and a line of the translation marker. Under the current technical conditions, meaningful translation lines cannot be
obtained automatically and need to be translated manually. Figure 1 takes Tibetan as an
example to show an example of interlinearized
annotation.
The original line is the original text of theethnic literature. The segment line is a unitof words (partly morphemes and phrases), a‘word‘ or ‘word suﬀix‘ for the input text. However, ‘word‘ is the most basic unit for the machine to understand the text. To satisfy the
deep analysis and mining of the text, marking the word boundary is the most basic task.
The materials of ancient texts marked withword boundaries help users understand ancient texts. They can be used for training todevelop automatic lexical analysis tools for ancient texts, providing resources to support the

information processing of ancient literature.
The transcription lines are ethnic texts transcribed in Latin alphabet or the international
phonetic alphabet. The aim is to create across-reference database of ethnic scripts insyllables. The annotation line: the meaningful words in the annotated lines are translated
into Chinese or English. Function words aremarked with the abbreviated form of theirgrammatical function in English, and the abbreviated form often uses a combination ofcapital letters, which comes from the English
word ‘AGENT‘ and semantically indicates theadministration of things. This internationallyaccepted grammatical mark is conducive tothe dissemination of national ancient literature materials to the world. The translationline is the Chinese or English translation of
the original line.
3.2 The Schema of Ancient Literature
Interlinearized Annotation

Designing symbols for interlinearized annotation requires consideration of the usage needsof ancient ethnic literature, which have been
discussed earlier. For the same ancient ethnic literature, different researchers have different needs. Such as syllogisms, word segmentation, and named entity recognition are common needs for researchers. Semantic annotation is a common need for linguistics-related
disciplines such as syntax and language information services. Named entity annotation(NER) is a common need for history and literature, etc. Chinese translation of meaningful
words is closely related to machine translation.
This paper focuses on syntactic and semantic
annotation and entity annotation, whose annotation materials can meet the needs of mostdisciplines. Grammatical and semantic annotation can reflect functional words’ grammatical meaning and semantic function. Entity annotation includes proper names such as person,
place, and time. Labels for function words are
generally composed of two or three letters, taking the first three letters of the English word.
If repetition is encountered, the abbreviatedletters are modified as appropriate. When agrammatical function requires more than one
English word to be represented, the appropriate combination of letters from multiple words
is selected. The combination of labels also fol225

Figure 1: The example of Tibetan interlinearized annotation.

lows specific rules, and the linguistic community usually adopts the Leipzig Terminology
Rules (The Leipzig Glossing Rules) system 3 ,which was jointly developed by the Department of Linguistics of the Max Planck Institute for Evolutionary Anthropology and theDepartment of Linguistics of the University ofLeipzig. In our study of interlinearized annotation of ancient civil texts, the Leipzig labeling system was employed as the primary basis,
with the addition of some labels. The tagging
system includes person and number, grammatical, tense, tone, mood, demonstrative, special
word classes, syntax, noun-pronoun correlative markers. The grammatical tags employed
for interlinear annotation will vary with the degree of refinement of the corpus, and the tags
listed here are only the main ones. Some minority language scripts require an extension of
the tagging system according to specific needs
but keep the basic system unchanged. The
NLP-based NER is adopted.

we have developed a semi-automated interlinearized annotation platform. The interlinearized annotation platform mainly has the following modules.
(1) Interlinearized annotation operation
module. This module completes the task ofinterlinear annotation in the original languageof ancient ethnic literature and accomplishes
four main functions:(a) Automatic conversion from the original
language to the transcription line. The conversion from the original Chinese text to the Latinalphabet depends on the mapping table between the original syllables and the Latin syllables. For example, the Tibetan to-Latin conversion collects about 28,000 syllables, including modern Tibetan, ancient Tibetan, Sanskrit syllables transcribed in Tibetan characters and punctuation marks. Also, it includes
syllables that conform to Tibetan spelling
rules but do not exist in existing literature.
(b) Automatic word segmentation. The interlinearized annotation is based on the anno4 Human-computer Interaction
tation of words (or phrases), and word segmenInterlinearized Annotation
tation is a necessary process.
Platform(c) Annotation line including meaningfulwordtranslations and grammatical labels. To
Corpus annotation is time-consuming andbe compatible with various text editors andlabor-intensive; however, annotated corpora
can provide essential resources for ancient lit- convenient for different researchers, the interlinear annotation results are stored as XML
erature research and are indispensable. With
format files, with a set of brackets {} to indienough training corpus, NLP algorithms can
assist corpus annotation, such as NER, rela- cate the four levels of interlinear correspondtion extraction, text classification, and ma- ing lexical entries, which are filled in with themorphological analysis before the split form,chine translation. The corpus annotation is
also a common task in NLP. The interlin- Latin transcription, morphological analysis after the Latin transcription and annotation inearized annotation of ancient ethnic literature
is a pioneering work, and by manually annotat- formation, respectively.
(d) Manual proofreading. Manual proofing a certain scale of training data, the NLP
readingneeds to do three checking aspects:
algorithms can assist in data annotation.
fillinginvacant paraphrases, correcting paraTo advance the research in this paper,
phrase errors due to multiple meanings, and3
https://www.eva.mpg.de/lingua/resources/glossing- annotation errors due to subtext errors.
rules.php

226

(e) Batch export and import of annotation
data: to reduce the workload of manual annotation, the import and export of data aresupported after a certain number of interlinear annotation data are completed.
(f) Interlinearized annotation corpus retrieval module. This module supports structured retrieval and can meet the needs of general literature information retrieval, such as
the retrieval of Latin transcriptions of ethnic
scripts, pairs of translated words and labels;
the extraction of cross-referenced word lists,
the extraction of named entities, and otherfunctions.

5

5.1

Automatic Interlinear Annotation
Method based on Multi-task
Learning
MethodIn this paper, we propose NLP method to promote the interlinearized annotation of ancientethnic literature for deep analysis and exploration. In the previous work, we have annotated a dataset of ancient Tibetan interlinearized annotation, so we take ancient Tibetan
as an example to introduce the automatic generation method of interlinearized annotation
based on deep learning, and the interlinearized
annotation of other ethnic ancient literature
is similar to ancient Tibetan. The interlinearized annotation dataset of ancient Tibetan
includes original, segmentation, Latin, annotated, and translation lines. Among them, thesegmentation lines are obtained by automaticword segmentation based on the original lines(Liu H D, 2012), the Latin lines can be transcribed directly by the Tibetan-Latin conversion table, and the annotation lines are generated based on the segmentation lines withcorresponding meaningful words and function
words annotated. The original line, the syllogism line, and the annotated line are all valuable for generating translation lines. Therefore, the ancient Tibetan interlinearized annotation model is mainly used to generate segmentation, annotated, and translation lines.
Translation lines are translations from Tibetan
sentences to Chinese sentences, which belong
to the research scope of machine translation.
Regarding the current research base, the generation of Chinese lines is more complex andrequires human intervention. The model’s input is the original line, and the output is thecontent of the segmentation line, the annotation line and the translation line. The information of the annotated line depends on thecontent of the segmentation line, and the information of the translation line depends on theinformation of both the segmentation line andthe annotated line. Therefore, the pipelinemodel (Li et al., 2020) is employed in modeling, and its architecture is shown in Figure
4-a. The pipeline approach consists of threemodels: (1) the word segmentation model: theinput of this model is the original line, andthe output is the result of the word segmentation line; (2) the annotation model: the inputof this model is the information of the original line and the word segmentation line, andthe output is the result of the annotation line;(3) the translation model: the input of thismodel is the original line, the word segmentation line and the annotation line, and the output is the result of the translation line. The
pipeline method splices the outputs and inputs
of different models, such as using the outputof the segmentation line as the input of the annotation line, which facilitates the implementation of the model. However, the pipeline
method suffers from problems such as error
propagation. For example, the segmentation
model can only utilize the information of theoriginal text line, but the information of theannotation line also has important value forthe segmentation model, and the error of thesegmentation line may cause the obvious errorof the annotation line information. However,since the models are independent, the error information cannot be effectively transferred tothe segmentation model, and this part of theinformation cannot be utilized.
Recently, multi-task learning models replaced pipeline-based models in several fields,such as segmentation and annotation models,
named entity recognition and entity linking
models (Nguyen and Grishman, 2015). The
advantage of multi-task learning models isthat they can make full use of the correlation
between different tasks, e.g., there is a strong
correlation between the Tibetan word segmentation and lexical annotation tasks, and theresults of word segmentation determine the227text block boundaries of linguistic annotation.
In contrast, the results of lexical annotationcan, in turn, verify whether there are errors
in the word segmentation results. Therefore,
multi-task learning approaches are widely applied due to the advantages in modeling multiple related tasks. Interlinearized annotation
requires the generation of corresponding segmentation lines, annotated lines, and translation lines based on the original text lines, atypical multiple-related task suitable for modeling using a multi-task learning framework.
Based on the above analysis, this paper
designs a multi-task model to conduct theword segmentation, annotation, and translation models. The model’s input is the original text line, and the shared coding layer
encodes the input information. Then, different upper-layer models are used to model the
output tasks (word segmentation, annotation,and translation lines). The word segmentation model, the annotation model, and thetranslation model share the embedding layer(Embedding) (Lai et al., 2016) and the encoding layer (Bi-directional Long Short-Term
Memory, BiLSTM) (An et al., 2018; An andLong, 2021), and the word segmentation line
is based on the original line, and the words
are segmented from each other using spaces.
Therefore, in this paper, the word segmentation is modeled as a sequential annotationtask, and the task layer uses Conditional Random Field (Sutton et al., 2012). The annotation line contains grammatical annotation
information and word translation information
for a sequence generation task, which is modeled as an encoder-decoder sequence generation task in this paper. The translation lineis the translation of the original line content
into Chinese, which is a typical machine translation task, and is also modeled as an encoderdecoder sequence generation task in this paper(Guo et al., 2019).
5.2

Experimental Settingstences, and we divide it into the training set,development set, and test set according to thescale of 8:1:1.
5.3 Experimental Settings
This section describes the implementation
framework and hyperparameters employed in
the experiments. We utilize Pytorch to implement a multi-task model. The dimension of Tibetan syllables is 100; the coding
layer is BiLSTM. We employ CRF to implement word segmentation tasks. In the annotation task, we decode the sequences using
a single-layer BiLSTM to generate grammatical function labels. We use a single-layer
BiLSTM to generate the translation line in
the translation task. This paper employs thepipeline-based model as the baseline, including the BiLSTM+CRF-based word segmentation model, BiLSTM+BiLSTM-based annotation model, and encoder-decoder-based translation model. These models are trained separately, with input and output data transfer
and information interaction. This paper employs Precision, Recall, and F1-value to evaluate the results of the segmentation line andannotation line. We employ BLEU (Papineni
et al., 2002) to evaluate the result of the translation line.
5.4 Experimental Result
We conduct three sets of experiments: the first
set of experiments adopts a multi-task learning approach, aiming to implement interlinearized annotation of ancient Tibetan literature, generating segmentation lines, annotatedlines, and translation lines based on the original lines; the second set of experiments is a
stripping experiment, using a multi-task learning approach to model segmentation lines andannotated lines, segmentation lines and translation lines, and annotated lines and translation lines, respectively, to analyze the effect of
joint learning of different tasks; the third setof experiments utilizes a pipeline model as abaseline model to compare the performance ofproposed models.

To verify the effectiveness of the interlinearized annotation model, we completed fourinterlinearized annotated ancient Tibetan lit5.5 Experimental Results
erature, Baxie (Ba S N, 1990), Weixie, ZhuThe input of this experiment is the original
mian Shi (Bu D, 2007), and Di Wu Shijitext line, and the output includes the segmen(Schneider, 2002), through the annotation
platform. The dataset consists of 12,284 sen- tation line (Seg), the annotation line (Ann),228

and the translation line (Tra). The experimental results are shown in Table 1. The
experimental results show that the multi-task
learning-based model (Multi) proposed in thispaper significantly improves all of the threetasks of segmentation lines, annotation lines,
and translation lines (6.7%, 15.6% and 32.6%,
respectively). The multi-task learning-basedmodel achieved better performance than thepipeline model (Pipe). Therefore, the multitask-based model can achieve better performance in interlinearized annotation task.
Task
Seg
Ann
Tra

Model
Mult
Pipe
Mult
Pipe
Mult
Pipe

P
74.2
68.2
66.2
54.1
-

R
67.8
64.7
60.4
55.2
-

F
70.9
66.4
63.2
54.6
-

a significant decrease, indicating that the information of the segmentation rows has an essential impact on the results of the annotatedline; (d) The results of translation lines in bothgroups B and C decreased compared with themulti-task learning model but better than thepipeline-based model, indicating that the information of both segmentation lines and annotated lines has auxiliary value for the translation model.
Task
A

BLEU
18.7
14.1

B
C

Model
Seg
Ann
Seg
Tra
Ann
Tra

P
72.1
60.3
71.8
50.3
-

R
66.2
59.4
66.0
48.2
-

F
69.0
59.8
68.8
49.2
-

BLEU
15.3
14.7

Table 2: The result of ablation study.

Table 1: The result of multi-task learning model.

6 Conclusion
5.6

Ablation Study

5.7

Ablation Study

The ablation study aims to analyze the effectof different task combinations. In this experiment, group A ( segmentation model + annotation model), group B ( segmentation model+ translation model), and group C (annotation
model + translation model). Table 2 shows
the results of the three experimental groupsof experiments. Based on the experimental results, we can draw the following conclusions:(a) In both groups of multi-task learning inwhich the segmentation model participates,
the segmentation result improves (F1 value
66.4%), indicating that the results of both theannotated lines and the translation lines canimprove the performance of the segmentation
model; (b) The segmentation model achieves
better results in the results of group A, indicating that the annotation line provides better
feedback to the segmentation model than thetranslation line; (c) In group A, the multi-task
model achieves better results than the pipelinemodel, indicating that the annotation model
can give effective feedback to the segmentation model. However, the performance of theannotated rows in group C experiments has

In this paper, we discuss the idea and visionof interlinearized annotation of ancient ethnic literature from the perspective of data resource normalization and sharing. Taking ancient Tibetan literature as an example, we propose accumulating corpus based on manual interlinearized annotation and then using machine learning to conduct automatic annotation. This research provides a new researchparadigm for developing and utilizing ancientethnic literature in China, especially the structured data of interlinearized annotation, which
lays a good foundation for ancient literaturedevelopment and utilization. In the future,
we plan to construct ancient language knowledge based on an interlinearized annotation
dataset.

7 Acknowledgments
This work is supported by the Natural Science Foundation of China (22BTQ010), theNational Natural Science Foundation of China(62076233) and the Innovation Project major
research of Chinese Academy of Social Sciences(2022MZSQN001).

229

References
Bo An, Bo Chen, Xianpei Han, and Le Sun. 2018.
Accurate text-enhanced knowledge graph representation learning. In Proceedings of the 2018
Conference of the North American Chapter ofthe Association for Computational Linguistics:
Human Language Technologies, Volume 1 (LongPapers), pages 745–755.

Zhao W N Liu H D, Nuo M H. 2012. Segt: A practical tibetan word segmentation system. Journal of Chinese Information Processing, 26(1):97–
103.
Ai S Liu J M, Tu R G. 2011. Research on statisticalmachine translation-based chinese-uyghur wordalignment. Computer Applications and Software,
28(4):57–59.

Bo An and Congjun Long. 2021. Neural dependency parser for tibetan sentences. Transactions
on Asian and Low-Resource Language Information Processing, 20(2):1–16.

Congjun Long, Bo An, and Shengyan Zhang.
2023. Research on the construction of knowledge graph of old tibetan inscriptions. Library
And Information Service, 67(8):141.

Tong J H Ba S N. 1990. Ba Xie, volume 10.
Sichuan Ethnic Publishing House.

Linglei Meng, Jingnan Xing, and Mingyan Tan.
2023. Exploration of cultivating a sense of community for the chinese nation in” data structures” course teaching. International Journal of
Education and Humanities, 7(3):136–139.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua
Bengio. 2014. Neural machine translation by
jointly learning to align and translate. arXiv
preprint arXiv:1409.0473.

Thien Huu Nguyen and Ralph Grishman. 2015.
Relation extraction: Perspective from convolutional neural networks. In Proceedings of the
1st workshop on vector space modeling for natural language processing, pages 39–48.

Mark Bender. 2015. Ethnic minority literature. A
Companion to Modern Chinese Literature, pages
261–275.
Pi W C Bu D. 2007. History of Budun Buddhism,
volume 9. Gansu Ethnic Publishing House.
�Jiang D. 2016. Chinese national language grammar annotation text series. (No Title).
Wenzhong Guo, Jianwen Wang, and Shiping Wang.
2019. Deep multimodal representation learning:
A survey. Ieee Access, 7:63373–63394.
Karnakhar Khatiwada. 2017. Interlinearized Texts
in Dhimal with Grammar Notes. Lincom Europa.
John Timothy King. 2008. A grammar of Dhimal.
Ph.D. thesis, Leiden University.
Siwei Lai, Kang Liu, Shizhu He, and Jun Zhao.
2016. How to generate a good word embedding.
IEEE Intelligent Systems, 31(6):5–14.
Randy J LaPolla and Dory Poa. 2001. Rawang
texts, volume 18. Lincom Europa.
Yuying Lei, Xilong Hou, Xiaoguang Wang, et al.
2022. The logic and approach of digital reconstruction of ancient books in the data intelligence. Digital Human Research, 2(2):46.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li.
2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1):50–70.
Ke W N. W Q L Li F G. 2007. Study on Ancient
Tibet, volume 1. Beijing Tsinghua University
Press.

Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages
311–318.
Tikaram Poudel. 2006. Rajbanshi grammar andinterlinearized text, volume 34. Lincom.
Jiang Q. 2000. A textual research on the compilation of qing wenjian in imperial four and fivestyles.
Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen
Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A
survey. ACM Computing Surveys, 55(11):1–37.
Hanna Schneider. 2002. Tibetan legal documentsof south-western tibet: structure and style. In
Proceedings of the Ninth Seminar of the IATS,
2000. Volume 1: Tibet, Past and Present, pages
415–427. Brill.
Niu X H Su L Y, Zhao Y P. 2018. The study
on ethnic-to-chinese scare-resource neural machine translation. Journal of Chinese Information Processing, 32(6):44–51.
Liu Q L Suo N J Z. 2002. Tibet Wangtongji, volume 2. Nationalities Publishing House.
Charles Sutton, Andrew McCallum, et al. 2012.
An introduction to conditional random fields.
Foundations and Trends® in Machine Learning,
4(4):267–373.

230

Sun-tik Wong. 2016. The critical study of politicalpbligation in zhou shu of shang shu and liji=”shang shu, zhou shu” ji” li ji”’zheng zhi yi wu’zhi
yan jiu. HKU Theses Online (HKUTO).
Dang Z Z X. 2014. Ancient tibetan dictionary.
Huang W Z. 2007. Research on Tibetan vows in the
8th and 9th centuries, volume 8. Nationalities
Publishing House.
Zhang Y Z. 2013. A collection of translated names– the origin of tibetan bilingual dictionary.
Wang Qian etc Zhao Yang, Zhou Long. 2019. The
study on ethnic-to-chinese scare-resource neuralmachine translation. Journal of Jiangxi Normal
University, 43(6):630–637.

231

Author Index
Mercelis, Wouter, 148
Mischer, Lisa, 160

An, Bo, 213, 222
Assenmacher, Matthias, 103
Attema, Jisk, 23

Naaijer, Martijn, 23
Nie, Ercong, 68
Nissim, Malvina, 49
Nuñez, Gilary Vera, 103

Beersmans, Marijke, 1
Bennett, Ellie, 193
Brooks, Creston, 170
Chang, Bolin, 122
Coeckelbergs, Mathias, 23
Cowen-Breen, Charlie, 170

Palladino, Chiara, 179
Pedrazzini, Nilo, 49
Peels, Saskia, 49
Picca, Davide, 88

de Graaf, Evelien, 1
De Vos, Ilse, 111

Qi, Yue, 117

Fantoli, Margherita, 1Feng, Minxuan, 122Frank, Anette, 30

Richard, Caroline, 88
Riemenschneider, Frederick, 30
Romanov, Maxim, 160

Gamba, Federica, 59
Garces Arias, Esteban, 103
Graziosi, Barbara, 170
Guzman-Soto, Hansel, 133

Sahala, Aleksi, 193, 203
Schmid, Helmut, 68
Schöffel, Matthias, 103
Schütze, Hinrich, 68
Shamsian, Farnoosh, 179
Sikkel, Constantijn, 23
Spanakis, Gerasimos, 39
Stopponi, Silvia, 49
Swaelens, Colin, 111

Häberlin, Alexander, 103
Hakimi, Hamid Reza, 160
Haubold, Johannes, 170
Heumann, Christian, 103
Hu, Dongxin, 138

Tate, Derrick, 13

Jin, Kai, 96

Van de Cruys, Tim, 1
Van Hal, Toon, 148
Van Peursen, Willem Th., 23
Vico, Gianluca, 39

Keersmaekers, Alek, 148
Koch, Philipp, 103
Krahn, Kevin, 13
Lamicela, Andrew C., 13
Lefever, Els, 111
Li, Bin, 117, 122
Li, Haonan, 80
Lindén, Krister, 203
Liu, Liu, 117
Liu, Wuying, 96
Liu, Yudong, 133
Long, Congjun, 222

Wang, Dongbo, 117, 122
Xu, Zhixing, 122
Yousef, Tariq, 160, 179
Yuan, Yiguo, 122
Zeman, Daniel, 59
Zhang, Yixuan, 80
Zhao, Dan, 96

McGillivray, Barbara, 49
233