The IMP project: developing
resources for historical SloveneTomaž Erjavec
Dept. of Knowledge TechnologiesJožef Stefan InstituteLjubljanaFirst ENeL workshop
September 29 2014
BledThe IMP project

Overview
1. Project background
2. Digital library
3. Hand-annotated corpus
4. Lexicon
5. Annotation tool

6. Conclusions

2

The IMP project

3Background
1. AHLib project (2004–08)• Deutsch-slowenische/kroatische Übersetzung 1848– 1918,
AAS / KFU (prof. Erich Prunč) + JSI
2. EU IP IMPACT (2010–2011)• Improving Access to Text,NUK (Alenka Kavčič Čolić, Ines Vodopivec) + JSI3. Google award (2011 + 2012)• Computational models for historical Slovene,ZRC SAZU (Matija Ogrin) + JSI• Funding for Wikisource (Miran Hladnik)

The IMP digital library

4The Texts and Goals of the Projects• AHLib• Slovene books 1848 – 1918 translated from German• Develop a corpus to study translation processes• IMPACT• Slovene books + 1 newspaper from 18th and 19th centuries• Proof-read texts and facsimiles for improving OCR andlexicon of historical words for improving IR• Google:• Two samples of very old Slovene books +Wikisource Slovene literary classics from (mainly) 1850 - 1918
• develop computational models for historical Slovene for better
language technologies

The IMP digital library

5

IMP Digital Library• Each text consists of:• meta-data• facsimile• proof-read transcription• All texts are uniformly encoded according to TEI

Guidelines for Electronic Text Encoding and Interchange,
TEI P5• No problems with further use or dissemination (CC-BY)

6The IMP digital library

IMP DL by source
Source
AHLib

Units
105

Pages
12.854

Units / source
3Words
2.783.569

KRN36555

551.599

NUK142.428

338.874

WIKI

500

30.29610.761.913

ZRC

3

75

20.631∑658

46.20814.456.586

AHLib

105
36

KRN

14

NUK
WIKI

500

ZRC

Words / year
800000700000600000500000400000300000200000100000

1584
1768
1773
1784
1789
1792
1799
1810
1830
1835
1837
1843
1845
1847
1849
1851
1853
1855
1857
1859
1861
1863
1865
1867
1869
1871
1873
1875
1877
1879
1881
1883
1885
1887
1889
1891
1893
1895
1897
1899
1901
1903
1905
1907
1909
1911
1913
1915
1917

0

The IMP digital libraryDistribution of DL texts

7

The IMP digital library

8

Annotation on each unit
• Meta-data (teiHeader):• id, responsibility, extent, availability …
• basic bibliographic information (two titles: original, modern)• taxonomy: medium (manuscript, book, magazine, newspaper), text
type (fiction, non-fiction, religious), translation status (original,translated)• tag usage, revision description• Facsimile:• images in several sizes, each page break linked to facsimile• Text structure:• divisions, headings, lists, tables, notes, poems, figures, line
breaks…
• Editorial interventions:• sic/corr, foreign

The IMP digital library

9

TEI P5 encoding: source description<sourceDesc><bibl><title type="orig" xml:lang="sl-bohoric">Genovefa</title><title type="reg" xml:lang="sl">Genovefa</title><author>Schmid, Christoph von</author><respStmt><resp xml:lang="sl">Prevajalec</resp><resp xml:lang="en">Translator</resp><name>Malavašič, Fran</name></respStmt><date>1841</date><publisher>J. Blasnik</publisher><pubPlace>V Ljubljani</pubPlace><extent>107</extent><idno>10242 oder 13150 → NUK - Narodna in univerzitetna knjižnica</idno><note type="TraDok" xml:lang="de"><ref target="http://itat2.uni-graz.at/pub/tradok/">TraDok</ref><lb/>
1841<lb/>

The IMP digital library

10

TEI P5 encoding: text body
<pb n="[3]" facs="#FPG04260-002" xml:id="pb.003"/><div type="level1" xml:id="div.2"><head xml:id="head.2">1. <lb/>Kako Ožbalt iz vojske domú pride in kaj ljudjé govorijo.</head><figure xml:id="figure.3"><figDesc>Ornamentna sličica. Okrašena črka v.</figDesc></figure><p xml:id="p.7">V nedéljo po poldne je bilo in v Zlati Vasi so mlajši fantini in dekleta pod staro
lipo sedéli in peli, ali pa se smejali, kadar jo je kdó iz pivnice prilomil, ki je pregloboko v kozarček
polukal. Nekteri kmetje s svojimi ženami so pa v gostivnici sedéli in pri bokalu prav židane volje bili,
kakor je že to navada, kadar sta vino in vol po ceni.</p><p xml:id="p.8">Kar jo primaha nék neznan človek v vas. Terdne in velike postave je bil in<choice><sic>kakil</sic><corr>kakih</corr></choice>
tridesét lét je mogel iméti; obléčen je bil v sivi suknji, na strani je imel veliko sabljo, na herbtu pa…

The IMP libraryUp-translation to TEI• AHLib:

Dedicated RTF-to-TEI web converter• IMPACT:XSLT stylesheets for PageXML-to-TEI conversion• Wikisource:
PHP converter for HTML-to-TEI• Latest: http://nl.ijs.si/tei/convert/

11

The IMP digital library

12Down-translation to HTML
• We use (slightly modified) TEI XSLT stylesheets forTEI-to-HTML• Each text is one HTML file, showing both the facsimileand (typeset) transcription• Indexes to books are by taxonomy, sorted by (one of)• author, title, date, signature (one index with title pages)

The IMP digital library

Example book: front

13

The IMP digital library

Example book: body

14

The IMP library

Real example…

15

The IMP library

Example index

16

The IMP library

Example index

17

The IMP library

IMP DL: summary• Large library of Slovene historical texts• Focus on the 19th and early 20th century• Rich (and somewhat varied) TEI encoding• HTML rendering, facsimile PDF, TEI• Basic indexes, no search (poor man‘s DL)• No ePub, PDF of transcription• Still too small• But is JSI the place to host a large & professionaly

curated cultural heritage library?

18

The IMP project

Part II. Linguistic annotation
ljubesenljubésen
ljubesinljubésinljubezen
ljubézen
ljubezinljubézinlubesanlubesenlubésenlubęsenlubesèn
lubésèn
lubęsènlubesęn
lubesnlubésnlubęsnlubeſnlubiesen

ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen

ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezen
ljubezenlubiesn ljubezen ljubezen
ljubeznih ljubeznih ljubezen
ljubesnijo ljubeznijo ljubezen
ljubésnijo ljubeznijo ljubezen
ljubeznijo ljubeznijo ljubezen
ljubéznijo ljubeznijo ljubezen
lubesnio ljubeznijo ljubezenlubęsnio ljubeznijo ljubezen
ljubezníjo ljubezníjo ljubezen
ljubesni ljubezni ljubezen
ljubésni ljubezni ljubezen
ljubesní ljubezni ljubezen
ljubezni ljubezni ljubezen
ljubézni ljubezni ljubezen
lubesne ljubezni ljubezen
lubeſne ljubezni ljubezen
lubèſne ljubezni ljubezen
lubesni ljubezni ljubezen
lubésni ljubezni ljubezenlubęsni ljubezni ljubezen
lubeſniljubezni ljubezen

19

The IMP corpora

20

The goo300k corpus
Gold-standard linguistically annotated corpus of historical
Slovene• 300.000 words, 1.100 pages• page-sampled from IMP DL• manually annotated• TEI encoded
Word-level annotations:• Tokenisation• Contemporary word-form• Contemporary lemma• Part-of-speech• Gloss for archaic words

The IMP corpora

goo300k text distribution

21

The IMP corpora

Linguistic annotation

22

The IMP corpora

Manual annotation• Team of students lead by a lexicographer• Several rounds of annotation and correction• Annotators' manual, Cookbook, FAQ,…
• Used INL CoBaLT editor

23

The IMP lexicon

24

IMP lexicon• Simply a dump of the hand annotated corpus• However, goo300k is too small for a (large) lexicon:• sampled another corpus from IMP DL, foo3M• also annotated in CoBaLT• but only word-forms that do not appear in goo300k• The lexicon is (of course) also encoded in TEI P5• Mounted in the Web as a set of HTML files(bespoke XSLT stylesheet)

The IMP lexicon

Hyperlinked lexicon on the Web25

The IMP lexicon

26

Size of the lexicon• XL: all words(goo300k is fully annotated!)• L: dictionary words(no numerals, typos, foreign words, proper nouns)• M: historical words(historical form ≠ contemporary form)• S: archaic words(with gloss)

The IMP tools

27

ToTrTaLe• Tool to automatically:
1. Tokenise the text: mlToken
2. Transcribe (modernise) the words: Vaam
3. PoS tag the words: TnT
4. Lemmatise the words: CLOG
• Use of background resources:• IMP lexicon + modern lexicon + transcription rules• tagger + lemmatiser models for contemporarySlovene• TEI I/O

The IMP tools

28

ToTrTaLe example„… združimo tiga[tega] kužniga[kužnega]...“<w lemma="združiti" ana="#Vmer1p">združimo</w><c> </c><choice><orig><w>tiga</w></orig><reg><w lemma="ta" ana="#Pd-msg">tega</w></reg></choice><c> </c><choice><orig><w>kužniga</w></orig><reg type="pattern" n="[ega@←iga@]"><w lemma="kužen" ana="#Agpmsg">kužnega</w></reg></choice>• Vmer1p = Verb, Type=main, Aspect=perfective, VForm=present,

Person=first, Number=plural

The IMP corpora

29Concordancers• The IMP DL was annotated with ToTrTaLe → IMP corpus• The IMP and goo300k corpora are available via twoconcordancers:• noSketchEngine: OS version of the popular (and commercial)

SketchEngine• CUWI: our front-end to the well-known IMS CWB corpus workbench• The concordancers offer:• powerful search query syntax (REs over words and annotations)• filters over meta-data (text types, year of publication, author, …)• various sorting options over concordances• construction of frequency lexica• collocations• saving results• etc.

The IMP corpora

noSketchEngine

30

The IMP corpora

CUWI31

The IMP corpora

More than just concordances

32

The IMP project

33

Conclusions• Presented IMP language resources, the result of threeconnected projects• Available at http://nl.ijs.si/imp/• A varied and connected BLARK for historical Slovene• Used and useful for language technologies and digitalhumanities

The IMP project34

HLT vs DH• Human language technologies:

digital library, lexicon, corpus• uniformly encoded in TEI P5• available under CC-BY• Digital humanities:• web mounted digital library and lexicon• goo300k and IMP corpus available throughconcordancers• Two birds with one stone (or is it two stones?)

The IMP project35

HLT R&D• Lexicon used for full-text search in dLib.si, the digital

library of Slovenia• Experiments investigating better modernisation, e.g.:Scherrer Y., Erjavec T. (2013) Modernizing historical Slovene words
with character based SMT. In: BSNLP 2013 - 4th Biennial Workshop
on Balto-Slavic Natural Language Processing, Sofia.

The IMP project36

Use in digital humanities• Server logs from 2013 and first half of 2014 show

2,000–3,400 accesses per month or about 100 per day• On-line survey, 52 respondents:• 2/3: elementary and secondary school teachers, next under- andpostgraduate students• 2/3: resources used about once a month• 90%: resources are useful for linguistic research, for teaching
Slovene and important for the Slovene society• 95%: resources are worthy of recommendation• improvements should be focused on:
linguists (14%), pupils and students (both 13%), elementary andsecondary school teachers (both 12%) and lexicographers (9%)

The IMP project37

The horror of cycles• Changing annotation guidelines• Automatic / manual annotation:

ToTrTaLe → Corpus → CoBaLT → Lexicon →ToTrTaLe• Text corrections:Corpus → CoBaLT → a tangle of XSLT →Corpus

The IMP project

Further work
• HLT experiments with text normalisation• ToTrTaLe Version 2• Correcting mistakes• Extend DL & corpus > 1918 (Wikisource)• Offer more output formats: ePub, PDF• Import into the „big“ DLs and dictionary portals

38