Unit 7. Digitising Texts
2020: MABUCHI Yōko, National Institute for Japanese Language and Linguistics
2021: KONDŌ Asuko, National Institute for Japanese Language and Linguistics

SummaryThis unit provides an exercise for digitising textual resources. Through these exercises, you willlearn how textual resources are digitised and used. Using a scanned text, you will practise
encoding the text and marking up its linguistic information by using XML.

1. Encoding Scanned Texts
What you need to encode a text
Before attempting to encode a text, you must first decide on your specifications. Thesespecifications will depend on your purpose for encoding the text (for what purpose will the text
be used and how will it be used?) The appropriate specifications will vary markedly, dependingon whether you want the digital data to reproduce the look of the physical manuscript asfaithfully as possible or if you instead want to make the textual content of the manuscript as
accessible as possible to researchers.
Outlined below are the main matters you should clarify in case you want to make the data
accessible to researchers. You should decide on these matters before determining yourspecifications for encoding the text:
What kind of data do you want? Do you want plain text (only the text itself ispresented), or do you want structured text (tagged / marked up text)?
How will you encode the text? The encoding format you need will depend on thecharacter set, by which you save the text and on software (operating system, textanalysis tools) by which the text will analysed.

What about the character set? What character set will you use? How will you define
graphemes and glyphs? How do you set the boundaries between characters? Does thedocument use standard Chinese characters, or does it contain many archaic forms of thesecharacters? Does it contain nonstandard variants? Does it include characters from morethan one language?
How will you deal with characters outside your character set? If your document
contains characters outside the character set (gaiji 外字), you must work out how
you will represent them.

Exercise for encoding a text
One approach for encoding text is to transcribe the document digitally (to manually type out thetext). Another approach is to scan the document and then use optical character recognition(OCR) to convert images of text into machine-encoded characters.
It was once unfeasible to encode documents featuring archaic orthography (such as kuzushi
characters or hentai hiragana). However, following the technological advances in kuzushi
character recognition, we now have OCR services9 compatible with such characters.
This unit outlines two approaches for encoding premodern texts as open data: 1) manuallytransliterating the text and 2) using OCR software that recognises kuzushi characters.

Premodern sources used in
the exercise

Source: Shunshoku Ume Goyomi 春色梅児誉美 (Archive ofthe National Institute for Japanese Language andLinguistics)
https://dglb01.ninjal.ac.jp/ninjaldl/bunken.php?title=umegoyomi
Section of source: Volume 1, folium 1 recto to folium 2 verso.

Specifications

Data format: Plain text (no ruby; line breaks are as in source
manuscript).
Character encoding: UTF-8
Character set: JIS X0213:2004
Processing of characters outside set: Use inclusive criteria ofJIS X0213:2004. For non-JIS Chinese characters, encode within
Unicode range. For non-Unicode characters, use the nearest

equivalent JIS character instead. If no such substitute isavailable, input ‘〓’ (tag: U+3013).
Other specs: Where text is lost due to pest damage or where thetext is illegible, input the tag ‘␣’ (open box: U+2423).

Archive of the National Institute for Japanese Language and Linguistics
https://dglb01.ninjal.ac.jp/ninjaldl/
For each work, this archive includes bibliographic information and, scanned images (JPEGs andPDFs), a transcript, and an IIIF manifest for each volume. The transcripts (ground truth) allow
students and researchers to access the sources easily. The source of interest in this unit isShunshoku Ume Goyomi.
The latter part of this source is contained on a NINJAL database for hentaigana glyphs (cited anddescribed below). This database displays scanned images overlaid with a transcript, apresentation format ideal for users who want to study the text. The source is also included in a
CODH database of kuzushi texts (also cited and described below). This database is ideal for
testing OCR-based encoding. AI-driven OCR services for kuzushi script have the potential torecognise kuzushi characters with reasonable accuracy.

Online resources on transcription and kuzushi characters
CODH database of kuzushi texts (Nihon kotenbaku kuzushiji dētasetto 日本古典籍くずし字データセット)
URL: http://codh.rois.ac.jp/char-shape/
Provided by: Center for Open Data in the Humanities
This database is provided on the Center for Open Data in the Humanities’ (CODH) dataset ofJapanese classics (Nihon kotenbaku dētasetto 日本古典籍データセット) and comprises
digitally encoded images of classic texts. The dataset is intended for machine-learning, but it also
helps humans to study kuzushi script. For the latter purpose, the database provides a list of text
images by transcript code (meaning that each text image is paired with the corresponding part ofthe manuscript), and a search engine. To enable the dataset to be used as learning data, CODH
has also provided an AI-driven OCR service for recognising kuzushi characters.

Notes9.

One example is an AI-driven OCR service provided by the Center for Open Data in the Humanities(http://codh.rois.ac.jp/kuzushiji-ocr).

NINJAL database of hentaigana glyphs (Kokugoken hentaigana jikei dētabēsu 国語研変体仮名字形データベース)
URL: https://cid.ninjal.ac.jp/hentaiganaDB /Provided by: National Institute for Japanese Language and Linguistics
This database includes images of hentaigana glyphs, a writing system used in printed books ofthe Edo period. Each image has a nested hierarchy: reading, letter, and Unicode. Users can viewthe hentaigana at these levels. Users can also use a function that links the scanned images ofcharacters with scanned images of the corresponding part of the manuscript. There is also anoption to view the text images overlaid with the transcript.

NINJAL hentaigana database for exchanging academic information (Gakujutsu jōhōhentaigana 学術情報交換用変体仮名)
URL: https://cid.ninjal.ac.jp/kana/home
Provided by: National Institute for Japanese Language and Linguistics
This database includes images and textual data pertaining to hentai hiragana texts, which wereselected with a view to establishing international standards for encoding hentaigana. Users canview text and search by currently used hiragana, letter, and Unicode.
Video resource: https://youtu.be/sn_xQqZ4C78
Explanatory PDF: https://www2.ninjal.ac.jp/openhouse2020/pdf/c-08.pdf

Search engine for linking wooden tablet database (mokkanko 木簡庫) with digital kuzushi
lexicon database (denshi kuzushiji jiten db 電子くずし字字典 DB)
URL: https://mokkanko.nabunken.go.jp/renkei/kensaku.php
Provided by: Nara National Research Institute for Cultural Properties, and the HistoriographicalInstitute of the University of Tokyo
This is a search engine for the two databases mentioned above. The first database contains
images of glyphs written on wooden tablets, held by the Nara National Research Institute forCultural Properties. The other database contains glyphs in ancient documents, records, andclassics held by the Historiographical Institute of the University of Tokyo. In the search engine,
you can input single characters, and the engine will retrieve the relevant images from bothdatabases showing the locations of these characters in the texts. The search engine links towebpages that give information about each database.

Mojizo
URL: https://mojizo.nabunken.go.jp
Provided by: Nara National Research Institute for Cultural Properties and the HistoriographicalInstitute of the University of Tokyo
Mojizo is an image-matching service for the two databases above. You upload an image of asingle character, and Mojizo displays matching characters from the databases.

2. Marking up Digital Text
Prerequisites to marking up text
As with encoding, before attempting to mark up a text, you must first decide on yourspecifications. It is a good idea to follow standard specifications (such as those outlined in TEIguidelines), as this will help ensure that the database is accessible.

The database should be useful to linguistics researchers; therefore, you will mark up the text
using Extensible Markup Language (XML). You will use XML to add original informationaltags at specific places to reproduce (and thus enable users to analyse) the original properties ofthe manuscript, including its physical properties (such as pagination, lines, annotative glossing),
orthography (such as the use of unbroken cursive text and nonstandard hiragana), and textual
properties (such as word frequencies and collocations).

XML syntax rules
XML has certain rules, and if you do not use the correct syntax, the document will not berecognised as XML, and neither will it be machine-readable. Outlined below are the formatting
rules.
XML tags:
To create an XML document, you mark up the text by adding information tags at specific placesin the text. The portion of text to be covered by a given tag begins with a start-tag and ends withan end-tag. A start-tag looks like this: <***> (*** = character string). An end-tag looks like this:</***>. The text within the boundary of a start-tag and end-tag is called an element. Further, thecharacter string (indicated by ***) is called the element name. You can decide on what to namethe element. You can use Roman letters or Japanese characters. For example, you could make thestart-tag <tag> or <タグ>, or you could make it <element> or <要素>. You may want to create atag that has no content (if you just want to add a tag without marking range). In this case, youcan make a self-closing tag by adding a backslash to the end of the opening tag, like this:<***/>. This is called an empty-element tag.
To add textual information to an element, leave a half-width (hankaku) space after the element
name and then input the following (the brackets are Japanese kagikakko 鉤括弧 brackets):「***=” (add information here)”. This added information is called an attribute. Elements cancontain multiple attributes (but they cannot contain multiple identical attributes).

<****>●●●●●</****><****/>
Opening tag

Closing tag

Empty tag***…「Element name」●●●●●…「Textual information」「value」┗━━━━Element━━━━┛┗Element┛<s no=”10”>XML What is a text 


no…「Attribute name」┗Attribute┛

10…「Attribute value」

A tag that starts an element must have the same name as the tag that ends it (note that tags arealso case sensitive). You can make nested elements (by placing a pair of opening and closing
tags inside a pair of opening and closing tags), but ensure that all the tags and element names
correspond correctly.

Incorrect markup (element names notmatching)
Correct markup
Incorrect markup (start-tag does notcorrespond with end-tag)

XML documents:
An XML document comprises three portions: 1) an XML declaration, 2) an XML document-typedefinition (optional), and 3) the data. The first line of the XML text should include the XML
declaration. The data portion should be enclosed in a root element, the element that encloses allother elements.

Document-type definition

Markup procedure
To ensure a smooth markup process, you should use an XML editor that can process the target
document quickly and accurately. For the target data file that you encoded with the proceduredescribed above, you will use an XML editor that allows you to extend the specifications so thatyou can add textual information in our early-modern Japanese source in compliance with TEI P5.
Target document:
The text data that you generated through the procedure described in the section ‘exercise forencoding text’.
Markup syntax
The following syntax conforms to TEI P5. The tags are as follows (attributes are inputted afterthe @ symbol):
TEI @xmlns="http://www.tei-c.org/ns/1.0"
teiHeadertext
front/backbodytitle
docAuthorpssp
speakerpb @n=”state page no. here”
pb @facs=”state page no. here”
lb @n=”state line number here”
rubyrbrt

The root element
Textual or bibliographic information (not used in this exercise)
The whole encoded text
Front and back matter (the target document has no such content)
The main text (which excludes any front or back matter)
The title of the document
The author of the document
A paragraphA sentence
Individual speech in a performance text or a passage presented assuch in a prose or verse text
The portion indicating the speaker of the <sp> element
Start of a new page
Start of a new page in facsimile image of page
Start of a new line
The span of text being glossed by <rb>
The ruby text
Pointer pointing to the base being glossed by <rt> (indicate direction
to base as follows: <rt @place=”above/below/left/right>)

Extendible specs (none of these are used in this exercise):
seg @type=”wari”
milestone @unit=”wlb”
milestone @unit=”rb”
char @xml:id=”UUnicode number
mapping @type=”standard

Segment for glossing
Line break in <seg> element
Break in cursive text
Information about nonstandard character/glyph
Equivalent standard character/glyph

Online resources on TEI markup
TEI P5https://www.tei-c.org/release/doc/tei-p5-doc/en/html/index.html
TEI-C workshop on Japanese and East Asian classics
https://github.com/TEI-EAJ/jpn_classical
TEI/XML-compliant Taishō shinshū daizōkyō (SAT)
https://21dzk.l.u-tokyo.ac.jp/SAT/sat_tei.html

Originally written by Yōko MABUCHI, 4 November 2020
Updated by Asuko KONDŌ, 30 November 2021