Unit 6. Digitally Representing TextualData
2020: MABUCHI Yōko, National Institute for Japanese Language and Linguistics
2021: KONDŌ Asuko, National Institute for Japanese Language and Linguistics

Summary
Books are packed with linguistic information. One way of using this information is to digitise it
or digitally represent the information and store the information in a form where it can be read on
a computer.
In this unit, you will learn about three common aspects of digitising a text: 1) digitisation
methods and formats, 2) categories and content of textual data, and 3) practical examples ofdigital text.

1. Digitising Texts
Why digitise texts?
(a) Digitisation makes texts accessible
Analogue texts can only be accessed in a certain location (the place where they are held) and at
certain times (during the library’s working hours), making it hard to access valuable textual
resources.
➡Digital texts can be shared online, allowing many people to access them simultaneously, at any

time, and from anywhere in the world.
(b) Digital archiving protects culturally important texts
Analogue texts are hard to preserve in their original state. They deteriorate over time and aresusceptible to pest damage and natural disasters.

➡When you digitise a text, you get a snapshot of the text in its current state. This allows you topreserve the text in that state (or at least to keep degradation to a minimum).
(c) Digitisation allows software-assisted processing and analysis
To analyse analogue texts, you will need to decipher and digest the contents according to yourability and capacity, which requires time and effort.
➡With digital data, you can use softwares to analyse or edit voluminous data in an instant.

How do you digitise a text?
(a) Scanning documents
In this process, you scan documents with a digital camera and store the files. This process is
primarily used for documents that are culturally valuable or contain valuable information. Manynational/public libraries and academic institutes upload their digital scans to make data publicly
available.
Examples of digital archives and digital databases of national documents:
National Diet Library Digital Collections https://dl.ndl.go.jp
National Archives of Japan Digital Archives https://www.digital.archives.go.jp
Database of Pre-Modern Japanese Works https://dl.ndl.go.jp

Center for Open Data in the Humanities http://codh.rois.ac.jp/pmjt/

Archive of the National Institute for Japanese Language and Linguistics
https://dglb01.ninjal.ac.jp/ninjaldl/(b) Encoding the text
In this step, you encode the text—you convert the bibliographic and textual data in the scanned
image into editable text. This allows you to search for terms in the text and analyse its contents.
In some cases, you will encode the data as plain text, which represents only the character string.
In other cases, you will encode it as structured or tagged text. In addition to representing thecharacters, structured or tagged text represents structural information.
When texts use modern script and contain few decorative elements (i.e. when they just display
character string), you can simply use optical character recognition (OCR), a process that
automatically converts images of text into machine-encoded digital text. However, suchencoding often proves unfeasible in cases where the text contains many decorative elements anda complex layout (such as when images and character string appear on the same page) or when
the text uses an archaic writing system (such as kuzushi-ji or hentai-hiragana). In such cases, you
may need to transliterate or reprint the text first.

2. Digitising Linguistic Data
Linguistic data and linguistic hierarchies
Books contain different kinds of data. Some of the data are linguistic—information pertaining to
the language in the text, including script, vocabulary, and semantic aspects. Language has nested
hierarchies, with each tier playing a different role (see Figure 1). The linguistic data you want todigitally represent and the form you want to represent it in will depend on the tier of the language
in which you are interested.

Figure 1. Hierarchy of language

Different types of linguistic information
The different kinds of linguistic information used at each tier by researchers in the fields oflinguistics and natural language processing are given below:

Structure of whole document

This information pertains to the structure of the entire text inthe book. It includes the scope of the text, phraseology used inheadings, chapter structure, paragraphing, itemised lists,
sentences, and clauses.

GrammarThis information pertains to inter- and intra-sentence
relationships. It includes conjunctions, cases, semantic roles,
and dependency.

Semantic information

This information pertains to the meanings of words andphrases. It includes meanings derived from context, where thewords and phrases themselves can have multiple meanings.

Morpholexical information

This information pertains to the words themselves. It includes
the span of word characters (where a word begins and ends),
pronunciation, the lemma (the form cited in dictionaries as theheadword), parts of speech, and conjugations.

Orthographic information

This information pertains to the orthographic style used in the
text. It includes text direction (whether the text is writtenvertically or horizontally), writing systems (Chinese script,
mixture of Chinese script and hiragana, mixture of Chinese
script and katakana, use of katakana for nouns, Roman
letters), use of modifiers and emphasis, font, delimiters,
content outside the string of text (ruby characters, notes,
punctuation), annotative glossing, and interlinear notes.

Typographical information

This information pertains to the individual characters in the
text. It includes information about logographs, character
morphology, letters, symbols, enclosed alphanumerics, andkumimoji 組み文字 (multiple glyphs within a single character
space), superscript, and subscript.

Phonemic information

This information pertains to how specific words arepronounced.

Digitising linguistic information(a) Structured text
In linguistics and language processing, the above information must be digitised (stored asmachine-readable data), along with the character string. Accordingly, the text must be encoded
as structured rather than as plain text. This requires a markup process, in which you add
informational tags at certain places in the text. Text that has undergone this process is called
structured or tagged text.

A typical example of structured text is an HTML document. An HTML document uses
Hypertext Markup Language (HTML), which used in website design. HTML is designed torepresent document structure. However, as it uses a fixed tag set, HTML is sometimes
inadequate for displaying information to be used for linguistic research and language processing.
Accordingly, researchers often mark up text using Extensible Markup Language (XML), whichallows you to define the markup format for the information of interest.
One advantage of XML is that it allows you to extend the definition freely (see Figure 2).
However, in the interests of sharing the data, it is better to use a common schema to structure thetext (see Figure 3). To that end, the Text Encoding Initiative (TEI) issues guidelines, providing aplatform for sharing text data in the field of humanities. The latest (as of November 2021) issueof the guidelines is known as P5.

Figure 2. Left: XML document with independently defined formats.1 Right: Source manuscript.2
Figure 3. Left: XML document that conforms to TEI P5.3 Right: Source manuscript.4

Notes
1.
2.

3.

4.

National Institute for Japanese Language and Linguistics. Meiroku Zasshi kōpasu 明六雑誌コーパス Ver
1.1., 2013. https://pj.ninjal.ac.jp/corpus_center/cmj/meiroku/
National Institute for Japanese Language and Linguistics. “Nihongoshi kenkyū ryō” 日本語史研究資料.
Meiroku Zasshi 明六雑誌 1. (f1 r.).
https://dglb01.ninjal.ac.jp/ninjaldl/show.php?title=meirokuzassi&issue=001
Kawase Akihiro 河瀬彰宏, Ichimura Tarō 市村太郎, Ogisō Toshinobu 小木曽智信. “TEI: P5 ni
motozuku kinsei kōgo shiryō no kōzōka to sono mondai ten” TEI：P5 に基づく近世口語資料の構造化とその問題点. Jinmonkon 2013 ronbunshū じんもんこん 2013 論文集 (p. 9, Figure 4), 2013.
https://ipsj.ixsq.nii.ac.jp/ej/?action=repository_uri&item_id=96411&file_id=1&file_no=1
National Institute for Japanese Language and Linguistics. “Nihongoshi kenkyū ryō” 日本語史研究資料.
Keiseikai futasuji michi 傾城買二筋道 (f5 r.).
https://dglb01.ninjal.ac.jp/ninjaldl/hutasuzimiti/001/jpg/htsz001-021.jpg(b) Character code
In many cases, textual data can be represented in character code. Premodern Japanese texts oftenfeature hentai hiragana or hentaigana (archaic variants of modern hiragana). Version 10.0.0 of
Unicode, released in June 2017, added 286 hentai hiragana—these characters can now bedigitally edited (see Figure 4).
What is a Unicode? A Unicode is an international standard for encoding and handling text inwriting systems used worldwide. The standard treats grapheme clusters as single character units
and assigns each of these character units an ID.

Figure 5. Example of text encoding under Unicode

3. Digitised Documents
What is a corpus?
A corpus is a database of structured language used for linguistic analysis. While text corpora
contain a set of plain texts with just character strings, tagged corpora contain a set of structured(marked up / tagged) texts. Corpora can be released just as text data. Alternatively, they can belinked to relational databases and provided with a concordancer (software that searches corpora
and shows the results in KWIC5) or a profiler (software that analyses words in a corpus to reveal
inter-word connections, collocations, and other information).

Corpora for Japanese Historical Texts
One corpus for Japanese historical texts is the Corpus of Historical Japanese (CHJ),6 which the
National Institute for Japanese Language and Linguistics has been developing and releasing inphases since 2012. The CHJ contains historically valuable Japanese texts dated from the Nara to
Taishō periods (and texts from the early Shōwa period). This corpus contains textual data for all
the main text, as well as the morphological data distinguishing between short and long units. It
also contains bibliographic information (including author information) and links to scans of theoriginal manuscript (or reference materials providing an equivalent to an original manuscript).
The corpus can be accessed via the Chunagon search engine (which can be used for free, but you
must create an account7). Table 1 lists the subcorpora contained in the CHJ and the text volume
in each subcorpus.

Notes
5.
6.
7.

KWIC stands for ‘key word in context’. It is a concordance search format that displays concordance lines
with words aligned around the search term.
https://pj.ninjal.ac.jp/corpus_center/chj/
https://chunagon.ninjal.ac.jp/useraccount/registerTable 1. Corpus of Historical Japanese (ver. 2021.3): Subcorpora and text volume
PeriodSubcorpus

Short
units

Long
units

Nara Period Series I: Man’yōshū
Nara Period Series II: Senmyō
Nara Period Series III: Norito
Heian Period Series
Waka-shū Series
Kamakura Period Series I: Folktales and Essays

90 k
21 k
11 k
1m
275 k

―
912 k
250 k

844 k

792 k

Kamakura Period Series II: Diaries and Travel
Literature
Muromachi Period Series I: Kyōgen

128 k

118 k

277 k

256 k

Muromachi Period Series II: Christian Material(Kirishitan Shiryō)
Edo Period Series I: Share-bon

138 k

128 k

218 k

―

Edo Period Series II: Ninjō-bon

412 k

―

Edo Period Series III: Chikamatsu-joruri

255 k

―

Edo Period Series IV: Essays and Travel Literature

16 k

―

Meiji Era / Taishō Era Series I: Magazines

14 m

―

Meiji, 1868–1912

Meiji Era / Taishō Era Series II: Textbooks

856 k

―

Taishō, 1912–26

Meiji Era / Taishō Era Series III: Early Meiji
Spoken Materials
Meiji Era / Taishō Era Series IV: Modern Novels

211 k

―

779 k

―

Nara, 710–94
Heian, 794–1185
Heian, Kamakura
Kamakura, 1185–1333

Muromachi, 1336–1573

Edo, 1603–1867

94 k
17 k

Morphological Information and Lexical Units
When assigning morphological information in a corpus, one faces the problem of delineating
morphological units. While alphabetic writing systems clearly show the boundaries between
separate lexemes, this is not always clear in Japanese text. Accordingly, Japanese corpora requiredifferent rules for determining where a word begins and ends. The academic applications of the
corpus will vary depending on how long you define a lexeme to be. Accordingly, corpora
released by the National Institute for Japanese Language and Linguistics assign morphologicalinformation in two levels: short unit and long unit (see Figure 6).

Short units: A short unit is equivalent to a lemma (the form of a word cited in dictionaries as theheadword). Short units can be defined consistently under coherent rules. They are useful as
search terms or search results. A single unit comprises one lexeme or a pair of lexemes (no more than two).
Generally, a short unit will be a native Japanese word (wago) or Chinese word (kango) or a pair of wago or
kango. In the case of foreign words, a unit is what is considered a word in the language from which the word
originated.

Long units: These are generally clausal units divided into independent words (words with
lexical meaning) and auxiliary words (words with a grammatical function). The corpus does notconsider compound words, compound phrases (compounds serving as postpositional particles or
auxiliary verbs), or derivative words as constitutive elements. Long units are useful as key terms
that typify the genre or subject area in question.

Figure 6. Examples of long and short units
For more on short units, see NINJAL’s explanatory video (only available in Japanese):https://youtu.be/s5akpYOzNIU
The video is based on the following resource: https://www2.ninjal.ac.jp/openhouse2020/pdf/c19.pdf

Text Databases
Even without building a corpus, text databases can prove useful for linguistic and language
processing. Many text databases allow you to search or edit the content using a text editor orreconfigure the text data and embed a generic concordancer.8

Examples of general text data provided as a text database for historical Japanese:‘NINJAL text dataset for historical Japanese’ https://textdb01.ninjal.ac.jp/dataset/‘NIJL database of ancient Japanese literature’ http://base1.nijl.ac.jp/~nkbthdb/
‘Database of selected Japanese classics’ http://base1.nijl.ac.jp/~selectionfulltext/‘Akihiro Okajima’s text files for Japanese literature’
http://www.ne.jp/asahi/nihongo/okajima/bungaku.htm

‘Japanese classic texts’ http://www2s.biglobe.ne.jp/~Taiju/classic.htm

Originally written by Yōko MABUCHI, 4 November 2020
Updated by Asuko KONDŌ, 30 November 2021

Notes
8.

Examples include KH Coder (https://khcoder.net) and Himawari(https://www2.ninjal.ac.jp/lrc/index.php?%C1%B4%CA%B8%B8%A1%BA%F7%A5%B7%A5%B9%A
5%C6%A5% E0%A1%D8%A4%D2%A4%DE%A4%EF%A4%EA%A1%D9).