Introduction of Digital History
Research at CODH
Asanobu KITAMOTO
Director, ROIS-DS Center for Open Data in theHumanities (CODH) andProfessor, National Institute of Informatics
http://codh.rois.ac.jp/ @rois_codh
2024/3/4

21th CODH Seminar

1

ROIS-DS Center
for Open Data in
the Humanities(CODH)

DeepenIncrease

ScholarMachinehttp://codh.rois.ac.jp/
1. Data-driven Humanities:Innovation in humanities researchusing computer science technologies
and tools.
2. Humanities Big Data: Innovationin non-humanities research usinghumanities data.
2024/3/4

21th CODH Seminar

CitizenOpen science isthe key to fill thegap between
disciplines andsolve big problems

Expand2

Digital History Research at CODH
Our main interests are in the technical side, and the goal is tocreate systems for historians and the public. In addition toJapanese history, we also work on eastern and western history.

1. Data structuring and historical big data
2. Geographic information and entity database
3. Knowledge representation and linked data
4. Criticism and public history for non-textual sources
2024/3/4

21th CODH Seminar3

Historical Big Data (HBD)
http://codh.rois.ac.jp/historical-big-data/WeatherNaturedata

Earthquake

Culturedata

Population

Historicalsources

Platform fortheEruption
integrated
Data
Diseasestructuring analysis of
Economy workflow
HBD.
Politics
Culture2024/3/4

21th CODH Seminar

4

Data structuring Workflow
Handwritten characters(printed, hand-written)

Analysis-ready data(curated data)

Digitizing
Bidirectional
linkageTranscribing
Plain text (unstructureddata)

Linking
Entity linking

Annotating
Markup text (semistructured data)

Linked data (structureddata)

Tidy tabular data(structured data)
Tidying

Entity search
21th CODH Seminar

Machine-readable tabular
data (structured data)

Entity space

Document space
2024/3/4

Digital image(unstructured data)

Quantifying

5

Massive Documents vs. Few Readers
http://codh.rois.ac.jp/pmjt/

2024/3/4

1 billion
documents

10000
readers

Estimated number ofold books anddocuments in Japan

Estimated number ofpeople with fluency
in reading Kuzushiji

21th CODH Seminar

6

KuroNet: Kuzushiji
recognition using deeplearning.
About one second for
one image.
Accuracy is 95% forbooks with the best
condition.
http://codh.rois.ac.jp/kuronet/

2024/3/4

21th CODH Seminar

7

Miwo: App for AI Kuzushiji Recognition, http://codh.rois.ac.jp/miwo/

Show a recognition result incharacters
2024/3/4

Show a recognition resultwith bounding boxes

Modify the error withreference to root characters.

21th CODH Seminar

Generate the text output
from the recognition result
8Tsukushi Project for Historical Japanese LLM
http://codh.rois.ac.jp/tsukushi/

1. Create text from premodern books using AI
Kuzushiji OCR.
2. Release the dataset of
historical Japanese text
for training LLM.
3. Use LLM for digital history,
such as translation andsummarization.
2024/3/4

21th CODH Seminar

9

Diachronic Transcription for Time-Series Data
http://codh.rois.ac.jp/bukan/

Question: how can
we transcribe
books over 200+years?
Solution: detectand transcribe thedifference to create
diachronic data.
2024/3/4

21th CODH Seminar

10

Text-based and Image-Based Collation
http://codh.rois.ac.jp/differential-reading/

Text-based Collation = Many tools are available

Text A

Text B

Transcription
OCRTranscription
OCR

Image A

Image B

Image-based Collation = No standard tools
Mainstream is “side-by-side comparison” by visual inspection
2024/3/4

21th CODH Seminar

11

Page Collation

http://codh.rois.ac.jp/bukan/

1. Read images from twobooks for comparison
using vdiff.js
2. You can move forward
or backward within abook.
3. You can move to thenext or previous bookby keeping the same
woodblock2024/3/4

21th CODH Seminar

12

Edo Maps Beta

http://codh.rois.ac.jp/edo-maps/

2024/3/4

21th CODH Seminar

From 29 sheets,
8719 place
names wereextracted.
13

Dataset of Historical Cities and Towns
https://geoshape.ex.nii.ac.jp/city/

2024/3/4

21th CODH Seminar

14Nihon Rekishi Chimei Taikei Placenames
https://geoshape.ex.nii.ac.jp/nrct/

We collaborated witha publisher to releasedata of about 80,000
place names as open
data.
2024/3/4

21th CODH Seminar

15

GeoLOD

https://geolod.ex.nii.ac.jp/

1. GeoLOD ID is an
identifier designed forplacenames.
2. Each identifier has
metadata to describe
relevant information.
3. Multiple services use
the same identifiers
for data integration.
2024/3/4

Places areconverted tothe gazetteer
format for
GeoLOD.
Name: Isobe Shrine
GeoLOD ID: G8AYsq
Lat: 35.676326
Lng: 139.774755
https://geolod.ex.nii.ac.jp/resource/G8AYsq
21th CODH Seminar

16

Digital Silk Road Project
http://dsr.nii.ac.jp/geography/Stein, Marc Aurel. “Innermost Asia Detailed Report of Explorations in Central
Asia, Kan-Su and Eastern Īrān.” NII “Digital Silk Road” / Toyo Bunko.
doi:10.20676/00000187.
2024/3/4

21th CODH Seminar

A map wasconsidered
trustworthy,
but that is thereason for
missing ruins
problem.
17Missing Ruins

MissingRuins
ProblemYārkhoto②③④
BezeklikKara-khōja⑤⑥⑦Toyuk①⑧Sirkip

LukchunChong-hassār,
Kichik-hassār① Oi-tam, ruined fort
② Bögan-tura③ Buluyuk ( Shipang, Sassik-bulak, Kazma )④ Murtuk-ruins
2024/3/4

21th CODH Seminar⑤ Yoghan-tura⑥ Chikkan-köl⑦ Bedaulat’s town, Bēsh-kāwuk, Kosh-gumbaz⑧ Yutōgh
18

Error Distribution of Old Maps

Error Distribution in Turfan Basin / White: Innermost Asia / Black: Serindia

Some ruins were reported by 20th expeditions, but are missing
in recent survey reports.
2024/3/4

21th CODH Seminar

19

Linking Entities Across Sources
Textual criticism

Textual source A
Place name S?

Textual criticism

Textual source B
Place name T

Digitally-enabled criticism

Place name UGeographic
And VisualRelationship

Place name V

Data Source C, D, E…
2024/3/4

21th CODH Seminar

20

North China Railway Archive
http://codh.rois.ac.jp/north-china-railway/

Collaborator: Institute for Research in Humanities,
Kyoto University, Center for Southeast AsianStudies, Kyoto University, and Toyo University.
2024/3/4

1. 35,000+ stock photo taken
between 1937 and 1945 by
North China Railway, aJapanese transportationcompany.
2. Collected for propaganda, andpublished in magazines.
3. Photographs illustrate not only
the railway construction, butalso people’s daily lives,cultural heritage, and so on.

21th CODH Seminar

21

Same-Composition Photography
https://mp.ex.nii.ac.jp/

National Diet Library
2024/3/4

21th CODH Seminar

22

Memorygraph AppGoogle PlayApp Store
2024/3/4

21th CODH Seminar

23

App for Public History
Community
Archive: Fill the gap
between thepresent and thepast using visualsources to realize
the value of thehistorical landscape.
Kyoto Sanjyo Event by Akira Takahashi, Osaka University
2024/3/4

21th CODH Seminar24

Visit our Website
http://codh.rois.ac.jp/2024/3/4

21th CODH Seminar25