Construction of a Metadata Database for
Efficient Development and Use of Language Resources
Hitomi Tohyama†, Shunsuke Kozawa†, Kiyotaka Uchimoto††,Shigeki Matsubara† and Hitoshi Isahara†††

Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
National Institute of Information and Communications Technology
3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{hitomi, kozawa,matubara}@el.itc.nagoya-u.ac.jp, {uchimoto, isahara}@nict.go.jp††

Abstract
The National Institute of Information and Communications Technology (NICT) and Nagoya University have been jointly constructing
a large scale database named SHACHI by collecting detailed meta information on language resources (LRs) in Asia and Westerncountries, for the purpose of effectively combining LRs. The purpose of this project is to investigate languages, tag sets, and formatscompiled in LRs throughout the world, to systematically store LR metadata, to create a search function for this information, and to
ultimately utilize all this for a more efficient development of LRs. This metadata database contains more than 2,000 compiled LRs suchas corpora, dictionaries, thesauruses and lexicons, forming a large scale metadata of LRs archive. Its metadata, an extended version of
OLAC metadata set conforming to Dublin Core, which contain detailed meta information, have been collected semi-automatically.
This paper explains the design and the structure of the metadata database, as well as the realization of the catalogue search tool.
Additionally, the website of this database is now open to the public and accessible to all Internet users.

1.

Introduction

The construction of LRs such as corpora, dictionaries,
thesauruses, etc., has boomed for years throughout theworld in its aim of encouraging research and developmentin the main media of spoken and written languages, andits importance has also been widely recognized. Of theorganizations willing to store and distribute LRs, there
exist some consortia fulfilling their function such as LDC1,
ELRA2, CLARIN,3 and OLAC4 (Hughes et al, 2005), in
Western countries, and GSK 5 which does so mainly inJapan. However, those released LRs are scarcely
connected with each other because of the difference
between written and spoken language as well as thedifference between languages such as Japanese, English,
and Chinese. This situation makes it difficult for
researchers and users to find LRs which are useful for
their researches. In the meantime, by connecting
systematically existing various LRs with Wrapper
Program, the attempt to realize multilingual translation
services has already begun (Ishida et al, 2008, Hayashi etal, 2008). Moreover, since language information tags
given to those LRs and their data formats are multifarious,
each LR is operated individually. As LR development
generally entails enormous cost, it is highly desirable thatthe research efficiency be enhanced by systematicallycombining those existing LRs altogether and extending
1

LDC :Linguistic Data Consortium, http://www.ldc.upenn.edu/.
ELRA: European LRs Association, http://www.elra.info/.
CLARIN: Common Language Resources and Technologies
Infrastructure, http://www.ilsp.gr/clarin_eng.html.
4
OLAC: Open Language Archives Community,
http://www.language-archives.org/.
5
GSK: Gengo Shigen Kyokai; Language Resource Association,
http://www.gsk.or.jp/ . (in Japanese)2

them, which will encourage an efficient development of
unprecedented LRs.
The National Institute of Information andCommunications Technology (NICT) and Nagoya
University, for the purpose of developing LRs efficiently,
have been constructing a large scale metadata databasenamed SHACHI 6 as their joint project by collecting
detailed meta information on LRs in Western and Asian
countries (Tohyama et al, 2008). This research project
aims to extensively collect metadata such as tag sets,formats, and recorded contents of LRs existing at home
and abroad and store them systematically. Meanwhile, wehave already developed a search system of LRs by the useof meta information and are attempting the experiment ofwidely providing meta information on our stored LRs to
those from researchers to common users. This metadata
database has been now open to the public in the Web andallows every Internet user to access it for the search andread information of LRs at will. Moreover, by exchanging
opinions on them with its users, efforts go into registering
new LR information and appending search functions to it
as required.
In this paper, Section 2 describes the purpose of themetadata database. Then, Section 3 explains thecollection of metadata and the extended meta elements
which characterize this database as one of its distinctive
features. Section 4 discusses three search functions ofSHACHI and their characteristics. Lastly, Section 5 gives
an outline of statistic information about LRs registered in
SHACHI.

3

6

1687

SHACHI: Metadata Database of Language Resources
SHACHI, http://shachi.org/shachi/en/index.php, (Shachimeans “orca” in English.

2.

Purpose of the construction of themetadata database

The purpose of the construction of the database isfourfold.
(1) To store language resource metadata:
It is convenient that all information about
language resources in the world be put together in
one place to be combined with each other bymaking an organic relation and to be strategicallydeveloped. Therefore, SHACHI semi-manually
collects detailed metadata of language resources
and constructs their detailed catalogues. Figure 1
shows a sample page of a LR catalogue stored in
SHACHI (ex. Euro WordNet). This catalogue
provides more detailed meta information than otherLR consortia do.
(2) To systematize language resource metadata:
According to detailed metadata obtained bystep (1), language resource ontology is tentatively
constructed by classifying types of language
resources (in this paper, it is called “ontology”). At
the moment, it is under investigation what is themost useful and functional ontology for users bydeveloping some ontologies such as human-madeontology, semi-automatically produced ontology,
and automatically produced ontology. It isconsidered that storing systematized LRs in theworld serves for the strategic development anddistribution of LRs as well as the expansion of
language processing technologies. Moreover, thisontology is experimentally applied to one of
SHACHI search functions. (See Sec.3.3 & Sec.4)(3) To make each language resource related to eachother:
Catalogues contained in SHACHI are morecharacteristic than those of other consortia in termsof its collection of detailed metadata. Those
detailed metadata enabled us to describe
characteristics of each language resource and to
expectably specify relationships among language
resources. Figure 3 shows a part of the SHACHI
search screen. It shows language resources found asa search result, the references to which theselanguage resources conform as well as otherlanguage resources whose formats are common totheirs. By statistically investigating therelationships of these language resources, it ispossible to design tagsets and data formats
according to world level standards as well as typesof language resources in demand.
(4) To statistically investigate language resources:
On the SHACHI site, users are able to perusestatistical information about language resource
metadata which are contained in SHACHI. By
statistically analyzing those metadata, users areable to grasp what kinds of language resources exist
in different part of the world and to understandcurrent tendencies of language resources whichhave been available to the public. (See Sec.5)(5) To promote the distribution of language
resources:
Since this metadata database enables users toeasily gain access to language resources inaccordance with their needs, owing to fully
equipped search functions, SHACHI will be able tosupport an effective use and an efficientdevelopment of language resources. (See Sec.4)
Some 2,000 resources of metadata have already beencollected in the database so far and they will be enlarged
by a further 3,000 by December 2008. To that end, it isindispensable for us to work in cooperation with language
resource consortia at home and abroad and to take theinitiative in contributing to Asian language resources.
Consortium
ULR
Asian Language Resource Catalogue
http://nlp.kuee.kyoto-u.ac.jp/
ChineseLinguistic Data Consortium
http://www.chineseldc.org/
Corpus-linguistics
http://www.corpus-linguistics.de/European Language Resources Association http://www.elra.info/fr/
Global WordNet Association
http://www.globalwordnet.org/
http://www.gsk.or.jp/
GSK (Gengo Shigen Kyokai )ICAME Corpus Collection
http://icame.uib.no/newcd.htm
Linguistic Data Consortium
http://www.ldc.upenn.edu/Catalog/
Natural Language Processing Portal Site
http://nlp.kuee.kyoto-u.ac.jp/NLP_Portal/lr-cat-j.html
Speech Information Technology
http://www.sitec.or.kr/English/& Industry Promotion Center
Speech Resources Consortium
http://research.nii.ac.jp/src/links/

Table 1. List of major language resource consortia which
SHACHI covers
3.

SHACHI metadata

3.1 Policy for collecting metadata
The LRs which our metadata database stores should
satisfy the following conditions:
1. Those resources should be stored in a digitalformat.
2. Those resources should be one of the following:
corpus, dictionary, thesaurus, or lexicon.
(Numeric data are not considered to be thesubject of collection for SHACHI.)
3. Those resources should be collected fromEnglish websites and its data must be open to the
public.
4. Those resources should be created by research
institutions,researchers,orbusinessorganizations.
It is essential to obtain and store information on
highly recognized and frequently used language resources
through collecting language resource metadata. Therefore,
during the construction of SHACHI, a web search through
Google was first conducted to investigate the actual stateof language resources that meet the above mentioned
conditions. LRs metadata database SHACHI contains
extensive data obtained from highly recognized LRs thatwere searched for on the Internet, and also covers meta
information provided by LR consortia such as ELRA,
LDC, and OLAC whose more detailed metadata are fed
into the database by semi-automatic means of importing.
Table 1 shows a list of major LR consortia whose detailed
LR information has already been registered in SHACHI.

1688

SHACHI Extensions

Qualifier
DC Qualifiers
Qualifiers used for more precise description of the resources
LEVEL 1LEVEL 2
DCMES ElemeDC Element Refinements
OLAC Extensions
SHACHI Extensions
1 TitleAlternative
2 Creator
3 Subject
Linguistic Subject (29)mono_multi_lingual (2)[olac:linguistic-field]
monolingual
multilingual
anthropological_linguistics
applied_linguistics
ResourceSubject (4)cognitive_science
corpuscomputational_linguistics
dictionary
discourse_analysis
thesaurus
forensic_linguistics
glossary
general_linguistics
historical_linguistics
history_of_linguistics
language_acquisition
language_documentation
lexicography
linguistics_and_literature
linguistic_theories
mathematical_linguistics
morphology
neurolinguistics
philosophy_of_language
phonetics
phonology
pragmatics
psycholinguistics
semantics
sociolinguistics
syntaxtext_and_corpus_linguistics
translating_and_interpreting
typology
writing_systems
OLAC-Language extension[olac:language]
4 Description Table Of Contents
Abstract
Landuage (of description)Price
5 Publisher Role [olac:role] (24)
6 Contributor annotator*authorcompiler
consultant
data_inputter
depositor
developer
editorillustrator
interpreterinterviewer
participant
performer
photographer
recorderresearcher
research_participant
responder
signerAttribute of *Speaker/Author(
singermother-tongue*speakerintonation
sponsorlevel
transcriberagetranslator
gender
7 DateCreatedValid
Available
IssuedModified
Date Accepted Date Copyrighted
Date Submitted
8 Type(DC Type Vocabruary)
Purpose(4)Discourse Type (10)lexicography[olac:discourse-type]analysisdramadeveloping_technologies
formulaic_discourse
educationinteractive_discourse
Style (2)language_play
speechoratory
writtennarrative
Form (2)procedural_discourse
fixedreportunfixedsinging
Sentence(3)unintelligible_speech
short
Linguistic Data Type (3)long[olac:linguistic-type]
mixed
lexiconAnnotation (3)primary_text
annotatedlanguage_description
plain
Annotaion_sample
Sample9 Format
ExtentMediumEncoding
MarkupMono_Multi_lingual (2) Applies to: subject
A resource using only one language:
The same language is use for themonolingual
subject language and to describe thesubject language.
A resource in several different
languages. Different language(s) are
multilingual
used for more than one subject
languages and to describe the subject
language(s).
ResourceSubject (3)dictionary

thesaurus

A list of the words of a language inwhich the words are arranged in groups
that have similar meanings.

lexiconA list of words on a particular subject.

Attribute (5)Attributed of a contributor

mother-tongue

The performer is whether a native or
non-native speaker of the language.

intonation

Dialectal status whether the performer
uses a standard language or a dialect.

level

Whether the performer has received aprofessional level of linguistic(speaking
or writing) training or no such training.

age

An age group the performer is in. When
there are many performers in aresource, the ratio between all the agegroups to which the performers belong.

genderSex of the performer. When there aremany performers in the resource, theratio of males and females.

Purpose(4)lexicography
analysis

Applies to: type
The creation of the resource is intended
for lexicography.
Thecreation of the resource is intended
for analysis.

developing_technolog

Thecreation of the resource is intended
for developing technologies.

education

The creation of the resource is intended
for the use in education.

Style (2)speechThe resource is of the spoken
language.

written

The resource is of the written language.

Form (2)fixedunfixedA collection of fixed forms ofexpressions.
The resource collects various forms ofexpressions.

Sentence(3)shortA collection of short sentenses.

long

A collection of long sentenses.
A collection of varying length ofsentences.

Functionality
10 Identifier

13 Relation

Applies to: subject
A list of the words of a language inwhich the definitions or meanings of
the words are explained either in thesame language or in a different
language.

mixed
Bibliographic Citation
11 Source
12 Language -Description

OLAC-Language extension[olac:language]

Is Version Of
Has Version
Is Replaced By
Replaces
Is Required By
Requires
Is Part Of
Has Part
Is Referenced By
References
Is Format Of
Has Format
Conforms To

Annotation (3)annotated
plain
annotaion_sample

Tagged corpus.
A corpus without annotations.
A sample of tagged data

SampleA sample of the language resource

Format (2)encoding
markup
Functionality

Software Functionality

Utilization

Applicability of the resource. The
described resource is utilized for thereferenced technology, education,
research or a product.

Utilization
14 Coverage Spatial
Temporal
15 Rights
Access Rights
LicenseAn encoded character set used by adigital resource.
A markup scheme used by a digital
resource.

Table 2. SHACHI metadata set
Table 3. Description of SHACHI extensions

1689

Requests for the participation in our project toresearch laboratories and institutes developing languages
resources by registering themselves in our database arescheduled to be made, as soon as the preparations for
publishing RSS feeds of our data and our input format of
metadata standard are completed. At the moment our web
page for registration is not open to the public.

3.2 Extensions of metadata element
Since users sometimes search for LRs without a clear
objective, it is necessary for language resource providers
to construct language resource ontology which enables
users to find what they want by indicating the affinity oflanguage resource attributes as well as their relationshipbetween their classified positions in order to provide themwith sufficient information. This database conforms to the
OLAC metadata set which is based on 15 kinds offundamental elements of Dublin Core and constitutes anextended vision of OLAC with 19 newly added metadata
elements which were judged to be indispensable fordescribing characteristics of LRs. Table 2 show the
SHACHI metadata set. “SHACHI Extension” in the third
row is meta elements which SHACHI recently adopted
and SHACHI’s originally adopted extended elements(See Table 3) are shown in the right edge of the table 2.
SHACHI provides usage information about how and inwhich situation language resource researchers utilizedeach language resource, which is also important for users.
In this SHACHI project, by using parsing techniques themethod, through which usage information about LRs is
automatically retrieved from academic article databases,
is conceived and presented (Kozawa et al, 2008). (See
“Utilization” in Figure1).

3.3 Systematization of LRs
The above-mentioned database aims not only to collectlanguage resource metadata but also to store themsystematically. Clear description of the relations among
LRs can be applied to the efficient development of LRs
and search tools for common users of database. We firstsurveyed the frequency of possible values of metadata
element choices, obtained the standard deviation of its
frequencyandgeneratedtheontologybyhierarchicalizing meta elements of our meta categories inputting the meta element whose standard deviation was
the smallest on the highest class. While ontology can beconstructed in various ways from different standpoints,
our ontology is particularly designed for users who want
to search for a language resource for a certain purpose toenable to find them efficiently by following thehierarchical classes of our ontology. This ontology isapplied during a trial in one of three search functions ofSHACHI. Those details are described next in Section
4-(C).

4. Catalogue search tools
The metadata database is composed of language resource
catalogs, a list of all language resource catalogs, acatalogue search tool by which users can retrieve theinformation from all LRs stored in the database from all
angles, and the statistic information of the metadata of its
LRs. Figure 2 shows a screen image of a search result
through SHACHI.
For the purpose of facilitating users of this metadata
database to find their intended language resource
catalogues, SHACHI provides three search functions:(a) Keyword search function
This tool is suitable for users who have clear
images to search for specified LRs and a technicalknowledge of language processing. It allows them to
input keywords as they want and to search all words
stored in SHACHI metadata archive. Boolean
Searching is adopted by this function.
(b) Facet search function
This tool is suitable for users who have a vague
idea of what kind of LR they want. It is equippedwith a choice of 15 kinds of metadata elements
selected from the SHACHI metadata set, from whichusers choose an element which seems to be the mostsuitable language resource for their objective. Then,they narrow down the target LRs one by one in order
to find the intended one. For example, with one click
on “age”, three choices such as “Children’sutterance?”, “Adults’ utterance?” and “Both are
OK?” will be shown. Then, users can narrow the
choice of LRs down according to their research
purposes. This function can be also used together
withkeywordsearchfunctionastheabove-mentioned in (a).
(c) Ontology search function
This tool was developed by adopting the idea
acquired by systematizing LRs registered in
SHACHI which was described in Section 3. Whenusing the ontology search function, users find theintended LRs by following the vertical relationshipof the ontology. It was ascertained that ontology
search function tool had the merit of enabling users
to discover LRs that have not been ever found by
keyword search and facet search functions. This fact
became clear according to the experiment conductedby using 10 subjects who were all research workers
in information processing or language processing
fields. It was also revealed that this tool could
indicate appropriate LRs more effectively than othertwo functions could. This function can be also used
together with keyword search function as theabove-mentioned in (a).

1690

Scrollable

Automaticacquisition ofusage information
for LRs

Figure1. A sample page of SHACHI catalog (ex. Euro WordNet)

Ontology search
Current query
LR’s title

Description

Relation to other LRs
Keyword search

Facet search

Figure 2. Catalogue search tool

1691

5.

Statistical data

It is possible to find the transition of characteristics thatLRs having been released so far possessed by observing
metadata of our collected LRs. This website provides thestatistical information on the LR metadata database
SHACHI on the spot. For example, Figure 3 is a graphic
chart which shows an increasing tendency of numbers of
LRs registered in SHACHI by language. The number ofChinese and Arabic LRs is notably increasing. Figure 4
shows an increasing tendency of written and spoken LRs.
It can be observed that the number of released speech LRs
has been increasing recently. Those kinds of information
help us to promote strategic development of LRs. On
SHACHI project, we are investigating what kind of LR isneeded and how it is efficiently developed by utilizingthem.

information on LRs such as corpora, dictionaries andthesauruses, each of which is fed into the databasethrough semi-automatic means, forming a large scale LRs
metadata archive. One of SHACHI’s characteristicfeatures is that with a collection of tag sets and format
samples given to LRs it has a desirable design for an
efficient development of LRs including thestandardization of tags and organic combination of LRs.
This collection of meta information has been reinforcedby manually entering even more detailed meta
information into the bottom-level category of the meta
element set. We are currently attempting to measure theaffinity between LRs and to systematically store andclassify LRs in the world (construction of LRs ontology).
From now on the SHACHI project is intended to promote
cooperation among other LRs consortia abroad as well asin Japan and to take the initiative in contributing to thedevelopment of LRs in Asia.

Acknowledgements
The construction of this database presented in this paper
was done in collaboration with Professor Kiyoaki Shiraiat the Japan Advanced Institute of Science andTechnology, Sachiko Waki and Cristobal Viveros as atranslator, Miho Ohnishi at the Graduate School of
Literature in Nagoya University, Takahiro Ono and Kenji
Sugiki at Matsubara Group of the Graduate School ofInformation Science in Nagoya University and all theregistration staff of Anchor Co., Inc. We would like tothank them for their generous support.

Referencess
Figure 3. Increasing Tendency by language（Indicated languages are those which have more than 15
LRs registered in SHACHI.）

Figure 4. Increasing tendency of
written and spoken LRs

6. Conclusion
In this research, we reported on the design of SHACHI, a
metadata database of LRs now being developed, theexpansion and construction of metadata for it, and an
actualization of a search function we have developed. At
present, it contains approximately 2,000 pieces of meta

Hayashi, Y., Declerck, T., Buitelaar, P., Monachini, M.
(2008). Ontologies for a Global Language
Infrastructure, ICGL, pp.105-112.
Hughes, B., and Kamat, A. (2005). A Metadata SearchEngine for Digital Language Archives, D-Lib
Magazine,
Volume11
Number2,
http://www.dlib.org/dlib/february05/hughes/02hughes.
html.
Ishida, T., Nadamoto, A., Murakami, Y., Inaba, R.,
Shigenobu, T., Matsubara, S., Hattori, H., Kubota, Y.,
Nakaguchi, T., and Tsunokawa, E. (2008). A Non-Profit
Operation Model for the Language Grid, ICGL,
pp.114--121.
Kozawa, S., Tohyama, H., Uchimoto, K., Matsubara, S.,
and Isahara, H. (2008). Automatic Acquisition of Usage
Information for Language Resources, In Proceedings of
the 6th edition of the Language Resources and
Evaluation Conference, in print.
Tohyama, H., Kozawa, S., Uchimoto, K., Matsubara, S.,
and Isahara, H. (2008). SHACHI: A Large Scale
Metadata Database of Language Resources. In
Proceedings of the 1st International Conference on
Global Interoperability for Language resources, pp.
205--212.

1692