SHACHI: A Large Scale Metadata Database of Language Resources
Hitomi Tohyama†, Shunsuke Kozawa†, Kiyotaka Uchimoto††,Shigeki Matsubara† and Hitoshi Isahara†††Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan{hitomi,kozawa,matubara}@el.itc.nagoya-u.ac.jp,††National Institute of Information and Communications Technology
3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{uchimoto,isahara}@nict.go.jp

Abstract
The National Institute of Information andCommunications Technology (NICT) andNagoya University have been jointly constructing a large scale database named
SHACHI by collecting detailed meta information on language resources in Asia
and Western countries, for the purpose ofeffectively combining language resources.
The purpose of this project is to investigate
languages, tag sets, and formats compiledin language resources throughout the world,
to systematically store language resource
metadata, to create a search function forthis information, and to ultimately utilize
all this for a more efficient development oflanguage resources. This metadata database
contains more than 1,700 compiled language resources such as corpora, dictionaries, thesauruses and lexicons, forming alarge scale metadata of language resourcesarchive. Its metadata, an extended versionof OLACmetadataSet conforming to Dublin Core, which contain detailed meta information, have been collected semiautomatically. This paper explains the design and the structure of the metadata database, as well as the realization of the catalogue search tool.

1

Introduction

The importance of the construction of language
resources such as corpora, dictionaries, thesauruses,

etc., has been widely recognized, and has boomed
for years throughout the world in its aim of encouraging research and development in the main
media of spoken and written languages. Among theorganizations willing to store and distribute language resources, there exist some consortia fulfilling their function such as LDC (Linguistic Data
Consortium), ELRA (European Language Resources Association), OLAC (Open Language Archives Community) and Chinese-LDC (Chinese
Linguistic Data Consortium) in Western countries
and China, and GSK (Gengo Shigen Kyokai; Language Resource Association) which does so mainly
in Japan. However, those released language resources are scarcely connected with each other not
only because of the difference between written
languages and spoken languages but also becauseof the difference between languages such as Japanese, English, Chinese, etc. Moreover, since language information tags given to those language
resources and their data formats are multifarious,
each language resource is operated individually.
As language resource development generally entails enormous cost, it is highly desirable that the
research efficiency be enhanced by combining
those existing language resources and systematically developing them altogether. For the purposeof fully integrating their language resources, NICT
and Nagoya University have been constructing alarge scale database named SHACHI as their joint
project by collecting detailed meta information on
language resources in Western and Asian countries.
This research project aims to extensively collect,and systematically store, metadata such as tag sets,formats, and language resource recordings existingat home and abroad. Meanwhile, we have alreadydeveloped a facet search function of language resources using meta information, and are performing the experiment of widely providing this meta
information on the stored language resources tothose from professional researchers to common
users. This paper outlines our language resource
database, named SHACHI, in the development
stage.
The structure of this paper is as follows: first we
will outline the purpose and design of SHACHI in
the second chapter. Next we will describe the collection of metadata in the third chapter, and the
database structure, as well as the fundamental statistics, in the fourth chapter. Finally we will explain our future task in the fifth chapter.

2

Design of the Metadata Database

2.1

Purpose of the Construction of SHACHIThe purpose of the construction of SHACHI isfourfold.
•
To investigate the actual conditions of tags
and format types of language resources existing at home and abroad.
•To systematically obtain and store metadata of international language resources according to the information obtained from theabove-mentioned. (This leads to the construction of language resource ontology.)(Hayashi, 2007)•

To conduct an investigation into the organic combination of language resources.
(This leads to the strategic development oflanguage resources.)•

To promote the distribution of language
resources.

Some 2,000 resources of metadata have alreadybeen collected in the database so far, and by December 2008 they will be further enlarged by 3,000.
To that end, it is indispensable for us to work incooperation with language resource consortia athome and abroad, and to take the initiative in contributing to Asian language resources.
Additionally, this database is obviously different
from those of other language resource consortia

since all of our detailed metadata are inputtedmanually. The database is notably characterized by
the attempt to make an ontological construction of
language resources throughout the world, as theaffinity of language resource types and that of theirtag sets are analyzed by applying natural language
processing techniques to those detailed metadata. It
seems certain that the realization of its ontologicalconstruction will contribute to a cutback in research and development costs, and to establishing alanguage resource infrastructure which meets present-day needs as an on-demand service. (NICT,2007).
2.2

Design for Collecting MetadataAmong organizations willing to store and distribute language resources, there exist some consortia
fulfilling their function such as LDC, ELRA,OLAC and Chinese-LDC in Western countries andChina, and mainly GSK in Japan.
As for websites, there are two attempting to systematically amass metadata of language resources
and promote their distribution, such as Language
Technology World (LTW 2007) owned by DFKI
(Deutsches Forschungszentrum for KunstilicheIntelligencz) and a page owned by OLAC (OLAC,
2007).
To return the benefit of developed information
processing technologies to society, it is highly desirable that the research be done in mutual cooperation among various language resource consortia
and be enhanced by mutually exchanging information. SHACHI will make this possible as its metadata enables us to collect more detailed meta information in accordance with the OLAC metadata
set by extending it. OLAC is creating a worldwide
virtual library of language resources by developing
consensus on the best current practice for the digital archiving of language resources, and by developing a network of interoperating repositories andservices. OLAC metadata is based on the complete
set of Dublin Core metadata set but a part of which
was extended. The metadata set of SHACHI is described in detail in the following third chapter.

Table 1. SHACHI metadata set
DC Qualifiers
LEVEL 1DCMES Elements
DC Element Refinements
1 titlealternative
2 creator
3 subject

Qualifiers for more precise description of the resources
LEVEL 2
OLAC Extensions
SHACHI Extensions

linguistic-field (29)language(OLAC-Language extension)

4 description
5 publisher
6 contributor

7 daterole(24)mother-tongue (2)nativenon-native
intonation (2)standard_dialect
dialectlevel (2)age (3)gender (3)discourse-type (10)ｐurpose(4)lexicography
analysisdeveloping_technologies
educationｓtyle (2)speechwrittenform (2)fixedunfixedsentence(3)
short
long
mixed
has-annotation (2)annotated
plain
annotation
sampleencoding
markupfunctionality

createdissued8 typelinguistic-type (3)9 formatextentmedium10 identifier
11 source
12 language
13 relation
14 coverage
15 rights

mono-multi-lingual (2)monolingual
multilingual
resource-subject (3)dictionary
thesaurus
lexiconprice

language(OLAC-Language extension)
DC relation refinements (13)temporalutilization

Table 2. Description of SHACHI extensions
SHACHI Extensions
mono_multi_lingual (2)monolingual
multilingual
ResourceSubject (3)dictionary
thesaurus
lexiconAttribute (5)mother-tongue
intonation
level
age
genderPurpose(4)lexicography
analysisdeveloping_technologies
education
Style (2)speechwrittenForm (2)fixedunfixedSentence(3)
short
long
mixed
Annotation (3)annotated
plain
annotaion_sample
SampleFormat (2)encoding
markup3
3.1

Description
Applies to: subject
A resource using only one language:. The same language is use for the
subject language and to describe the subject language.
A resource in several different languages. Different language(s) are used formore than one subject languages and to describe the subject language(s).
Applies to: subject
A list of the words of a language in which the definitions or meanings of thewords are explained either in the same language or in a different language.
A list of the words of a language in which the words are arranged in groups
that have similar meanings.
A list of words on a particular subject.
Attributed of a contributor
The performer is whether a native or non-native speaker of the language.
Dialectal status whether the performer uses a standard language or adialect.
Whether the performer has received a professional level of
linguistic(speaking or writing) training or no such training.
An age group the performer is in. When there are many performers in aresource, the ratio between all the age groups to which the performers
Sex of the performer. When there are many performers in the resource, theratio of males and females.
Applies to: type
The creation of the resource is intended for lexicography.
Thecreation of the resource is intended for analysis.
Thecreation of the resource is intended for developing technologies.
The creation of the resource is intended for the use in education.
The resource is of the spoken language.
The resource is of the written language.
A collection of fixed forms of expressions.
The resource collects various forms of expressions.
A collection of short sentenses.
A collection of long sentenses.
A collection of varying length of sentences.
Tagged corpus.
A corpus without annotations.
A sample of tagged data
A sample of the language resource
An encoded character set used by a digital resource.
A markup scheme used by a digital resource.

Functionality

Software Functionality

Utilization

Applicability of the resource. The described resource is utilized for thereferenced technology, education, research or a product.

Collecting Metadata
Extensions of Metadata Element

The metadata set of this language resource database follows 15 kinds of elements of Dublin Core
(title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage, rights), which is an extended version of OLAC metadata set, Table 1
shows the SHACHI metadata set. The shadedboxes “SHACHI Extension” are the elements
added to those of Dublin Core. As for the defini-tion of extended metadata elements, Table 2 isprovided for showing it with the proviso that only
one of the extended elements named “Utilization”
is adopted on trial. This element is to provide information on the use of language resources which
is semi-automatically retrieved from scholarly papers (Kozawa, 2007). Those items were added tothis metadata, because it seemed that they gave us
clues for finding the attributes of a corpus in collecting metadata of language resources such as
corpora. Furthermore, we encourage our registrants
to register as minutely as possible the information
corresponding to each metadata element by them-language resources through collecting language resource metadata. Therefore, during
the construction of SHACHI, a web searchthrough Google was first conducted to investigate the actual state of language resourcesthat meet the above mentioned conditions.
Then, a search for language resources using
key words such as “corpus”, “dictionary”,“thesaurus”, “translation” or “multilingual”,
or key phrases such as “corpus so-and-so”(e.g. such as ‘PennTreeBank’), was done andas a result those which ranked highly and fitted our key words were retrieved. Table 3
shows the result. According to this research,
WordNet was found to be the most widelydistributed language resource used as a thesaurus. WordNet was originally compiled incertain European languages, and is now also
being compiled in Chinese, Korean and other
Asian languages. On the other hand, it was
found that WordNet in Japanese has not beenon the Internet yet. Research done throughthe Internet such as that described here isconsidered to be important in identifying thetypes of language resources to be developedin the future.

selves instead of automatically. In consequence,
the information sometimes contains a lot of key
words and somewhat longer sentences, which
however enlarges the data subject to the key word
search. We believe that its information will provide
important clues to measure the affinity (similarity)of each language resource through language processing technologies in the future. The developmentof this project must contribute to the constructionof an international language resource ontology.
As for the meta elements of language attributes,
160 values are currently introduced in our metadata set, which conforms to ISO 639 (ISO, 2007).
The way to describe the date and the time conforms to ISO 8601. Moreover, this database will be
intended to conform to ISO TC46/SC4, the International Standard for Information and Documentation & Technical Interoperability.
3.2

Policy for Collecting Metadata

The language resources which SHACHI storesshould satisfy the following conditions:
1. The resources should be stored in a digitalformat.
2. The resources should be one of the followings: corpus, dictionary, thesaurus, or lexicon.
(Numeric data are not considered to be thesubject of collection for this database.)
3. Those resources should be collected fromEnglish websites and its data must be open to
the public.
4. Those resources should be created by research institutions, researchers, or businessorganizations.
In addition to the above conditions, we are primarily collecting metadata of language resources
which contain a large volume of data, are wellknown to the public, and are considered to be important for improving information processing technology. As expected we will call for participationin our project from research laboratories and institutes developing languages resources, by registering them in our database, as soon as preparations
for publishing RSS feeds of our data and our input
format of metadata standards are completed. Its
registration page is also to be open to the public
but is not yet on view.
•

Preferred Collection of Highly Recognized Language
It is essential to obtain and store information
on highly recognized and frequently used•

Collecting the Metadata from Major
Organizations
It is important to gather language resource
metadata from all over the world to performa study about language resources, their promotion of distribution, and their strategic
development. SHACHI not only covers
metadata from major language resource
consortia in Japan, Western countries, andChina, but also conducts semi-automaticways of registering detailed metadata in accordance with SHACHI metadata sets. Table 4 shows the list of major language resource consortia which this database covers.

4

Construction of SHACHISHACH is composed of language resource catalogs, a list of all language resource catalogs, catalogue search tool by which users can retrieve theinformation from all language resources stored in
SHACHI from all angles, and the statistical information of the metadata of the language resourcesof SHACHI.

Table 3. The results of investigation on highly recognized and frequently used language resources
Monolingual Dictionary
Webster's Revised Unabridged
Dictionary (1913)
Oxford Advanced Learner's
Dictionary
LONGMAN Dictionary ofContemporary English
Wiktionary
Collins COBUILD Advanced
Learner's Dictionary
The Swedish PAROLE Lexicon
American Heritage Dictionary of the
English Language
EDR Electronic Dictionary
Technical Guide

Multilingual Dictionary

Parallel Corpus

Thesaurus

The EDICT Dictionary File

Aligned Hansards corpus

WordNetCollins ROBERT French Dictionary

European Parliament Proceedings
Parallel Corpus
1996-2003
OPUS - an open source parallel
corpusUN Parallel Text

Equine Multilingual Dictionary

Hong Kong Parallel Text

MeSH

The Papillon Project

COMPARAThe English-Norwegian Parallel
CorpusCRATER Multilingual Aligned
Annotated Corpus

Global WordNet

WebLSD
Oxford-Hachette French Dictionary

CJK Lexical Resources

SANSEIDO Daijirin

Multilingual Dictionary of Proper
Nouns CJKE-DPN
Multilingual Glossary of technicaland popular medical terms in nine
European Languages
EIGIROA Japanese Lexicon (Japanese link)

EDR Bilingual Dictionary

Le Petit Robert French monolingual
dictionary

EuroWordNet
Hindi WordNet
Roget's

UMLS Metathesaurus
Merriam-Webster's Collegiate
Thesaurus

The JRC-Acquis Multilingual Parallel
CorpusERIC Thesaurus

Polyglot Bible

Art & Architecture Thesuarus

NICT JLE Corpus

The European multilingual thesaurus
on health promotion in 12 languages

Table 4. List of major language resource consortia which SHACHI covers
Consortium
Asian Language Resource Catalogue
ChineseLDC (ChineseLinguistic Data Consortium)
ELRA(European Language Resources Association)
Global WordNet Association
GSK (Gengo Shigen Kyokai)
ICAME Corpus Collection on CD-ROM
LDC(Linguistic Data Consortium )Natural Language Processing Portal Site
SiTEC
(Speech Information Technology & Industry
Speech Resources Consortium

The input format of metadata is, of course, notopen to the public, but will be released in the future for use by research organizations and language resource consortia scattered all over theworld. Because of this, we intend to provide outfits and interfaces which will allow researchers toregister their language resource metadata without
restraint, and which will enrich the quality of
SHACHI by extending the contents of its database.
4.1

Outline of Catalogue

This catalogue allows SHACHI users to obtainoutlines of the language resources at a glance
without referring to the linked official websites(See Figure 1).
SHACHI catalogue allows SHACHI users to
grasp outlines of the language resources at a
glance without referring to the linked official
websites.
4.2

Catalogue Search Tool

A catalogue search tool has been developed forthe purpose of allowing users who visit the SHACHI site to find a language resource that meets
their needs. This search tool is loaded with a key

ULR
http://nlp.kuee.kyoto-u.ac.jp/

http://www.chineseldc.org/
http://www.elra.info/fr/
http://www.globalwordnet.org/
http://www.gsk.or.jp/
http://icame.uib.no/newcd.htm
http://www.ldc.upenn.edu/Catalog/
http://nlp.kuee.kyoto-u.ac.jp/NLP_Portal/lr-cat-j.html
http://www.sitec.or.kr/English/
http://research.nii.ac.jp/src/links/

word search function as well as a facet searchfunction. Figure 2 shows an image screen of thesearch tool. As for the key word search function, it
allows users to input key words as they want, andto search all words stored in the SHACHI metadata archive. On the other hand, the facet searchfunction is equipped with choices of 15 kinds of
metadata elements selected from 25 kinds that
SHACHI stores. This function helps users to obtain the desired language resource by selecting, inorder, an element which is closest to their needs
and then allows them to narrow down theirchoices. SHACHI especially brings its ability intofull play for users who have no idea of adequate
key words or have only a vague idea of how tofind their desired information, since it can provide
a large scale database with more detailed metadatathrough its characteristic search function(Bontcheva, 2007). On an actual image screen of
the search results, titles of language resource candidates and a list of their descriptions are displayed, and at the bottom the relationships among
other language resources are indicated (See Figure2). We intend to conceive a method which enables
relationships among language resource candidates

Figure 1. A sample page of SHACHI catalog (A part of catalogue; ex Brown Corpus)

shown in the search result to be visualized andmeasured.
4.3

Statistical Data

By observing metadata of the collected language
resources, it is possible to find the transition ofcharacteristics possessed by language resourcesthat have been released so far. This website provides on the spot statistical information derived
from the language resource metadata database
SHACHI. It gives us the information, for example,
that Arabian, Korean and Chinese language resources have been increasing recently, and that theprices of language resources have been rising.
Thus, the SHACHI project investigates what kindsof language resources are needed and how they
can be efficiently developed.

5

Future Work

These days new language resources are being created one after the other, and those resources arealso modified on a daily basis. In order to grasp
the current conditions of those resources, collecting and updating metadata at regular intervals is
inevitably required. In addition, it is indispensable
for us to collaborate with other research institu-

tions such as GSK in order to collect language
resource metadata efficiently and to develop thehalf-automated way of collecting and updating
information via Web conversation (an online
communication tool). On the other hand, we willconceive of an automatic information retrieval
method by which we will obtain information on
the situation of language resource diffusion, andvarious ways to utilize this information collected
through the results of web searches and quoted
information from papers.

6. Conclusion
The National Institute of Information and Communications Technology and Nagoya University
have been collecting meta information extensively
in order to construct a new type of language resource which uses metadata in their researchthrough an organic combination of language resources. In this paper, the design of SHACHI (ametadata database of language resources now being developed), the expansion and construction of
metadata for it, and the actualization of a searchfunction we have developed were reported.

LR’s nameCurrent Query

Key word search

Facet Search

Figure 2. Catalogue search tool

SHACHI contains extensive data obtained fromhighly recognized language resources searched foron the Internet, and also covers meta information
provided by language resources consortia such as
ELRA, OLAC and LDC. Furthermore, this continuing work to register more detailed metadata ismaking SHACHI an even larger scaled database.
At present, it contains approximately 1,700 piecesof meta information on language resources, forming the world largest language resource metadataarchive. One of SHACHI’s characteristic features
is that the manner in which it collects tag sets andformat samples given to language resources, has adesirable design for the strategic development oflanguage resources including the standardization oftags and the efficient development of language resources. This collection of meta information has
been further reinforced by the manual entering ofeven more detailed meta information into the bottom-level category of the meta element set. We
have already measured the affinity between language resources and have systematically stored andclassified language resources throughout the world.
We believe this will lead to the construction of alanguage resource ontology.

Acknowledgements
We would like to thank Sachiko Waki, Miho Ohnishi of
Nagoya University, Kenji Sugiki and Takahiro Ono of
Matsubara’s SLP Group at Nagoya University and all
the staff of ANCHOR Co., Inc. for their full cooperationin constructing the database.

References
Hayashi, Y. 2007. Conceptual Framework of an Upper
Ontology for Describing Linguistic Services, Proceedings of IWIC2007, pp.246-260.
ISO official site, 2007.
http://www.iso.org/error/sitedown.html
Kozawa, S. Toyama, H. Uchimoto, K. Matsubara, S.
2007. Automatic Acquisition of Expressions Representing Purposes and Methods of Using A Language
Resource from Academic Articles, Proceedings ofWorkshop on Informatics 2007, pp.65-70, (in Japanese).
LTW (Language Technology World). 2007. official site,
http://www.dfki.de/lt//publications_show.php?id=148
NICT Language Grid project official site, 2007 . official
site, http://langrid.nict.go.jp/indexj.htm.
OLAC (Open Language Archives Community), 2007.
official site, http://www.lt-world.org/