Challenges in Digital Libraries
Key Issues Learned from
Metadata-Centric Projects at Tsukuba
Shigeo Sugimoto
Research Center for Knowledge Communities
Graduate School of Library, Information and Media Studies
University of Tsukuba
sugimoto@slis.tsukuba.ac.jp
2005-8-25

IADLC, Nagoya

1Outline• Introduction• Overview of Digital Libraries– Looking Back R&D activities– Digital Libraries at Libraries• Metadata-Centric Projects at Tsukuba
and Some Lessons Learned• Challenges in Digital Libraries• Summary
2005-8-25

IADLC, Nagoya

2

Introduction• Digital Library: a perspective– DL: integration of many new information technologies
in accordance with types of collections, users andenvironments• Technology × Collection × Environment– Technology oriented view vs. Service oriented view• “Challenges in Digital Libraries”– Many challenging issues in digital libraries– “Interoperability, Preservation, and Theory”(from conversation with Prof. Edward Fox atICADL’02 in Singapore)
2005-8-25

IADLC, Nagoya

3

Overview of Digital Libraries

Looking Back DL Projects and Programs• IT oriented Programs– Digital Library Initiative (Phase 1, 2), USA• DLI-1(1994-98): NSF, NASA, DARPA• DLI-2(1998-2003): NSF, NASA, DARPA, NLM,LoC, NEH• Working Groups for Key Topics in DL (EU-NSF)– Information Discovery, Metadata, Preservation, IPR
issues, etc.

• Digital Libraries for Education– National Science Digital Library (NSDL)
2005-8-25

IADLC, Nagoya

4Overview of Digital Libraries

Looking Back DL Projects and Programs (cont’d)• Development of Digital Collections– Historical and Rare Materials• American Memory by Library of Congress• Making of America by Cornell, Michigan, UC Berkeley• Digital Library from the Meiji Era by NDL, Japan,etc.

– Scholarly Repositories• ePrint Archive, NDLTD, etc.
• Open Archives Initiative– Cooperative activity by repositories
2005-8-25

IADLC, Nagoya

5Overview of Digital Libraries

Looking Back DL Projects and Programs (cont’d)• Metadata for DL and Internet– Dublin Core
• Metadata for Resource Discovery/Description
across Domains– Domain Specific Schemas• Preservation, Education, Government,Accessibility, Geospatial Information, e-Commerce,
Rights Management, Video, etc.

– Semantic Web• Metadata Framework• Ontology
2005-8-25

IADLC, Nagoya

6Overview of Digital Libraries

Fundamental Changes in our Information
Environment• The Internet and WWW– The Internet and WWW are important part ofinformation lifeline.
– Information seeking behavior has been changed byInternet search engines and directory services, e.g.
Google and Yahoo!.

• Online Resources for Libraries– Scholarly Resources• Electronic Journals• Scholarly Repositories• Digitized Collections– Learning Resources– Governmental Resources
2005-8-25

IADLC, Nagoya

7

Overview of Digital Libraries

Fundamental Changes in our Information
Environment (cont’d)• Statistics from 2005 White Paper
Information and Communications inJapan, Ministry of Internal Affairs andCommunications– Internet Users: 80M– PC: 61M, Mobile Phone: 58M– PC only: 21M, Mobile Phone only: 15M

2005-8-25

IADLC, Nagoya

8Overview of Digital Libraries

DL Functions in Library Services• Building Collections of Primary Resources– Electronically Published Resources• E-Journals• Repositories, etc.

– Digitized Resources• Navigational Services (Secondary/Meta ResourceCollection)– Subject Gateways, Portals– Reference Services• Preservation of Digital Resources• User Environment Building– Personalization– Hybrid Library
2005-8-25

IADLC, Nagoya

9Metadata-Centric Projects at TsukubaOutline• Subject Gateway Projects– ULIS-DL• Gateway for Library and LIS Resources• Simple Dublin Core– IPL-Asia• Resources for Public Library Users in Chinese, Japaneseand Korean (CJK) languages• Metadata in CJK– Collaboration with Okayama Prefectural Library• Digital Okayama Dai-Hyakka– Regional Portal– On Going Activity• Gateway to Digital Collections in Public Libraries in Japan– Created approximately 5000 records for resources published in170 library Web sites
2005-8-25

IADLC, Nagoya

10Metadata-Centric Projects at Tsukuba• Metadata Schema Projects– Metadata Schema Registry• Collaboration with DCMI

– Layered Model for Metadata Interoperability– On Going Activity• Metadata Framework for Context Sensitive ResourceSelection and Adoption• Why Metadata-Centric Projects?
– Value addition by metadata– Semantic Web Technologies, i.e., RDF/XML, OWL, etc.
2005-8-25

IADLC, Nagoya

11

ULIS-DL

Building a Core Subject Vocabulary• Outline of ULIS-DL– Subject gateway for resources published by libraries
and LIS institutions.
– Metadata records created based on Simple Dublin
Core.
– ULIS-DL has a retrieval function but no directory styleinterface to browse and navigate the contents.
– Subject terms are given as free terms, i.e. nocontrolled vocabulary.

• Goal of our research– To create a subject vocabulary for directory styleinterface of ULIS-DL
2005-8-25

IADLC, Nagoya

12

ULIS-DL

Building a Core Subject Vocabulary• Status as of 2003
– 26,000 metadata records– 16,000 distinct text strings In the Subject element of
the raw metadata records, including typos andsynonyms• Issue: How to choose appropriate subject terms• Methodology to build a Core Subject Vocabulary– Extract terms of Subject elements which appear morethan once.
– Measure coverage/uncoverage ratio of the term sets
(Candidate Term Set-n).

• Result: approximately 90% of collected sites iscovered by 1025 subject terms.
2005-8-25

IADLC, Nagoya

13

ULIS-DL

Building a Core Subject Vocabulary

CTS-2
CTS-3
CTS-4
CTS-5

Subjectterms

Excludedrecords

Uncoverage
ratio

3979
2045
1366
1025

1519
2083
2590
2801

6%
8%
10%
11%

• Findings– 90% of the whole resources can be covered by CTS-5.
– “1000 words” is a manageable size for manual encoding in XML.
– Includes NDC terms, proper nouns, ULIS-DL specific terms, etc.
2005-8-25

IADLC, Nagoya

14

2005-8-25

IADLC, Nagoya

15

IPL-Asia• IPL-Asia– Provides information about CJK resources useful as apublic library resources.
– Provides resource information in CJK languages.

• Lessons Learned– Domain oriented subject vocabularies need not be large
but need community-specific terms, e.g., school activities
and regional activities.
– User interface for children requires to represent subjectterms in accordance with their ages.
– Costs
2005-8-25

IADLC, Nagoya

16Digital Okayama Dai-Hyakka (DODH)• Regional Portal by the Library of OkayamaPrefecture• Metadata Creation by Librarians and Nonprofessionals, e.g. School Teachers, Students,and Volunteers.
• Three Subject Vocabularies– NDC (Nippon Decimal Classification)– Kid’s Vocabulary• Multiple labels in accordance with user ages

– Prefectural Resource Vocabulary• Small Set of Subject Terms Usable for the Nonprofessionals designed in accordance withRegional Needs.
2005-8-25

IADLC, Nagoya

17

Multiple Labels for a Single Term“としょかん”“図書館（トショカン）”senior graderselementary school

junior graderselementary schoolpresentation labels“Library”

presentation labels
a subject term

“図書館”

“図書館”

junior high school

high school and above• Multiple labels for a single concept in accordancewith type of audience to improve accessibility– Encoding in XML is straightforward, e.g. OWL, XML
TopicMaps.
2005-8-25

IADLC, Nagoya

18

Comparison of Subject Vocabularies- Okayama’s Case • Comprehensive and conventional subject
vocabularies are not always useful for domainspecific resources.
• Comparison between Subject Vocabularies usedin Okayama– Prefecture government’s subject vocabulary forgovernmental resources (Prefecture Vocabulary)– A subject vocabulary for children (Kids Vocabulary)– Three vocabularies used in DODH
• NDC, PV, and KV• Mappings between all pairs of these three vocabularies
2005-8-25

IADLC, Nagoya

19

Distribution of Terms in the NDC term space- Okayama’s Case NDC

000 100 200 300

400 500 600

700

800

900

total

#Kidsterms

1788196 58 54

286266443

#NDC
in KV

7734427 27

202644169#Prefterms

15212 171 30 56

441711349#NDC
425 34 11 18 25 15 1
1 116
in PV
NDC: 000=Generalities, 100=Philosophy, 200=History, 300=Social Sciences,
400=Natural Sciences, 500=Technology, 600=Industry, 700=The Arts,
800=Language, 900=Literature#NDC in KV/PV: the number of NDC terms in x00 used in the KV/PV mapping
2005-8-25

IADLC, Nagoya

20Subject Vocabularies
Resources for
Government andSocial Activities

PVEducational andLearning Resources

KVSocial Science
TechnologyIndustries

Natural Science
Arts

Comprehensive

NDC

General Resources
2005-8-25

IADLC, Nagoya

21

Some Lessons Learned from
Subject Gateway Projects• Subject vocabularies were always the central issue forthese projects.
– Conventional and comprehensive vocabularies are not always
useful because of the narrow domain.
– (Reasonably) small subject vocabulary seems useful for endusers.
– Domain-specific/regional vocabularies are useful, however
vocabulary maintenance technology should be investigated.
• Semantic Web technologies• Issues– Interoperability vs. Domain Specificity– Maintenance– Reusability and Customizability of Metadata Vocabularies
2005-8-25

IADLC, Nagoya

22

A Layered Model of Metadata
Schema Model• Metadata Schema– Element Set, Application Profiles, XML Bindings, etc.

• Splitting metadata schema into Three Layers inMetadata Schema– Semantics• Metadata Vocabularies– Abstract syntax• syntax definition neutral to implementation technology• Application Profile– Concrete syntax• syntax definition to encode metadata instances
2005-8-25

IADLC, Nagoya

23

A Concrete Syntax: a Catalog Card
Title

Challenges in Digital Libraries

CreatorSugimoto, Shigeo
杉本, 重雄／すぎもと, しげお
Metadata Schema, Interoperability
Nagoya University Library
Nagoya, Japan

Creator
Subject
Publisher
Publishing
Location
Date
Language

2005-8-25

2005-8-25
EnglishIADLC, Nagoya

24

A Concrete Syntax: a DB record
Title (en)Challenges in Digital Libraries

Creator (en)Sugimoto, Shigeo

Creator (ja)杉本, 重雄Subject (en)Metadata Schema

Subject (en)Interoperability

Publisher (en)Nagoya University Library

Publishing Location (en)Nagoya, Japan

Date (ISO-8601)

2005-8-25

Language (ISO-639)

en（pronunciation: すぎもと, しげお）

Metadata DB(Catalog DB)

2005-8-25

IADLC, Nagoya

25

A Concrete Syntax: an XML instance<dc:title> Challenges in Digital Libraries</dc:title><dc:creator> Sugimoto, Shigeo</dc:creator><dc:creator xml:lang=“ja”><rdf:value>杉本, 重雄 </rdf:value><uliscore:pronunciation>すぎもと, しげお</uliscore:pronunciation></dc:creator>…
Metadata DB(Catalog DB)

2005-8-25

IADLC, Nagoya

26

An Abstract SyntaxSyntactic features neutral to implementation syntax

Title: Mandatory
Creator: Mandatory if Applicable, Repeatable
Subject: Optional, Repeatableetc.

2005-8-25

IADLC, Nagoya

27

Semantics - Definition of Terms
Title
Element Name: Title
Label: Title
Definition: A name given to the resource.
Comment: Typically, Title will be a name by which
the resource is formally known.
CreatorElement Name: Creator
Label: Creator
Definition: An entity primarily responsible for making
the content of the resource.
Comment: Examples of Creator include a person, …
2005-8-25

IADLC, Nagoya

28

Application Profile
termA:Mandatory

termA

termB

termC:
termX:
Optional
Mandatory
Repeatable Repeatable

termC

Metadata Vocabulary 1(Metadata Element Set)

termXtermZ:
Mandatory
if applicable

termY

termZ

Metadata Vocabulary 2(Metadata Element Set)

A Structural View of an Application Profile
2005-8-25

IADLC, Nagoya

29

Abstract Syntax and Concrete Syntax<rdf:Description about=”foo”><mv1:A>an example.</mv1:A><mv2:X>bar</mv2:X>...

<meta name=”mv1:A”
content=”an example”><meta name=”mv2:X”
content=”bar”>

Description in asyntax defined inan application...
termA:
Mandatory

termA

termB

termC:
Optional
Repeatable

termX:
Mandatory
Repeatable

termC

termX

Metadata Vocabulary 1(Metadata Element Set)

Application Profile:Terms used in an
application andstructural constraints

termZ:
Mandatory
if applicable

termY

termZ

Metadata Vocabulary 2(Metadata Element Set)

Application Profile

2005-8-25

IADLC, Nagoya30

A Layered Model
split semantics and syntax into layers
Layer 3Concrete
SyntaxLayer 2
Abstract
SyntaxLayer 1
Semantics

XML
implementationin an XML
SchemaDCMI Library
Application
ProfileDCMES(ElementsandQualifiers)

An Oracle
implementation

OpenArchives
Initiative
SchemaIEEE-LOM

RDF
implementation

IPL Asia
SchemaULIS
Core
SchemaULIS
element
extension

Layered Model of Metadata Schema
2005-8-25

IADLC, Nagoya

31

Metadata Schema Registry• Primary Function– Store reference descriptions of metadata schemas• Metadata Terms, i.e. vocabularies, element sets
• Application Profiles– Provide the reference descriptions not only to humanusers but also to machines via the Internet• Searching and browsing functions for human users• Application program interfaces (APIs) via the Web for machines• Metadata Schema in an Understandable Form both for humans
and machines• Collaborating Distributed Schema Registries– Enhance information sharing
2005-8-25

IADLC, Nagoya

32

Metadata Schema Registry
Search from collaboratingregistries
Metadata SchemaRegistry

Internet

Search and Get
Descriptions

Metadata SchemaRegistry

Application Profiles
2005-8-25

Search & Browse

Metadata Terms
IADLC, Nagoya

33

Layered Modeland Metadata Schema Registry
Application profile designers get definitions of metadata
terms from Registry A and/or B.
Layer 3Layer 2

Application Profile A

Application Profile B

Layer 1

Registry A

Element Set A
e.g. Dublin
Core MetadataElement Set

2005-8-25

Registry B

Element Set Be.g. Tsukuba
Element Set

IADLC, Nagoya

ULIS-DL
SubjectVocabulary34

Layered Modeland Metadata Schema Registry
Application designers get an application profile and define
implementation syntax.

Layer 3

Implementation Syntax A1
for Application Profile A

Implementation Syntax A2
for Application Profile A

Layer 2Registry
Layer 1

Application Profiles
Metadata Terms

2005-8-25

IADLC, Nagoya

35

A Metadata Framework for Context
Sensitive Resource Selection- an on going study • Find and access a resource in accordance withuser characteristics and user environment– Users with Disabilities• Identification of appropriate resources in accordance withuser characteristics– Size of Displays• PC, PDA, mobile phone– Environment• In-door, out-door• Band-width of Network Connection– etc.
2005-8-25

IADLC, Nagoya

36Some Lessons Learned• Metadata Schema Interoperability– A good model is required.
– The layered model gives a framework. Refinement ofthe model is required.

• Metadata schema registry– Sharing information about metadata schemas– Maintenance of metadata vocabularies andapplication profiles– Software tools connected to the registry• e.g. Schema editor, Schema maintenance tool, Software toolgenerator
2005-8-25

IADLC, Nagoya

37

Challenges in Digital Libraries• Key Issues based on the experiences inthe metadata-centric projects– Interoperability– Model and Theory– Digital Preservation– Globalization vs. Localization,Universality vs. Domain-Specificity– Accessibility and Adaptability– Metadata
2005-8-25

IADLC, Nagoya

38

Challenges in Digital Libraries• Interoperability– General and basic requirements• Single access point for a network of DLs– Difficulties in various aspects technologicallyand non-technologically• Model and Theory– Guidelines for development and maintenanceof DLs– Guidelines for cooperation between DLs
2005-8-25

IADLC, Nagoya

39

Challenges in Digital Libraries• Digital Preservation– Temporal Interoperability– Digitized Resources and Born Digital Resources• Preservation Levels– Bit String, Functions, Look & Feel, Technologies• OAIS Reference Model– Web Archiving• Hidden/Deep Web problem• Internet Archive / Intranet Archive (Institutional Archive)– Negotiation between providers and archives

2005-8-25

IADLC, Nagoya

40

Challenges in Digital Libraries• Digital Preservation (cont’d)– Maintenance of (Preserved) Resources• Migration• Broken Link Problem– Metadata for Preservation• Costs• Metadata Schema Issues– Resource Identification for Preservation• Guidelines to determine to identify resources for preservation• Dynamic Resources– “single-source multi-use” resources– Documents dynamically composed of fragments• Preservation of functional features– Acceptable degradation of functional features
2005-8-25

IADLC, Nagoya

41

Challenges in Digital Libraries• Globalization vs. Localization,
Universality vs. Domain-Specificity– Metadata Issues• Domain specific vocabulary vs. Common vocabulary• Dumb-down principle– A principle introduced for semantic interoperability by DCMI
– A framework for semantic refinement of a metadata element
without loosing interoperability• Layered Model– A framework to understand and enhance metadata interoperability– User Issue• User Interface issues, e.g., language, style
2005-8-25

IADLC, Nagoya

42

Challenges in Digital Libraries• Accessibility and Adaptability– Guidelines for Web Accessibility– Resource Selection by Matching User
Characteristics and Resource Characteristics– Resource Adaptation in accordance withUsers and User Environments• Metadata– The Key and Common Component among theChallenging Issues in DL
2005-8-25

IADLC, Nagoya

43

Summary• There are many challenging issues for further
development of DLs in addition to the issues listedin this presentation.
• Interoperability is the most fundamental andchallenging issue for DL.
– DLs have to cope with contradictory requirements• legacy data vs. new type of resources, media and users• Regional/Domain-specific Community vs. Global Communityetc.

– Collaboration among libraries is crucial to clarify theissues that should be solved and to promote technology
development and knowledge sharing.
2005-8-25

IADLC, Nagoya44