NASA / TM-2001-211049

Buckets: Smart Objects for Digital Libraries

Michael L. Nelson

Langley Research Center, Hampton, Virginia

August 2001

https://ntrs.nasa.gov/search.jsp?R=20010089150 2018-05-15T10:41:49+00:00Z
The NASA STI Program Office ... in Profile

Since its founding, NASA has been dedicated to

the advancement of aeronautics and space

science. The NASA Scientific and Technical

Information (STI) Program Office plays a key

part in helping NASA maintain this important

role.

The NASA STI Program Office is operated by

Langley Research Center, the lead center for

NASA's scientific and technical information. The

NASA STI Program Office provides access to the

NASA STI Database, the largest collection of

aeronautical and space science STI in the world.

The Program Office is also NASA's institutional

mechanism for disseminating the results of its

research and development activities. These

results are published by NASA in the NASA STI

Report Series, which includes the following

report types:

TECHNICAL PUBLICATION. Reports of

completed research or a major significant

phase of research that present the results of

NASA programs and include extensive

data or theoretical analysis. Includes

compilations of significant scientific and

technical data and information deemed to

be of continuing reference value. NASA

counterpart of peer-reviewed formal

professional papers, but having less

stringent limitations on manuscript length

and extent of graphic presentations.

TECHNICAL MEMORANDUM. Scientific

and technical findings that are preliminary

or of specialized interest, e.g., quick release

reports, working papers, and

bibliographies that contain minimal

annotation. Does not contain extensive

analysis.

CONTRACTOR REPORT. Scientific and

technical findings by NASA-sponsored

contractors and grantees.

CONFERENCE PUBLICATION. Collected

papers from scientific and technical

conferences, symposia, seminars, or other

meetings sponsored or co-sponsored by

NASA.

SPECIAL PUBLICATION. Scientific,

technical, or historical information from

NASA programs, projects, and missions,

often concerned with subjects having

substantial public interest.

TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific

and technical material pertinent to NASA's

mission.

Specialized services that complement the STI

Program Office's diverse offerings include

creating custom thesauri, building customized

databases, organizing and publishing research

results ... even providing videos.

For more information about the NASA STI

Program Office, see the following:

• Access the NASA STI Program Home Page

at http'//www.sti.nasa.gov

• E-mail your question via the Internet to

help@sti.nasa.gov

• Fax your question to the NASA STI Help

Desk at (301) 621-0134

• Phone the NASA STI Help Desk at

(301) 621-0390

Write to:

NASA STI Help Desk

NASA Center for AeroSpace Information

7121 Standard Drive

Hanover, MD 21076-1320
NASA / TM-2001-211049

Buckets: Smart Objects for Digital Libraries

Michael L. Nelson

Langley Research Center, Hampton, Virginia

National Aeronautics and

Space Administration

Langley Research Center

Hampton, Virginia 23681-2199

August 2001
Available from:

NASA Center for AeroSpace Information (CASI)

7121 Standard Drive

Hanover, MD 21076-1320

(301) 621-0390

National Technical Information Service (NTIS)

5285 Port Royal Road

Springfield, VA 22161-2171

(703) 605-6000
ABSTRACT

BUCKETS: SMART OBJECTS FOR DIGITAL LIBRARIES

Michael L. Nelson

Old Dominion University, 2000

Director: Dr. Kurt Maly

Current discussion of digital libraries (DLs) is often dominated by the merits of

the respective storage, search and retrieval functionality of archives, repositories, search

engines, search interfaces and database systems. While these technologies are necessary

for information management, the information content is more important than the systems

used for its storage and retrieval. Digital information should have the same long-term

survivability prospects as traditional hardcopy information and should be protected to

the extent possible from evolving search engine technologies and vendor vagaries in

database management systems. Information content and information retrieval systems

should progress on independent paths and make limited assumptions about the status or

capabilities of the other.

Digital information can achieve independence from archives and DL systems

through the use of buckets. Buckets are an aggregative, intelligent construct for publishing

in DLs. Buckets allow the decoupling of information content from information storage

and retrieval. Buckets exist within the Smart Objects and Dumb Archives model for DLs

in that many of the functionalities and responsibilities traditionally associated with

archives are "pushed down" (making the archives "dumber") into the buckets (making

them "smarter"). Some of the responsibilities imbued to buckets are the enforcement of

their terms and conditions, and maintenance and display of their contents. These

additional responsibilities come at the cost of storage overhead and increased complexity

for the archived objects. However, tools have been developed to manage the complexity,
ii

and storageis cheap and getting cheaper;the potential benefits buckets offer DL

applicationsappearto outweightheir costs.

Wedescribethemotivation,designandimplementationof buckets,aswell asour

experiencesdeployingbucketsin two experimentalDLs. We alsointroducetwo modified

forms of buckets:a "dumb archive"(DA) andthe BucketCommunicationSpace(BCS).

DA is a slightly modifiedbucketthat performs simple set managementfunctions. The

BCSprovides a well-known locationfor bucketsto gain accessto centralizedbucket

services,suchas similarity matching,messagingand metadataconversion. We also

discussexperienceslearnedfrom usingbucketsin theNCSTRL+ andUniversalPre-print

Server(UPS) experimentaldigital libraries. We concludewith comparisonsto related

work anddiscussionaboutpossibleareasfor futurework involving buckets.
iii

ACKNOWLEDGMENTS

This dissertationwas madepossible through the assistance, encouragement and

patience of many people. Foremost among these people are the members of my

committee. Kurt Maly provided the direct advisement, insight and strategic vision

necessary for the definition, refinement and wide adoption of the research results.

Stewart Shen and Mohammad Zubair were constant supporters and more than

occasionally devil's advocates during our weekly meetings. Frank Thames provided much

of my initial motivation to pursue a Ph.D., and David Keyes' encouragement is the reason

I chose to obtain it at Old Dominion University.

Many fellow students at Old Dominion University have positively affected my

research. Xiaoming Liu, Mohamed Kholief, Shanmuganand Naidu, Ajoy Ranga, and

Hesham Anan are among those that have made design or coding suggestions, developed

supporting technologies, and ferreted out many bugs.

NASA Langley Research Center has provided me with the opportunity and

resources to perform digital library research and development. These current and former

NASA colleagues have provided technical, financial and moral support in the breadth of

my digital library activities at Langley: David Bianco, Arleen Biser, David Cordner,

Delwin Croom, Sandra Esler, Gretchen Gottlich, Nancy Kaplan, Mike Little, MingHokng Maa, Mary McCaskill, Daniel Page, Steve Robbins, Joanne Rocker, George

Roncaglia, and Melissa Tiffany.

A number of people outside Old Dominion University and Langley Research

Center played significant roles in supporting the development and adoption of buckets.

Among these people are: Herbert Van de Sompel (University of Ghent), Marcia Dreier

(Air Force Research Laboratory) and Rick Luce (Los Alamos National Laboratory).

Finally, I would like to thank Rod Waid for the creation of the lovable "Phil"

character that eventually evolved into our research group's mascot, and Danette Allen for

her patience and support.
iv

TABLE OF CONTENTS

LIST OF TABLES .................................................................................................

LIST OF FIGURES .............................................................................................

Chapter

1. INTRODUCTION ..................................................................... 1

2. MOTIVATION AND OBJECTIVES .............................................. 6

2.1 Why Digital Libraries? ...................................................... 6

2.1.1 Digital Libraries vs. the World Wide Web .................... 8

2.1.2 Digital Libraries vs. Relational Database Management

Systems ............................................................ 9

2.2 Trends in Scientific and Technical Information Exchange ............... 10

2.3 Information Survivability ................................................... 13

2.4 Objectives and Design Goals ............................................... 15

2.4.1 Aggregation ........................................................ 15

2.4.2 Intelligence ......................................................... 16

2.4.3 Self-Sufficiency ................................................... 16

2.4.4 Mobility ........................................................... 17

2.4.5 Heterogeneity ..................................................... 18

2.4.6 Archive Independence ............................................ 18

3. BUCKET ARCHITECTURE ....................................................... 19

3.1 Overview ...................................................................... 19

3.2 Implementation .............................................................. 24

3.2.1 Bucket Methods .................................................. 26

3.2.2 File Structure ..................................................... 30

3.2.3 Terms and Conditions ........................................... 33

3.2.4 Internal Bucket Operation ....................................... 38

3.2.5 Metadata Extensions ............................................. 39

3.3 Discussion ..................................................................... 41

3.3.1 Bucket Preferences ............................................... 41

3.3.2 Systems Issues ................................................... 43

4. DUMB ARCHIVES .................................................................. 47

4.1 Overview ...................................................................... 47

4.1.1 The SODA DL Model .......................................... 47

4.1.2 Archive Design Space ............................................ 49

4.1.3 Publishing in the SODA Model ................................ 50

4.2 Implementation ............................................................... 51

PAGE

vii

viii
V

4.2.1 Implemented Methods ..........................................

4.2.2 Changes from a Regular Bucket .................................

4.3 Discussion .....................................................................

4.3.1 DA Examples .....................................................

4.3.2 DBM Implementation Notes ...................................

4.3.3 Open Archives Initiative Dienst Subset Mapping ...........

5. BUCKET COMMUNICATION SPACE ..........................................

5.1 Overview ......................................................................

5.1.1 File Format Conversion .........................................

5.1.2 Metadata Conversion ............................................

5.1.3 Bucket Messaging ................................................

5.1.4 Bucket Matching .................................................

5.2 Implementation ..............................................................

5.2.1 Implemented Methods ..........................................

5.2.2 Changes from a Regular Bucket ................................

5.3 Discussion ....................................................................

5.3.1 Performance Considerations ....................................

5.3.2 Current Limitations ..............................................

6. BUCKET TESTBEDS ...............................................................

6.1 NCSTRL+ ....................................................................

6.1.1 Dienst ..............................................................

6.1.2 Clusters ............................................................

6.2 Universal Preprint Server ...................................................

6.2.1 Lightweight Buckets .............................................

6.2.2 SFX Reference Linking in Buckets .............................

7. RELATED WORK ....................................................................

7.1 Aggregation ...................................................................

7.1.1 Kahn/Wilensky Framework and Derivatives .................

7.1.2 Multivalent Documents .........................................

7.1.3 Open Doc and OLE ..............................................

7.1.4 Metaphoria ........................................................

7.1.5 VERS Encapsulated Objects ....................................

7.1.6 Aurora .............................................................

7.1.7 Electronic Commerce ............................................

7.1.8 Filesystems and File Formats ..................................

7.2 Intelligence ...................................................................

7.3 Archives ......................................................................

PAGE

52

52

53

53

56

57

60

60

61

61

62

62

63

63

68

69

69

72

74

74

74

76

77

78

79

83

83

83

83

84

84

84

85

85

86

86

87
vi

PAGE

7.4BucketTools ................................................................. 88

8. FUTUREWORK ..................................................................... 92

8.1AlternateImplementations................................................ 92

8.1.1Buckets............................................................ 92

8.1.2DumbArchives ................................................... 93

8.1.3BucketCommunicationSpace.................................. 93

8.2ExtendedFunctionality..................................................... 93

8.2.1Pre-definedPackagesandElements............................. 94

8.2.2XML Metadata................................................... 94

8.2.3MoreIntelligence................................................. 95

8.3Security,AuthenticationandTerms& Conditions ..................... 95

8.4New Applications........................................................... 97

8.4.1Discipline-SpecificBuckets..................................... 97

8.4.2UsageAnalysis................................................... 97

8.4.3SoftwareReuse................................................... 98

9. RESULTSAND CONCLUSIONS.................................................. 99

REFERENCES............................................................................ 102

APPENDICES

A. BUCKETVERSION HISTORY ...............................................

B. BUCKET API .....................................................................

C. DA API .............................................................................

D. BCSAPI ...........................................................................

114

118

150

156
vii

LIST OF TABLES

TABLE PAGE

1. System configurations used for bucket testing .................................... 25

2. Bucket API ........................................................................... 27

3. Reserved packages ................................................................... 32

4. Directives available in T&C files ................................................... 37

5. Bucket preferences .................................................................... 42

6.The archive design space ............................................................... 50

7. DA API ................................................................................. 52

8. OAi ---, DA mapping .................................................................. 59

9. BCS API ................................................................................ 64

10. UPS participants ..................................................................... 78
viii

LIST OF FIGURES

FIGURE PAGE

1. DL component technologies ........................................................... 9

2. Pyramid of publications for a single project/concept .............................. 11

3. Pyramid of publications rests on unpublished STI ................................ 12

4. STI lost over time ..................................................................... 15

5. Model of a typical NASA STI bucket ............................................... 21

6. Sample project bucket ................................................................ 22

7. Sample course bucket ................................................................. 23

8. Output of the "display" method ..................................................... 28

9. Thumbnails in the "display" method ............................................... 31

10. Bucket structure ...................................................................... 32

11. RFC- 1807 metadata .................................................................. 40

12. The three strata of DLs .............................................................. 48

13. The SODA publishing model ....................................................... 51

14. Population of the DA ................................................................ 54

15. DA query (?method=da put&adate=<20000101) ............................... 55

16. DA query (?method=da put&adate=19940101-20000101&subject=cs) .... 55

17. DA query (?method=da_put&subject=phys) .................................... 56

18. NACA bucket before similarity matching .......................................... 66

19. NACA bucket after similarity matching ........................................... 67

20. Sample similarity matching matrix .................................................. 70

21. Partitioning of the similarity matching matrix ..................................... 72

22. NCSTRL+ lineage .................................................................... 74

23. A UPS bucket with SFX buttons ................................................... 80

24. SFX interface .......................................................................... 81

25. Creation tool ........................................................................... 89
ix

FIGURE PAGE

26.Managementtool ...................................................................... 90

27.Administrationtool ................................................................... 91
CHAPTER ONE

INTRODUCTION

Although digital libraries (DLs) pre-date the World Wide Web (WWW) (BernersLee, Cailliau, Groff, &Pollermann, 1992), the popularity and prevalence of the WWW has

focused attention on DLs for both the general user and research communities. The WWW

provides ubiquitous access to distributed information content. However, finding

information in the WWW can be difficult. It is estimated that the best WWW search

engines contain less than 35% of the total indexable WWW, with some as little as 3%

(Lawrence & Giles, 1998). DLs are seen as a way to define gardens of information in a

vast, untamed forest of spurious information resources. DLs are now commonly used in

science, technology, arts and humanities. In some cases, they provide an on-line analogue

of traditional libraries, but without the geographic or temporal limitations. In other cases,

DLs are being used to create and disseminate collections of information that had not been

previously feasible or possible to collect in traditional libraries.

We begin with the observation that information content is more important than

the systems used to store and retrieve it. While this seems obvious enough, this fact is

often obscured during discussions of DLs. Instead, the focus of DL discussions is

primarily on the merits of specific relational database mangers (RDBMs), search engines,

the programming language or systems used, and other implementation specific details.

This is because when a specific DL implementation is chosen, the services it provides

(e.g., searching, browsing, document access) are often vertically integrated with the

content it services, sometimes done purposefully, in an attempt to control the intellectual

property rights to the object. However, such tight integration is at odds with the goals of

easily transitioning to future DL systems and concurrent support of multiple DL access

to a single collection of data objects. Even in many DL systems that have the direct goal

The journal model for this dissertation is the Journal of the American Society for

Information Science.
of having an open architecture, with multiple searching, browsing and other user interfaces

possible, there is an assumption of tightly tying the data objects to a single service that

controls their access. For example, in the open architecture DL proposal of Lagoze &

Payette (1998), the integration of repository and object is explicitly stated:

"The repository service provides the mechanism for the deposit,

storage, and access to digital objects. A digital object is considered

contained within a repository if the URN of that object resolves to the

respective repository (and, thus, access to the object is only available

via a service request to that repository)."

Our approach begins with promoting the importance of the information objects

above that of the DL systems used for their storage, discovery, and management. Within

the context of DLs, we make the information objects "first-class citizens". We propose

decoupling information objects from the systems used for their storage and retrieval,

allowing the technology for both DLs and information content to progress independently.

Paepcke (1996) argues that "searching is not enough" and that DLs need to provide a

wide range of value-added services, far more than DLs currently provide. We agree with

this position, and feel that dismantling the current stovepipe of "DL-archive-content" is

the first step in building richer DL experiences for users.

To demonstrate this partitioning between DLs, archives and information content,

we introduce "buckets". Buckets are aggregative, intelligent, object-oriented constructs

for publishing in digital libraries. They are partially similar in design to Kahn-Wilensky

Digital Objects (DOs) (Kahn & Wilensky, 1995), but with a few significant differences

and are optimized for DL applications. Although buckets could accurately be described as

"archivelets", the name "buckets" was chosen for several reasons: First of all it is easy to

pronounce and has a strong visual metaphor for its aggregation capability. Most

importantly, the target user community (not all of which are computer scientists) warmed

to it more than variations on "object", "package" and other popular computer science

terms.
Buckets exist within the "Smart Objects, Dumb Archives" (SODA) DL model

(Maly, Nelson, & Zubair, 1999). The SODA DL model dictates that functionalities

traditionally associated with archives are pushed down into the buckets, making the

buckets "smarter" and the archives "dumber". Some of a bucket's responsibilities

include: storing, tracking, and enforcing its own terms and conditions (T&C);

maintenance, display and dissemination of its contents; maintaining its own logs of

actions and errors; and informing appropriate parties when certain events occur. Buckets

provide mechanism, not policy. Buckets have no assumptions about their content, T&C,

their deployment profile or other matters. However, the mechanisms that buckets and

their related tools provide should be sufficient to implement an organization's policy.

The motivation for buckets came from previous experience in the design,

implementation and maintenance of NASA scientific and technical information (STI) DLs,

including the Langley Technical Report Server (LTRS) (Nelson, Gottlich, & Bianco, 1995;

Nelson & Gottlich, 1994), the NASA Technical Report Server (NTRS) (Nelson, Gottlich,

Bianco, et al., 1995), and the NACA Technical Report Server (NACATRS) (Nelson,

1999). Buckets can trace their evolution back to the NACATRS project, which featured

what we now call "proto-buckets". Objects in the NACATRS had many of aggregation

features of buckets, but lacked the additional features such as intelligence and did not have

a well-defined application programming interface (API).

In early user evaluation studies on these DLs, one reoccurring theme was detected.

While access to the technical report (or re/pre-print) was desirable, users particularly

wanted access to the raw data collected during the experiments, the software used to

reduce the data, and the ancillary information that went into the production of the

published report (Roper, McCaskill, Holland, et al., 1994). The need for NASA research

projects to deliver not just a report, but also software and supporting technologies was

identified as early as 1980 (Sobieski, 1994), but NASA's treatment of non-report STI has

remained uneven. Reports continue to receive the primary focus, and the interest and

capacity to archive and disseminate other information types (data, notes, software, audio,
video)ebbsandflows. Theinteresthereisto createasetof capabilitiesto permit DLs to

accommodaterequestsfor substantially more information than just finalized reports.

However, rather than setup separateDLs for each information type or stretch the

definitionof a traditionalreportto includevariousmulti-mediaformats,the desirewas to

defineanarbitrary digitalobjectthat couldcaptureandpreservethepotentially intricate

relationshipbetweenmultiple informationtypes.

Additionally, our experienceswith updating the DLs and makingthe content

accessiblethrough other DLs and web-crawlersled to the decision to make the

informationobjectsintelligent. Wewantedthe objectsto receivemaximumexposure,so

we did not want them "trapped" inside our DLs, with the only method for their

discoverycomingfrom ourDL interface.However,theDL shouldhavemorethanjust an

exportabledescriptionof how to accessthe objectsin the DL. The information object

shouldbeindependentof theDL, with thecapabilityto existoutsideof the DL and move

in andoutof differentDLs in thefuture.However,to not assumewhich DL wasusedto

discoverand accessthe bucketsmeansthat the bucketsmust be self-sufficient and

performwhatevertasks arerequiredof them, potentially without the benefit of being

arrivedat througha specific DL. Multiple implementationsof buckets are possible.

However,for thebucketimplementationpresentedhere,the followingrequirementsmust

bemet forthecomputerhostingthebuckets:

a hypertext transferprotocol (http) (Fielding,Gettys, Mogul, et al., 1999)

serverthatimplementsthecommongatewayinterface(CGI) specification.

aPerl5 interpreter(Wall, Christiansen,& Schwarz,1996)that the bucketcan

find.

As longasthesetwo requirementsaremet,thebucketswill be ableto function.

Thebucketshavea"bunker"mentality:evenif thevarioussearchengines,DLs andother

resourcesnormally usedfor their discovery moves,breaks,or otherwisedegenerates,
bucketsshouldcontinueto function. The well beingof a bucketdependson the lowest

possiblecommondenominator:a CGI http serverandPerl interpreter,andnot on more

complexandpossiblytransientDL services.

The outlinefor the rest of this thesis is as follows: ChapterTwo provides the

motivationfor DLs andbuckets,anddesigngoalsof buckets. ChapterThreediscusses

thebucketarchitectureand implementation. ChapterFour discussesthe dumb archive

architecture and implementation. Chapter Five discusses the architecture and

implementationof theBucketCommunicationSpace.ChapterSixdescribeshow buckets

wereusedin two prototype DLs: NCSTRL+ andtheUniversalPreprintService(UPS).

ChapterSevencomparesand contrastsbucketswith relatedwork, and Chapter Eight

discussessomeof the possiblefuture work. ChapterNine providesthe conclusionsand

summary.
CHAPTER TWO

MOTIVATION AND OBJECTIVES

2.1 Why Digital Libraries?

The preservation and sharing of its intellectual output and research experiences is

the primary concern for all research institutions. However, in practice information

preservation is often difficult, expensive and not considered during the information

production phase. For example, Henderson (1999) provides data showing for the period

of 1960-1995 that "knowledge conservation grew half as much as knowledge output", as a

result of research library funding decreasing relative to increasing research and

development spending (and a corresponding increase in publications). In short, more

information is being produced, and it is being archived and preserved in fewer libraries,

with each library having fewer resources. Though eloquent arguments can be presented

for the role for and purpose of traditional libraries and data can be presented for the

monetary savings libraries can provide (Griffiths & King, 1993), the fact remains that

traditional libraries are expensive. Furthermore, the traditional media formats (i.e. paper,

magnetic tapes) housed in the traditional libraries are frail, requiring frequent upkeep and

are subject to environmental dangers (Lesk, 1997; United States General Accounting

Office, 1990). DL technologies have allowed some commercial publishers to become

more involved with library functions, serving on the WWW the byproducts of their

publishing process (PostScript, PDF, etc.). However, ultimately the goals of publishers

and the goals of libraries are not the same, and the long-term commitment of publishers to

provide library-quality archival and dissemination services is in doubt (Arms, 1999).

While not a panacea, an institution's application of DL technologies will be an integral

part of their knowledge usage and preservation effort, in either supplanting or

supplementing traditional libraries.
All of this has tremendousimpact on a U.S. Governmentagencylike NASA.

Beyond attention grabbingheadlinesfor its various spaceprograms,NASA ultimately

producesinformation. The deliverablesof NASA's aeronauticaland spaceprojects are

information for either a targetedset of customers(e.g., Boeing) or for scienceand

posterity. The informationdeliverablescanhavemany forms:publicationsin the open

literature;a self-publishedtechnicalreport series;and non-traditionalSTI mediatypes

suchasdataandsoftware. NASA contributionsto the openliteraturearesubjectto the

samewideninggap in conservationand output identified by Henderson(1999). For

some,theNASA report seriesis either unknown or hard to obtain (Roper,McCaskill,

Holland,et al., 1994). For sciencedata,NASA haspreviously beencriticizedfor poor

preservationof this data(United StatesGeneralAccountingOffice, 1990). However,

NASA has identifiedandis addressingtheseproblemswith ambitiousgoals. From the

NASA STIProgramPlan(NASA, 1998):

"By theyear 2000,NASA will captureanddisseminateall NASA STI

andprovideaccessto moreworldwide mission-relatedinformationfor

its customers.Whenpossibleandeconomical,this informationwill be

provided directly to the desktop in full-text format and will include

printed material,electronic documentation,video, audio, multimedia

products, photography, work-in-progress, lessons-learned data,

researchlaboratory files, wind tunnel data, metadata,and other

informationfromthescientificandtechnicalcommunitiesthat will help

ensure the competitiveness of U.S. aerospace companies and

educationalinstitutions."

Althoughtemperedwith thephrase"possibleandeconomical",it is clearthat the

expectationsaremuchhigherthansimply automatingtraditionallibrary practices. Much

of theSTIidentifiedabovehashistoricallynotbeenincludedin traditionallibrary efforts,

primarily becauseof themismatchin hard-andsoft-copy mediaformats. However,the

ability to now documenthe entireresearchprocessandnotjust the finalresultspresents

entirely new challengesabout how to acquireand managethis increasedvolume of

information. To effectively implementthe abovemandate,additionalDL technologyis

required.
2.1.1 Digital Libraries vs. the World Wide Web

A common question regarding DLs is "Why not just use existing WWW

tools/methods?" Indeed, most DLs use the WWW as the access and transport

mechanism. However, it is important to note that while the WWW meets the rapidity

requirement of STI dissemination, it has no intrinsic management or archival functions.

Just as a random collection of books and serials do not make a traditional library, a

random collection of WWW pages does not make a DL. A DL must possess acquisition,

management, and maintenance processes. These processes will vary depending on the

customers, providers and nature of the DL, but these processes will exist in some format,

implicitly or explicitly.

There have been proposals to subvert the traditional publication process with

authors self-publishing from their own WWW pages (Harnad, 1997). However, while

this availability is useful, pre-prints (or re-prints) linked from a researcher's personal

home page are less resilient to changes in computer infrastructure, organization changes,

and personnel turnover. Ignoring the socio-political issues of (digital) collegial

distribution, there is an archival, or longevity, element to DLs which normal WWW usage

does not satisfy. The average lifetime of a uniform resource locator (URL) has been

estimated at 44 days (Kahle, 1997), clearly insufficient for traditional archival

expectations. Uniform Resource Names (URNs) can be used to address the transient

nature of URLs. URNs provide a unique name for a WWW object that can be mapped to

a URL by a URN server. The relationship between URNs and URLs is the same as

Internet Protocol (IP) names and IP addresses, respectively. CNRI Handles (Sun &

Lannom, 2000), Persistent URLs (Purls) (Sharer, Weibel, Jul, & Fausey, 1996) and

Digital Object Identifiers (DOIs) (Paskin, 1999) are some common URN

implementations. However, no URN implementation has achieved the ubiquity of URL

use, and significant maintenance is required to keep a large collection of URNs current. In

summary, a DL defines a well-known location for STI to be placed, managed, and
accessed. Given the prevalence of the WWW, the well-known location that a DL

provides is likely to be WWW accessible.

WWW (http) Access

(most common)

non-WWW

Access

(now uncommon)

Digital Library Services

(searching, browsing, citation anlaysis

usage analysis, alerts)

Vector

and/or

Boolean

Search Engines

(traditional IR)

RDBMS

File

Systems

Other

Technologies

Content

FIG. 1. DL component technologies.

2.1.2 Digital Libraries vs. Relational Database Management Systems

Perhaps the second most common question after "Why not just use the WWW?"

is "Why not just use a database?" The answer to the database question is subtler.

Generally relational databases are less well suited for more generalized information

retrieval (IR) requirements typical of library applications, which often feature boolean or

vector search engines. Two main differences between traditional IR systems and relational

databases management systems (RDBMS) is that the data objects in IR systems are

documents, which are less structured than the tables of relations which are the data

objects in RDBMs (Frakes & Baeza-Yates, 1992). Also, retrieval in an IR system is

probabilistic, as opposed to deterministic in a RDBMS (Frakes & Baeza-Yates, 1992).

Some commercial and professional society DLs are constructed with web pages indexed
10

by traditional IR search engines, including the Institute for Electrical and Electronics

Engineers (IEEE) Computer Society DL, which uses the "Autonomy" search engine and

D-Lib Magazine DL which uses the "Excite" search engine.

However, it is possible to use a RDBMS to build a DL, especially if high-quality,

structured metadata is available from which tables can be built. This is the approach of

the IBM "DB2 DL" commercial product and the Association for Computing Machinery

(ACM), which uses an "Oracle" RDBMS for its DL. A DL is the union of its content and

the services it provides on that content. A traditional IR search engine or a RDBMS,

insofar as they provide only a single service (searching), are just components of a DL, not

the DL itself. The relationship between the WWW, traditional IR search engines, DLs,

RDBMS, and other technologies is illustrated in Figure 1.

2.2 Trends in Scientific and Technical Information Exchange

Rapidity and breadth of communication have always been significant requirements

in the exchange of STI. Scientific journals evolved in the 17th century to replace the

system of exchanging personal letters between scientists, which evolved because of

unacceptable delays in publishing books (Odlyzko, 1995). However, journals are no

longer used for rapid communication, but rather as "a medium for priority claiming,

quality control and archiving scientific work." (Bennion, 1994). To achieve rapid

communication of STI, different disciplines have adopted various models. Starting in the

1960's, "Letters" journals began to appear in some disciplines to offer more rapid

dissemination of research results, while in other disciplines the pre-print or technical

report emerged as the rapid dissemination vehicle (Vickery, 1999). In computer science,

the technical report is a common unit of exchange. In disciplines such as high-energy

physics, the pre-print culture is well established. Paul Ginsparg, a physicist active in

digital libraries, notes that "The small amount of filtering provided by refereed journals

plays no effective role in our research." (Ginsparg, 1994). While noting that not all

disciplines embrace the pre-print / technical report culture equally, Odlyzko (1995) states

"it is rare for experts in any mathematical subject to learn of a major new development in
11

theirareathroughajournal publication"andalsorelatescommentsby computerscientists

RobPike("that in his areajournals havebecomeirrelevant")andJoanFeigenbaum("if it

didn't happenat aconference,it didn't happen").

Ajournal articleis oftenonly a fractionof theavailabletechnicalliteratureabouta

givensubject.Theses,dissertations,conferencepapers,andtechnicalreports areknown

as "grey literature" and receivevarying degreesof peerreview. "White literature,"

availablethroughstandardpublicationschannelsandprocesses,is often supportedby a

largerbody of grey literature. The role of the largeamountof grey literature and its

relation to the smaller amount of white literature, and the issues associatedwith

integratingthe two havebeenpresent sincethe post-World War II U.S. Government

sponsoredresearchboom(Bennington,1952;Gray,1953;Scott,1953). David Patterson,

co-inventorof the RISCcomputerchip, notedthat in oneof his first researchprojects,

the output was 2 journal articles, 12 conferencepapers, and 20 technicalreports

(Patterson,1994). If we considerthispyramidof publications(Fig. 2) to be typical, then

ajournal articleactuallyfunctionsasanabstractof alargerbody ofSTI.

tim

JournalArticles

ConferencePapers

TechnicalReports

FIG.2.Pyramidof publicationsfor asingleproject/concept.

It is estimatedthat thereareover 100,000domestictechnicalreportsproduced

annually(Esler& Nelson, 1998).The result is that even if there are 20,000 primary

researchjournals (Bennion,1994),they do not representthe entirety of STI. These

numbersdo not include 1) confidential,secret,proprietary, and otherwise restricted

reports;or 2)non-report STI, suchas computersoftware,datasets, video, geographic
12

data,etc. Indeed,anecdotalevidencesuggeststhattheWWW is notjust arapid transport

mechanismfor white andgrey literature,but collectionsof WWW pagesarebecominga

newunit of STIexchangeaswell. Figure3 showsthe Pyramid of Publicationsdescribed

in Figure2restingonalargerbodyof unpublishedSTI.

tim_

JournalArticles

/ _ Conference Papers

_ware raw data notes VmidaegO]_ Technical RepOrts

FIG. 3. Pyramid of publications rests on unpublished STI.

Schatz and Chen (1996) give a summary of the Digital Library Initiative (DLI)

projects focusing on building large digital libraries of non-report STI. However, these

efforts can be summarized as propagating a "separate but equal" philosophy with regards

to non-report STI. Instead of integrating software, datasets, etc. into the same DL, which

contains the reports, separate DLs are created for the new collection. The researcher is

still left to reconstruct the original information tuple by integrating search results from

various DLs. The DLI2 initiative (Lesk, 1999; Griffin, 1999), a follow-on to the 1994-

1998 DLI, is funding a broader range of DL projects, including a great number with focus

on non-report literature. However, these projects still do not focus on redefining the

output of the STI research process. We consider "separate-but-equal" DLs to be harmful.

For example, no matter how sophisticated a video DL becomes, the video should never be

de-integrated from the data sets that supplement the video, the software used to process

the data sets, and the report that documents the entire project. The limitations of current

STI exchange mechanisms can be summarized as follows:
13

highly focused on journal articles, despite their decreasing value to researchers

and practitioners in some fields;

inadequate acquisition of grey literature, the grist of technical exchange; and

inability to integrate non-publication media, such as datasets, software, and

video.

These limitations are largely side effects of the hard copy distribution paradigm.

As STI exchange moves toward electronic distribution, existing mechanisms should not

merely be automated, but the entire process should be revisited.

2.3 Information Survivability

The longevity of digital information is a concern that may not be obvious at first

glance. While digital information has many advantages over traditional printed media,

such as ease of duplication, transmission and storage, digital information suffers unique

longevity concerns that hard copy does not, including short life spans of digital media

(and their reading devices) and the fluid nature of digital file formats (Rothenberg, 1995;

Lesk, 1997). The Task Force on Archiving of Digital Information (1996) distinguished

between: refreshing, periodically copying the digital information to a new physical media;

and migrating, updating the information to be compatible with a new hardware/software

combination. Refreshing and migrating can be complex issues. The nature of refreshing

necessitates a hardware-oriented approach (perhaps with secondary software assistance).

Software objects cannot directly address issues such as the lifespan of digital media or

availability of hardware systems to interpret and access digital media, but they can

implement a migration strategy in the struggle against changing file formats. An

aggregative software object could allow for the long-term accumulation of converted file

formats. Rather than successive (and possibly lossy) conversion of:

Formatl --) Format2 --) Format3 --) .... --) FormatN
14

We shouldhavetheoptionof:

Formatl --) Format2

Formatl --) Format3

Formatl --) ....

Formatl --) FormatN

With eachintermediateformatstoredin thesamelocation. Thiswouldallowus to

implementthe "throw away nothing"philosophy, without burdeningthe DL directly

with increasingnumbersof formats.

For example,a typical researchproject at NASA Langley ResearchCenter

produces information tuples: raw data, reduceddata, manuscripts,notes, software,

images,video, etc. Normally, only the reportpart of this informationtuple is officially

publishedandtracked. The report might referenceon-line resources,or evenincludea

CD-ROM, but theseitemsarelikely to belost, degrade,or becomeobsoleteovertime.

Someportions suchas software, can go into separatearchives(i.e., COSMIC - the

official NASA softwarerepository)but this leavesthe researcherto locatethe various

archives,thenre-integratetheinformationtupleby selectingpiecesfrom thedifferent,and

perhaps,incompatiblearchives. Most often, the software and other items, such as

datasetsaresimplydiscardedor effectivelylostin informal,short-livedpersonalarchives.

After 10 years, the manuscript is almost surely the only surviving artifact of the

informationtuple. Thefatetypical of variousSTI datatypesis depictedin Figure4.

As anillustration,COSMIC ceasedoperationin July 1998;its operationswere

turnedoverto NASA's technologytransfercenters.However,at the time of this writing

there appears to be no operational successorto COSMIC. Unlike their report

counterpartsin traditional librariesor evenDLs suchasLTRS, the softwarecontentsof

COSMIChavebeenunavailablefor severalyears,if notcompletelylost.

Additionalstepscanbetakento insurethesurvivabilityof the informationobject.

Datafiles could be bundledwith the applicationsoftwareused to processthem, or if
15

commonenough,differentversionsof theapplicationsoftware,with detailedinstructions

aboutthehardwaresystemrequiredto runthem,couldbeapartof the DL. Furthermore,

theycouldincludeenoughinformationto guidethefutureuserin selecting(or developing)

thecorrecthardwareemulator.

Project _ m uscript software it own tipbr site awayI User New raw data Project

images • filing cabinent

project archival---------_ reuse

FIG. 4. STI lost over time.

2.4 Objectives and Design Goals

The objectives of this research are as follows:

1.

.

3.

Develop "buckets" - a collection of mechanisms and protocols to aggregate,

protect, manage, and mobilize content and basic services.

Develop a reference implementation of buckets based on http, CGI and Perl.

Evaluate the concept and reference implementation in different application

domains.

The development of buckets is guided by a number of design goals. As suggested

by the SODA model, buckets have unique requirements due to their emphasis on

minimizing dependence on specific DL implementations. The design goals are:

aggregation, intelligence, self-sufficiency, mobility, heterogeneity and archive

independence.

2.4.1 Aggregation

As outlined in the discussion given above, DLs should be shielded from the

transient nature of data file formats and the information object should be allowed to

evolve independently of the system(s) users employ to discover the information object.
16

Furthermore,a trendwasnotedthatDLs wereoftenbuilt aroundthe now obsoletemedia

boundariesof traditionallibraries(Esler& Nelson, 1998):technicalreports existedin a

technicalreportDL, imagesexistedin animageDL, softwareexistedin a softwareDL. A

thesisof this study is that sinceall of theseobjectswere createdat the sametime and

potentially havesubtlerelationshipsbetweenthem,they shouldbeableto be storedin

the sameinformationobject. In NASA DLs therewas aninformation hemorrhaging:a

suiteof informationobjectswould be prepareddigitally, but sincethe DLs could only

accepta singleinformationobject(report or re-print), the other objectswere left to be

archivedanddistributedinformally,if at all.

With decreasingcostsof physical storagemedia,the cost of not savingdatasets

becomesmoreexpensivethansavingthem. Somedata,suchas time-dependentsatellite

dataor pilot-in-the-loop flight simulations,cannotbe replacedor recreated. Buckets

providea way to aggregateall therelatedinformationobjects,which couldbeuseful for

future, possibly unknown uses into a single containerobject that provides easeof

maintenance.Bucketscan also be used to aggregatethe successivemigrationsof an

informationobjectfrom onehardware/softwaresystemto thenext.

2.4.2 Intelligence

Yet another design goal of buckets is that they be autonomous and active, not

passive and tied to a server. Buckets should be able to perform and respond to actions on

their own and have them be active participants in their own state and existence. Buckets

do not necessarily have to reveal their intelligence in interaction with users, but rather in

interaction with tools and other buckets. Making information objects intelligent opens the

door for whole new realms of applications. Although some bucket applications are

obvious, such as making information objects computational entities and self-arranging,

most others remain undiscovered.

2.4.3 Self-Sufficiency

For maximum autonomy, the default configuration for buckets is to contain all

their code, data, user and password files, and everything else they need physically inside
17

thebuckets.Optimizationsexistin which codeandpasswordfiles canbe"factored" out

of the buckets,resultingin storagesavingsand easiermanagement.However, these

savingscomeat thecostof decreasedautonomyandmobility (seebelow). Givenproper

tools,it isexpectedself-sufficientbucketscanbeeasilymanagedandtheincreasedstorage

overheadisnegligiblegiventhatstorageis "cheap".

As for data,bucketscanstoredataphysically insidethe bucket,or simply store

"pointers"to dataobjectsthat existoutsidethebucket. Internal datastorageis preferred,

however somedata (e.g.,databaseviews) makesenseto store as dynamic pointers.

Although bucketscanstore either physical copiesor pointers, bucketsobviously can

makenoguaranteesaboutthelong-termsurvivability of itemsthat lie outsidethebucket.

Buckets should provide the mechanismsto implementthe internal vs. externaldata

storagepoliciesfor specificapplications.

2.4.4 Mobility

Related to self-sufficiency, buckets can be mobile. That is, they can physically

move from place to place since they contain all the code and support files they need.

Furthermore, placing a bucket on a host should require no modifications to the http

server. For technical reports and re-prints, the need for mobility is not obvious, beyond

the role that it plays in assisting information refreshing. However, an application

proposed by the Air Force illustrates the power of mobility. In their plan, buckets are

used to represent people and the buckets store supplemental human resources (HR)

information (papers published, personnel reviews, CV materials, etc.) As people move

between Air Force installations, their HR bucket moves with them, "plugging-into" the

HR system at the host installation.

Mobility can be used in other situations where we wish to move the buckets in

response to a particular workflow model. Rather than requiring the bucket to be anchored

in a particular spot, it would be possible for a bucket to travel from place to place, and be

local to the system that it is sampling data from. After collecting data at its various
18

stops,it couldthenbemovedto a locationwhereit is visibleto a DL, andbeindexedand

foundby users.

2.4.5 Heterogeneity

A significant requirement for buckets is that they all do not have to look or act the

same. It is possible for different installations to locally modify the buckets created at that

site to reflect their specific publishing policy or take advantage of known characteristics

of the data they store. Similarly, it is possible for buckets to evolve differently over time,

with new methods being added, deleted or overridden as appropriate. Furthermore, it

should be possible to publish buckets with entirely different structure and functionality,

based on what discipline the buckets support. Intuitively, an earth science bucket and a

biomedical engineering bucket should at least have the option of looking and acting

differently. However, buckets should retain enough basic methods so their version and

features can be dynamically discovered.

2.4. 6 Archive Independence

To the extent reasonable, buckets should work with any type of archive and/or

DL. Similarly, they should not break any archive or DL. In fact, archives and DLs are

not required for bucket operation. It should be possible for buckets to be indexed in any

number of DLs. Archives, DLs, search engines, etc. are not intrinsic to the operation of

buckets - they are add on services that can be used in management and resource discovery

and should be completely decoupled from the buckets themselves.
19

CHAPTER THREE

BUCKET ARCHITECTURE

3.1 Overview

A bucket is a storage unit that contains data and metadata, as well as the methods

for accessing both. It is difficult to overstress the importance of the aggregation design

goal. In our experience with other NASA DLs, data was often partitioned by its semantic

or syntactic type: metadata in one location, PostScript files in another location, PDF files

in still another location, etc. Over time, different forms of metadata were introduced for

different purposes, the number of available file formats increased, the services defined on

the data increased, new information types (software, multimedia) were introduced, the

logging of actions performed on the objects became more difficult. The result of a report

being "in the DL" eventually represented so much DL jetsam - bits and pieces physically

and logically strewn across the system. We responded to this situation with extreme

aggregation.

The first focus of the aggregation was for the various data types. Based on

experience gained while designing, implementing and maintaining LTRS and NTRS, we

initially decided on a two-level structure within buckets:

buckets contain 0 or more packages

packages contain 0 or more elements

Actual data objects are stored as elements, and elements are grouped together in

packages within a bucket. In LTRS and NTRS, a two-level architecture was sufficient for

most applications, so this two-level architecture was retained as a simplifying assumption

during bucket implementation. Future work will implement the semantics for describing

arbitrarily complex, multi-level data objects.
20

An elementcanbe a "pointer" to anotherobject:anotherbucket,or any other

arbitrary network object. By havingan element"point" to otherbuckets,bucketscan

logically contain other buckets. Although bucketsprovide the mechanismfor both

internalandexternalstorage,bucketshavelesscontrol overelementsthat lie physically

outside the bucket. However, it is left as a policy decisionto the user as to the

appropriatenessof includingpointersin anarchivalunit suchasabucket. Bucketshave

nopredefinedsizelimitation,eitherin termsof storagecapacity,or in termsof numberof

packagesor elements. Bucketscanuse a CNRI handle,a URN implementation,for a

globallyuniqueid. Bucketsareaccessedthrough1 or moreURLs. For an exampleof

how a singlebucket canbe accessedthroughmultiple URLs, considertwo hosts that

shareafile system:

http://hostl.foo.edu/bar/bucket-27/

http://host2.foo.edu/bar/bucket-27/

Both of these URLs point to the same bucket, even though they are accessed

through different hosts. Also, consider a host that runs multiple http servers:

http://hostl.foo.edu/bar/bucket-27/

http://hostl.foo.edu:8080/bucket-27/

If the http server running on port 8080 defines its document root to be the

directory "bar", then the two URLs point to the same bucket.

Elements and packages have no predefined semantics associated with them.

Authors can model whatever application domain they desire using the basic structures of

packages and elements. One possible model for bucket, package, and element definition is

based on NASA DL experiences. In Figure 4, packages represent semantic types

(manuscript, software, test data, etc.) and elements represent syntactic representations of
21

the packages (a . ps version, . pdf version, . dr2 version, etc.). Other bucket models

using elements and packages are possible. For example, we have used buckets for entire

research projects (Fig. 6) and university classes (Fig. 7) as well as for STI publications.

Though the display of the two buckets is different, the two-level architecture of packages

and elements is evident.

Buckets have the capability of implementing different policies as well: one site

might allow authors to modify the buckets after publishing, and another site might have

buckets be "frozen" upon publication. Still another site might define a portion of the

bucket to receive annotations, review, or contributions from the users, while keeping

another portion of the bucket frozen, or only changeable by authors or administrators.

Buckets provide mechanism, not policy.

Packages

inside the

bucket

CNRI Handle Access Methods

(unique id)

Terms and Conditions ]

Metadata (RFC 1807, Dublin Core) ]

Manuscript .ps .pdf.tex .doc • ]

Software .tar .c .java _-_"-__

images .gif .jpeg

I data sets .xls .tar

Elements inside

the package

FIG. 5. Model of a typical NASA STI bucket.
22

_ii_iiiiiiiiiiiiiiii!ii":_iiiiiiiiiiiiiiiiiiiiiiii _ _i__i__iiiiiiiiiiiiiiiiiiiii_iiii_i"::iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii _ !i __:_::_"iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_!ii:iiiiiiiiiiiiiiiiiiiiiiiii_i_i_":i _ _ _:iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii!i i_ :_:_iiiiiiiiiiiiiiiiiiiiiiii_i_i_i:: _ _::iiiiiiiiiiiiiiiiiiii __!i ':::_ _ ..:.___: __:_:_:_:_:_iii i !i!iiiiiiiiiii iiiiiiiiiiiiiiii! ii i i iiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii ! iiiiiiiiiiiiiiiii iiiiiiiii !i iiiiiii

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiii__ii_i_i!i_

FIG. 6. Sample project bucket.

Another focus of aggregation was including the metadata with data. In previous

experiences, we found that metadata tended to "drift" over time, becoming decoupled

from the data it described or "locked" in specific DL systems and hard to extract or share

with other systems. For some information types such as reports, regenerating lost

metadata is possible either automatically or by inspection. For other information types

such as experimental data, the metadata cannot be recovered from the data. Once the

metadata is lost, the data itself becomes useless. Also, we did not want to take a

proscriptive stance on metadata. Although the bucket itself has to ultimately chose one
23

metadata format as canonical for storing and modifying its internal structure information,

buckets needed to be able to accommodate multiple metadata formats. Buckets do this by

storing metadata in a reserved package and using methods for reading and uploading new

metadata formats as elements in the metadata package. As a result, buckets can

accommodate any number of past, present or future metadata formats.

[itlei introduction to Digit_d Libr_iee

Kurt FA_I9 (m_lg@c_,odu.edu) _d MiChOLel Nelson (m.l.nel_on@l_rc.na$_,gov)

Spring 1999

_p0_i ODU C$ 495/595

Digit0LILibr_Lrie_ (DL_) are incro_singl9 popUl0Lrro_e_ch _ro_ that encomp_us_ more than

tr_lition_l intonn_ion rotriee_l or d_,a, se method_ and techniques. We will couer _ brief

hi_tor9 of DL deeelopment_ with empha, si_ on ',_'orld '_'ide Web implement_ion_. C_,se

_tudie_ _ill be performed on e_iou_ DL_. The cl_,s_ _ill focu_ he_uilQ on project wol'k._

e_pecioLIIy writing _d/or coding, At the end of the ¢ouroe_ _tudent$ will be prepared to

develop_ evalu_e_ or apply digital libr'ar9 technologiee in their work environment,

idi ncstdpiue,odu,c_//dl-cl_u'_

Gene_Infom_iO

__H__ (1 I_)

Referonce_ used in -_,|ides (incomplete) (9 KB)

._P__i_e__t _!__a__e___

Nefl_on'_ homep_0_lle

_e_rch Lo_ {M_ WoO) 125 Ke)

Time |_oq (NS WoN) (24 It,B)

_(TSO Ke)

_(2 Ke)

_(203 Ke)

_l!-____e_______i__n__,rtL_, (1 KB)

(_4 Key

_v'_ne_0_', _(2 KB)

N_La__e____0____l.(127 Ke)

FIG. 7. Sample course bucket.

The final aggregation focus was on the services defined on buckets and the results

of those services. The default state is for everything the bucket needs to display,

disseminate, and manage its contents is contained within the buckets. This includes the
24

sourcecodefor all the methodsdefinedon thebucket, the userids and passwords,the

accesscontrol lists, the logsof actionstakenon thebucket,Multipurpose Internet Mail

Extensions(MIME) (Borenstein& Freed, 1993) definitions and all other supporting

technologiesnecessaryfor the bucket to function. The self-sufficiencyand mobility

designgoalsdictatethata bucketcannotmakemanyassumptionsaboutthe environment

thatit will residein andshouldrequireno servermodificationsto function.

3.2 Implementation

The buckets described in this chapter are version 1.6. Appendix A lists the full

bucket history. Buckets are currently written in Perl 5 and use http as the transport

protocol. However, buckets can be written in any language as long as the bucket API is

preserved. Buckets were originally deployed in the NCSTRL+ project (Nelson, Maly,

Shen, & Zubair, 1998), which demonstrated a modified version of the Dienst protocol

(Lagoze, Shaw, Davis, & Krafft, 1995). Owing to their Dienst-related heritage, bucket

metadata is stored in RFC-1807 format (Lasher & Cohen, 1995), with package and

element information stored in NCSTRL+ defined optional and repeatable fields. Although

buckets use RFC-1807 as their native format, they can contain and serve any metadata

type. Dienst has all of a document's files gathered into a single Unix directory. A bucket

follows the same model and has all relevant files collected together using directories from

file system semantics. The bucket is accessible through a CGI script that enforces terms

and conditions, and negotiates presentation to the WWW client.

Aside from Perl 5, http, and CGI, buckets make no assumptions about the

environment in which they will run. Mobility is one of the design goals of buckets, and a

corollary of that is that buckets should not require changes in a "reasonable" http server

setup; where "reasonable" is defined to be allowance of the index, cg± convention.

Once these assumptions have been met, buckets by default take care of everything

themselves with no server intervention, including MIME typing, terms and conditions,

and support libraries. Although bucket development was conducted under Solaris (Unix),

buckets have been tested on a variety of system configurations (Table 1).
25

TABLE 1.Systemconfigurationsusedfor buckettesting.

Architecture OperatingSystem Perl http server

Sparc Solaris2.7 5.005_03

Sparc Solaris2.7 5.005_03

Sparc RedHat6.0(Linux 5.005_03

2.2.5-15)

Intelx86 WindowsNT 4.0

(1381/ SP 5) 5.005_03

Intel x86 Mandrake Linux 6.2 5.005 03

MIPS R10000 IRIX 6.5 5.004 04

RS/6000 AIX 4.2 5.002

PowerPC 604 Linux 2.0.33 5.004 01

(MkLinux)

Apache 1.3.9

NCSA httpd 1.5.2

Apache 1.3.6

Active Perl Apache 1.3.12

Apache 1.3.6

Apache 1.3.4

Apache 1.3.12

Apache 1.2.6

The biggest difficulty in mobility across multiple platforms is locating the Perl

interpreter. Buckets use the Unix-style "#!" construct to specify which interpreter

should be used to process the script. For example, the first line in index, cg± script in

bucket version 1.6 is:

#!/usr/local/bin/perl

Which explicitly specifies where Perl is expected to be found. On Unix systems,

this is generally not a problem, since any of the following values are generally at least

symbolic links to the canonical location of the Perl interpreter:

/usr/local/bin/perl

/usr/bin/perl

/bin/perl
26

However, for Windows NT systems, Perl generally exists in a different location

altogether, and the default Unix values are less likely to work. It is possible on the

Windows NT version of Apache to bind the Perl interpreter to all scripts ending in . cg±,

but for testing on our Windows NT system, the first line of the index, cg± script was

changed to be:

#!\Perl\bin\perl.exe

For greater Unix portability there is a standard trick to gain slightly more

portability. It is possible to replace the first line of the ±ndex. cg± script to contain:

#!/bin/sh -- # -*- perl -*-

eval 'exec perl --S $0'

if O;

This invokes the Bourne shell and determines where Perl exists on the host by

using the first value it finds in the $PATH environment variable. However, since this

depends on the Bourne shell, it is even less likely to work on Windows NT systems than

the current #! value. A general purpose bootstrapping procedure to specify the Perl

interpreter has not been found.

3.2.1 Bucket Methods

Communication with buckets occurs through a series of bucket messages defined

by the bucket API. The list of defined bucket methods is given in Table 2, and the

bucket's detailed API is in Appendix B. Note that these are methods defined for our

generic, all-purpose buckets. It is expected that local sites will add, delete and override

methods to customize bucket structure details to their own requirements. It is important

to note that regular users are not expected to directly invoke methods - the users require

no special knowledge of buckets. All the user needs is the initial URL pointing to the

bucket, and then the applicable methods for accessing its contents are automatically built

into the bucket's HTML output. The other creation and management-oriented methods

are expected to be accessed by a variety of bucket tools.
27

TABLE 2. Bucket API.

Method Description

add element

add method

add_package

add_principal

add tc

delete bucket

delete element

delete_log

delete method

delete_package

delete_principal

delete tc

display

get log

get_preference

get_state

id

lint

list_logs

list methods

list_principals

list source

list tc

metadata

pack

Adds an element to a package

Adds a method to the bucket

Adds a package to the bucket

Adds a user id to the bucket

Adds a T&C file to the bucket

Deletes the entire bucket

Deletes an element from a package

Deletes a log file from the bucket

Deletes a method from the bucket

Deletes a package from the bucket

Deletes a user id from the bucket

Deletes a T&C file from the bucket

Displays and disseminates bucket contents

Retrieves a log file from the bucket

Retrieves a preference(s) from the bucket

Retrieves a state(s) from the bucket

Displays the bucket's unique id

Checks the buckets internal consistency

Lists all the log files in the bucket

Lists all the methods in the bucket

Lists all the user ids in the bucket

List the method source

Lists all the T&C files in the bucket

Displays the metadata for the bucket

Returns a "bucket-stream"
28

set metadata

set_preference

set state

set version

unpack

version

Uploadsametadatafile to thebucket

Changesabucketpreference

Changesabucketstatevariable

Changestheversionof thebucket

Overlaysa "bucket-stream"into thebucket

Displaystheversionof thebucket

_::::::::::: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

ii iiiiiiii iiiiiiiiiiiii®iiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiii iiiiiiiiiii iiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiii!i!iiiii 

::::::::::::::::: ................................... _............................. } ::,_,:.4_: ================================= ......................... ::

title: A self-synchronizing stroboscopic Schlieren system for the study of unsteady air flows

anthers: Lawrence, Leslie F, SchmJdl, Stanley F and Looschen, Floyd W

0_liZ_i0ll: NAOA Ames Aeronanticat Laborotor9 (Moffett Field, Calif., United States)

d_e: October 1951

NACA TN-_'509

A self-synchronizincj stroboscopic schlieron s9stem developed for the visualization of unsteady air flows

about aerodynamic bodies in wind tunnels is described. This instrument consist essentially of a conventional

stroboscopic schlieron s9stem modified by the addition of electronic and optical elements to permit the

detailed e_amination of phenomen_t of cyclic nature,bet of fluctuating frequency. An additional feature of the

device makes possible the simo_lion of continuous slow motion, at _ud_itt"at' 9 chosen tales, of parlicular flow

features.

id: ncatq_los.adu.cs//n_a-ln-2589

Repb_i

PDF oenian (90fi KB)

Sc_med _i_es

FIG. 8. Output of the "display" method.
29

Our referenceimplementationof bucketsimplementsthe bucketAPI usinghttp

encodingof messages.Bucketsappearas ordinaryURLs and casualusers shouldnot

realizethatthey arenot interactingwith a typical web site. If no methodis invokedvia

URL arguments,the "display" methodis assumedby default. This generatesa humanreadabledisplay of the bucket's contents. For example,a bucket version of a NACA

TechnicalNotecanbereachedat:

http://www.cs.odu.edu/-nelso_m/naca-tn-2509/

which is the same as:

http://www.cs.odu.edu/-nelso_m/naca-tn-2509/

?method=display

Both of which will produce the output in figure 8. These URLs could be reached

through either a searching or browsing function within a DL, or they could be typed in

directly from above - buckets make no assumptions on how they were discovered. From

the human readable interface the "display" method generates, if users wish to retrieve the

PDF file, they click on the PDF link that was automatically generated:

http://www.cs.odu.edu/-nelso m/naca-tn-2509/

?method=display&pkg_name=report.pkg

&element_name=naca-tn-2509.pdf

which would cause the WWW browser to launch the PDF reader application or plug-in.

Similarly, if the users wished to display the scanned pages, selecting the automatically

created link would send the following arguments to the "display" method:

http://www.cs.odu.edu/-nelso m/naca-tn-2509/

?method=display&pkg_name=report.pkg

&element_name=report.scan
30

which would produce the output seen in figure 9. To the casual observer, the bucket API

is transparent. However, if individual users or harvesting robots know a particular URL

is actually a bucket, they can exploit this knowledge. For example, to extract the

metadata in default (RFC-1807) format, the URL would be:

http://www.cs.odu.edu/-nelso_m/naca-tn-2509/

?method=metadata

which would return the metadata in a structured format, suitable for inclusion in an index

being automatically built by a DL. If a user or agent wishes to determine that nature of a

bucket, a number of methods are available. For example, to determine the bucket's

version, the message is:

http://www.cs.odu.edu/-nelso_m/naca-tn-2509/

?method=version

And to see what methods are defined on a bucket:

http://www.cs.odu.edu/-nelso_m/naca-tn-2509/

?method=list methods

However, if a harvester is not bucket-aware, it can still "crawl" or "spider" the

bucket URLs as normal URLs, extracting information from the HTML human-readable

interface generated by the "display" method (assuming the "display" method is not

restricted by T&C). Buckets offer many expressive options to the users or services that

are bucket-aware, but are transparent to those who are not bucket-aware.

3.2.2 File Structure

Buckets take advantage of the package/element construct for their internal configuration.

In addition to the user data entered as packages and elements, the bucket keeps its own

files as elements in certain reserved packages. Thus, methods such as "add_element",

"delete_element" and so forth can be used to update the source code for the bucket,

update the password files, etc. Table 3 lists the predefined packages and some of the
31

elementsthey contain. By convention,thesepackagesbeginwith anunderscore("_")

character.Figure10providesamodelrepresentationof the structureof a typical bucket,

with internalpackagesandelementson the left and user-supplieddatapackageson the

right.

N: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::=====================================================

":_:::::::::::'_:::::::::::::::::::::::::::::::::::::::::: iiiiiii i _"-" _==============================='" _:::::::::::::::::::" "':::::::::::::+:_==============================_ _:::::::::::::::::::::::: _ "::::::::::::" ===============================_ :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

|iUei A self-synchronizin(j stroboscopic $chliereo system for the study of unsteady air flows

emthor_i Lawrence_ Leslie F _ Schmidt, Stanley F and Looschen_ Floyd W

O_iZ_i0a: I_iAOA Ames Aeronautical Laborolor9 (Moffett Field, Calif,, United St_es)

d_e: October 1951

rep0_i FJ,O,(_,o,TI.J-2509

p_e_i 31

A self-synchroaizin9 stroboscopic schlieren system deueloped for the uisualiz_ion of unsteady air flows

about aerodynamic bodies in wind tunnel_ i_ described. Thi_ instrument con_i_t essentially of • conventional

.............................................................. stroboscopic schlieren system modified by the addition of electronic and optical elements to permit the detailed

ex_rnin_ion of phenomen_t of cyclic n_ure,but of tluctu_ing frequency. An additional fe_ure of the device

makes possible the simualtion of continuous slow motion, _ e_r'oitrap/chosen rates, of particular flow features.

idi ocstrplus.odu.cs/An_t-to-2509

I _ IP_vi_us I FLrst Pa_-8 I

FIG. 9. Thumbnails in the "display" method.
32

TABLE 3. Reservedpackages.

Package ElementsWithin thePackage

_http.pkg cgi-lib.pl- StevenBrenner'sCGIlibrary

encoding.e- alist of MIME encodingtypes

mime.e- alist of MIME types

_log.pkg

_md.pkg

_methods.pkg

_state.pkg

_tc.pkg

access.log- messagesreceivedby thebucket

[handle].bib- aRFC-1807bibliographicfile

othermetadataformatscanbestoredhere,but

the.bibfile is canonical

1file perpublicmethod

1file perstoredstatevariable

1file per.tc(termsandcondition)file

passwordfile

.htaccessfile

Bucket

I

index.cgi

i mothodspkg II httppkg II 'ogpkgI

source files I http I logs for method dependency

files

i mdpkgIIsOtopkg I I tcpkg

metadata bucket state terms and

conditions

report.pkg I apendix.pkg

soflware.pkg ] testdata.pkg ]

default bucket packages sample bucket payload

FIG. 10. Bucket structure.
33

3.2.3 Terms and Conditions

Bucket terms and conditions are currently implemented using http and CGI

facilities. Although it is possible to access a bucket from the command line, this would

effectively bypass any security measures implemented by the bucket. Buckets hosted on

shared machines must use file system protections to prevent users from bypassing the

bucket API. Building from the CGI facilities, buckets implement simple access control

lists (ACLs) that restrict access based on username/password pairs, Internet Protocol (IP)

hostnames, and IP addresses. It is also possible to apply these restrictions to entire

methods, entire packages, or package/element pairs. The first example given below

assumes the buckets are entirely self-contained and nothing has been factored out. For

collection-wide T&C, factoring out the often repetitive T&C files as well as the user ids

and passwords allows for easier maintenance. However, the bucket operation remains the

same, only the location of where the bucket looks (internally vs. a shared location) is

changed. Creation of user, hostname or address groups is supported as well. However,

the bucket does not directly process these groups, it first flattens the groups to a list of

values, and then they are processed normally. Factoring and group support are

orthogonal and can be combined.

The ACLs are stored as elements (with a . tc extender) in the _tc. pkg package.

The filename for the . tc file specifies which method it protects. For example, to protect

the entire method "list_tc", the file _tc. pkg/list_tc, tc could contain the following

entries, or "directives":

user:nelson

user:maly

host:.*.larc.nasa.gov

host:.*.cs.odu.edu

Which would require the http-based user authentication for either "nelson" or

"maly" to be given, and for the originating computer to have either a . larc. nasa. gov
34

or a .cs.odu.edu IP address. To protect all elements of a package from being

displayed, a _tc. pkg/display, tc file could contain:

addr: 128.155.*

package: data.pkg

This would require access to any of the elements within the package "data.pkg" to

originate from a machine on the 128.155 subnet. Other packages and elements can be

retrieved from any machine by any user. To restrict who can delete a specific element

from, the file _tc. pkg/delete_element, tc would contain:

user:nelson

package:report.pkg

element:report.pdf

element:report.ps

This would prevent the two specified elements of the report, pkg package

from being deleted by anyone other than the user "nelson". Note that if in the above

example, the user : line was deleted, and no other user :, host : or addr : line was

present, this would have the same effect of preventing anyone from deleting the above

elements. The delete_element.tc file would have to be changed before those

elements could be deleted.

To specify T&C for the entire bucket, an index, tc file is used. An index, tc

file uses the same syntax and can be used in conjunction with method T&C files. If

index, tc and add_package, tc both exist, then calls to "add_package" would have

to satisfy both T&C files, with index, tc being enforced first.

For ease of management, it is possible to define groups of users, hostnames and

address. These bucket groups are not implemented with http groups, but rather as text

files with the following syntax:

groupl: userl user2 user3

group2:user2 user4
35

Terms and conditions are enforced using CGI mechanisms. This includes checking

the environment variables REMOTE_HOST, REMOTE_ADDR, and REMOTE_USER.

The CGI environment on the http server automatically sets

REMOTE HOST and REMOTE ADDR. REMOTE USER is set when CGI user

authentication is done. When a bucket method is invoked, before the bucket loads the

appropriate code to execute that method, the bucket checks in the _tc. pkg package to

see if there is a . tc file for the called method. If the . tc file exists, its contents are

read. If host: or addr: lines are present, and if the package and element arguments

passed in are also listed in the . tc file, then REMOTE_HOST and REMOTE_ADDR are

compared with the host: and addr: lines for a match. If a match is not made, an

error message is returned and execution halts. If a match is made, execution continues

with no interruption.

Utilizing the username/password function (i.e., REMOTE_USER) is slightly more

complicated. If the .tc files have a user: line, then current bucket execution is

redirected. That is, if the display method requires REMOTE_USER to be set, then:

http://dlib.cs.odu.edu/bucket/?method=display

isredirectedto:

http://dlib.cs.odu.edu/bucket/restricted/?method=display

Every bucket has a /restricted/ redirect capability that invokes the CGI

username / password authentication by means ofa . htaccess file in the restricted

directory. The . htaccess file lists the usernames that it will accept. The passwords

are stored encrypted in a separate, Unix-style password file. All of this is kept in the

tc. pkg package. If the user authenticates to a recognized username, and that

username is one that is required in the . tc file, then execution continues. If not, an error

message is generated and execution halts.
36

To further illustrate the operation of bucket TiC, consider an example where we

wish to restrict all access to a large number of buckets so that only *. cs. odu. edu and

*. larc. nasa. gov machines can access them. Since all the buckets will have the same

TiC profile, we will use both factoring and groups for easier management of the TiC

files in the buckets and the specification. The buckets' preferences (discussed in the next

section) will be changed so they look for their . tc files outside the bucket. The

to_server preference would be changed from internal to a file system location

such as:

/usr/local/buckets/tc/group-lO/

The bucket would look inside this directory, and see the presence of an

index.to file that would specify TiC for the entire bucket - all methods. The

index, tc file would have contents similar to:

host_group: hgroup-lO

If the buckets' host_group preference has been changed from internal to:

/usr/local/buckets/tc/

Then the file /usr/local/buckets/tc/host_group would have entries

similar to:

hgroup- 1 :

hgroup-2 :

hgroup- 10 :

blearg.larc.nasa.gov

jaguar.cs.odu.edu .*

lion.cs.odu.edu

.nasa.gov

.*.larc.nasa.gov .*.cs.odu.edu

The bucket preferences and the values in the various . tc files and group files are

all manageable from the Administration Tool (chapter seven). While the TiC mechanism

described above is sufficient for a large number of applications, they are not rich enough
37

for a full array of needs anticipated for NASA DLs. In addition, the terms and conditions

can be defeated if the values of REMOTE_HOST, REMOTE._ADDR, or REMOTE_USER

are forged.

Table 4 lists all the directives that T&C files recognize. The inform: directive,

although present in the T&C file, does not specify access control. Instead, it specifies

who should be informed when an action takes place. If an action is successful, the bucket

generates an email message informing the recipient that the action was successfully

completed. If the action was not successful, the bucket generates an email message

indicating the failure of the action along with an HTML form that will allow the recipient

to reattempt the action. The inform: directive can be used in conjunction with other

directives.

TABLE 4. Directives available in T&C files.

Directive Arguments

user:

group:

addr:

addr_group:

host:

host_group:

package:

element:

inform:

principal names

principal groups

IP address (can be a Perl regular expression)

IP address groups

IP hostname (can be a Perl regular expression)

IP hostname groups

name of a package in this bucket

name of an element in this package (must be used with a package:

directive)

email addresses
38

3.2.4 Internal Bucket Operation

In this section, we examine what happens internally when a CGI-based bucket

receives a message. The details are specific to the current implementation of Perl-based

buckets; other implementations are free to implement the bucket API differently.

A user contacts a http server and specifies the bucket they wish to communicate

with. This can be done by either explicitly naming the index, cgi:

http://foo/bar/bucket/index.cgi

or implicitly:

http://foo/bar/bucket/

The CGI parsing is done with cgi-lib.pl (Brenner, 2000) that is stored inside the

bucket in the_http, pkg package. CGI.pm (Stein, 1998) is not used for two reasons:

1) it is not part of the standard Perl library, so it would have to be carried in the bucket as

well; 2) it did not perform well as well as cgi-lib.pl in command line operation (used

frequently in testing and debugging). The index, cgi script parses the input string to

determine which method the user is requesting. If no method is specified, the display

method is assumed. The index, cgi script then looks in the _tc. pkg package to see

if an index, tc or method, tc file exists (where method is the name of the method

that is being invoked). If either (or both) file(s) exists, the T&C are enforced as described

in the previous section. If the T&C are satisfied, the index, cgi script then performs a

run-time include of the source code of the method. The index, cgi script then calls the

function with the same name as the method invoked, which is assumed to be defined in

the source code included at run-time. This way only the code for the invoked method is

accessed. This procedure also allows for the index, cgi script to be written so that it

makes no assumptions about the methods that are available, allowing methods to be added

and deleted for specific bucket instantiations. This entire process can be encapsulated in
39

thisPerlcodesnippet,where$method is a variable containing the name of the requested

method:

Smethod

if (-f

}else

file = "$method dir/$method.pl";

m

$method file) {

&tc($method); #

# if we made it out of &tc,

require "$method file"; #

&$method; #

{

# method not found in bucket

&unsupported($method);

check tc

we must be ok...

run-time include

calls the method

If the "display" method is called with specific package and element arguments,

then the named file is returned. However, this is not done through "normal" http

operations - to enforce data hiding, packages have . htaccess files that prevent any

direct access of their elements. The index, cg2 script opens the file for reading, sets

the correct MIME type by checking the element _http. pkg/mime, e, and then writes

the file to STDOUT. If the file being returned to the user is an HTML file, the relative

URLs are re-written to access elements within the bucket. This is necessary because of

the inherent conflict between URLs, which are tightly tied with file location, and the

bucket's data hiding, which prevents access of specific file locations. If the element being

requested by the "display" method call is a URL to a location outside of the bucket, the

bucket will log that a "display" call was made, where the intended location is, and then

issue an http status code 302 (redirect) to the client.

3.2.5 Metadata Extensions

The metadata file in a bucket plays an extremely important role. Not only does it

hold the traditional bibliographic citation material, it also encodes the structure of the

bucket's contents. This structure is read and processed when the bucket's "display"

method is called and the bucket reveals its structure in a human readable, HTML format.
4O

RFC-1807 is an extensible format. To describe the two-level bucket structure,

two tags have been defined: "PACKAGE::" and "ELEMENT::". All previously defined

RFC-1807 tags are also available with the "PACKAGE" and "ELEMENT" prefix:

"PACKAGE-TITLE::", "ELEMENT-END::", etc. Currently only the values for the

"PACKAGE-TITLE::" and "ELEMENT-TITLE::" tags are revealed during a "display"

method call, however this is likely to change in the future. Figure 11 shows the RFC1807 metadata for a bucket:

BIB-VERSION:: X-NCSTRL+I.0

ID:: ncstrplus.odu.cs//naca-tn-2509

TITLE:: A self-synchronizing stroboscopic Schlieren system for the study of

unsteady air flows

REPORT:: NACA TN-2509

AUTHOR:: Lawrence, Leslie F

AUTHOR:: Schmidt, Stanley F

AUTHOR:: Looschen, Floyd W

ORGANIZATION:: NACA Ames Aeronautical Laboratory (Moffett Field, Calif.,

United States)

DATE:: October 1951

PAGES:: 31

ABSTRACT:: A self-synchronizing stroboscopic schlieren system developed

for the visualization of unsteady air flows about aerodynamic bodies in

wind tunnels is described. This instrument consist essentially of a

conventional stroboscopic schlieren system modified by the addition of

electronic and optical elements to permit the detailed examination of

phenomena of cyclic nature,but of fluctuating frequency. An additional

feature of the device makes possible the simualtion of continuous slow

motion, at arbitrary chosen rates, of particular flow features.

PACKAGE:: report.pkg

PACKAGE-TITLE:: Report

ELEMENT:: naca-tn-2509.pdf

ELEMENT-TITLE:: PDF version

ELEMENT-END:: naca-tn-2509.pdf

ELEMENT:: report.scan

ELEMENT-TITLE:: Scanned pages

ELEMENT-END:: report.scan

PACKAGE-END:: report.pkg

PACKAGE:: staff.pkg

PACKAGE-TITLE:: For LaRC Staff

ELEMENT:: report.tiffs

ELEMENT-TITLE:: TIFFs

ELEMENT-END:: report.tiffs

ELEMENT:: maintenance.html

ELEMENT-TITLE:: Maintenance page

ELEMENT-END:: maintenance.html

PACKAGE-END:: staff.pkg

END:: ncstrplus.odu.cs//naca-tn-2509

FIG. 11. RFC-1807 metadata.

The values for ...... PACKAGE:: , PACKAGE-END:'". , "ELEMENT::" and

"ELEMENT-END::" correspond to the actual filesystem names inside the bucket. Just
41

aselementscanonly existwithin packages,"ELEMENT" tagsandprefixedtagsmustbe

containedwithin theirrespective"PACKAGE::" / "PACKAGE-END::" tag pairs.

3.3 Discussion

The previous sections describe the "normal" operations of a bucket. However, a

bucket's operation can be transformed by the setting of bucket preferences. In this

section, we list what is possible through bucket preferences, as well as examining a

number of systems issues with the current bucket implementation.

3.3.1 Bucket Preferences

Bucket preferences can be checked and set through the "get_preference" and

"set_preference" methods, respectively. Preferences allow individual buckets to tailor

their operation to reflect their unique requirements, but yet retain a standard, public way

of being changed in the future. Table 5 lists the currently defined preferences and gives a

short explanation of their function.

Inspection of Table 5 will reveal that many preferences exist so the method source

code, user names and passwords, and T&C files can all be "factored out" of the bucket.

The default model is the bucket carries all of this internally, thus allowing for greater

mobility and independence. However, this level of freedom comes at the cost of increased

storage and complexity in managing multiple copies of source code, passwords, etc. So

we provide the mechanism to factor out all the pieces that do not need to be internally

stored in the bucket. However, these need not be permanent decisions - a mostly

homogenous collection of buckets can all share from a central store their source code and

other items that have been removed. However, specific buckets that require different

functionality or a higher level of independence can have their preferences changed so they

return to the default model of internal storage for some or all things.
42

TABLE 5. Bucketpreferences.

Preference Default Description

Value

access.log on

addr_group internal

bcs_server (none)

expanding off

framable off

group

host_group

intemal

internal

This is the name of the single default log, and by default

logging is set to "on". Logging can be turned off by setting

this value to "off'.

By default, the bucket expects to internally store the file

that maps addr group names to lists of IP addresses.

Filesystem pathnames are other acceptable values.

A bucket can choose which Bucket Communication Space

server it communicates with. The current default is just a

sample value, and is likely to be site dependent. URLs are

acceptable values here.

The bucket display by default lists all elements in all

packages at once. By setting this preference to "on", the

elements will not be visible until the package name is

"clicked", revealing its contents.

By default, the bucket "display" method includes JavaScript

to keep the bucket from being "trapped" inside a frame.

Setting this preference to "on" allows buckets to exist inside

frames.

By default, the bucket expects to internally store the file

that maps group names to lists of user names. Filesystem

pathnames are other acceptable values.

By default, the bucket expects to internally store the file

that maps host_group names to lists of IP hostnames.

Filesystem pathnames are other acceptable values.
43

maxdata

method server

passwd

sfx server

tc server

thumbnail inc 10

rement

5000000

intemal

internal

(none)

internal

This is the default value for maximum file size of an

uploaded file. Any integer value greater than or equal to

zero is acceptable.

By default, the bucket expects to find the source code for

methods inside the bucket. Filesystem pathnames are other

acceptable values.

By default, the bucket expects to find the password file

(stored in Unix "/etc/passwd" format) inside the bucket.

Filesystem pathnames are other acceptable values.

The location of a Special Effects (SFX) reference linking

server. This value is just a placeholder; the nature of SFX

insures that this needs to be set to a site-specific value.

URLs are acceptable values.

By default, the bucket expects to internally store the T&C

files for the bucket methods.

other acceptable values.

When displaying thumbnails

Filesystem pathnames are

of scanned pages, this

preference determines how many thumbnails to show at a

time. Any integer greater than 1 is an acceptable value.

3.3.2 Systems Issues

There are a number of systems-related issues concerning the current Perl

implementation of buckets, which might not be present in alternate bucket

implementations. These issues include: interaction between buckets and http caching;

http server permissions and file permissions; and resource consumption by buckets.

Because they depend on CGI, client or server http caches should automatically

not store responses to bucket messages. While this can result in lower performance for
44

the user for repeatedaccessto DL objects,given the potentially dynamic nature of

buckets,not cachingresponsesis desirable.

Anothercommonissuein bucketoperationis thattheownerof thefiles that make

upthebucketandtheuserid of http serverdo not alwaysmatch. Sinceit is possibleto

changethebucketthroughbucketmethods,thehttp serverneedsto be ableto add,delete

andmodify files thatmakeupthebucket. Therearefour waysto accomplishthis:

The files that comprisethe bucketcanbe world writable. While this allows

the http serverto write to thebucket,it alsomakesthebucketvulnerableto

anyonewith file systemaccessto the bucket. This methodwould only be

reasonableif the interactiveloginson the machinehostingthe bucketswere

limited to trustedparties.

The index, cg± script can be setuid, so when it is invoked, it runs as the

owner of the script, not the caller of the script (in this case, the http server).

However, for security purposes on general-purpose machines many system

administrators do not allow setuid programs outside of file partitions used by

the operating system. Furthermore, many operating system kernels have a

potential security flaw via a race condition in invoking setuid scripts, so the

Perl interpreter will not run setuid scripts unless the kernel has been patched

or the scripts have been wrapped with a C program (Wall, Christiansen, &

Schwartz, 1996).

The http server can be run as a user (or group member) that has write access

to the files in the bucket. However, most http servers on the standard port

(80) are run as "nobody" or some other account with minimal privileges and

no interactive login. However, it is possible to run a http server on a nonstandard port that is run as the owner of the files in the bucket. This will

work, but it does leave open the possibility that if attackers were to

compromise the http server, they could gain access to a privileged account and

not a limited one such as "nobody".
45

Currentversionsof Apache,a popularfree-sourcehttp server,havea "setuid

module". This allows the installerof apacheto decideif all CGI programs

shouldrunnot asthe sameownerof the http server,but asthe ownerof the

CGIfile. Thisisanelegant,generalsolutionif a site is runninganhttp server

with this capability.

Note that bucketsmakeno assumptionshow the problem of http userid and

bucketfile userid is solved- only thatit is solved. If anhttp serverdoesnot havewrite

permissionto a bucket's files, attemptsto update the bucketwill fail. Assumingfile

systempermissionspermit,readattemptswill continueto functionnormally.

The current implementationof buckets should still be consideredresearch

prototypes. As such, they consumestorageresourcesmore greedily than a stable,

productionversionwould. Bucket version 1.6 currently requires68 inodesand 144

kilobytesstoragefor an"empty" bucket. Inodesareusedby theUnix filesystemto store

informationonindividualfilesanddirectories.Inodesarefinite, but additionalinodescan

be allocatedby a systemsadministrator. The inodeandkilobyte requirementsof the

currentimplementationarea non-trivial overheadimposedby the buckets. However,

thereareotherfactorsto consider:

Theseareresearchprototypes, and as suchare "wasteful" in the nameof

convenience.The sourcecodehasfull documentationand other featuresnot

required for use in a non-developmentsetting. Many of the inodes are

consumedto store simplepreferences(e.g.,"on" or "internal") where in a

production system thesecould be compressedinto a single file or datastructure. Although suchoptimization hasnot beenvigorouslypursued(see

chaptersix for optimizationperformedfor theUPS project),it is anticipated

that50%of theinodesand30%of thekilobytesrequiredcouldbereduced.

Furthermore,the storagerequirementis small when comparedwith the large

aggregationsof data that they are designedto hold. For example,in the
46

NACATRSdigital library, theaveragestoragerequirementper scannedpageis

approximately80KB (Nelson,1999). Thus, the KB requiredfor a bucketis

less than two scannedpages. 144KB should not be an issuewhen using

bucketsto store 100scannedpagereports,potentially with largesupporting

datasetsor software.

Storageischeapandgettingcheaper.Lesk(1997)reportsthatstorageis about

4.5 MB /US $1.0. A quick glance through a current Computer Shopper (a

popular computer hardware mail order retailer) reveals an average of about 20-

25 MB / US $1.0, which fits the profile of storage capacity doubling roughly

every 1.5 years. The exact numbers are not as important as the trend: with

each refresh or migration, the bucket storage overhead problem will decrease

relative to the amount of storage available at a fixed price.

For DL applications where buckets are likely to be largely homogeneous,

factoring of source code, T&C, and authentication information is available to

reduce the inode and kilobyte requirements. For example, factoring out just

the method source code of a version 1.6 bucket can save 31 inodes and 67

kilobytes.

So while it is true that buckets do impose additional storage requirements, it is felt

that the small additional cost is more than offset by the additional capabilities that

buckets provide.
47

CHAPTER FOUR

DUMB ARCHIVES

4.1 Overview

Buckets are the smart objects in the Smart Object, Dumb Archive DL model. To

complement the buckets, dumb archives exist primarily to aid in the discovery and group

management of buckets. It is possible to use buckets in other DL models, but SODA

provides the most striking demonstration of the shift in responsibilities.

4.1.1 The SODA DL Model

We present a model that defines DLs as composed of three strata (Fig. 12):

digital library services - the "user"

browsing, usage analysis, citation

information (SDI), etc.

archive - managed sets of digital objects. DLs

newly published digital objects, for example.

digital object - the stored and trafficked digital content. These can be simple

files (e.g., PDF or PS files), or more sophisticated objects such as buckets.

functionality and interface: searching,

analysis, selective dissemination of

can poll archives to learn of

DLs are built by Digital Library Service Providers (DLSPs) that:

identify a user group

identify archives holding buckets of interest and individual bucket owners

negotiate terms and conditions with publishing organizations (archive and

individual bucket owners)

create indices of appropriate subsets through extracting bucket metadata

create DL services such as search, browse, and reference linking

create user interaction services such as authentication and billing
48

In most DLs, the digital library services(DLS) and the archivefunctionality are

tightly coupled.A digital object is placedin an archive,and this placementuniquely

determinesin which DL it appears.Webelievethatif thereis not a 1-1mappingbetween

archivesandDLs,but ratheraN-M mapping,the capacityfor interoperabilityis greatly

advanced.A DL candrawfrom manyarchives,andlikewise,anarchivecancontributeits

contentsto manyDLs.

Digital

Objects

in Archives

Digital Library Service Providers

Objects

chive 1 Archive 2 • • • Archive N O O out of Archives

O O

0 0 0 0 0 0 _ublishers

!

0 0 0 0 0 0

FIG. 12. The three strata of DLs

However, since we can no longer be sure which DL will be used for the discovery

and presentation of an object, it is necessary to evolve the notion of the object and to

imbue it with greater functionality and responsibility. DL objects should be selfsufficient, intelligent, and aggregative and capable of enforcing their own terms and

conditions, negotiating access, and displaying their contents.

Much of the traditional functionality associated with archives (terms and

conditions, content display, etc.) has been "pushed down" into the objects, making the

objects "smarter" and the archives "dumber". To demonstrate a SODA DL, a reference
49

implementation,NCSTRL+ (fully describedin ChapterSix),hasbeenconstructedwhich

implementseachof the3 stratalisted aboveusingthe Dienst protocol andhttp services.

The DLSs areprovided by using the basiccoreof Dienst for searching,browsing and

similarservices.The archivefunctionality was originally implementedusing a modified

versionof Dienst,becauseabucket-basedarchivesystemwasnot originallyavailable.

TheobservationthatmotivatestheSODA modelfor DLs is thatdigital objectsare

moreimportant thanthe archivesthat hold them.Many DL systemsandprotocols are

reachinga point where DL interoperability and object mobility are hinderedby the

complexity of the archivesthat hold the objects.The goal of the current work is to

increasethe responsibilitiesof objects,and decreasethe responsibilitiesof archives.If

digitalobjectsthemselveshandlepresentation,termsand conditionsand their own data

management,it will beeasierto achieveinteroperabilitybetweenheterogeneousDLs as

well asincreaseobjectmobility andlongevity.As a consequence,moreDLSPsshouldbe

encouragedto builddigital librariesfor varioususercommunities.

4.1.2 Archive Design Space

Archives exist primarily to assist DLs in locating objects -- they are generally not

for direct user access. It appears that many digital libraries and their associated access

protocols (e.g., Dienst and the Repository Access Protocol (RAP) (Lagoze & Ely, 1995))

have become unnecessarily complex. For example, the Dienst protocol contains a built-in

document object model, and this limits its applicability in different domains and makes it

more difficult to transition to evolving document object models. It is the archived objects,

not archives, that should be responsible for the enforcement of terms and conditions,

negotiation and presentation of content, etc. Although it is expected that some archive

implementations will retain portions of the above functionality - indeed, SOSA (Smart

Objects, Smart Archives) may become the most desirable DL model -a "dumb archive"

model is used here to illustrate the full application of smart objects (buckets). When

archives become "smart" again, it will with other functionalities, not duplication of bucket
50

functionality.Using this terminology,Table 6 illustrateshow the archivedesignspace

partitions.

TABLE 6. Thearchivedesignspace.

SmartArchives DumbArchives

SmartObjects SOSA: Smart Objects, Smart SODA: Smart Objects, Dumb

Archives Archives

DL Example:noneknown DL Example:NCSTRL+

DOSA: Dumb Objects, Smart DODA: Dumb Objects, Dumb

Archives Archives

DL Example:NCSTRL DL Example: any anonymous

FTPserverwith .ps.Zfiles

DumbObjects

4.1.3 Publishing in the SODA Model

Separating the functionality of the archive from that of the DLS allows for greater

interoperability and federation of DLs. The archive's purpose is to provide DLs the

location of buckets (the DLs can poll the buckets themselves for their metadata), and the

DLs build their own indexes. And if a bucket does not "want" to share its metadata (or

contents) with certain DLs or users, its terms and conditions will prevent this from

occurring. For example, it is expected that the NASA digital publishing model will begin

with technical publications, after passing through their respective internal approval

processes, to be placed in a NASA archive. The NASA DL (which is the set of the

NASA buckets, the NASA archive(s), the NASA DLS, and the user communities at each

level) would poll this archive to learn the location of buckets published within the last

week. The NASA DL could then contact those buckets, requesting their metadata. Other

DLs could index NASA holdings in a similar way: polling the NASA archive and

contacting the appropriate buckets. The buckets would still be stored at NASA, but they

could be indexed by any number of DLs, each with the possibility for novel and unique
51

methodsfor searchingor browsing.Or perhapsthe DL collectsall the metadata,then

performsadditionalfiltering to determineapplicability for inclusion into their DL. In

additionto anarchive'sholdingsbeingrepresentedin manyDLs, a DL couldcontainthe

holdingsof manyarchives.If all digitally availablepublicationsareviewedasa universal

corpus,thenthis corpuscouldberepresentedin N archivesandM DLs, with eachDL

customizedin functionandholdingsto theneedsof its userbase.Figure13illustratesthe

SODApublishingmodel.

User Population ._

DL_ Building

FromArchives

and Buckets

Archives

Managing

Buckets

All Known

Bucke_

(marchives

and ou O

FIG. 13. The SODA publishing model.

4.2 Implementation

A few simple prototypes of a DA were built as standalone services, but

eventually the decision was made to extend an existing bucket with new methods so it

could function as the DA. Not only did this allow for rapid development of the DA, but

it also showcases the flexibility in modifying buckets for different purposes. It should

also be noted that although DA was created to keep track of buckets, there is nothing in

its implementation that requires the objects it tracks to be buckets. For example, it would

be possible to use DA for an archive of PDF files.
52

4.2.1 Implemented Methods

A goal of the DA was to be very simple, performing only set management

routines. As such, only five new methods are defined. Table 7 highlights those methods,

and they are explained in detail in Appendix C.

TABLE 7. DA API.

Method Description

da_put

da delete

da list

da info

da_get

insert a data object into the archive

remove a data object from the archive

display the holdings of the archive

display metadata about the archive

redirects to the object's URL or URN

The DA does not disable any of the currently defined bucket methods. Some of

the methods may be unnecessary, but they were left in they were left in for

completeness. For example, end users are not meant to interact directly with DAs; DAs

exist to aid in the construction of DLs. However, the "display" method was left in the

DA because: 1) a user might "stumble" across a DA, and it should be able to generate a

human readable display; and 2) an archive might have need to store human consumable

information in regular packages and elements - for example links to all the DLs that

harvest from the archive. It might be advantageous for DA's to have their standard

methods overridden with implementations tailored to archive application. However, a

DA's main traffic is expected to remain DLs calling the various da_* methods.

4.2.2 Changes From a Regular Bucket

The five da_* methods are stored as regular methods in the standard

methods.pkg package. However, DAs also have a DA-specific package,

holdings, pkg, which contains library source files as well as the databases generated to

store the objects in the DA. A tool for duplicating a DA's holdings could simply retrieve
53

(modulothecorrectT&C) theknown elementsfrom this packageto geta "copy" of the

DA's contents.

Similarly,a regularbucket could be changedinto a DA through the "pack" /

"unpack" methods to extract and replicate the contents of the holdings, pkg package

and the five DA methods. Furthermore, if desirable for a specific application, a bucket

could serve "double duty" - responding to da_* methods from DLs, and all the while

serving "regular" data contents to users interacting with the bucket through the normal

bucket methods.

4.3 Discussion

Even for the limited goals of a dumb archive, the current implementation only

scratches the surface of the work that could be done. In this section, some of the systems

issues of DA implementation are discussed, and an outline is given on how other archive

protocols could be implemented using DA.

4.3.1 DA Examples

Although interaction with a DA should occur through a software tool interface, we

can examine the methods used to populate and interact with the DA. Consider an

installed DA:

http://dlib.cs.odu.edu/da/

This will appear as a regular bucket to someone that goes directly to the above

URL. If it is known that the URL is a DA, then the items registered with the archive can

be listed with:

http://dlib.cs.odu.edu/da/?method=da_list

The above URL will produce a list of ids and URLs for all items registered with

the archive, or a null list if nothing is registered. Items can be placed in the archive:
54

http://dlib.cs.odu.edu/da/?method=da_put&id=reportl

&url=http:%2f%2foo.edu%2frl&adate=19991220

&pdate=19940429&subject=cs

Where "adate" and "pdate" are accession date and publication date, respectively,

and are in the format YYYYMMDD. "subject" is a string describing an arbitrary subject

classification system, and "url" is an optional encoded URL that maps to the id. If a

URN implementation or other id scheme is not used, it is possible to use URL values in

the "id" field and not use the "url" argument. Figure 14 shows the URLs used to

populate a small DA, and figures 15, 16, and 17 show the results of calls to "da_list" on

that archive with various arguments.

http://res-ups.cs.odu.edu/-nelso_m/da-l. Ol/?method=daput&id=testl

&adate=20000214&subject=aero&url=http:%2f%2ffoo.edu%2freportl

http://res-ups.cs.odu.edu/-nelso_m/da-l. Ol/?method=daput&id=test2

&adate=19951225&subject=aero&url=http:%2f%2ffoo.edu%2freport2

http://res-ups.cs.odu.edu/-nelso m/da-l. Ol/?method=daput&id=test3

&adate=19951225&subject=cs&url=http:%2f%2ffoo.edu%2freport3

http://res-ups.cs.odu.edu/-nelso_m/da-l. Ol/?method=daput&id=test4

&adate=19930iOl&subject=cs&url=http:%2f%2ffoo.edu%2freport4

http://res-ups.cs.odu.edu/-nelso m/da-l. Ol/?method=daput&id=test5

&adate=19991111&subject=phys&url=http:%2f%2ffoo.edu%2freport5

http://res-ups.cs.odu.edu/-nelso m/da-l. Ol/?method=daput&id=test6

&adate=19991111&subject=phys&pdate=20000iOl&url=http:%2f%2ffoo.edu%2freport6

http://res-ups.cs.odu.edu/-nelso_m/da-l. Ol/?method=daput&id=test7

&adate=19991111&subject=phys&pdate=19991229&url=http:%2f%2ffoo.edu%2freport7

FIG. 14. Population of the DA.
55

 iii iiiiii iiiiiiiiiii iiiiiiiiiiiii iiiiiiii

FIG. 15. DA query (?method=da put&adate=<20000101).

X X X X£ "_

_' ii_ii_ _ _; :_._:..... _::-" _:_ _:_:_.... _: :::_:_::::,.":::_ .......................... i

_i_i_i_i_i_i_i_i_i_i_i_i@i@iq_i_i_iii_@iiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiii_iiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiii_@iiiiiiiiiiii_iiiiiiiiiiiiiiiiii

_$ii::i::::::i!!_i::i::i::i::i_i_::i::i_i::i::_ii::i_ttp ://_es-ups. cs. odu. edu/-r,::i::iii_/i::i_!_{_J_ii

::?:_:]_:::: :::: :::::: i@.::i::i::ii_ __::i::i::i::i::i::i::ii_i_i::i::i::i::i::i::i::#:_i::i::i::i::+_i::i::i::i::ii_ :._+_#:i::i::_i::i

FIG. 16. DA query (?method=da put&adate = 19940101-20000101 &subject=cs).
56

iliiiiiiiii_i_iii_i_iiii_iiiii_iiiii__//_oo-_o. oo.o_.o_/_o__!liii__ 1

iii ii_iii_iii_i_iiiiiiiiii_iiiiii_iiiiiii_iiiiiiiii_iiiii

FIG. 17.DA query (?method=da_put&subject=phys).

4.3.2 DBM Implementation Notes

At first glance, it is tempting to implement the DA functionality using the package

and element constructs of the bucket. Packages could be defined for each of "da_put"

arguments, and an element with record's id in those packages would contain the values for

those arguments. However, as discovered during the UPS implementation (detailed in

Chapter Six), the Solaris operating system will allow only 32,767 inodes within a single

directory (Sun Microsystems, 1999). The current bucket implementation would have

scalability difficulties with archives containing more than 32,767 records. To alleviate

this problem, a different internal data structure was used for the DA functionality instead

of the package / element semantics.

The internal data structures used by DA are implemented with variants of the

Berkeley Database Management (DBM) library (Olson, Bostic & Seltzer, 1999). An
57

indexis built for eachthepossibleargumentsto "da put": "id", "url", "adate","pdate",

"subject",and"metadata"(seeAppendix C for a detaileddescription). DA attemptsto

usethe Gnuversion,GDBM, but this library is not always availableon standardUnix

distributions. If GDBM is not available,it will use the NDBM version, which is

availableonall systems.

GDBM is preferableto NDBM (and the other standardversions,SDBM and

ODBM) becausethe GDBM doesnot havethelimitation of the othersof the key + hash

for anentryhavingatotalsizeof 1024bytes. As such,if GDBM is not availableon the

system running DA, typical valuesfor "metadata"will exceed1024 bytes and the

"da put" will notbe successful.

The DBM librariesprovide a convenientandlightweightdatabasemechanismfor

theDA, but it doescomeat a cost to thebucket. Whereas"normal" bucketsaremobile

and can move from serverto server,a DA has mobility only within homogeneous

architectures. DBM files are binary and differ for various machinearchitectures.

Furthermore,DBM variantsarenot interchangeable, so if a DA beganwith GDBM, it

couldnot automaticallyreadthedatafilesusingNDBM. However,giventhe permanent

natureof archives,for mostapplications,non-mobilearchiveswill notbe a problem. Not

only is the DA an exampleof a modifiedbucket, but it is also an exampleof how

specializationimpactsthegeneralbucketrequirements.

4.3.3 Open Archives Initiative Dienst Subset Mapping

The currently evolving Open Archives initiative (OAi) aims at making the

technology available for information providers to open up their archives for digital library

service providers to harvest their contents, apply their value-added processing, and

present them to their targeted customer base (Van de Sompel & Lagoze, 2000). The OAi

is a DODA DL model, with sophisticated user services expected to be built from the

harvesting of multiple OAi-compliant archives. While previous attempts within the DL

community at defining common functionality for archives have generated "limited

consensus" (Scherlis, 1996), the OAi has bounded archive expectations by focusing on
58

whatis achievablein theshortterm. The OAi hasdefineda smallsubsetof the popular

Dienstprotocol,known asthe "Open Archives DienstSubset"(Davis,Fielding,Lagoze,

& Marisa,2000),which has the solepurpose of aidingserviceproviders in harvesting

archives.

The OAi is an evolvingprotocol, with version 2 expectedin December2000.

Oncethe OAi protocol hasstabilized,it is anticipatedthat it will be easyto implement

this protocol througha mappinginto DA commands.A site usinganOAi enabledDA

will be ableto respondto genericOAi harvestingrequests,but also havethe additional

capabilitiesof theDA, suchasT&C - which arecurrently not part of the OAi protocol,

orreturnmetadatain non-extensiblemarkuplanguage(XML) encodings.Table8 shows

thecurrentlydefinedOAi Dienstsubset"verbs" andtheir DA equivalents.

Someof the conceptsdo not directly map becausethe OAi Dienst heritage

emphasizesthepreeminenceof archivesandDLs, wherebucketsemphasizethe objects

themselvesas the canonicalsource. For example,DAs can provide the metadatavia

"Disseminate" as a convenienceto the harvester,however the bucket remains the

canonicalsourceof metadataaboutitself. Similarly,if theharvesterwantsthemetadatain

acertainformat,theexpectedprocedurerelativeto bucketsisto askthe bucketitself, not

thebucket's archive. However,the DA couldbe modifiedto performtheseserviceson

theharvester'sbehalfsincenotall harvesterswill bebucket-aware.

Anotherareaof concernis that Dienst encodesits messagesin non-standardCGI

format,requiringmodificationof the http serverconfigurationto successfullytrap the

incominghttp requestsandroutethemto the correctlocation. In comparison,all bucket

messages(includingDA messages)are entirely encapsulatedin the http messageand

requireno servermodifications.While not difficult to develop,a scriptwouldbe required

to traptheincomingDienstmessagesandre-routetheminto theDA format.
TABLE 8. OAi ---,DA mapping.

59

OAi Dienst DA

SubsetVerb Method

Discussion

Disseminate da list

List-Contents da list

List-Meta- da info

Formats

List-Partitions da info

Structure n/a

"da_list" with the "id" (or "url") and "metadata"

argumentcurrently fulfills the same purpose of the

"Disseminate"verb. "Disseminate" does support the

possibilityof retrievingdifferentmetadataformats,while

"da_list" only returns what was originally uploaded.

"da_list" couldbe modifiedto perform this serviceas a

convenienceto the harvester (calling the bucket's

"metadata"methodontheharvester'sbehalf).

"da_list" as currently implementedis not as generalas

"List-Contents",but the "da_list" arguments"adate"and

"subject" provide the same functionality. If other

partitions (or "clusters") for the DA are defined,they

couldbeincludedin the samemanneras"subject". Note

that "List-Contents" does not by default support the

conceptexpressedin the "pdate"argumentto "da_list".

DAs couldlist their nativemetadataformat(s)aspart of

the"da info" method.

DAs couldeitherlist the partitions supportedaspart of

the "da_info" method,or it couldjust the list the predefinedpartitions(or "clusters")of theDA.

"da_list" couldbemodifiedto providethis capability,but

metadataconversionis really the provenanceof the

bucket(assistedthroughtheBCS).
60

CHAPTER FIVE

BUCKET COMMUNICATION SPACE

5.1 Overview

The Bucket Communication Space (BCS) is partially motivated by Linda, the

parallel communication library (Carriero & Gelernter, 1989). In Linda, processes

effectively pass messages by creating "tuples" that exist in "tuple space". These data

objects are created with the "eval" primitive, and filled with data by processes using the

"out" primitive. Processes use "rd" and "in" for reading and reading-removing operations,

respectively. These primitives allow processes to communicate through tuple space,

without having to know the details (e.g. hostnames, port numbers) of where the processes

are. The messages written to tuple space can have regular expressions and control logic to

specify who should read them. When a "in" tuple sees an "out" tuple and the conditions

of the former match that of the latter, the message is communicated to the receiving

process and the tuple is removed from tuple space. Though it imposes a performance

overhead, the Linda environment provides a useful layer of abstraction for inter-process

communication.

We wished to provide something similar for buckets: buckets communicating with

other buckets without having to know the details of bucket location. This is especially

important if the buckets are mobile, and a bucket's location is not guaranteed to be static.

The BCS also provides a method for centralizing functionality that cannot be replicated in

individual buckets. This could be either because of efficiency concerns (the resulting

bucket would be too bloated) or implementation limitations (a service is not available on

the architecture that is serving the bucket). Buckets need only know how to communicate

to a BCS server, which can handle their requests for them.

Buckets maintain the location of their BCS server through a bucket preference.

This allows for the specification of a single BCS server, with no provisions for if that
61

BCSserveris not available. Currently, no detailedplans havebeenmadefor complex

BCSarchitectures.Thereis nobuilt-in conceptof a masterBCSserverfor all buckets,

localizedBCSservers,rings of BCS serversor any otherarchitecturalpossibilities. If

thesearchitecturesareto bebuilt, it will involvethe modificationof the BCSbucketsto

recognizepeer BCSbuckets,masterBCSbuckets,etc. However, thesemodifications

shouldbetransparento thedatabucketsthemselves,with databucketsstill only tracking

thelocationof theirentryinto thebucketcommunicationspace.

The BCSmodelopensup manypossibleserviceareas.A subtleelementof the

BCSis that buckets,not people,areresponsiblefor the provision and coordinationof

these services. We provide proof-of-concept implementations for four significant

services:file format conversion,metadataconversion,bucket messaging,and bucket

matching.

5.1.1 File Format Conversion

File format conversion provides bi-directional conversion of image (e.g. GIF,

JPEG) formats and page description formats (e.g., PostScript, PDF). Format conversion

is an obvious application - additional formats will become available after a bucket's

publication and the ability to either place them in the bucket or dynamically create them

will be useful in information migration.

5.1.2 Metadata Conversion

Metadata conversion is similar to file format conversion, providing conversion

between some of the more popular metadata formats (e.g., Refer, RFC-1807, bibtex).

Metadata conversion is extremely important because although buckets ultimately have to

choose a single format to operate on, it is unreasonable to assume that all applications

needing metadata from the bucket should have to choose the same format. Being able to

specify the desired format to receive from a bucket also leaves the bucket free to change

its canonical format in the future.
62

5.1.3 Bucket Messaging

Messaging allows multiple buckets to receive a message if they match specific

criteria. While point-to-point communication between buckets is always possible, bucket

messaging provides a method for discovering and then sending messages to buckets.

Messaging provides functionality closer to the original inspiration of Linda, and can be

used as the core of a "bucket-multicasting" service that sends pre-defined messages to a

subset of registered buckets. This could be used in turn to implement a metadata

normalization and correction service, such as that described by French, Powell,

Schumann, & Pfaltz (1997) or Lawrence, Bollacker, & Giles (1999).

5.1.4 Bucket Matching

The most compelling demonstration of the BCS is bucket matching. Matching

provides the capability to create linkages between "similar" buckets. Consider a technical

report published by the Old Dominion University computer science department that is

also submitted to a conference. The report exists on the DL maintained by the

department and the publishing authority is: ncstrl.odu_cs. If the conference paper is

accepted, it will eventually be published by the conference sponsor. For example, say the

conference sponsor is the Association for Computing Machinery, whose publishing

authority would be ncstrl.acm. Although the conference paper will surely appear in a

modified format (edited and perhaps abbreviated), the technical report and the conference

paper are clearly related, despite being separated by publishing authority, date of

publication, and editorial revisions. Two separate but related objects now exist, and are

likely to continue to exist.

How best to create the desired linkage between the two objects? It is easy to

assume ncstrl.acm has neither the resources nor the interest to spend the time searching

for previous versions of a manuscript. Similarly, ncstlrl.odu_cs cannot link to the

conference bucket at the creation time of the technical report bucket, since the conference

bucket did not exist then. It is unrealistic to suggest the relevant parties will go back to

the ncstrl.odu_cs archive and create linkages to the ncstrl.acm bucket after six months to a
63

year have passed. However, if both buckets are registeredin the same bucket

communicationspace(byway of sendingtheir metadataor fulltext), they can"find each

other" without humanintervention. Whena match,or near match (the thresholdfor

"match" beinga configurableparameter)is found, the bucketscaneither automatically

link to eachother,or informahumanreviewerthat a potentialmatchhasbeenfound and

requestapprovalfor thelinkage.

Thistechniquecouldalsobeusedto find relatedwork from differentauthorsand

evenduplications(accidentalorplagarious).In the testrunsusingtheNACA portion of

theUniversalPreprintService(seechaptersix), find multi-part reportswerefound (e.g.

Part 1, Part 2), TechnicalNotes (archivalequivalentof a computer sciencetechnical

report)thatwereeventuallypublishedasReports(archivalequivalentof ajournal article),

and a handful of errors where duplicate metadatawas erroneouslyassociatedwith

multiple reports.

5.2 Implementation

While the current BCS implementation lacks the elegance of Linda, it is easy to

implement. Similar to DA, instead of developing an entirely new application for the BCS,

buckets were modified to have BCS-specific methods. Also similar to the DA, none of

the standard bucket methods were removed in the BCS, even though it is not envisioned

that end users working directly with the BCS. Although presented as two separate

buckets, it is possible for a single bucket to be both a BCS server and a dumb archive.

However, BCS differs from DA in that the DA makes no assumption that the objects in

the DA are buckets, but the BCS does assume that all of the objects it has registered are in

fact buckets.

5.2.1 Implemented Methods

Table 9 includes a short summary of the BCS methods, and Appendix D covers

them in detail. "bcs_register", "bcs_unregister", and "bcs_list" are used to manage the

internal data structures for inclusion in the BCS. The BCS uses the DBM variants,

GDBM or NDBM, for its internal storage just as the DA does.
64

"bcs_convert_image"is simply a wrapper to the ImageAlchemy conversion

program(ImageAlchemy, 2000). Any conversionprogramcould be used,suchas the

popularfreewareproductImageMagick(ImageMagick,2000). In fact,it would havebeen

preferableto useImageMagick,notonly becauseit wasfreebut alsobecauseit includesa

Perl module for easy conversionand manipulationfrom inside a script. However,

ImageMagickwas not installedon the developmentmachines,so ImageAlchemy was

usedinstead. It would alsobepossibleto implement"bcs_convert_image"usinga suite

of tools insteadof just one,or to implementa more sophisticatedformat conversion

environment,suchastheTypedObjectModel (TOM) ConversionService(Ockerbloom,

1998).AlthoughImageAlchemysupportsover100imageformats,thecurrentversionof

"bcs_convert_image"only implementsthe popular TIFF, GIF, JPEG,PNG, PostScript

andPDF formatsfor demonstrationpurposes.

TABLE 9. BCSAPI.

Method Description

bcs_convert_image

bcs convert metadata

bcs list

bcs match

bcs_message

bcs_register

bcs_unregister

convertsanuploadedimageto aspecifiedformat

convertsan uploadedmetadatafile to anothermetadatafile

format

listsall thebucketsregisteredwith theBCS

finds& createslinkagesbetweenall "similar" buckets

identifiesbucketsthatmatchaspecificcriteria,andsendsthem

amessage

registersthebucketinto theBCS

unregistersthebucketfrom theBCS

"bcs_convert_metadata"is awrapperfor our own metadatatranslationprogram,

mdt (Nelson,et al, 1999).In the courseof implementingvariousDL projects,a host of

metadatatranslation scripts have been developed- some generalized,some highly
65

specialized. Furthermore,therearea numberof other metadatatranslationprograms

freelyavailable,suchas"bp" (Jacobsen,1996)and"InterBib" (Paepcke,1997).Many of

theseprogramshaveoverlappingformat coverageandnoneperformall conversionswith

equalproficiency. Ideally,"bcs_convert_metadata"shouldbeconstructedfrom theunion

of the best metadataconversionprograms,not just a single one. However, for

demonstration,only mdt is used and the following formats aresupported:refer (Lesk,

1978),bibtex (Knuth, 1986),RFC-1807,Dublin Core(Weibel,Kunze,Lagoze,& Wolfe,

1999),andtheOpenArchivesMetadataSet(OAMS)(VandeSompel& Lagoze,2000).

"bcs_message"searchesthroughall theregisteredbuckets,lookingfor thosethat

matcharegularexpressionpassedin asanargument."bcs_message"caneitherreturnthe

uniqueids / URLs of the matching buckets, and/or send the matching buckets a message

(also passed in as an argument).

"bcs_match" searches through all the registered buckets, either comparing all of

them against all of them, or a list (passed in as an argument) of buckets against all of them.

"bcs_match" considers only the metadata passed in during registration when computing

similarity. To determine similarity, "bcs_match" uses the cosine correlation with

frequency term weighting (CCFTW), first used by Salton & Lesk (1968). Adapting

Harman's (1992) definition of CCFTW to document-document comparison (instead of

document-query), similarity is defined as:

where

similarity (dj,dk) =

n

(tdij " tdik)

i=l

_/ n tdij 2 n

_ tdik 2

i=l i=l

tdij = the ith term in the vector for document j

tdik = the ith term in the vector for document k

n = the number of unique terms in documents j and k
66

TheCCFTWreturnsanumberbetween0 and 1. For the testbedof 3036NACA

documents,it was informally determinedthat auseful thresholdfor similarity was 0.85.

Numbersmuchbelow0.85did notappearsimilaron inspection. Conversely,numbersat

or above0.93werealmostalwaysthe"same"documentpublishedin anotherversion(i.e.,

NACA TN vs.NACA Report). Thesimilarity thresholdis tunableparameter- different

corporamayrequiredifferentthresholds.

 iiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiii iiiiii

i_liiiiiiiiiiiiiiiiiii_::_i_i _iiii_i_i_iiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiii::%ii_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii _................................................................................................................................................. ii_i_i_i_i_i _iiiiii_::::::: _:_:_:i_iii_i_iii_iii_i_iiiiiiiiiiiiiiiiiiiiiiii_i_i_iiiiiii_i_iiiiii_

H|le: Prostate distribution over an NACA 23012 airfoil with an NACA 23012 extemai-airtail flap

auth0_: 'd/enzin_er, Carl J

0_niZ_i0n: NACA Langle9 Memorial Aerooauticai Lai_oratow Lan_lle9 Field Y_ United St_e_

1938

p_eS: 15

_t_¢: Report present_ the re_ult_ of pres_uro-distribution tests of an NACA 23012 airfoil with an NACA 23012

e:ctemai airfoil flap made in the 7 b 9 10-foot wiad tunnel, The pre_ure_ were measured on the upper and lower

_urfaces at one chord _ectioo on both the main airfoil and on the flap for several different Hap del-lection_ and at

.............................. _everai an_Jle_ of attack, A te_t in_tailation w_ used in which the airfoil w_ mounted horizontad9 in the wind

tunnel between vertical end planes so that t'no-dimenaiouai flow w_._ approximated, The data are presentad in

the form of pressure-distribution di_roans and a_ graph_ of calculated coefficient_ for the airfoil-and-flap

combioatian and for the Hlap alone,

up_ id: up_.at_s/cNTIIS,NACA.,naca-report-614

_o|es: (Translated from the 25th AnaivereacQ Number of the Techai_che Hoch_cule, Danzig_ 1904-1929_ pp. 329-343)

_:_._.._'._.!_!!.

il ........................................... I............................................................................................................................................

FIG. 18. NACA bucket before similarity matching.
67

To initiate similarity matchingfor the contentsregisteredwith the BCS, the

followingmessagewouldbesent:

http ://dlib. cs. odu. edu/bcs / ?method=bc s_match

&threshold= 0.9 0&report=on& link=on

The "threshold" argument resets the definition of relevancy for the value returned

from the CCFTW function. The "report" argument stores the results of the matching in

the BCS bucket for later perusal (the default value is to discard the results), and the "link"

argument instructs the BCS server to attempt the buckets that are found to be similar (the

default action is to report the findings but not link). An example of a NACA report before

and after similarity matching is shown in figures 18 and 19.

i::i::i::Ni :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::: _ NN N_::N::i NN _::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::_::__i_i_i_iil

iiiiliiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

tiilei Pressure distribution over an NACA 23012 airfoil with an NACA 23012 e_ternai-airfoil .ap

an_he_ waneaiger,c_d

NACA Langle9 Memorial Aerenanticai Laporator9 Langle9 Field Ve United States

d_ei 1938

_t_ti flaport presents the results of pressure-distribuUan tests of an NACA 23012 airfoil with an NACA

23012 external airfoil flap me.de in the 7 b9 10-foat wind tunnel, The pressures were measured on the

upper and lower surf_es at one chord section on both the main airfoil and an the flap for severat

dilIereat flap deflections and at several an(lies of attack, A test installation was used in which the airfoil

was mounted horizantail9 in the wind tunnel between uedicai end planes so thai two-dimeasionai flow

was approximated. The data are presented in the form of pressure-distribution di_rams and as _raphs

of calculated coefficients for the airfoil-and-flap combination and for the flap alone.

UP_-id', ups,atre//NTflS,NACA,,naca-report-614

(TPanslated from the 25th Aaniver_acy Number of the Techaiache Hachacule, Danzig, 1904-1929, pp,

320-343)

The UPS protoproto contains data from _.._',._'_._.:?[_ NQ.}!_'_!:., _'{_!_'.[_, _._'_._, _"!_._ and 88._,

Implementation coordinated b,_ "!:h_:_.g4_._:!, _!_h_!._!._#._, and !.!#._._:_..V_.._.._._e!,

Sponsored bQ _h#_e_,_!_R_:3_:_!_i_:_:_;_e_#_#_t!_ and e_:!_,

NCSTflL+ based digital libraQ deuelaped b_/Old Dominion Uniuersit_/and NASA Lan(_leQ flaseacch Center,

FIG. 19. NACA bucket after similarity matching.
68

Othersimilaritymeasuresarepossible,includingthe inverteddocumentfrequency

(IDF) measure (Sparck Jones, 1972) and its variations (Sparck Jones, 1979; Croft &

Harper, 1979), and the 2-Poisson model (Bookstein & Swanson, 1974). Also, it could be

possible to implement the algorithms used in duplication and plagiarism detection

systems such as SCAM (Shivakumar & Garcia-Molina, 1995), MDR (Monostori,

Zaslavsky, & Schmidt, 2000), and dup (Baker, 1995a). However, the purpose of

"bcs_match" was not to test which measures are best, but rather to simply prove the

BCS could perform the service.

If a DL is used to discover a bucket with BCS similarity links, the search results

page of the DL will likely have many of the same buckets listed there that are listed as

similarity links in the bucket. However, depending on the search criteria (such as

searching on authors or dates instead of abstract keywords), the search results and

similarity links could also differ significantly. The similarity links provide a fixed

navigation structure for the corpus that does not change as search criteria change. Also,

the presence of BCS similarity links does not preclude the existence of a compliementary

value-added service that performs dynamic searches into other DLs for documents similar

to the current bucket. A dynamic similarity service would offer more flexibility in finding

relevant documents, but it also assumes the continued existence and accessibility of the

service.

5.2.2 Ckanges From a Regular Bucket

A BCS bucket is similar to a regular bucket, but with the seven new bcs_*

methods in the methods, pkg package. Also, a new package, bcs.pkg, is added to

contain all the support libraries, programs and data files for BCS operation.

The resulting BCS bucket is larger and even less mobile than the DA. A BCS

bucket carries two of its support programs with it: mdt and Image Alchemy. mdt is a

Perl program, so it is portable, but Image Alchemy is binary program (approximately

3MB) that is obviously not portable between architectures. BCS buckets also use DBM
69

variations for their registration data structures, and thus inherit their portability

limitations. However, it is unlikely that BCS buckets will be expected to be mobile, and

since a site will probably not have more than one BCS bucket (or at most a few), their

increased size should not be a problem.

5.3 Discussion

It should be stressed that the newer BCS buckets lag behind regular buckets in

their development and maturity, regular buckets having the benefit of several years of

testing in production environments. Combined with the fact that BCS buckets perform

more sophisticated tasks, a review of BCS operation should be considered proof-ofconcept of BCS operation, and not the final representation of their capability and

performance profile.

5.3.1 Performance Considerations

Two of the BCS methods have significant time requirements for their operation,

and are the first targets for optimization. "bcs_message" performs a linear search through

all the registered buckets searching for those that match the requested regular expression.

If the BCS bucket used an existing search engine, or implemented its own inverted files for

this purpose, "bcs_message" would run in O(logn) time.

Even more inefficient is the "bcs_match" method, which currently runs in O(n2).

This is because the default case is to compare everything to everything else. With the

3036 NACA documents in UPS as the testbed, the similarity matching was run on all of

the documents. 3036 documents require 9,217,296 comparisons. However, since

similarity is bi-directional, only half that many were computed. Similarity matching for a

corpus can be thought of as filling a matrix similar to Figure 20. The diagonal is all l's

(since documents are always completely similar to themselves), and the bottom half of

the matrix is simply a duplication of the top half.

The implementation eventually optimized to the point where it could complete

approximately 576,000 comparisons per hour while running on the NACA collection,

finishing the NACA documents in approximately 8 hours. The largest collection
70

"bcs_match"hasactuallybeentestedonis 6,867documents(UPSNACA (3036)+ UPS

Math(3831)). Similarity matchingon this collectionranin approximately42 hours,for

approximately 561,000 comparisonsper hour. The final results found no similar

documentsbetweenthetwo collections,and159matchesin theNACA collectionand35

matchesin theMath collection.

Computingsimilarity is hard. For example,eventhoughDienst providesa user

option for rankingthe searchresults by relevancy, it remainsunimplemented.The

commonlyavailableWideAreaInformationServer(WAIS) searchengine(Kahle,Morris,

Davis,Tiene,Hart,& Palmer,1992)implementationsarbitrarily limit the set of returned

documentsto bein therangeof 200 - 450, which allows similarity computationto be

tractable.

id-1 id-2 id-3 id-4 ... id-n

id-1

id-2

id-3

id-4

id-n

1 0.298 0.783 0.267 ... 0.459

"', 1 0.976

i%

not computedsame as above

the diagonal

0.732 ... 0.432

0.868 ... 0.291

1 0.870

'" ,,. 1 0.904

"%1_ 1

FIG. 20. Sample similarity matching matrix.
71

An obviousoptimizationwould be to useinverted files, andperform similarity

matchingon only thosedocumentsthat havea minimumlevel of intersectionbetween

them. This is similarto what a searchenginedoes:findingthe documentsthat havethe

samekeywords as the query, and performing similarity matching only on those

documents.However,"bcs_match"wouldnot gainthesamegreatreductionin the search

spacebecausetypical queriesareonly a fewwords,so filtering throughaninvertedfile is

likely to produceonly a small numberof documents. In "bcs_match",the query is

actuallyanentiredocument,so the searchspacewould not bereducedthe sameamount

asarelativelysmallquery.

Another optimization, somewhatrelatedto the above,is to use a clustering

technique(Rasmussen,1992)to partition the corpusinto a smallernumberof "related"

sections,andperformsimilarity matchingonly within thosepartitions. This is illustrated

in thesimilarity run with the combinedNACA andMath collections. Thatrun took 42

hours,andfoundnomatchesbetweenthe two collections.Whenthe similarity matching

is runonthetwo collectionssequentially,bothrunscanbe completedin approximately8

hourseach.

Both clusteringand inverted files addressthe similarity matchingproblem by

reducingthe searchspace,clearly a necessaryoptimization for an O(n2) algorithm.

However,clusteringmerelypostponestheproblem. If thesizeof the collectiongrowsto

30,000,000documents,andclusteringtechniquesareemployedto producepartitions of,

for example,30,000documents,then similarity matchingwill still requireO(n2) within

that cluster. Theproblemcouldbe further postponedif moreefficientimplementations

of"bcs_match"couldyield dramaticimprovementsover570,000comparisonsperhour.

Another optimizationapproachwould be to exploit the parallelizablenatureof

similarity matching. Considerpartitioningthe similarity matrixsuchthat regionsof the

matrix were assignedto separatecomputers(Fig. 21). No communicationbetween

computershandlingdifferentregionsis necessary;they couldsimplereport their results

backto theBCSserverthatwouldthencollatetheir results. This couldbeaccomplished
72

by harvestingidleworkstation cycles(Kaplan& Nelson, 1994;Baker, 1995b),or even

througha specializedscreensaversimilar to the popular SETI@Home,which taps the

power of idle personal computers (Sullivan, Werthimer, Bowyer, Cobb, Gedye, &

Anderson,1997). This approachto similarity matchingcouldbepursuedindependently

of otherpossibleoptimizations.

Host 2 ]

not computedsame as above

the diagonal

FIG. 21. Partitioning of the similarity matching matrix

5.3.2 Current Limitations

The BCS puts a solid foundation in place, but as yet individual buckets and the

BCS buckets themselves have not tapped its real potential. The BCS does not yet have

the "killer app" needed to unequivocally demonstrate its usefulness. The similarity

matching is a good candidate, but the performance limitations of current implementation

make it less than compelling. Designing and implementing the "perfect" similarity

matching solution is a significant undertaking in its own right. We hope the easily

extensible nature of the BCS buckets will encourage others to optimize existing, or

develop new BCS services.
73

Anotherlimitation is the messagespassedthrough the BCS (or evenbucket-tobucket)aredefinedin termsof thealreadyexistingbucketAPI. Thatis, thereis currently

no way to constructa messagefor a bucketto specify somethingother than what is

availablethroughthebucketAPI. An unexploredrealmwould be defininggeneralbucket

messages,perhapsencodedin the KnowledgeQuery Manipulation Language(KQML)

(Finin, Fritzson, McKay, & McEntire, 1994),to specify abucket's beliefs, desiresand

intentions. In short,bucketsareintelligent,but not asintelligentasthey canbe,andthe

BCSis theinfrastructurethatwill facilitatetheintelligencegrowthof buckets.
CHAPTER SIX

BUCKET TESTBEDS

74

6.1 NCSTRL+

Bucket development was begun within the context of the NCSTRL+ project.

NCSTRL+ is the result of several years of research and development in digital libraries

(Fig. 22). In 1992, the ARPA-funded CS-TR project began (Kahn, 1995) as did LTRS.

In 1993, WATERS (Maly, French, Fox, & Selman, 1995) shared a code base with LTRS.

In 1994, LTRS launched the NTRS, and the CS-TR and WATERS projects formed the

basis for the current NCSTRL (Davis & Lagoze, 2000). In 1997, NCSTRL+ was begun,

drawing from the contents of NCSTRL and NTRS.

, cs-TR, Ncs , [_[[:_l NCSTRL+ _"_

I I I I I I

1992 1993 1994 1995 1996 1997

FIG. 22. NCSTRL+ lineage.

NCSTRL+ was used as the primary testbed for buckets and other DL

technologies from 1997-1999. NCSTRL+ contains approximately 1000 buckets drawn

from LTRS as well as a handful of buckets constructed for various Old Dominion

University research projects. NCSTRL+ also provides distributed searching into the

NCSTRL collection.

6.1.1 Dienst

Dienst was chosen to implement NCSTRL+ instead of other digital library

protocols such as TRSkit (Nelson & Esler, 1997) because of Dienst's success over several

years of production in NCSTRL. Dienst appeared to be the most scalable, flexible, and
75

extensibleof digital library systemssurveyed(Esler & Nelson, 1998),but scalability

limitationswere discoveredin the referenceimplementation(Van de Sompel,Krichel,

Nelson,etal., 2000b).However,thiswasalimit of thereferenceimplementation,not the

protocol itself. Dienst hasalsobeenusedin a variety of DL experiments,suchas the

Networked Digital Library of Thesesand Dissertations(Fox, Eaton,McMillan, et al.,

1997),the ElectronicLibrary for Grey Literature(part of the MeDoc project) (Adler,

Berger,Bruggemann-Klein,et al., 1998), the EuropeanTechnicalReferenceLibrary

(Andreoni, Bruna Baldacci,Biagioni, et al., 1998), the ACM-sponsored Computing

ResearchRepository (CoRR) (Halpern& Lagoze,1999),and most recently the Open

Archives initiative dataharvestingprotocol (Van de Sompel & Lagoze,2000). For

NCSTRL+, Dienst 4.1.8is used. Dienst 5.0 has somesignificantarchitecturalchanges

that makeit less suitablefor theseparticularresearchpurposes. Most notableis the

switch from a distributed searchto a centralizedsearchmechanism.Full distributed

searchingwas abandonedfor the productionversion of NCSTRL because of the low

availability of theentire distributedsystem. One study from 1997-1999found that at

leastoneof thenodesin NCSTRLwasalwaysunavailable(Powell& French,2000).

While Dienst is disciplineindependent,it is currently discipline monolithic. It

makesnoprovision for knowledgeof multiple subjectswithin its system. While it is

possibleto set up a collectionof Dienst serversindependentof NCSTRL, thereis no

provision for linking such collections of serversinto a higherlevel meta-library. A

collection servicehas beenproposed that would allow for partitioning of a server's

holdings(Lagoze& Fielding,1998),but thecollectionserviceisnotin productionuse.

Dienst consistsof 5 components:1) Repository Service;2) Index Service;3)

Meta-Service;4)UserInterfaceService;and5)Library ManagementService.Eachof the

serviceshasa list of valid "verbs" thattheserviceunderstands,andsomeof theverbs can

takearguments.Dienstuseshttp asa transportprotocol. Thestandardformat is:

http://machine.name:port/Dienst/Service/Version/

Verb/Arguments
76

An example of a valid Dienst request is:

http://repository.larc.nasa.gov:8080/Dienst/Meta/

2.0/Publishers/

This contacts the Meta-Server service at repository.larc.nasa.gov and requests a

list of publishing authorities that this machine contains. Dienst names objects in

collections using handles, a URN implementation from the Corporation for National

Research Initiatives (CNRI). NCSTRL+ uses the experimental and unregistered handles

of "ncstrlplus.larc" and "ncstrplus.odu.cs". Meta-data for objects is stored in RFC-1807

format.

The basic architecture of NCSTRL has a single entry point ("home page") for user

access. Each publishing authority (in practice, an authority generally corresponds to a

university department or laboratory) runs its own copy of the Dienst software. The

home page gathers the queries and dispenses the queries in parallel to each server, gathers

the results, and displays the correlated results to the user. To assist with performance

and reliability, Dienst now employs a Regional Meta-Server (RMS) to partition all

NCSTRL participants into geographic regions. The various RMSs share their data with

the Master Meta Server (MSS) at Cornell (the home of Dienst and NCSTRL). A Merged

Index Server (MIS) provides a single index of all the metadata outside a region. A search

query is sent to all standard sites within a region, and to the region's MIS for metadata

outside the region.

6.1.2 Clusters

While Dienst is a successful production quality DL protocol, it has some inherent

limitations that prevent additional features from being added. Among these is the

inability to subdivide collections along anything other than institutional boundaries.

Clusters are a way of aggregating logically grouped sub-collections in a DL along

some criteria. NCSTRL+ provides 4 clusters: organization, archival type, terms and
77

conditions,and subject category. Organizationis the "publishing authority" that is

includedby defaultin Dienst. Archival typeincludesspecifiesthesemantictype,suchas

pre-print, technicalreport, software,datasets,etc. The terms and conditions cluster

specifiesthe accessrestrictionsassociatedwith the object, such as free, passwordor

monetarychargerequired,etc. For subject category,the NASA STI categorieswere

adoptedandmodifiedslightly to belessaerospace-centric(Tiffany & Nelson, 1998). A

two-level hierarchy, there are 11 top-level categories,and each category has

approximately10sub-levelcategories.This provides subjectclassificationthat is broad

andlightweight.

6.2Universal Preprint Service

TheUniversalPreprintService(UPS),which has sincebeenrenamedthe Open

Archivesinitiative(OAi), is a muchlargerDL testbedintroducedin October1999andis

basedon NCSTRL+ software. The UPS prototype was a feasibility study for the

creationof cross-archiveend-userservices.With the premisethat userswould prefer to

have accessto a federationof digital libraries,the main aim of the project was the

identificationof thekey issuesin actually creatingan experimentalend-userservicefor

dataoriginatingfrom important existing,production archives. This includeda total of

almost200,000bucketsharvestedfrom six existingproductionDLs. Table 10providesa

list of thearchivesandtheir contributedcontent.A full discussionof theresults from the

UPSprojectcanbe foundelsewhere(VandeSompel,Krichel, Nelson,et al., 2000a;Van

deSompel,Krichel, Nelson,et al., 2000b). The two key bucket-relatedtechnologiesare

lightweightbucketsandSFXreferencelinkingwithin buckets.
78

TABLE 10. UPSparticipants.

Archive / DL Records in DL Buckets in UPS Buckets Linked to

Full Content

arXiv 128943 85204 85204

www.arxiv.org

CogPrints 743 742 659

cogprints.soton.ac.uk

NACA 3036 3036 3036

naca.larc.nasa.gov

NCSTRL 29680 25184 9084

www.ncstrl.org

NDLTD 1590 1590 951

www.ndltd.org

RePEc 71359 71359 13582

netec.mcc.ac.uk

Totals: 235361 187115 112516

6.2.1 Lightweight Buckets

Only the metadata was harvested from the six archives - not the actual content

itself. While harvesting the full content would have been technically possible, it would

have been storage intensive and would have added little to the cross-archive

demonstration. The resulting buckets were dubbed "lightweight buckets", since they

contained only the metadata and pointers back to the content in the original archives.

However, the lightweight buckets proved to be useful containers for additional material

and value added services that could not be added to the original archive. Although the

BCS was not completed at the time of the UPS demonstration, the lightweight buckets

have since served as mount points for BCS value-added services, such as bucket matching.

There is an entire class of applications where it is desirable to aggregate information about
79

anobject,buttheoriginalobjectcannotbemoveddueto storageconstraintsor intellectual

propertyrestrictions.For UPS,metadatawas aggregatedin multiple formats,(RFC-1807

andReDIF (Cruz & Krichel, 1999)),and usedthe bucketsas attachmentsfor services

suchastheSFXreferencelinking service.

6.2.2 SFX Reference Linking in Buckets

The SFX reference linking service (Van de Sompel & Hochstenbach, 1999) is a

dynamic layer of abstraction between bibliographic information objects and potential

library databases and services. The traditional library has an array of commercial

databases and services, such as:

ISI's Current Contents

ISI's Journal Citation Reports

Ulrich's International Periodicals Directory

Books in Print

Online Public Access Catalogs (OPACs)

Serial Catalogs

Publisher Full-Text Databases (Elsevier, Springer-Verlag, IEEE, etc.)

The number of these services available at a library depends on the nature of the

library, their budget, customer profile, and other factors. The list of services is dynamic,

with services being added and deleted as they become available, fall into disuse or move.

Given all this, static linking between an object and the services applicable to an object is

not feasible. SFX provides a dynamic lookup of the services that are likely to be

available, given the nature of the bibliographic information and a set of heuristics defined

by the local library. For example, a book should produce links to "Books in Print" and

perhaps "Amazon.com", but not "Journal Citation Reports".

The SFX reference linking service was placed in buckets by way of using "SFX

buttons". A button was available for both pre- and post-publication versions of the

work, if both versions were known to be available. Figure 23 shows a UPS bucket with
80

both pre- and post-publication SFX buttons. Of the six constituent archives comprising

UPS, only arXiv, RePEc and NCSTRL received SFX buttons. The SFX server did not

have enough interesting services to warrant SFX buttons for the buckets from the other

three archives. The buttons themselves link to a SFX server, which then queries the

calling bucket to retrieve the bucket's metadata in ReDIF format. The SFX server then

presents an interface to the user showing the various services that are applicable to the

bucket (Fig. 24). The user can correct misspellings in authors' names, volume numbers or

other fields that may have been parsed incorrectly before submitting the request to get

that service.

Experimental Search for Chargino and Neutralino Production in

Supersymmetry Models with a Light Gravitino

au_hor_i Dzero Collaboration, B. Abbott and et al

1997-08-04

_!i_t_n_tu_ Published in Phys.Rev.Lett. 80 (1998) 442-447

Fermilab-Pub-g7/273-E

#bs-_;-a_i We search for inclusive high $E_T$ diphoton events with large

missing transverse energy in Sp\bar{p}$ collisions at

$\sqrt{s}$=l.8 TeV. Such events are expected From pair

production of charginos and neutralinos within the framework of

the minimal supersymmetric standard model with a light gravitino.

No excess of events is observed. In that model, and assuming

gaugino mass unification at the GUT scale, we obtain a 95% CL

exclusion region in the supersymmetry parameter space and lower

mass bounds of 150 GeV/c$A2$ for the lightest chargino and 75

GeV/cSA2$ for the lightest neutralino.

u ps. xxx. hep- ex//x xx. xxx. h ep- ex. 970800 S

nOtes_ 11 pages, 4eps figures, To be Published in Phys. Rev. Letters

_ pre-published version)

(published version)

FIG. 23. A UPS bucket with SFX buttons.
81

::i::i::i::iN_::i::i::i::i::i::EiNI_i::i::i::i::i::_@:i_i_::i::i::i::i::i::i__I_i::i::i::i::i::iHiNF_

h.it

E_peiii

Supe

_hBrsii Dzer_

$997

Ferm

_t_-a_ii we

$\sq_

prod_

the r

No e:

gaug

exclu_

massii

ups,

#Ote_ii 11

FIG. 24. SFX interface.

Since a SFX server provides an interface to a locally defined set of value-added

services (which are often subscription based), each local site is expected to have its own

SFX server. This introduces the complication of having to tell the bucket which SFX

server a particular user should be referencing. Buckets can set a default SFX server

through a bucket preference. SFX server values can also be passed in as arguments to the

"display" method, or by using http cookies. The order of precedence is:
82

1. http argumentto the"display" method("sfx")

2. http cookie("sfx url")

3. bucketpreference("sfx_server")

Thepossiblevaluesfor theSFXserverwill beevaluated,andthelink to the server

is dynamicallybuilt in theHTML displayto theuser. In the UPSprototype, theNCSA

http serverrequired by the Dienst software did not support cookies, so a bucket

preferencewasusedto specify a SFX serverhostedby the University of Ghentfor the

duration of the demonstration. It would normally be the responsibility of the DL

softwareto set either the cookie,or pass in the argumentto the "display" method to

correctly specify the SFX server. If this is not possible,the University of Ghenthas

developed"cookie pusher"scripts that allow a clientto overcomethe limitation of http

cookiesonly beingsentto thesitethat setthem. Using acookiepusher,aclient coulduse

a cookieto point to their localSFX serverevenwhen visiting previouslyunvisited,nonlocalbuckets.

All otherSFXdemonstrationshaveinvolved the modificationof the DL software

to presentSFX buttons during the searchingand displaying of results. The UPS

implementationof SFXreferencelinking demonstratesthatbucketscanbeusedasmount

pointsfor value addedservices,includingthosedevelopedby otherresearchgroups,and

requiringlittle ornomodificationof theDL software. This is especiallyimportant if the

DL softwareis a commercial,non-opensourceproduct. The value-addedservicesare

attachedto thedataobjectitself, so no matterhow thebucketis discovered,the services

will beavailableto theuser.
83

CHAPTER SEVEN

RELATED WORK

7.1 Aggregation

There is extensive research in the area of redefining the concept of "document" or

providing container constructs. In this section we examine some of these projects and

technologies that are similar to buckets.

7.1.1 Kahn/Wilensl_ Framework and Derivatives

Buckets are most similar to the digital objects first described in the

Kahn/Wilensky Framework (Kahn & Wilensky, 1995), and its derivatives such as the

Warwick Framework containers (Lagoze, Lynch, & Daniel, 1996) and its follow-on, the

Flexible and Extensible Digital Object Repository Architecture (FEDORA) (Daniel &

Lagoze, 1997). In FEDORA, DigitalObjects are containers, which aggregate one or more

DataStreams. DataStreams are accessed through an Interface, and an Interface may in

turn be protected by an Enforcer. Interaction with FEDORA objects occurs through a

Common Object Request Broker Architecture (CORBA) (Vinoski, 1997) interface. No

publicly accessible, FEDORA implementations is known to exist at this point, and it is

not known what repository or digital library protocol limitations will be present.

7.1.2 Multivalent Documents

Multivalent documents (Phelps & Wilensky, 2000) appear similar to buckets at

first glance. However, the focus of multivalent documents is more on expressing and

managing the relationships of differing "semantic layers" of a document, including

language translations, derived metadata, annotations, etc. One of the more compelling

demonstrations of Multivalent documents is with geospatial information, with each

valence representing features such as rivers, political boundaries, road infrastructure, etc.

There is not an explicit focus on the aggregation of several existing data types into a single

container. Multivalent documents provide a unique environment for interacting with
84

informationthat mapswell to the semanticsof havingmultiple "layers". Although not

yet attempted,Multivalent documentscouldresideinsidebuckets,effectively combining

thebenefitsof bothtechnologies.

7.1.3 Open Doc and OLE

OpenDoc (Nelson, 1995) and OLE (and its many variations) (Brockschmidt,

1995) are two similar technologies that provide the capability for compound documents.

Both technologies can be summarized as viewing the document as a loose confederation of

different embedded data types. The focus on embedded documents is less applicable to

our digital library requirements than that of a generic container mechanism with separate

facilities for document storage and intelligence. OpenDoc and OLE documents are more

suitable to be elements within a bucket, rather than a possible bucket implementation.

7.1.4 Metaphoria

Metaphoria is a WWW object-oriented application in which content is separated

from the display of content (Shklar, Makower, Maloney, & Gurevich, 1998).

Metaphoria is implemented as Java servlets that aggregate derived data sources from

simple data sources, with possible multiple layers of derived data sources. A simple data

source could be an ASCII file, a WWN¢ page, or an SQL query. Metaphoria parses the

content and makes it available through multiple representations, or document object

models. It has additional presentation enriching capabilities, such as caching and session

management. Metaphoria provides a complex server environment where the main focus is

the dynamic reconstitution and presentation of data sources. As such, Metaphoria could

sit "above" the bucket layer, where it would be used as a highly sophisticated

presentation mechanism for viewing collections of buckets.

7.1.5 VERS Encapsulated Objects

The Victorian Electronic Record Strategy (VERS) focuses on VERS Encapsulated

Objects (VEOs) as a way of preserving the governmental records of Australian state of

Victoria (Waugh, Wilkinson, Hills, & Dell6ro, 2000). VEOs are designed to insure the

long-term survivability of the archived object, with as much encapsulation and textual
85

encodingof its contentsaspossible,evengoingasfar asexpressingbinary dataformatsin

Base64encoding(Borenstein& Freed,1993). A significantdifferencebetweenbuckets

andVEOs is the latter arepurely for archivalpreservation. VEOs areactually XML

objects,andthus haveno computationalcapability of their own. They rely on another

serviceto instantiateandreadthem.

7.1.6 Aurora

The Aurora architecture defines a framework for using container technology to

encapsulate content, metadata and usage (Marazakis, Papadakis, & Papadakis, 1998).

Aurora defines the containers in which arbitrary components can execute, providing a

variety of potential services ranging from shared workspaces, pipelining of electronic

commerce components, and workflow management. Aurora's encapsulation of metadata,

data and access is similar to that of buckets. The Aurora framework of services are

defined in terms of a CORBA-based implementation, and the range of services available in

Aurora reflect the richness and complexity of CORBA.

7.1.7 Electronic Commerce

Two representative electronic commerce (or e-commerce) solutions are "DigiBox"

(Sibert, Bernstein, & Van Wie, 1995) and IBM's "cryptolopes" (Kohl, Lotspiech, &

Kaplan, 1997). Cryptolopes define a three-tier architecture designed to provide potential

anonymity between both the users and providers of information through use of a middle

layer clearinghouse. The goal of DigiBox is "to permit proprietors of digital information

to have the same type and degree of control present in the paper world" (Sibert,

Bernstein, & Van Wie, 1995). As such, the focus of the DigiBox capabilities are heavily

oriented toward cryptographic integrity of the contents, and not on the less stringent

demands of the current average digital library.

E-commerce solutions are highly focused on providing "superdistribution" (Mori

& Kawahara, 1990), where information objects are opaque and can be distributed widely,

but are only fully accessible through use of a key (presumably for sale from a service).
86

Thereappearto benohooksfor DigiBox or cryptolope intelligence.Both arecommercial

endeavorsandarelesssuitableforresearchin value-addedDL services.

7.1.8 Filesystems and File Formats

To a lesser extent, buckets are not unlike some of the proposals from various

experimental filesystems and scientific data types. The Extensible File System (ELFS)

(Karpovich, Grimshaw, & French, 1994) provides an abstract notion of "file" that

includes both aggregation, data format heterogeneity, and high performance capabilities

(striping, pre-fetching, etc.). While ELFS is designed primarily for a non-DL application

(i.e., high-performance computing), it is typical of an object-oriented approach to file

systems, with generic access APIs hiding the implementation details from the

programmer.

The Hierarchical Data Format (HDF) and similar formats (netCDF, HDF-EOS,

etc.) is a multi-object, aggregative data format that is alternatively: raw file storage, the

low-level I/O routines to access the raw files, an API for higher level tools to access, and a

suite of tools to manipulate and analyze the files (Stem, 1995). While HDF is mature and

has an established user base, it is largely created by and for the earth and atmospheric

sciences community, and this community's constraints limits the usefulness of HDF as a

generalized DL application. It is worth noting, however, that buckets of HDF files

should be entirely possible and appropriate.

7.2 Intelligence

Intelligent agent research is an active area. There are many different definitions of

what constitutes an "agent". From Birmingham (1995), we use the following definition:

"Autonomy: the agent represents both the capabilities (ability to

compute something) and the preferences over how that capability is

used. [...]

Negotiation: since the agents are autonomous, they must negotiate

with other agents to gain access to other resources or capabilities.

[...]"
87

Using this definition, it is clear that buckets satisfy the autonomy condition, since

buckets perform many computational tasks that are influenced by their individual

preferences. However, the current implementation of buckets only weakly satisfy the

negotiation condition, since only a handful of transactions have actual negotiation. An

example of such a transaction is the case when a bucket requests metadata conversion

from the BCS; there is a negotiation phase where the requesting bucket and the BCS

server negotiate the availability of metadata formats. However, the direction is clear that

buckets are becoming increasingly intelligent, so they will eventually be considered

unequivocally as true agents.

In practice, the information environment application of intelligent agents has

generally dealt with assistants to aid in searching, search ordering, finding pricing bargains

from on-line sales services, calendar maintenance, and other similar tasks. Birmingham

(1995) defines the three classes of agents in the University of Michigan section of the

NSF funded DLI: User Interface Agents, Mediator Agents, and Collection Interface

Agents. Other projects are similar: agents to help DL patrons (Sanchez, Legget, &

Schnase, 1997), retrieval and access agents (Salampasis, Tait, & Hardy, 1996), and DL

construction/authoring (Sanchez, Lopez, & Schnase, 1998). There appear to be no other

projects that attempt to make archival objects intelligent. Note that making the archived

object intelligent does not preclude the use of other agents in a DL environment (search

agents, collection agents, etc.). In fact, increasingly intelligent buckets should be able to

assist the traditional DL agents in performing their tasks.

7.3 Archives

There has been an increased amount of interest regarding the nature of archives,

specifically in the separation of the roles of providing (or "publishing", or "archiving")

data and of discovering (or "searching") data. In early DL projects, there was often little

distinction but with DLs reaching larger scales and the greater interest in interoperability,

such vertically integrated DLs are no longer feasible. The current highest profile archive
88

projectis theOpenArchivesinitiative, presentedin ChapterSix. However,anumberof

otherprojectshavedemonstratedthis separationofrolesaswell.

TheGuildfordProtocol(Cruz& Krichel, 1999)hasbeenin usein the Economics

communityfor quitesometimewithin the RePEcproject (a participantin UPS). RePEc

is uniquein that it specifiesnouser services,but only focuseson the coordinationand

propagationof distributedcollectionsof metadata.SeveralDLs havebeenbuilt from the

metadataharvestedfrom RePEc. Dienst hasa Repository service,a portion of which

forms the basis for the OAi harvestingprotocol. Unfortunately, the full Dienst

Repository servicealsocontainsthe conceptof a documentmodel. The inclusionof a

documentmodelin an archivalserviceis too heavy and limits the applicationsof that

archival service. Stanford has proposed archival awarenessalgorithms, in which

distributedarchivescanmaintainconsistencyin thefaceof updatesanddeletions(Crespo

& Garcia-Molina,1997).

There are a number of other possible implementationsfor archival services.

Althoughusinga RDBMS or the light-weighdirectory accessprotocol (LDAP) (Yeong,

Howes,& Kille, 1995)seemsto be an obvious implementation,there appearto be no

sucharchiveimplementationwithin theDL community.

7.4 Bucket Tools

Related to buckets, but being developed separately by another team at Old

Dominion University, are tools for bucket creation, administration and simple workflow

management. The need for high quality tools for the creation and management of buckets

is obvious: no matter how expedient and useful buckets may be, if they are not created in

sufficient quantities they will not be adopted on a large scale. The bucket tools are

needed to hide the details of bucket creation from ordinary users, and if bucket tools use

the bucket API for all of their actions, the tools should be applicable across bucket

implementations.

For batch creation of buckets, a number of scripts have been developed to

automate the process. However, they tend to be specialized for the DL they are
89

populatingandarenot really worthy of beingcalleda "tool". A suiteof threeintegrated

toolshasbeencreatedthatcanbeinstalledata localsiteto providea simpleauthor-based

publishingworkflow. Thesetools shouldbe considereda referenceimplementationof

possibletools for buckets- additionalimplementationsor tools of different naturesare

possible.Theprocessbeginswith anauthoraccessingthe CreationTool (Fig. 25),which

is usedto createandpopulate a bucket. The bucketsarekept in a temporary staging

archive,whereonly the author canaccessthe bucket. The population canoccurover

manysessions,with theauthorsavingtheintermediatebucketattheendof eachsession.

_iii_iiiiiiiiiiii_iiiiiiiiiiiiiii_iiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiii_iiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiii_iiiiiI

_f_::"""""......................................... "............................... _!!XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX::_::_::_::_::_::_:_" .............................................

NCSTRL+ Creation Tool :iill..........

............... : .......................... _t.45,A a_;.'d C_>!_ _o_,i_io_ U_}iv_,_y

FIG. 25. Creation tool.
90

Whenthe authoris donewith the bucket, it can be submitted for review by

management.Thisphysicallymovesthebucketfrom its temporarystagingarchiveto the

managementarchive.Themanagereceivesanemailstatingthatanewbuckethasarrived.

Themanagerwill thenusetheManagementTool(Fig. 26)to reviewthependingbuckets.

Themanagercanapprove,disapproveor deferactiononthelist of buckets. Disapproved

bucketswill gobackto theauthor'stemporarystagingarchive,andapprovedbucketswill

bepromoted to the site's official archive. It is the official archivethat DLs know to

harvestfrom, andplacementinto thisprivilegedarchiveconstitutes"publishing".

i_iii _i _iii _i_iii _iii _iii_iii _ii _iii_i __ii _ii _i _i _iii_i _i _iii _i_i_i _iii _i __iii _i _i _ii _i_ii _iii_iii _i_iii _ _iii_i _iii _i _i _i_iii _i_i___m'

ii_::i::i::i::i_:!!_J_ _ ..........._------_-------_::_::_::-;:_ ................. _--_--------------_::_::_ P !................7_- .........•............•............:............................................ g .................................iiiii_:_iiiiii_iiiiiiiii_:_:_::i:_:_:_:_:_i_:_::i:_:_:iiii_:_:_:_:_::_:i:_:_:_

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_!_i!iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

FIG. 26. Management tool.
91

Finally, the Administration Tool (Fig. 27) provides a mechanismfor the

maintenanceof alreadypublishedbuckets. This tool provides a standardinterfaceto

multi-castmessagesto multiplebuckets,for suchfunctionsas harvestingtheir log files,

updatingpreferences,addinggeneralvalue-addedservices,andall suchfunctionsthat are

notpart of thecreation/approvalprocess.

_iiiii_iiiiiii_ii_iiiiiii_iiiiiiiiiiiiiiiiii_N_iiiiiiii_iiiiii_N_iiiiiiiiiiiiiiii_iiiiiiii_iiiiiiii_iiiiiiiiiiiiiiiiiiii_!_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

NOSTRL+Administration Tool "

ii

Name Of Group

Delete C_oup

FIG. 27. Administration tool.
92

CHAPTER EIGHT

FUTURE WORK

8.1 Alternate Implementations

The lessons learned from implementing buckets and the supporting technology,

along with their success and popularity in NCSTRL+ and UPS, point to many areas of

future work, of both practical and academic interest. Perl and CGI are good development

platforms. Perl is expressive, commonly available and reasonably fast. However, there

are a number of other interesting languages that deserve study. Plain CGI is low

performance, often running at 10% - 50% of the performance of regular http requests for

small transactions, with Perl CGI programs performing slightly worse than C CGI

programs (McGrath, 1996).

8.1.1 Buckets

There are three, possibly overlapping, areas for alternate implementation for

buckets: rewriting buckets in another language, replacing CGI, and making the API

available through something other than http.

Buckets could be rewritten in another scripting language like Python (Lutz, 1996)

or Tcl (Ousterhout, 1994). Buckets currently exploit some of the Perl idioms, and its

possible that Python, Tcl or another similar language could easily add new capabilities.

An obvious area to explore is writing a Java (Arnold & Gosling, 1996) implementation of

buckets - given the proliferation of Java compilers embedded in a variety of applications

and even hardware devices. The run-time performance gains of implementing buckets in a

compiled language such as C/C++ would probably be negated by the lack of mobility

resulting from "compiled buckets".

Replacing CGI would be a big performance win in high-traffic DLs. One example

relevant to Perl buckets is that they could be written to use rood perl, where the Perl

interpreter is actually embedded into the Apache http server (Stein, MacEarchern, Mui,
93

1999). This eliminatesthe costly overheadof CGI. Similarly, Javabuckets could use

servlets(Hunter,Crawford,& Ferguson,1998)for performancegains.

If thebucketAPI was availableto runningprogramsthrougha mechanismmore

familiarandnaturalfor applicationsprograms,suchasCORBA, thenthe capabilitiesof

bucketswould be greatly expanded. It would be easierfor running programs to

automaticallyretrieve and fill buckets as they are running without convertingtheir

requeststo http. Probablythesimplestway to providethis capability would bethrough

aCORBA gatewaythatconvertedmessagesto/frombucketmethods.

8.1.2 Dumb Archives

The current implementation of an archive service is a modified bucket which has

its methods extended to include basic set management functionality. While it would be

possible to add more sophisticated methods to DA, and the orthogonal approach of using

a modified bucket has an aesthetic appeal, it is also possible to implement archive services

with other technologies as well. DA functionality could be implemented using a RDBMS

or LDAP. The OAi harvesting protocol (Van de Sompel & Lagoze, 2000) is a subset of

the Dienst protocol, including only those verbs related to archive management. The

proposed collection service for Dienst (Lagoze & Fielding, 1998) could also be the

foundation for an implementation of the Smart Object, Smart Archive (SOSA) model.

8.1.3 Bucket Communication Space

While alternate archive implementations could be constructed or adapted without

requiring specific knowledge about buckets, this would be more difficult for the BCS,

since it performs services specific to buckets. A modified bucket might not be the most

efficient implementation for the BCS, but unlike for the DA, there are no obvious

candidates for alternate BCS implementations.

8.2 Extended Functionality

Contrasting with implementing the same functionality in different languages or

environments, there are a number of new functionalities that could be implemented in the
94

short term. Theseincludeusingpre-definedbucketpackagesand elementsandXML

metadatasupport.

8.2.1 Pre-defined Packages and Elements

Some functionality improvements could be made not through new or modified

methods, but through conventions established on the current infrastructure. One

convention already adopted was the use of a BCS_S±m±lar±ty. pkg package to hold

the resulting links of the BCS similarity indexing. Other possible uses include: standard

element names for bucket checksums (entire bucket, packages or elements) to insure the

integrity of elements; standard packages (or elements) for bibliographic citation

information, possibly in multiple encodings; or standard package or element names for

previous revisions of bucket material. Conventions are likely to be adopted as need and

applications arise.

8.2.2 XML Metadata

Perhaps the most urgent improvement for buckets involves removing the RFC1807 historical dependency and using XML as its canonical metadata set. The current

modified RFC-1807 format dictates a generic, but inflexible two-level bucket structure.

A challenge for XML support is that until XML parsers are ubiquitously available (e.g.,

included in the Perl standard library), the bucket will have to internally be able to parse

XML files. Switching to XML should allow the modeling of arbitrarily complex data

objects, and the use of a "bucket style sheet" should allow the display method to more

easily change the bucket presentation without specific code changes in the "display"

method itself. XML support would also allow the use of the Resource Description

Framework (RDF) (Miller, 1998). RDF would allow for greater metadata expressiveness,

facilitating the sharing of semantic metadata models within a standard framework.

Adding lightweight XML parser support, along with style sheets and generalized

handling of metadata will be a difficult task. All of this is possible, but the parser must be

compact enough to not greatly increase the bucket's storage requirement and not

introduce dependencies that would damage the mobility and self-sufficiency of buckets.
95

8.2.3 More Intelligence

There are a number of functions that buckets for which buckets already have

hooks in place, but have not yet been fully automated. For example, the "lint" method

can detect internal errors and misconfigurations in the bucket, but it does not yet attempt

to repair a damaged bucket. Similarly, a bucket preference could control the automatic

updating of buckets when new releases are available, while still maintaining the bucket's

own configuration and local modifications. The updated bucket could then be tested for

correct functionality, and rolled back to a previous version if testing fails. The option of

removing people from the bucket update cycle would ease a traditional administration

burden.

Buckets could also be actively involved in their own replication and migration, as

opposed to waiting for human intervention for direction. Buckets could copy themselves

to new physical locations so they could survive physical media failures, existing either as

functioning or dormant replicates. Should the canonical bucket be "lost" somehow,

buckets could vote among themselves to establish a new priority hierarchy. Distributed

storage projects such as the Archival Intermemory (Goldberg & Yianilos, 1998) or

Internet2 Distributed Storage Infrastructure Project (Beck & Moore, 1998) could serve as

complementary technologies for implementing migratory buckets.

8.3 Security, Authentication and Terms & Conditions

While every effort has been made to make buckets as secure and safe as possible, a

full-scale investigation by an independent party has not been performed. A first level of

investigation would be in attacking the buckets themselves, to determine if the buckets

could be damaged, made to perform actions prohibited by their T&C files, or otherwise be

compromised. A second level of investigation would be examining if buckets could be

compromised through side effects resulting from attacks on other services. Currently,

buckets have no line of defense if the http server or the system software itself is attacked.

Having buckets employ some sort of encryption on their files that is decoded

dynamically would offer a second level of security, making the buckets truly opaque data
96

objectsthat could withstand at least somelevel of attack if the system software was

compromised.

Authenticationis currentlydonethroughthe standardhttp procedures,relyingon

the serverto correctly set the value of REMOTE_ADDR, REMOTE_HOST, and

REMOTE_USER. Authentication alternativesusing Kerberos(Steiner,Neuman, &

Schiller,1988),MD5 (Rivest, 1992), or X.509 (CCIT, 1998) should be exploredso

bucketscanfit into a variety of large-scaleauthenticationschemesin use at various

facilities.

TheT&C modelusednow is simple,anddoesnot allow for complexexpressions,

especiallynestingof directives.For example,adisplay, tc file couldcontain:

user:nelson

user:maly

host:.*.larc.nasa.gov

host:.*.cs.odu.edu

package:datasets.pkg

package:software.pkg

The above imposes all the user and host restrictions on both packages. There is

currently no way to have different restrictions on different packages or elements within a

single T&C file. XML is a natural format to implement the upgraded T&C files, so that a

T&C file could be constructed to easily describe the hierarchy and relationship of the

desired T&C. An example would look something like:

<?xml version="l.O"?>

<package name="datasets.pkg">

<user name="nelson" \>

<host name=".*.larc.nasa.gov"\>

</package>

<package name="software.pkg">

<user name="maly" \>

<host name=".*.cs.odu.edu"\>

</package>
97

8.4 New Applications

Buckets provide a base level of functionality that is immediately useful.

However, reflection on the capabilities buckets provide soon causes one to think of the

additional capabilities that buckets could provide in the future.

8.4.1 Discipline-Specific Buckets

Buckets are currently not specific to any discipline; they have a generic "one-sizefits-all" approach. While this is attractive for the first generation of buckets since it

excludes no disciplines, it also does nothing to exploit assumptions and extended features

of a specific discipline. Intuitively, an earth science bucket could have different

requirements and features than a computational science bucket. Given a scientific

discipline, it could be possible to define special data structures and even special methods

or method arguments for the data, such as geo-spatial arguments retrieving data from

earth-science buckets or compilation services for a computational science bucket.

Generalized XML support in the bucket (discussed above) would simplify

tailoring buckets to different ontologies. Buckets could begin as generic buckets, then

acquire specific "skins" (in computer-game parlance) that would dictate their look and feel

as well as their functionality.

8.4.2 Usage Analysis

There are several DL projects that focus on determining the usage patterns of their

holdings and dynamically arranging the relationships within the DL holdings based on

these patterns (Bollen & Heylighen, 1997; Rocha, 1999). All of these projects are similar

in that they extract usage patterns of passive documents, either examining the log files of

the DL, or instrumenting the interface to the DL to monitor user activity, or some hybrid

of these approaches. An approach that has not been tried is for the objects themselves to

participate in determining the usage patterns, perhaps working in conjunction with

monitors and log files. Since the buckets are executable code, it is possible to not just

instrument the resource discovery mechanisms, but the archived objects also. We have
98

experienceinstrumentingbucketsto extractadditionalusagecharacteristics,but we have

not combinedthis strategywith thatof theotherprojects.

8.4.3 Software Reuse

Buckets could have an impact in the area of software reuse as well. If a bucket

stores code, such as a solver routine, it would not have to be limited to a model where

users extract the code and link it into their application, but rather the bucket could

provide the service, and be accessible through remote procedure call (RPC)-like

semantics. Interfaces between distributed computing managers such as Netsolve

(Casanova & Dongarra, 1998) or NEOS (Czyzyjk, Mesnier, & More) and "solver

buckets" could be built, providing simple access to the solver buckets from running

programs. Data, and the routines to derive and manipulate it, could reside in the same

bucket in a DL. This would likely be tied with a discipline specific application, such as a

bucket having a large satellite image and a method for dynamically partitioning and

disseminating portions of the data.

Or users could temporarily upload data sets into the bucket to take advantage of a

specialized solver resident within the bucket without having to link it into their own

program. This would be especially helpful if the solver had different system

requirements, and it could not easily be hosted on a user's own machine. However, the

traditional model of "data resides in the library; analysis and manipulation occurs outside

the library" can be circumvented by making the archived objects also be computational

objects.
99

CHAPTER NINE

RESULTS AND CONCLUSIONS

Buckets were born of our experience in creating, populating and maintaining

several production DLs for NASA. The users of NASA DLs repeatedly wanted access

to data types beyond that of the technical publication, and the traditional publication

systems and the digital systems that automated them were unable to adequately address

their needs. Instead of creating a raft of competing, "separate-but-equal" DLs to contain

the various information types, a container object was created capable of capturing and

preserving the relationship between any number of arbitrary data types.

Buckets are aggregative, intelligent, WWW-accessible digital objects that are

optimized for publishing in DLs. Buckets implement the philosophy that information

itself is more important than the DL systems used to store and access information.

Buckets are designed to imbue the information objects with certain responsibilities, such

as the display, dissemination, protection and maintenance of its contents. As such,

buckets should be able to work with many DL systems simultaneously, and minimize or

eliminate the necessary modification of DL systems to work with buckets. Ideally,

buckets should work with everything and break nothing. This philosophy is formalized

in the SODA (Smart Object, Dumb Archive) DL model. The objects become "smarter" at

the expense of the archives (who become "dumber"), as functionalities generally

associated with archives are moved into the data objects themselves. This shift in

responsibilities from the archive into the buckets results in a greater storage and

administration overhead, but these overheads are small in comparison to the great

flexibility that buckets bring to DLs.

This research has successfully met the objectives as stated in Chapter Two. First,

the concept of "buckets" was introduced, which is the collection of mechanisms and

protocols for aggregating and mobilizing content and services on the content. A well-
100

defined bucket API (Appendix B) is the result of previous DL experience and study of

the bucket concept. Secondly, a reference implementation of buckets was developed that

is written in Perl that uses http and CGI mechanisms for transport of bucket messages.

This reference implementation fully implements the bucket API. Other research projects

are investigating implementing the bucket API using other technologies, including Java

servlets and the Oracle RDBMS. Lastly, the bucket concept and the Perl-based reference

implementation were demonstrated in a variety of application and DL deployments.

Research project buckets, university class buckets, and traditional STI publication

buckets were created. STI publication buckets were demonstrated in great numbers in the

NCSTRL+ and UPS experimental DLs. To facilitate the adoption of buckets, other

projects have introduced support tools for buckets, most notably a Creation Tool,

Management Tool and Administration Tool.

Buckets have demonstrated their flexibility in a number of ways. First, for the

UPS project "light-weight buckets" emerged as a useful variation of buckets and it was

easy to augment buckets with value-added services such as the SFX reference linking

service, and then later similarity matching links. The extensibility of buckets was further

demonstrated when the creation of archive services (DA) and the Bucket Communication

Space were implemented using modified buckets. Since any ordinary bucket could be

turned into a DA or BCS bucket through a well-defined series of transformations, this

approach showed the orthogonality of buckets in a variety of applications. Buckets are

general purpose, stand-alone, WWW-accessible DL workhorses.

There are a number of projects that have similar aggregation goals as buckets.

Some are from the DL community, and others are from e-commerce and computational

science. Most do not have the SODA-inspired motivation of freeing the information

object from the control of a single server. The mobility and independence of buckets are

not seen in other DL projects. Most DL projects that focus on intelligence or agency are

focused on aids to the DL user or creator; the intelligence is machine-to-human based.
101

Buckets are unique because the information objects themselves are intelligent, providing

machine-to-machine (or, bucket-to-bucket) intelligence.

Buckets are already having a significant impact in how NASA and other

organizations such as Los Alamos National Laboratory, Air Force Research Laboratory,

Old Dominion University, and the NCSTRL steering committee are designing their next

generation DLs. The interest in buckets has been high, and every feature introduced

seems to raise several additional areas of investigation for new features and applications.

First and most important, the creation of high quality tools for bucket creation,

management and maintenance in a variety of application scenarios is absolutely necessary.

Without tools, buckets will not be widely adopted. Other short-term areas of

investigation include optimized buckets, alternate implementations of buckets, disciplinespecific buckets, XML support, and extending authentication support to include a wider

variety of technologies. Long-range plans include significant utilization of bucket

mobility and bucket intelligence, including additional features in the Bucket

Communication Space. Buckets, through aggregation, intelligence, mobility, selfsufficiency, and heterogeneity, provide the infrastructure for information object

independence. The truly significant applications of this new breed of information objects

remain undiscovered.
102

REFERENCES

Adler, S., Berger, U., Bruggemann-Klein, A., Haber, C. Lamersdorf, W., Munke, M.,

Rucker, S. & Spahn, H. (1998). Grey literature and multiple collections in

NCSTRL. In A. Barth, M. Breu, A. Endres & A. de Kemp (eds.), Digital libraries

in computer science: the MeDoc approach (pp. 145-170), Berlin: Springer.

Andreoni, A., Bruna Baldacci, M., Biagioni, S., Carlesi, C., Castelli, D., Pagano, P. &

Peters, C. (1998). Developing a European technical reference digital library. In S.

Abiteboul & A.-M. Vercoustre (eds.), Research and advanced technology for

digital libraries, third European conference, ECDL '99 (pp. 343-362), Berlin:

Springer.

Arms, W. A. (1999). Preservation of scientific serials: three current examples. Journal of

Electronic Publishing, 5(2). Available at http://www.pres.umich.edu/jep/05-

02/arms.html

Arnold, K. J., & Gosling, J. (1996). The Java programming language. Reading, MA:

Addison-Wesley.

Baker, B. S. (1995a). On finding duplication and near-duplication in large software

systems. Proceedings of the second IEEE working conference on reverse

engineering (pp. 86-96), Toronto, Canada. Available at http://cm.belllabs.com/cm/cs/doc/95/2-bsb-3.pdf

Baker, M. (1995b). Cluster computing review. Syracuse University Technical Report

NPAC SCCS-748. Available at

http://www.npac.syr.edu/techreports/html/0700/abs-0748.html

Beck, M. & Moore, T. (1998). The Internet2 distributed storage infrastructure project:

an architecture for Internet content channels. Computer Networking and ISDN

Systems, 30(22-23), 2141-2148. Available at http://dsi.internet2.edu/pdf-docs/i2-

chan-pub.pdf

Bennington, J. (1952). The integration of report literature and journals. American

Documentation, 3(3), 149-152.

Bennion, B. C. (1994, February/March). Why the science journal crisis? ASIS Bulletin,

25 -26.
103

Bemers-Lee, T., Cailliau, R., Groff, J.-F., & Pollermann B. (1992).World-Wide Web: the

information universe. Electronic Networking: Research, Applications and Policy,

2(1), 52-58.

Birmingham, W. P. (1995). An agent-based architecture for digital libraries. D-Lib

Magazine, 1(7) July 1995. http://www.dlib.org/dlib/July95/07birmingham.html

Bollen, J. & Heylighen F. (1997). Dynamic and adaptive structuring of the World Wide

Web based on user navigation patterns. Proceedings of the Flexible Hypertext

Workshop (pp. 13-17), Southhampton, UK. Available at

http://www.c3.1anl.gov/-jbollen/pubs/Bollen97.htm

Bookstein, A. & Swanson, D. R. (1974). Probabilistic models for automatic indexing.

Journal of the American Society for Information Science, 25, 312-319.

Borenstein, N. & Freed, N. (1993). MIME (multipurpose Internet mail extensions) part

one: mechanisms for specifying and describing the format of Internet message

bodies. Internet RFC- 1521. Available at ftp://ftp.isi.edu/in-notes/rfc 1521 .txt

Brenner, S. (2000). The cgi-lib.pl homepage. Available at http://cgi-lib.berkeley.edu/

Brockschmidt, K. (1995). Inside OLE 2. Redmond, WA: Microsoft Press.

Carriero, N. & Gelernter, D. (1989). Linda in context. Communications of the ACM,

32(4), 444-458.

Casanova, H. & Dongarra, J. (1998). Applying Netsolve's network-enabled solver. IEEE

Computational Science & Engineering, 5(3), pp. 57-67.

CCITT (1998). The directory authentication framework. CCITT Recommendation

X.509.

Crespo, A. & Garcia-Molina, H. (1997). Awareness services for digital libraries. In C.

Peters & C. Thanos (eds.), Research and advanced technology for digital libraries,

first European conference, ECDL '97 (pp. 147-171), Berlin: Springer.

Croft, W. B. & Harper, D. J. (1979). Using probabilistic models of document retrieval

without relevance information. Documentation, 35(4), 285-295.

Cruz, J. M. B. & Krichel, T. (1999). Cataloging economics preprints: an introduction to

the RePEc project. Journal of Internet Cataloging, 3(2-3).
104

Czyzyk, J., Mesnier, M. P. & More, J. J. (1998). The NEOS solver. IEEE

ComputationalScience& Engineering,5(3),pp.68-75.

Daniel, R. & Lagoze,C. (1997). Distributed active relationships in the Warwick

framework. Proceedingsof the secondIEEE metadataworkshop, Silver Spring,

MD.

Davis,J.R. & Lagoze,C. (1994). A protocolandserverfor adistributeddigitaltechnical

report library. Cornell University Computer ScienceTechnicalReport TR94-

1418.Availableat

http://ncstrl.cs.cornell.edu/Dienst/UI/1.0/Display/ncstrl.cornell/TR94-1418

Davis,J.R.,Fielding,D.,Lagoze,C.& Marisa,R. (2000).

OpenArchivesDienstsubset.Availableat

http://www.openarchives.org/sfc/sfc_dienst.htm

The SantaFe convention:the

Davis, J. R. & Lagoze,C. (2000). NCSTRL: designand deployment of a globally

distributed digital library. Journal of the American Society for Information

Science,51(3),273-280.

Esler, S.L. & Nelson,M. L. (1998).Evolution of scientific and technical information

distribution. Journalof the AmericanSocietyfor InformationScience,49(1),82-

91.Availableat http://techreports.larc.nasa.gov/ltrs/PDF/1998/jp/NASA-98-jasissle.pdf

Fielding,R.,Gettys,J.,Mogul J.C., Frystyk,H., Masinter,L., Leach,P. & Berners-Lee,

T. (1999). Hypertext transfer protocol -HTTP/1.1. Internet RFC-2616.

Availableatftp://ftp.isi.edu/in-notes/rfc2616.txt

Finin, T., Fritzson, R., McKay, D. & McEntire, R. (1994). KQML as an agent

communicationlanguage. Proceedingsof the third internationalconferenceon

information and knowledgemanagement(pp. 447-455), Gaithersburg,MD.

Availableathttp://www.cs.umbc.edu/kqml/papers/kqml-acl.ps

Fox, E. A., Eaton, J. L., McMillan, G., Kipp, N. A., Mather, P.,

Schweiker,W. & DeVane,Brian. Networked digital library

dissertations.D-Lib Magazine,3(9). Availableat

http://www.dlib.org/dlib/september97/theses/09fox.html

McGonigle, T.,

of theses and

Frakes,W. B. & Baeza-Yates,R. (1992). Information retrieval: data structures &

algorithms.UpperSaddleRiver,NJ:Prentice-Hall.
105

French,J. C., Powell, A. L., Schulman,E. & Pfaltz, J. L. (1997). Automating the

constructionof authorityfiles in digital libraries:a casestudy. In C. Peters& C.

Thanos (eds.), Researchand advancedtechnology for digital libraries, first

Europeanconference,ECDL ' 97(pp.55-71),Berlin:Springer.

Ginsparg,P. (1994).First stepstowardselectronicresearchcommunication.Computers

in Physics,8, 390-396.

Goldberg,A. V. & Yianilos,P.N. (1998). Towardsanarchivalintermemory.Proceedings

of the IEEEforum on researchand technologyadvancesin digital libraries(pp.

147-156),SantaBarbara,CA.

Gray, D. E. (1953). Organizing and servicing unpublished reports.

Documentation4(3), 103-115.

American

Griffin, S. M. (1999). Digital Library Initiative - phase2. D-Lib Magazine,5/(7-8).

Availableathttp://www.dlib.org/dlib/july99/07griffin.html

Griffiths, J.-M.& King, D. W. (1993). Speciallibraries:increasingthe informationedge.

Washington,DC: SpecialLibrariesAssociation.

Halpern,J.Y. & Lagoze,C. (1999). TheComputingResearchRepository:promotingthe

rapid disseminationandarchivingof computerscienceresearch.Proceedingsof

thefourthACM conferenceondigital libraries(pp.3-11),Berkeley,CA.

Harman,D. (1992). Rankingalgorithms. In W. B. Frakes& R. Baeza-Yates(Eds.),

Informationretrieval:dataStructures& algorithms(pp. 363-392),Upper Saddle

River,NJ:Prentice-Hall.

Harnad,S.(1997). How to fast-forwardserialsto the inevitableand the optimal for

scholarsandscientists.SerialsLibrarian,30,73-81.Availableat

http://www.cogsci.soton.ac.uk/-harnad/Papers/Harnad/harnad97.1earned.serials.ht

ml

Henderson,A. (1999). Information scienceandinformationpolicy: the useof constant

dollars and other indicators to manageresearchinvestments. Journal of the

AmericanSocietyfor InformationScience,50(4),366-379.

Hunter,J., Crawford,W. & Ferguson,P. (1998). Javaservletprogramming.Sebastopol

CA: O'Reilly & Associates.

ImageAlchemy (2000).Availableathttp://www.handmadesw.com/his/specs.html
106

ImageMagick (2000). Available at

http://www.wizards.dupont.com/cristy/ImageMagick.html

Jacobsen, D. (1996). bp, a Perl bibliography package. Available at

http://www.ecst.csuchico.edu/-jcabosd/bib/bp/

Kahle, B., Morris, H., Davis, F., Tiene, K., Hart, C., & Palmer, R. (1992). Wide area

information servers: an executive information system for unstructured files,

Electronic Networking: Research, Applications, and Policy, 2(1), 59-68.

Kahle, B. (1997). Preserving the Internet. Scientific American, 264(3).

Kahn, Robert E. (1995). An introduction to the CS-TR project. Available at

http://www.cnri.reston.va.us/home/describe.html

Kahn, R. & Wilensky, R. (1995) A framework for distributed

cnri.dlib/tn95-01. Available at

http://www.cnri.reston.va.us/home/cstr/arch/k-w.html

digital object services.

Kaplan, J. A. & Nelson, M. L. (1994). A comparison of queueing, cluster and distributed

computing systems. NASA Technical Memorandum 109025. Available at

http ://techreports. larc.nasa, gov/ltrs/PDF/tm 109025 .pdf

Karpovich, J. F., Grimshaw, A. S. & French, J. C. (1994). Extensible file systems (ELFS):

an object-oriented approach to high performance file I/O. Proceedings of the ninth

annual conference on object-oriented programming systems, languages and

applications (pp. 191-204), Portland, OR.

Kohl, U., Lotspiech, J. & Kaplan, M. A. (1997). Safeguarding digital library contents and

users. D-Lib Magazine, 3(9). Available at

http://www.dlib.org/dlib/septemeber97/ibm/lotspiech.html

Knuth, D. E. (1986). The TeXbook. Reading, MA: Addison-Wesley.

Lagoze, C. & Ely, D. (1995). Implementation issues in an open architectural framework

for digital object services. Cornell University Computer Science Technical Report,

TR95-1540. Available at

http ://ncstrl.cs.cornell. edu/Dienst/UI/1.0/Display/ncstrl.cornell/TR95-1540

Lagoze, C., Shaw, E., Davis, J. R., & Krafft, D. B. (1995). Dienst: implementation

reference manual. Cornell University Technical Report TR95-1514. Available at

http://ncstrl.cs.cornell.edu/Dienst/UI/1.0/Display/ncstrl.cornell/TR95-1514
107

Lagoze,C., Lynch C. A., & Daniel,R. (1996). The Warwick framework:a container

architecturefor aggregatingsets of metadata. Cornell University Computer

ScienceTechnicalReportTR-96-1593.Availableat

http://ncstrl.cs.cornell.edu/Dienst/UI/1.0/Display/ncstrl.cornell/TR96-1593

Lagoze,C.& Fielding,D. (1998). Defining collectionsin distributeddigital libraries.

Lib Magazine,4(11).Availableat

http://www.dlib.org/dlib/november98/lagoze/1 llagoze.html

DLagoze, C. & Payette, S. (1998). An infrastructure for open-architecture digital libraries.

Cornell University Computer Science Technical Report TR98-1690. Available at

http ://ncstrl.cs.cornell. edu/Dienst/UI/1.0/Display/ncstrl.cornell/TR98-1690

Lasher, R. & Cohen, D. (1995). A format for bibliographic records. Internet RFC-1807.

Available at ftp ://ftp.isi. edu/in-notes/rfc 1807.txt

Lawrence, S., Bollacker, K. & Giles, C. L. (1999). Distributed error correction.

Proceedings of the fourth ACM conference on digital libraries (p. 232), Berkeley,

CA.

Lawrence, S. & Giles, C. L. (1998). Searching the World Wide Web. Science, 280, 98-

100. Available at http://www.neci.nj.nec.com/Mawrence/science98.html

Lesk, M. E. (1978). Some applications of inverted indexes on the UNIX system. Bell

Laboratories Computing Science Technical Report 69.

Lesk, M. E. (1997). Practical digital libraries: books, bytes & bucks. San Francisco, CA:

Morgan-Kaufmann Publishers.

Lesk, M. E. (1999). Perspectives on DLI2 - growing the field. D-Lib Magazine, 5(7-8).

Available at http://www.dlib.org/dlib/july99/071esk.html

Lutz, M. (1996). Programming python. Sebastopol CA: O'Reilly & Associates.

Marazakis, M., Papadakis, D. & Papadakis, S. A. (1998). A framework for the

encapsulation of value-added services in digital objects. In C. Nikolaou & C.

Stephanidis (eds.) Research and advanced technology for digital libraries, second

European conference, ECDL '98 (pp. 75-94). Berlin: Springer.

Maly, K., French, J., Fox, E. & Selman, A. (1995). Wide area technical report service:

technical reports online. Communications of the ACM, 38(4), 45.
108

Maly, K., Nelson, M. L., & Zubair, M. (1999). Smart objects, dumb archives: a usercentric, layered digital library framework. D-Lib Magazine, 5(3). Available at

http://www.dlib.org/dlib/march99/maly/03maly.html

McGrath, R. E. (1996). Performance of several Web server platforms. National Center

for Supercomputing Applications Technical Report. Available at

http ://www.ncsa.uiuc.edu/InformationServers/Performance/Platforms/report.html

Miller, E. (1998). An introduction to the Resource Description Framework. D-Lib

Magazine, 4(5). Available at http://www.dlib.org/dlib/may98/miller/O5miller.html

Monostori, K., Zaslavsky, A. & Schmidt, H. (2000). Document overlap detection

systems for distributed digital libraries. Proceedings of the fifth ACM conference

on digital libraries (pp. 226-227), San Antonio, TX.

Mori, R. & Kawahara, M. (1990). Superdistribution: the concept and the architecture.

Transactions of the IEICE, E73(7). Available at

http ://www.virtualschool.edu/mon/ElectronicProperty/MoriSuperdist.html

NASA (1998). NASA Scientific and Technical Information (STI) program plan.

Available at http://stipo.larc.nasa.gov/splan/

Nebel, E. & Masinter, L. (1995). Form-based file upload in HTML. Internet RFC-1867.

Available at ftp ://ftp.isi. edu/in-notes/rfc 1867.txt

Nelson, C. (1995). OpenDoc and its architecture. The X Resource, 1(13), 107-126.

Nelson, M. L. & Gottlich, G. L. (1994) Electronic document distribution: design of the

anonymous FTP Langley technical report server, NASA-TM-4567, March 1994.

Available at http://techreports.larc.nasa.gov/ltrs/PDF/tm4567.pdf

Nelson, M. L., Gottlich, G. L., & Bianco, D. J. (1994). World Wide Web implementation

of the Langley technical report server. NASA TM- 109162. Available at

http ://techreports. larc.nasa, gov/ltrs/PDF/tm 109162.pdf

Nelson, M. L., Gottlich, G. L., Bianco, D. J., Paulson, S. S., Binkley, R. L., Kellogg, Y.

D., Beaumont, C. J., Schmunk, R. B., Kurtz, M. J., Accomazzi, A., & Syed, O.

(1995). The NASA technical report server. Internet Research: Electronic Network

Applications and Policy, 5(2), 25-36. Available at

http://techreports.larc.nasa.gov/ltrs/papers/NASA-95-ir-p25/NASA-95-irp25.html
109

Nelson,M. L. & Esler,S.L. (1997). TRSkit: a simpledigital library toolkit.

InternetCataloging,1(2), 41-55.Availableat

http://techreports.larc.nasa.gov/ltrs/PDF/1997/jp/NASA-97-jic-mln.pdf

Journalof

Nelson,M. L., Maly, K., Shen,S.N. T., & Zubair,M. (1998). NCSTRL+: addingmultidiscipline and multi-genresupport to the Dienst protocol using clusters and

buckets. Proceedingsof the IEEEforum onresearchandtechnologyadvancesin

digital libraries(pp. 128-136),SantaBarbara,CA.Availableat

http://techreports.larc.nasa.gov/ltrs/PDF/1998/mtg/NASA-98-ieeedl-mln.pdf

Nelson, M. L. (1999). A digital library for the National Advisory Committee for

Aeronautics.NASA/TM- 1999-209127.Available at

http://techreports.larc.nasa.gov/ltrs/PDF/1999/tm/NASA-99-tm209127.pdf

Nelson,M. L., Maly, K., Croom,D. R., & Robbins,S.W. (1999). Metadataandbuckets

in the smartobject,dumbarchive(SODA) Model, Proceedingsof the third IEEE

meta-dataconference,Bethesda,MD. Availableat

http://www.computer.org/proceedings/meta/1999/papers/53/mnelson.html

Ockerbloom,J. (1998). Mediating amongdiversedata formats. Ph.D. Dissertation,

CarnegieMellonUniversity,CMU-CS-98-102.Availableat

http://reports-archive.adm.cs.cmu.edu/anon/1998/abstracts/98-102.html

Odlyzko, A. M. (1995). Tragic loss or good riddance?The impending demise of

traditionalscholarlyjournals.InternationalJournalof Human-ComputerStudies,

42,71-122.

Olson,M. A., Bostic,K. & Seltzer,M. (1999). BerkeleyDB. Proceedingsof the 1999

USENIX annualtechnicalconference,Monterey,CA.

Ousterhout,J.K. (1994). Tcl andthe Tk toolkit. Reading,MA: Addison-Wesley.

Paepcke,A. (1996). Digital libraries:searchingis not enough. D-Lib Magazine2(5).

Availableat http://www.dlib.org/dlib/may96/stanford/O5paepcke.html

Paepcke,A. (1997). InterBib: bibliography-relatedservices. Available at http://wwwdiglib.stanford.edu/-testbed/interbib/

Paskin,N. (1999). DOI: currentstatusandoutlook. D-Lib Magazine,5(5).Availableat

http://www.dlib.org/dlib/may99/05paskin.html
110

Patterson, David A. (1994). How to have a bad career in research/academia. Keynote

Address at the First Symposium on Operating System Design and

Implementation, Monterey, CA. Available at

http://http.cs.berkeley.edu/-patterson/talks/bad.ps

Phelps, T. A. & Wilensky, R. (2000). Multivalent documents. Communications of the

ACM, 43(6), 83-90.

Powell, A. L. & French, J. C. (2000). Growth and server availability of the NCSTRL

digital library. Proceedings of the fifth ACM conference on digital libraries (pp.

264-265), San Antonio, TX. Available at

http://www.cs.viriginia.edu/-cyberia/papers/DL00.pdf

Rasmussen, E. (1992). Clustering algorithms. In W. B. Frakes & R. Baeza-Yates (Eds.),

Information retrieval: data structures & algorithms (pp. 363-392), Upper Saddle

River, N J: Prentice-Hall.

Rivest, R. (1992). The MD5 message-digest algorithm. Internet RFC-1321. Available at

ftp ://ftp.isi.edu/in-notes/rfc 1321 .txt

Rocha, L. M. (1999). TalkMine and the adaptive recommendation project. Proceedings

of the fourth ACM conference on digital libraries (pp. 242-243), Berkeley, CA.

Available at http://www.c3.1anl.gov/_rocha/d199.html

Roper, D. G., McCaskill, M. K., Holland, S. D., Walsh, J. L., Nelson, M. L., Adkins, S.

L., Ambur, M. Y., & Campbell, B. A. (1994). A strategy for electronic

dissemination of NASA Langley publications. NASA TM-109172. Available at

http ://techreports. larc.nasa, gov/ltrs/PDF/tm 109172.pdf

Rothenberg, J. (1995). Ensuring the longevity of digital documents. Scientific American,

272(1), 42-47.

Salampsasis, M., Tait, J. & Hardy, C. (1996). An agent-based hypermedia framework for

designing and developing digital libraries. Proceedings of the third forum on

research and technology advances in digital libraries (pp. 5-13), Washington DC.

Salton, G. & Lesk, M. E. (1968). Computer evaluation of indexing and text processing,

Journal of the Association of Computing Machinery, 15 (1), 8-36.

Sanchez, J. A., Legget, J. J., & Schnase, J. L. (1997). AGS: introducing agents as services

provided by digital libraries. Proceedings of the second ACM international

conference on digital libraries (pp. 75-82), Philadelphia, PA.
111

Sanchez,J. A., Lopez, C.A.., & Schnase,J. L. (1998).An agent-basedapproachto the

construction of floristic digital libraries. Proceedingsof the third ACM

internationalconferenceondigital libraries(pp.210-216),Pittsburgh,PA.

Schatz,B., & Chen,H. (1996). Building large-scaledigital libraries. IEEE Computer,

29(5),22-26.

Scherlis,W. L. (1996). Repository interoperability workshop: towards a repository

referencemodel. D-Lib Magazine,2(10).Availableat

http://www.dlib.org/october96/workshop/10scherlis.html

Scott, E. W. (1953). New patterns in scientific researchand publication. American

Documentation,4(3),90-95.

Sharer,K., Weibel,S.,Jul, E. & Fausey,J. (1996). Introduction to persistentuniform

resourcelocators. Proceedingsof INET 96, Montreal, Canada.Available at

http://purl.oclc.org/OCLC/PURL/1NET96

Shivakumar,N. & Garcia-Molina,H. (1995). SCAM: a copy detectionmechanismfor

digitaldocuments.Proceedingsof the secondinternationalconferencein theory

andpracticeof digital libraries(pp. 155-163),Austin, TX.

Shklar,L., Makower,D., Maloney, E. & Gurevich(1998). An applicationdevelopment

frameworkfor thevirtual Web. Proceedingsof the fourth internationalconference

on information systems,analysis,and synthesis, Orlando, FL. Available at

http://www.cs.rutgers.edu/-shklar/isas98/

Sibert,O.,Bernstein,D. & Van Wie, D. (1995). DigiBox:a self-protectingcontainerfor

informationcommerce.Proceedingsof the first USENIX workshopon electronic

commerce,New York,NY.

Sobieski,J. (1994). A proposal:how to improveNASA-developedcomputerprograms.

NASA CP-10159,pp. 58-61.

Sparck Jones,K. (1972). A statistical interpretation of term specificity and its

applicationin retrieval. Journalof Documentation,28(1),11-20.

Sparck Jones, K. (1979). Experiments in relevance weighting of search terms.

InformationProcessingandManagement,15(3), 133-144.

Stern,I. (1995). ScientificdataformatinformationFAQ.Availableat

http://www,faqs.org/faqs/sci-data-formats/
112

Stein,L. (1998). Official guideto programmingwith CGI.pm. New York, NY: John

Wiley & Sons.

Stein,L., MacEachern,D. & Mui, L. (1999). Writing Apachemodulesin PerlandC: the

ApacheAPI androod perl.SebastopolCA: O'Reilly & Associates.

Steiner,J.G.,Neuman,C.& Schiller,J.I. (1988). Kerberos:an authenticationservicefor

opennetworksystems.Proceedingsof thewinter 1988USENIX conference(pp.

191-202),Dallas,TX.

Sullivan,W. T. III, Werthimer,D., Bowyer, S., Cobb, J., Gedye,D. & Anderson,D.

(1997). A new majorSETIproject basedonProject Serendipdataand 100,000

personalcomputers. In C.B. Cosmovici,S. Bowyer, & D. Werthimer (Eds.),

Astronomicaland biochemicalorigins and the searchfor life in the universe,

Bologna,Italy: EditriceCompositori.Availableat

http://setiathome.berkeley.edu/woodypaper.html

SunMicrosystems,Inc. (1999). The maximumnumberof directoriesallowedon Solaris

is limited by theLINK_MAX parameter.InfoDoc # 19895.

Sun,S.X. & Lannom,L. (2000). Handlesystemoverview. InternetDraft.Availableat

http://www.ietf.org/internet-drafts/draft-sun-handle-system-04.txt

Task Forceon Archivingof Digital Information (1996). Preservingdigital information.

Availableat http://www.rlg.org/ArchTF/

Tiffany, M. E. & Nelson, M. L. (1998). Creatinga canonicalscientificand technical

information classification system for NCSTRL+. NASA/TM-1998-208955.

Availableat

http://techreports.larc.nasa.gov/ltrs/PDF/1998/tm/NASA-98-tm208955.pdf

United StatesGeneralAccountingOffice (1990). NASA is not properly safeguarding

valuabledatafrom pastmissions,GAO/IMTEC-90-1.

Van de Sompel,H. & Hochstenbach,P. (1999). Referencelinking in a hybrid library

environment:part 2: SFX, a genericlinking service. D-Lib Magazine 5(4).

Availableat

http://www.dlib.org/dlib/apri199/vande sompel/04/vande sompel-pt2.html
113

Van de Sompel,H., Krichel, T., Nelson, M. L., Hochstenbach,P., Lyapunov,V. M.,

Maly, K., Zubair,M., Kholief, M., Liu, X. & O' Connell,H. (2000a). The UPS

prototype: an experimentalend-userservice acrosse-print archives. D-Lib

Magazine,6(2).Availableat

http://www.dlib.org/dlib/februaryOO/vandesompel-ups/O2vandesompel-ups.html

Van de Sompel,H., Krichel, T., Nelson, M. L., Hochstenbach,P., Lyapunov,V. M.,

Maly, K., Zubair,M., Kholief, M., Liu, X. & O' Connell,H. (2000b).The UPS

prototype project:exploringthe obstaclesin creatingacrosse-print archiveenduserservice,Old DominionUniversity Computer ScienceTechnicalReport TR

2000-01.Availableat

http://ncstrl.cs.cornell.edu/Dienst/Ul/1.0/Display/ncstrl.odu_cs/TR_2000_01

VandeSompel,H. & Lagoze,C. (2000). TheSantaFeConventionof the OpenArchives

initiative. D-Lib Magazine,6(2).Availableat

http://www.dlib.org/dlib/february00/vandesompel-oai/02vandesompel-oai.html

Vickery, B. (1999). A century of scientific and technical information. Journal of

Documentation,55(5),476-527.

Vinoski, S. (1997). CORBA: integrating diverse applications within distributed

heterogeneousenvironments.IEEECommunicationsMagazine,4(2),46-55.

Wall, L., Christiansen,T. & Schwartz,R. L. (1996). ProgrammingPerl. Sebastopol,CA:

O'Reilly & Associates,Inc.

Waugh,A., Wilkinson,R., Hills, B., & Dell6ro, J. (2000).Preservingdigital information

forever. Proceedingsof the fifth ACM conferenceon digital libraries(pp. 175-

184),SanAntonio, TX.

Weibel,S.,Kunze,J.,Lagoze,C.& Wolfe, M. (1998).Dublin Coremetadatafor resource

discovery. InternetRFC-2413.Availableat ftp://ftp.isi.edu/in-notes/rfc2413.txt

Yeong,W., Howes,T. & Kille, S.(1995). Lightweightdirectoryaccessprotocol. Internet

RFC-1777.Availableat ftp://ftp.isi.edu/in-notes/rfc1777.txt
114

APPENDIX A

BUCKET VERSION HISTORY

Version Date Kilobytes Inodes Comments

"proto buckets" January 1996 n/a n/a

of the

NACATRS

version 0 July 1997 n/a n/a

version 1.0 July 1998 n/a n/a

version 1.1 August 1998 n/a n/a

version 1.11 September n/a n/a

1998

version 1.12 September n/a n/a

1998

version 1.13 October 1998 n/a n/a

Not really a bucket, but the

concept for buckets grew out

of the experiences from this

project.

First digital object to be

identified as a bucket. Used

only for research purposes:

refining the bucket concept &

defining the API. Structural

design is completely different.

complete re-write of version

0; the design of the current

buckets traces to this version.

First implementation of the

current T&C design.

Significant change in parsing

of metadata. Name collisions

handled.

T&C changes.

Fixed problems with selfdeleting buckets in version

1.12
115

version1.2

version1.3

version1.3.1

version1.3.2

UPSversion1.6

(basedonversion

1.3.2)

November

1998

97

July 1999 118

July 1999 125

July 1999 125

October1999 97

40

53

58

58

56

The first public release of

buckets. Hasonly a basicset

of methodsand simple T&C

support. Display of metadata

is improved. Bucket more

tolerant of variations in

internalstructure.

Method set expanding to

influence appearance and

behaviorof bucket. Packages

are locked out from http

browsing(truedatahiding).

Can now distribute different

typesof metadataif they have

been pre-loaded. More

appearance/behaviormethods

evolving.

Minor bug-fix.

Final versionof the template

used in the UPS project.

Basedon the 1.3.2 template,

theUPStemplatewas slightly

optimized for storage

efficiency, and introduced

someof the new functionality

in laterbucketversions.

version1.4 December 134 62 Codefactoring now possible.

1999 Many of the appearanceand
116

version1.5 February 145 68

1999

version1.5.1 February 148 70

1999

version1.5.2 February 149 70

1999

version1.5.3 March 1999 148 70

version1.5.4 March 1999 147 70

version1.5.5 April 1999 143 66

version1.6 April 1999 144 68

behavior models have been

collapsed into preferences.

"display" method borrows

heavily from UPS look and

feel.

"pack" and"unpack" methods

implemented to assist with

bucket mobility. "display"

method can take several

argumentsfor customizingits

output.

Group T&C support for IP

addresses and hostnames

added.

Minor bugfix.

Minor bugfixes.

Moreminorbugfixes.

Naming of metadatachanged

for the "display" method to

be inline with that usedin the

NCSTRL+ project.

Buckets now BCS aware,

especially with respect to

metadataconversion.Buckets

can now send email when

eventsoccur. Many bugfixes

andoptimizations.
117

Prior to version 1.2, source code releases were not preserved. Source code and detailed

release notes versions after 1.2 can be found at:

http://dlib.cs.odu.edu/bucket-shop/
118

APPENDIX B

BUCKET API

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

add element

element_name, pkg_name, element_bib, upfile

text/plain

?method=add_element&element_name=bar.pdf&

pkg_name=foo.pkg&element_bib=X&upfile=X

"add_element" adds a specified element to a specified

package in the bucket. "element_name" is the name the

element will be stored as in the bucket, and "pkg_name" is

the name of the package that the element will be put into.

"element_bib" contains the RFC-1807 description of the

element, and "upfile" contains the actual file, uploaded as

described in RFC- 1867 (Nebel & Masinter, 1995).

delete_element, display
119

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

add method

target, upfile

text/plain

?method=add_method&target=foo&upfile=X

"add_method" adds a method to a bucket by uploading Perl

source code into the bucket. "target" specifies the name of

the new method, while "upfile" contains the source code

uploaded as described in RFC-1867. The bucket performs no

error checking on the uploaded source code; if the user can

satisfy the T&C for "add_method", it is assumed they know

what they are doing.

delete_method, list_methods
120

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

add_package

pkg_name, pkg bib

text/plain

?method=add_package&pkg_name=foo.pkg&pkg bib=X

"add_package" creates a new package in the bucket. On a

subsequent "display" of the bucket, the new package will

appear at the end of the list of previous packages.

"pkg_name" is the name of the package to be added. If the

package name does not include an extender of".pkg", one will

be added. "pkg_bib" contains the RFC-1807 description of

the package.

delete_package
121

Method:

Arguments:

Returned MIME Type:

Example(s):

add_principal

principal, passwd, epasswd

text/plain

?method=add_principal&principal=bob&passwd=secret

Discussion:

See Also:

?method=add_principal&principal=bob&epasswd=4Rals3 Q

"add_principal" adds a user with a password to the bucket's

internal list of recognized principals that can be named in its

T&C files. "principal" is the name of the user to be defined.

"passwd" is the clear text version of the password, and

"epasswd" is the encrypted (with Unix crypt(3)) version of

the password. Only one of the two password arguments

needs to be supplied.

delete_principal, list_principal
122

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

add tc

target, value

text/plain

?method=add_tc&target=delete_bucket&value=X

"add_tc" uploads (as described in RFC-1867) a T&C file that

defines who can invoke the method named by 'target".

"value" contains the actual file contents; the syntax of which

is described in chapter three.

delete_tc, list_tc
123

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete bucket

none

text/plain

?method=delete bucket

"delete bucket" deletes the entire bucket - no confirmation is

requested. This is a very dangerous method, and because of

this it is disabled in the standard bucket distribution.

delete_element, delete_package
124

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete element

element_name, pkg_name

text/plain

?method=delete_element&element_name=bar.pdf&pkg_name

=foo.pkg

"delete_element" deletes the element named b y

"element_name" and "pkg_name". It will also delete the

RFC-1807 metadata associated with the named element.

add_element, delete_package
125

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete_log

log

text/plain

?method=delete_log&log=access.log

"delete_log" deletes the log named by the argument "log".

This method is intended to be used by tools or cron jobs to

autoally harvest and then prune bucket logs, which would

otherwise grow without bound.

get_log, list_logs
126

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete method

target

text/plain

?method=delete_method&target=add principal

"delete_method" deletes the method named in "target" from

the bucket. Calling "delete_method" with arguments of

"add method" then "delete method" would insure a static

bucket whose methods could not be changed through the

API.

add_method, list_methods
127

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

deletepackage

pkg_name

text/plain

?method=delete_package&pkg_name=foo.pkg

"delete_package" deletes the entire package named in

"pkg_name". Any elements residing in the package will be

deleted at the same time; no confirmation will be requested.

The relevant metadata fields will be deleted as well.

add_package
128

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete_principal

principal

text/plain

?method=delete_principal&principal=bob

"delete_principal" will delete the user named by the

"principal" argument from the bucket's internal list of named

principals. The user's password will also be deleted.

add_principal, list_principals
129

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

delete tc

target

text/plain

?method=delete_tc&target=display.tc

"delete_tc" deletes the T&C file named by "target".

that "edit_tc" functionality would be accomplished by

"list tc" / "delete tc" / "add tc" series of calls.

add_tc, list_tc

Note

a
130

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

display

none, bold, view, sfx, redirect, pkg_name, element_name,

page, thumbnail

text/html; other MIME types as appropriate

?method=display

?method=display&bold=aircraft+engine

?method=display&view=staff

?method=display&sfx=http://sfx.foo.edu/

?method=display&redirect=http://www. foo.edu/

?method=display&pkg_name=foo.pkg&element_name=bar.p

df

?method=display&pkg_name=foo.pkg&element_namebar.sca

n&page=0001 .gif

?method=display&pkg_name=foo.pkg&element_namebar.sca

n&thumbnail = 1

"display" is easily the most complex bucket method. When

called with no arguments, it generates an HTML humanreadable listing of the bucket contents. "bold", "view", and

"sfx" can all be used to describe the normal HTML bucket

listing. "bold" takes a list ofkeywords that displays them in

bold during the bucket display. "view" defines an alternate

display criteria for the bucket, which can be used to

implement role based displays. "sfx" provides the location

of a SFX server. "redirect" causes the bucket to generate an

http 302 response and redirect the browser to the specified

URL. When "pkg_name" and "element_name" are specified,

the bucket returns the named element, and gives it a MIME
131

See Also:

type based on the bucket's own internal listing of file

extenders and MIME types. If "element_name" ends in

".scan", then either "page" or "thumbnail" can be specified.

"thumbnail" will generate a listing of N thumbnail GIFs

which correspond to scanned pages of a document, and where

N specified by the preference "thumbnail_increment".

"page" shows only the large GIF of a scanned page.

none
132

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

get log

log

text/plain

?method=get_log&log=access. log

"get_log" returns the entire log specified in the "log"

argument. Currently, buckets only maintain a single log by

default, but this could change in future versions. Also, local

implementations of buckets are free to implement their own

logs.

delete_log
133

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

getpreference

none, pref

text/plain

?method=get-preference

?method=get-prefrence&pref=logging

"get_preference" with no arguments returns the current

values of all the buckets defined preferences. If a single

preference is specified in the argument "pref", then only its

value is displayed.

set_preference
134

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

get_state

state

text/plain; or other as appropriate

?method=get_state&state=approved

"get_state" returns the value of the bucket state variable

specified in the argument "state". Buckets do not internally

use these state variables - they are for use by external tools

that wish to leave the bucket in a certain state. State

variables are of type text/plain, but they can be any MIME

type.

set state
135

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

id

none

text/plain

?method=id

"id" returns the id for the bucket as specified in the RFC1807 "ID::" metadata line.

none
136

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

lint

none

text/plain

?method=lint

"lint" performs a series of internal checks on the bucket.

These include: comparing the packages and elements listed in

the metadata to those physically in the bucket; verifying that

all packages are closed to http browsing; verifying that files

are writable by the http server; and comparing the URL used

to access the bucket with that expect in the metadata.

See Also: none
137

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

list_logs

none

text/plain

?method=list_logs

"list_logs" returns a list of all the logs defined for the bucket.

Currently, buckets only maintain a single log by default, but

this could change in future versions. Also, local

implementations of buckets are free to implement their own

logs.

delete_log, get_log
138

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

list methods

none

text/plain

?method=list methods

"list methods" returns a list of all the methods defined for

the bucket. This list is expected to be different for locally

modified buckets, which could add or delete methods from

the default set.

add_method, delete_method
139

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

listprincipals

none

text/plain

?method=list_principals

"list_principals" lists all the defined users for the bucket.

Passwords are obviously not included in the display.

add_principal, delete_principal
140

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

list source

none, target

text/plain

?method=list source

?method=list_source&target=display

"list_source" returns the Perl source code used by the bucket.

If no arguments are given, the source code for the "index.cgi"

is returned. Otherwise, the source code for the method

specified in the argument "target" is returned.

add_method, delete_method, list_methods
141

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

list tc

none, target

text/plain

?method:list tc

?method=list_tc&target=display

"list_tc" with no arguments lists all the methods for the

bucket that have T&C files and lists the file contents. If no

T&C are defined, nothing is returned. If the argument

"target" is supplied, "list_tc" will return only the T&C file

for the method specified by "target", or nothing if no T&C

are defined for that method.

add_tc, delete_tc
142

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

metadata

none, format

text/plain (or text/xml as appropriate)

?method=metadata

?method=metadata&format=oams

"metadata" invoked with no arguments returns the metadata

in RFC-1807 format. If you specify a different metadata

format in the "format" argument, it will first look to see if

thatrmat is stored internally in the bucket, and if so determine

if it is clean. If the bucket does not have the format, or it is

dirty, it will contact the BCS and attempt to convert the

RFC-1807 format to the requested format, if the BCS can do

that conversion. If the BCS cannot convert to that format, an

error is returned.

set metadata
143

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

pack

none, name, type, format

application/tar (or other MIME types as appropriate)

?method=pack

?method=pack&name=foo.pkg

?method=pack&type=payload

?method=pack&format=tar

"pack" takes a number of interchangable arguments, but all

have default values so "pack" can be invoked sans arguments.

"type" specifies one of: bucket (entire bucket, default value),

package (package name specified in "name"), payload (user

portion of the bucket only), or ride (internal structure of the

bucket only). "format" currently only recognizes the value

"tar", but this should change in the future (though "tar" will

remain the default value). "pack" will generate a stream of

either the entire bucket or just the requested part of a bucket.

This stream can be used to overwrite an existing bucket,

similar to the Unix fork0/exec 0 model.

unpack
144

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

set metadata

name, upfile

text/plain

?method=set metadata&name=metadata.oams&upfile=X

"set metadata" writes the metadata file named in "name" and

supplied in "upfile" to the bucket. This method is useful in

either overwriting the RFC-1807 metadata (perhaps to

correct errors), or to upload other metadata formats; either

from BCS or entirely different formats that the BCS does not

know about.

metadata
145

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

set_preference

pref, upfile

text/plain

?method=set_preference&pref=framable&up file=no

"set_preference" writes the preference named in "pref" and

takes the value specified in "upfile".

get_preference
146

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

set state

state, upfile

text/plain

?method=set_state&state=approved&up file=yes

"set state" writes the state variable named in "state" and

takethe value specified in "upfile". "upfile" does not have to

be of type text/plain.

get_state
147

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

set version

value

text/plain

?method=set version&value=2.0

"set_version" sets the version of the bucket to the text string

specified in "value".

version
148

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

unpack

format, type, upfile

text/plain

?method=unpack&up file=X

?method=unpack&format=tar&upfile=X

?method=unpack&format=tar&type=bucket&upfile=X

"unpack" takes the bucket stream specified in "upfile" and

puts it into the bucket. "format" can be specified, but the

only currenlty defined format is "tar". "type" specifies one

of: bucket (entire bucket, default value), package (package

name specified in "name"), payload (user portion of the

bucket only), or ride (internal structure of the bucket only).

pack
149

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

version

none

text/plain

?method=version

"version" returns a text string to describe what type of

bucket it is. There is no structure imposed on what this

string can be.

set version
150

APPENDIX C

DA API

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

da delete

id

text/plain

?method=da delete&id=1234

"da_delete" removes the object specified by the argument

"id" from the archive. "id" has no built in assumptions

regarding what type of unique identifier is used: CNRI

handles, DOIs, URLs, etc. all could be used. Currently,

"da_delete" does not return an error if "id" is not present in

the archive.

da_put, da_list
151

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

da_get

id, url

text/plain

?method=da_get&url=http://foo.edu/1234/

?method=da_get&id = 1234

"da_get" is an optional method for the DA; it is used

primarily to build a model where the archive still contains

some control over the access of the bucket, so that rather

than going directly to the bucket, the archived is asked to

redirect the user to the bucket. The buckets could be

modified to only accept responses originating from an

archive, and the DA could have regular bucket T&C

controlling the behavior of"da_get". Either argument "id" or

"url" can be specified, and "url" has precedence over "id" if

both are specified. The archive issues an http status code

302 for URL redirection.

See Also: none
152

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

da info

none

text/plain

?method=da info

"da_info" currently takes no arguments and simply returns

the element stored in holdings.pkg/info.txt. The

purpose of this method is to return a machine readable

description of the archive, its capabilities and its holdings.

Human readable descriptions would be available through the

standard bucket method "display". "da_info" should be

expanded to take arguments as to which format it would like

the archive information (and the MIME type set

accordingly), with possible conversion from the BCS.

none
153

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

da list

id, url, adate, pdate, subject, metadata

text/plain

?method=da list

?method=da list&metadata=on

?method=da list&id = 1234

?method=da_list&url=http ://foo. edu/1234/

?method=da list&adate = 19990101

?method=da_list&pdate = 19930430

?method=da_list&pdate=> 19931231

?method=da list&adate = 19891231-20000101

?method=da_list&adate=subj ect=computer_science

"da_list" is the primary method of the DA. This is the

method that will be used by digital library services to

determine what contents an archive has, what contents have

changed since a specific date, and so forth. There are many

arguments, all of which can be combined in various forms to

select which ids (and URLs) will be returned. If no argument

is specified, all the archive's contents are returned. If a

specific "id" or "url" is requested, then "da_list" is used for

an existence test, yielding the id/url if the object exists, and

nothing if it does not. There are three pre-defined "clusters"

(in NCSTRL+ terminology) defined for the DA: "adatae"

(accession date), "pdate" (publication date), "subject". Both

date fields follow the format of YYYYMMDD. All dates are

non-inclusive. Dates can be specific days, or modified with

"<", ">", or "-" for less than, greater than, and range,
154

See Also:

respectively. "<" and ">" must precede the date, and "-"

must have a valid date on either side. The date modifiers

cannot be combined. Regular expressions in the dates (i.e.

"1999.*" for the entire calendar year 1999) are not

supported. "subject" can be any text string from any subject

classification system; there are no syntactic restrictions on

"subject". If"metadata" is set to any value, the metadata for

the object(s) (ifuploaeded) will be returned as well.

da_delete, da_put
155

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

da_put

id, url, adate, pdate, subject, metadata

text/plain

?method=da_put&id = 1234&url=http://foo.edu/1234/&adate

= 19990215 &pdate = 19580413 &subj ect=aeronautics&metada

ta=X

"da_put" places an object in the DA. There are many

arguments, but only "id" is mandatory - the others are

optional. "id" can be a unique identifier in any format, "url"

is a regular URL, "adate" (accession date) and "pdate"

(publication date) are date strings in the format

YYYYMMDD, and "subject" can be from any

subject/discipline nomenclature. "metadata" is the objects

metadata uploaded as per RFC-1867.

da_delete, da_list
156

APPENDIX D

BCS API

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

bcs_convert_image

in_format, out_format, upfile

text/plain

?method=bcs_convert_image&in_format=ps&out_format=p

df&upfile=X

"bcs_convert_image" takes the file uploaded as per RFC1867 in the argument "upfile" and converts it to the type

specified in the argument "out_format". "upfile" is assumed

to be of the type specified in "in_format" - no checking is

done to verify that "upfile" is of type "in_format".

"in_format" and "out_format" currently have the following

types defined (alternate values in parentheses):

jpeg (jpg)

tiff (tit)

png

ps

pdf

bcs convert metadata
157

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

bcs convert metadata

in_format, out_format, in_file

text/plain or text/xml

?method=bcs convert metadata&in format=rfcl807&out fo

rmat=oams&in file=X

"bcs_convert_metadata" takes the metadata file uploaded as

per RFC-1867 in "in_file" and returns it converted to the

metadata format specified in "out_format". The format of

"in_file" is specified in "upfile", but no checking is done to

verify the accuracy between "in_file" and "in_format".

"in_format" has the following values defined:

refer

dublincore

rfc1807

bibtex

"out_format" has the following values defined:

refer

dublincore

rfcl807

bibtex

oams

See Also: bcs_conve__image
158

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

bcs list

id, url

text/plain

?method=bcs list

?method=bcs_list&url=http://foo.edu/1234/

?method=bcs list&id=1234

"bcs_list" with no arguments lists all the ids (and URLs) of

all the buckets that are registered with that BCS server. If

either of "url" or "id" is specified, "bcs_list" acts as a test for

existence; if the bucket identified by either "url" or "id" is

registered with this BCS server, "bcs_list" will return its id

or URL and will return nothing if the bucket is not registered.

bcs_register, bcs_unregister
159

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

bcs match

threshold, link, report, ids

text/plain

?method = bcs match

?method=bcs match&threshold=0.70

?method = bcs match&link=on

?method = bcs_match&report=on

?method = bcs match=ids=1234+2345+1122

"bcs_match" performs similarity matching on registered

buckets using the cosine correlation with frequency termweighting measure. "bcs_match" can be a computational

expensive task and run for a long time (see Chapter Five for

details). Because the run time of a "bcs_match" session can

be much longer than an average WWW browser session,

"bcs_match" forks off a copy of itself to run on the server so

that it cannot be killed from the browser. There are a number

of arguments to "bcs_match", all of which can be combined

with each other. "threshold" defines a number between 0-1

for determining what constitutes "similar" documents (the

default value is 0.85). If "link" is set to any value,

"bcs_match" will upon completion of the similarity matching

attempt to automatically create the linkages between the

similar buckets. It will attempt to create a

BCS_Similarity.pkg package if one does not exist, and

then add the similar bucket, if not linked already (default

action is not to link). If "report" is set to any value,

"bcs_match" will record the output of this run in the element
160

See Also:

matching in the package bcs.pkg (default action is not

to record the output). If "ids" has 1 or more values, then

"bcs_match" will only compare the specified ids against the

entire list of registered buckets (the default action is to

compare all buckets against all buckets).

none
161

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

bcs_message

search, replace, mesg, repeat

text/plain

?method=bcs_message&search=NASA+Lewis

?method=bcs_message&search=NASA+Lewis&replace=NA

SA+Glenn

?method=bcs_message&search=foo&mesg = destroy bucket

?method=bcs_message&search=foo&mesg=destroy bucket&

repeat=l

"bcs_message" identifies buckets for communication

purposes. "search" is a mandatory argument that specifies

the regular expression to search for in all registered buckets.

If no further arguments are given, "bcs_message" returns the

ids or URLs of the buckets that have that regular expression.

If "replace" is specified, the "search" regular expression is

overwritten both in the registry and the bucket with the

regular expression in "replace". If "message" is specified,

"bcs_message" sends all matching buckets the bucket

message specified in "message". If "repeat" is given an

integer value, the "replace" or "mesg" actions are repeated up

to (possibly less, depending on "search") "repeat" times.

Users should be aware when using "repeat" that no

assumptions can be made on the order of how "bcs_message"

finds the regular expression specified in "search".

none
162

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

bcs_register

id, url, metadata

text/plain

?method=bcs_register&id = 1234

?method=bcs_register&id = 1234&url=http://foo.edu/1234/

?method = bcs_register&id = 1234&metadata=X

"bcs_register" places the bucket specified by "id" or "url" in

the bucket communication space. "id" is mandatory, but

"url" is optional. A bucket cannot the subject of

"bcs_message", "bcs_match" and other BCS methods until it

has been registered. Although optional, "metadata" contains

the metadata, uploaded as per RFC-1867, that will be used in

"bcs_message" and "bcs_match".

bcs_list, bcs_unregister
163

Method:

Arguments:

Returned MIME Type:

Example(s):

Discussion:

See Also:

bcs_unregister

id

text/plain

?method = bcs unregister&id=1234

"bcs_unregister" removes the bucket specified by "id" from

the bucket communication space. The corresponding URLs

and metadata associated with that id will also be removed.

bcs_list, bcs_register
REPORT DOCUMENTATION PAGE Form Approved OMBNO.0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data

sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other

aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and

Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188),

Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

August 2001 Technical Memorandum

4. TITLE AND SUBTITLE

Buckets: Smart Objects for Digital Libraries

6. AUTHOR(S)

Michael L. Nelson

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

9.SPONSORING/MONITORINGAENCYNAME(S)ANDADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

5. FUNDING NUMBERS

WU 992-16-05

8. PERFORMING ORGANIZATION

REPORT NUMBER

L-18106

10. SPONSORING/MONITORING

AGENCY REPORT NUMBER

NASA/TM-2001-211049

11. SUPPLEMENTARY NOTES

Also published as a PhD dissertation for the Old Dominion University, Computer Science Department, Norfolk,

Virginia, August 2000.

12a. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified-Unlimited

Subject Category 82 Distribution: Standard

Availability: NASA CASI (301) 621-0390

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

Current discussion of digital libraries (DLs) is often dominated by the merits of the respective storage, search

and retrieval functionality of archives, repositories, search engines, search interfaces and database systems.

While these technologies are necessary for information management, the information content is more important

than the systems used for its storage and retrieval. Digital information should have the same long-term

survivability prospects as traditional hardcopy information and should be protected to the extent possible from

evolving search engine technologies and vendor vagaries in database management systems. Information content

and information retrieval systems should progress on independent paths and make limited assumptions about the

status or capabilities of the other.Digital information can achieve independence from archives and DL systems

through the use of buckets. Buckets are an aggregative, intelligent construct for publishing in DLs. Buckets

allow the decoupling of information content from information storage and retrieval. Buckets exist within the

Smart Objects and Dumb Archives model for DLs in that many of the functionalities and responsibilities

traditionally associated with archives are "pushed down" (making the archives "dumber") into the buckets

(making them "smarter"). Some of the responsibilities imbued to buckets are the enforcement of their terms and

conditions, and maintenance and display of their contents.

14. SUBJECT TERMS 15. NUMBER OF PAGES

WWW, Digital Libraries, Information Retrieval, STI, Archives 177

16. PRICE CODE

17. SECURITY CLASSIFICATION

OF REPORT

Unclassified

NSN 7540-01-250-5500

15. SECURITY CLASSIFICATION

OF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATION

OF ABSTRACT

Unclassified

20. LIMITATION

OF ABSTRACT

UL

Standard Form 295 (Rev. 2-59)

Prescribed by ANSI Std. Z-39-18

298-102