# Presentation: 926463

## METS at UC Berkeley

- Generating METS Objects

## Background

- Kinds of materials:
- primarily imaged content & tei encoded content
- archival materials: manuscripts and pictorial collections
- oral histories
- Kinds of Metadata
- Structural metadata: physical structure
- Descriptive metadata
- BasicTechnical metadata about digital files and how they were produced

## Tools For Producing METS Objects

- GenDB
- Gathers structural, descriptive and technical metadata
- GenX
- Generates METS objects from GenDB

## GenDB

- Consists of:
- Relational database (Currently SQL Server)
- Locally developed software for gathering metadata and facilitating digital processing

## GenDB Database Structure

- Structural Metadata

- Div 2

- Div 3

**Object 1**

**Object 2**

- (root)

- (parent = div 1)

- (parent = div 1)

- Div 1

- Div 2

- Div 3

- (root)

- (parent = div 2)

- (parent = div 1)

- Div 4

- (parent = div 2)

- ...

**Structural Md Table**

**Notes:**
At core of Database structure is a structural metadata Table
Entries in the structural metadata Table correspond to divisions of METS structMap.
Divisions pertaining to the same Object are bound together by a common Object ID
Each division contains a pointer to its parent division.  This enables the divisions pertaining to a particular Object to be reorganized into a hierarchically structured METS structMap

## GenDB Database Structure

- Descriptive Metadata

- Div 2

- Div 3

**Object 1**

**Object 2**

- Div 1

- Div 2

- Div 3

- Div 4

- Core Desc Md

- Core Desc Md

- Core Desc Md

- Core Desc Md

- Core Desc Md

- Core Desc Md

- Core Desc Md

- Name 1

- Name 2

- Name 3

- Note 1

- Note 2

- Note 3

**Name Table**

**Note Tables**

**Structural Md Table**

**Notes:**
There are many descriptive metadata tables and linking tables in GenDB
Structural metadata table itself contains core descriptive metadata pertaining to each div reprsented.
Divs in the Structural metadata table link to descriptive metadata tables via their div ids.
Entries in some descriptive metadata tables are shared across objects and divs.  For example, divs in different objects can link to the same entry in the name table.
Entries in some descriptive metadata tables, like the note tables, are specific to a particular div.

## GenDB Database Structure

- Content File/Technical Md

- Div 2

- Div 3

**Object 1**

**Master Image Table**

**Derivative Image Table**

**Structural Md Table**

- Drv 1

- Drv 2

- Drv 3

- Mstr 1

- Mstr 2

- Technical Md

- Technical Md

- Drv 4

- Technical Md

- Technical Md

- Technical Md

- Technical Md

**Notes:**
Diagram just shows provision for image content files
Entry for each div in the Structural Metadata table can link to an entry for its associated Master image in the Master Image Table
Entry for each master image in the Master table links to entries for its associated derivative images in the derivative tables
Entries in the Master and derivative tables contain the content file names and associated technical metadata pertaining to the image

## Populating the Database Tables

**Web interface**: manual input of structural and descriptive metadata
**Digitization Management modules**
- Generate work orders to guide digitization process
- Import content file information and technical metadata coming out of digitization process** **
**Batch loader**: batch input based on TEI encodings, legacy metadata

## Web Interface: WebGenDB

- Web
- Interface

- SQL Server
- Database

- Java Servlet

- Java Server

- XML Config
- Files

- rmi

- jdbc

**Notes:**
Web-based WebGenDB interface supports manual input of structural and descriptive metadata, as well as processing information. 
Java servlet drives the web interface on basis of XML configuration files
XML Configuration files allow the user interface to be configured differently for each repository participating in a project; and for each object type the repository needs to input.
Java server mediates between the database and the user interface, and constitutes the logic layer.  It extracts the metadata from the database and packages it up in a more user-comprehensible format.  It receives the updated metadata, and commits the updates to the database. 
This effectively shields the user from the complexities of METS, MODS, MIX. 
Includes tools to facilitate the analysis of object structure. 

## Digitization Management Modules

- Web
- Interface

- Java Servlet

- Java Server

- SQL Server
- Database

- Imaging/
- Transcription
- WorkOrders

- Vendor

- Technical MD
- Spreadsheets

**Notes:**
WebGenDB Digitization management modules support the digitization process, as well as  gathering content file and technical metadata coming out of this process.
Using the WebGenDB web interface, project managers generate imaging or transcription work order for an objects in the database that are ready for processing.
Work Order and material goes to Imaging and/or transcription vendor.  
Vendor does imaging/transcription and produces spreadsheet with the content filenames and technical metadata about digitization process.
Technical metadata and content file names imported back into the database. 

## Batch Loader

- Web
- Interface

- SQL Server
- Database

- Java Servlet

- Java Server

- Java Batch
- Loader

- XML Batch
- Load File

- TEI Docs

- XSLT

**Notes:**
Descriptive, structural, and content file metadata may enter the database via a batch load.
Batch load file must be in XML format defined by a locally developed schema.
One possible source of a batch load file would be a set of existing TEI documents for which we want to enhance the descriptive metadata, and wrap in METS objects.  Such a batch load file would be generated via an xslt transformation run against the TEI documents.
Another possible sources of a batch load would be an export from a legacy database.
Project manager can invoke the batch loader from the WebGenDB web interface, and initiate the batch load process.
Batch loader loads the data and metadata from the batch load file using same GenDB server program and calls as are used for manual input (as shown on previous slide)

## WebGenDB

**The concepts that drove the design**
- Shielding user from METS complexity
- Highly configurable
- Unicode support
- Access driven by login privileges
- Use of Open Source software and components
- Distributed approach

**Notes:**
Shielding from METS complexity:
1)METS and auxiliary standards very complex:
          Hand coding precluded
     Shield users from complexity as much as possible
2) Metadata (esp. descriptive) may need to be expressed in different ways 
Highly configurable
1) Projects differ in metadata needs
2) Projects differ in resources, sophistication of participants
Unicode support:
Increasing number of project requires full Unicode support, i.e. Stone Rubbings, CalCulture
Access Driven by login privileges
1)Projects have different resources and participants of various degrees of skills.
2)Need to restrict to a project manager settings definition and control value lists access
3)In the future more will be added to manager features
4)Possible to develop in the future different levels of access for managers themselves.
Open Source
Long tradition of using and developing open source products
Cost efficient
Distributed Approach
Wanted to create a very flexible setup
     distributed repositories
     load balancing
2) At this time only used on one machine

## XML Configuration Files

- Three levels
- Common to all projects elements
- Common to all screens in a project elements
- Specific to a screen in a project
- Define fields common to all projects
- Define fields used in specific project
- Define screens by project & object type

**Notes:**
Variables common to all projects
Images used in the display
RMI call 
Control Value lists which can be modified through the interface
Location of project specific files
File taken at servlet initialization and referred from the web.xml file
Variables common to all screens in a project
1)Fields used in the project with their specifications:
        Label associated in the screen
	Type of field: checkbox, free text, controlled value, non editable
         Size of the input box for free text or display max for CV
	Other admin info for editing of CV fields
Specific to one screen
List of the screens for that object type (Main, AddMetadata, Edit)
For each of the screens
       all the fields it
       the field order and position in the screen
3) Chosen over JSP because of the projected heavy interrelation between fields
4) Screens defined for each of the participants in the repository
    Based on material type of the object: text, 3-D objects, cartographic

## Relation among XML files

**AlProjects.xml**

**Proj1.xml**

**Proj2.xml**

**ObjectType1.xml**

**ObjectType2.xml**

**ObjectType1.xml**

**ObjectType2.xml**

**Notes:**
AllProjects.xml reference in the web.xml read at initialization time.
		 Contains references to ProjX.xml files

ProjX.xlm project identifiers are set at login time
                   It’s read at user login time
                   Contains references to ObjectTypeX.xml

ObjectTypeX.xml end of the chain
                         it’s read every time that a new object is displayed (created or edited)

## Project XML file example

**<ObjectType>**
**   ****<name>workorder</name>**
**   ****<fileLocation>**
**         ****/data/_w/GenDB/WEB-INF/classes/edu/berkeley/library/propertyFiles/CalCultureWorkOrderScreensFile.xml**
**</fileLocation>**
** ****</ObjectType>**
**  ****<Field>**
**     ****<name>Image</name><type>checkbox</type><label>Image </label><size>1</size>**
** ****</Field>**
**  ****<Field>**
**      ****<name>Text</name><type>checkbox</type><label>Text </label><size>1</size>**
** ****</Field>**
**  ****<Field>**
**      ****<name>Title</name><type>text</type><label>Title </label><size>60</size>**
** ****</Field>**

**Notes:**
First line is the reference to the specific object type xml file

Following lines are examples of the definition of fields for the project

## Software used

- MSSQL running on NT
- Tomcat 4.1.2 implementing servlets 2.3
- Jsdk 1.4
- Xalan 2.4
- Xerces 1.0.3
- FOP 0.12.1
- JDOM beta 8
- Opta 2000

## Relationship of GenDB to METS

- Metadata not directly stored in METS, MODS or MIX schema formats.
- Much of the database structure was developed before these standards emerged
- Database structure and content adjusted to be compatible with all these formats

## GenX: From GenDB to METS

- Allows Digital Publishing Group staff to select the objects in the GenDB database that are ready for export and to export them as METS objects.

## GenX Architecture

- App
- Interface

- GenDB

- Java Application

- METS XML
- Repository

- JDBC

**Notes:**
GenX consists of a Java application with an application interface.
Using this application our Digital Publishing Group staff can review the database and select those objects that are ready for export.
GenX uses JDBC calls to gather the metadata for objects to be exported from the GenDB database and packages these up into METS objects.

## GenX Output

- METS output corresponding to version 1.3
- Descriptive metadata exported to METS descMD in MODS 2.0 format
- Technical Metadata exported to METS techMD in MIX format
- Planned:
- Text technical md to METS descMD in NYU TextMD
- Rights to METS rightsMD in ODRL subset

## Links

- GenDB Web Interface Demo
- http://sunsite2.berkeley.edu/GenD
- login: demo
- password: demo
- Developers:
- rbeaubie@library.berkeley.edu
- ghill@library.berkeley.edu
- jhassan@library.berkeley.edu