# Presentation: 913081

## GPO Federal Digital SystemArchitecture and Design

- Deng Wu
- Paul Nelson
- Johnny Gee

## Agenda

- System Architecture
- Conceptual Model
- Data Model and OAIS Implementation
- Application Architecture
- Application Security
- FDsys Ingest Process Model
- Data Processing and Search
- Collections, Packages and Granules
- Architectural Philosophy
- Search Features
- Accessing Documents
- Repository Design
- Q & A

## System ArchitectureDeng Wu

## System Architecture

- Conceptual Model

## Data Model

- OAIS Implementation
- SIP, AIP, DIP
- Metadata Standards: MODS, PREMIS, METS
- FDsys Package Model

**Package**

- -- packaging scheme
- - content files
- - metadata

**SIP**

**AIP**

**ACP**

- CMS-Based Logical
- Implementation

- Self-Describing
- Implementation

- IS A

**DIP**

- Self-Describing
- Implementation

## Data Model (cont.)

- SIP & ACP

**package**
**folder-1**

**package**
**folder-2**

- rendition
- folder-2

- rendition
- folder-1

- content
- files

- rendition
- folder-1

- content
- files

- granule
- folder

- granule
- files

- rendition
- folder-2

- content
- files

- content
- files

**SIP / ACP**

## Data Model (cont.)

- AIP

**package**
**folder-1**

**package**
**Folder-2**

- rendition
- folder-2

- content
- files

**mods.xml**

**aip.xml**

**premis.xml**

- rendition
- folder-1

- content
- files

**AIP**

## Application Architecture

## Application Security

**Documentum**

**Oracle**
**Internet**
**Directory**

**Microsoft**
**Active**
**Directory**

**FDsys Applications**

- -- Content Originator
- - Service Provider
- - EPA
- - DOE

- LDAP

- LDAP

- Authentication &
- Authorization

- synchronize

- Roles and groups to enforce application security control
- User access privileges depend on user’s roles and groups
- Content Originator, Service Specialist, Preservation Specialist, etc.
- Users, roles and groups all managed in LDAP

- -- Service Specialist
- - Preservation Specialist
- - U.S. Senate
- - U.S. House
- - GPO

## FDsys Ingest Process Model

## Data ProcessingandSearchPaul Nelson

## Collections, Packages, and Granules

## What is a collection?

- Within FDsys:
- A group of documents which are all *processed* the same
- Identified with a “Processing Code”
- Examples:  Federal Register, Congressional Bills, Congressional Record, Public and Private Laws
- To the public:
- A group of documents which logically belong together
- Identified with a “Collection Code”
- Examples: Federal Register, Congressional Bills, Congressional Record, Public and Private Laws

## Processing Codes and Collection Codes

- Metadata is used to determine the collection code at index time
- Example Where the Two are not the Same:
- Presidential Budgets
- BUDGET2005, BUDGET2006    BUDGET
- Processed individually but searched together
- Congressional Reports printed in the congressional record
- Will have two collection codes:  CR, CRPT
- Will show up in both collections!
- Possibility for “virtual collections” of the future
- For example:  All documents related to Education

## Packages

- Roughly Equivalent to a bound, paper document
- Examples:
- One issue of the Federal Register
- One issue of the Congressional Record
- A single Congressional Bill
- One issue of the Weekly Compilation of Pres. Docs.
- One volume of the Code of Federal Regulations (!)
- One volume of the United States Code
- The 9-11 Report
- A single congressional committee report

## Packages Example

## Granules

***“******The most usefully searchable unit”***
***Examples:***
***A single FR article***
***A single CR unit of business (separated by                           )***
***A CFR section (e.g. ******§******57.402 in Title 40)***
***An entire bill***
***An entire report***
***A Pres. speech***

## Implications of Granules

- Search over granules
- Search results are *much more accurate*
- Retrieve individual granules
- Can view a single granule rather than sifting through an entire issue or volume
- Granules can be digitally signed
- To ensure authenticity of the data
- No single page retrieval
- Instead, you get the best granule
- PDF viewer automatically jumps to the page you asked for

## Architectural Philosophy 

## FDsys is a Data Driven Architecture

*FDsys is a Data Driven Architecture*

***Raw Content***

- group into packages

***Packages***

- Extract Metadata

***content***

***metadata***

- Search

- createMODS

- Browse

- contentdelivery

## Metadata Flow Diagram

***Metadata Flow Diagram***

## Parsing Introduction

- Runs regular expressions to extract metadata
- Regular Expression: **(Public Law|Pub. L.|PL|P. L.) (1[0-9][0-9])-([0-9]+)**
- Example:  **Pub. L. 109-130**
- Produces: **<publicLaw congressNum="109">130</publicLaw>**
- Produces an instance of **fdsys.xml**
- Parsed metadata is also available in the MODS

## Federal Register Example Metadata

***agencies***

***title***

***action***

***summary***

***dates***

***contact***

***FR Doc Number***

***Billing Code***

## Extracted Metadata Example

** ****<descMdGroup id="id-05-10658">**
**   ****<title>Editorial Modifications of the Commission’s Rules</title>**
**   ****<printPageRange first="31372" last="31374"/>**
**   ****<migratedDocID>fr01jn05-12</migratedDocID>**
**   ****<collectionSpecific>**
**     ****<accessId>05-10658</accessId>**
**     ****<granuleClass>RULE</granuleClass>**
**     ****<agency order="1">FEDERAL COMMUNICATIONS COMMISSION</agency>**
**     ****<effectiveDate>2005-04-14</effectiveDate>**
**     ****<billingCode>6712-01-P</billingCode>**
**     ****<frDocNumber>05-10658</frDocNumber>**
**     ****<action>Final rule.</action>**
**     ****<summary>This document amends twelve sections ... updating the postal address of the Arecibo Radio Astronomy Observatory.</summary>**
**     ****<dates>Effective April 14, 2005.</dates>**
**     ****<contact>Rodney Small, Office of Engineering and Technology, (202) 418-2452.</contact>**
**     ****<cfr title="47">**
**       ****<part number="25"/>**
**       ****<part number="73"/>**
**       ****<part number="74"/>**
**     ****</cfr>**
**     ****<tocSubject1>Practice and procedure:</tocSubject1>**
**     ****<tocDoc>Commission’s rules; editorial modifications, </tocDoc>**
**   ****</collectionSpecific>**
**</descMdGroup>**

## Using the Table of Contents

- Subject oriented table of contents
- Federal register:
- Subject headings
- Weekly Compilation of Presidential Documents:
- Category headings
- Congressional Record:
- Daily digest

***subject headings used******to annotate granules***

## TOC Metadata Example

** ****<descMdGroup id="id-05-10658">**
**   ****<title>Editorial Modifications of the Commission’s Rules</title>**
**   ****<printPageRange first="31372" last="31374"/>**
**   ****<migratedDocID>fr01jn05-12</migratedDocID>**
**   ****<collectionSpecific>**
**     ****<accessId>05-10658</accessId>**
**     ****<granuleClass>RULE</granuleClass>**
**     ****<agency order="1">FEDERAL COMMUNICATIONS COMMISSION</agency>**
**     ****<effectiveDate>2005-04-14</effectiveDate>**
**     ****<billingCode>6712-01-P</billingCode>**
**     ****<frDocNumber>05-10658</frDocNumber>**
**     ****<action>Final rule.</action>**
**     ****<summary>This document amends twelve sections ... updating the postal address of the Arecibo Radio Astronomy Observatory.</summary>**
**     ****<dates>Effective April 14, 2005.</dates>**
**     ****<contact>Rodney Small, Office of Engineering and Technology, (202) 418-2452.</contact>**
**     ****<cfr title="47">**
**       ****<part number="25"/>**
**       ****<part number="73"/>**
**       ****<part number="74"/>**
**     ****</cfr>**
**     ****<tocSubject1>****Practice and procedure****:</tocSubject1>**
**     ****<tocDoc>****Commission’s rules; editorial modifications, ****</tocDoc>**
**   ****</collectionSpecific>**
**</descMdGroup>**

## TOC Searches

- Find all articles in the Federal Register which are listed as “meetings” in the Contents
- Find all entries from the body of the Congressional Record listed under “Measures Passed” in the Daily Digest
- Find all entries from the body of the the Congressional Record identified as “The Patriot Act” in the Daily Digest
- Find all presidential documents identified as “Communications to Congress” in the contents

## Table of Contents Example

## Congress members and committees are normalized

- Congress Member Codes
- Congressional Committee Codes

**<congCommittee authorityId="ssga00" chamber="S" **
**	****	****	****	****congress="109" type="S">**
**  ****<name type="authority-standard">**
**     ****Committee on Homeland Security and Governmental Affairs</name>**
**  ****<name type="authority-short">**
**     ****Homeland and Governmental Affairs</name>**
**</congCommittee>**

** ****<congMember authorityId="308" chamber="S" congress="109" party="D"**
**	****	****	****	****	**** ****role="SPONSOR" state="ND">**
**  ****<name type="parsed">Mr. Dorgan </name>**
**  ****<name type="authority-fnf">Byron L. Dorgan</name>**
**  ****<name type="authority-lnf">Dorgan, Byron L.</name>**
**  ****<name type="authority-other">Byron Leslie Dorgan</name>**
**</congMember>**

## Display the official names of congress members, presidents, and committees

- Search on a committee even if the name changes over time
- Search by congress member state codes or party affiliations
- “Final all documents sponsored by a senator from Maryland”

## Search Features

## Search Results

*Search Results*

***volume***

***firstpage***

***title***

***collection***

***section***

***action******(first 20 chars)***

***publishdate***

***teaser***

***link to ******content-detail***

## Navigators

## Collection Browsing

*Collection Browsing*

***number range***

***law type***

***congress***

***List of******Laws***

## Advanced Search Form

## Data Mapping Examples

**Internal Data Storage**

- 110
- V
- FR
- RULE
- congnum
- 2006-02-01
- [2006-02-01;]
- accode=PPLcongnum=110billnumber=1234publishdate=2008-01-01

**Web Application**

- 110th Congress (2007-2008)
- Part V
- Federal Register
- Rules and Regulations
- Congress Number
- February 1st, 2006
- After February 1st, 2006
- Public and Private Laws. 110th Congress. H.R. 1234. January 1, 2008.

## Content Detail

## Query Language:  Syntax

- Simple queries:  Barbara Mikulski and the Bay
- NOTE:  Defaults to AND
- Exact Phases:  “Department of Defense”
- Boolean operators:
- 		(Mikulski or Cardin) and bay not seasoning
- or, and, ( ), not
- Proximity Operators:  house near/10 action
- near/#,  before/#,  adj
- Date and numeric ranges
- range(110,130)
- range(2008-01-01, 2009-01-01)

## Query Language:  Examples

- hearing
- “congressional hearing”
- congressional adj hearing
- congressional hearing
- ways and means
- “ways and means”
- ways “and” means
- congressional or congress
- congress and (report or meeting or notice)
- congnum:range(103,110)

- congressional not report
- congressional –report
- +cardin congressional committee
- congresional not (committee report)
- congressional not (committee or meeting)
- representative near/10 cardin
- representative before/10 cardin

## The Field Operator

- Fields can be targeted with the field: operator
- For R1C2:  A selected list of fields are available
- Each collection has its own list of searchable fields
*Note:  More fields available with field: than are available on the advanced search form*
- Examples:
- title:(environmental controls)
- congressnum:110
- title:”executive communications”
- congressnum:110 and billnumber:range(1000,1500)

## Search over MODS

- Expected for a near-term future release
- Can perform complex queries directly over the MODS
- Since the MODS contains all extracted metadata, this means just about any search is possible
- Examples:
- mods:identifier:(@type:citation and “70 FR 14232”)
- collection:BILLS and mods:congMember:(@congress:house and @state:MD and @role:SPONSOR) and mods:USCode:(@title:14 and section:@number:673)

## Relevancy Ranking

- Priority 1:  Citations
- Public Law 108-5
- 70 FR 31389
- H.R. 1413
- Priority 2:  Words in the title
- Priority 3:  Names (agencies, organizations, congress members, committees)
- Priority 4:  Other summary metadata (abstracts)
- Priority 5:  Referenced documents (i.e. U.S.C. references, Stat. L. references)
- Priority 6:  Everything else

## Citations are Normalized and Expanded for Search

- All citation methods are normalized based on metadata
- Citations are then expanded to include all common variations
- HR 1042
- H.R. 1043
- House Bill 1043
- All variations are indexed, a search on any variation will retrieve the desired record

## Other search features

- Spell Check
- Basic English plus common proper names
- To be expanded in the next release
- Synonyms
- None in this release
- Capability installed and available for next release
- Wildcards
- Standard with software
- legis*
- presid*al
- theat??

## Accessing Documents

## Access IDs

- Federal Register	_*Example:*_
- Package:  Issue Date	FR-2008-01-03
- Granule:  FR Document Number 	06-1325
- Congressional Bills
- Package:  Congress number, bill type, bill number, bill version code	BILLS-109hr1323rfs
- Weekly Compilation of Presidential Documents
- Package:  Issue Date	WCPD-2008-01-03
- Granule:  Issue Date + page number + count	WCPD-2008-01-03-Pg423-2
- Public and Private Laws
- Package:  congress, type, number	PLAW-108publ5

## Things you can download

- Granules
- The PDF of the entire granule (not just page at a time)
- An HTML rendition of the granule
- Granule MODS
- Packages
- PDF files of the entire package, the front matter, the end matter, and any nested granule
- PDF files of any nested granule
-  ZIP file of all publicly available package contents
- Package level MODS
- Package PREMIS data
- Possibly other renditions (e.g. XML)

## Package-Level URLs

- Package Content Detail
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/content-detail.html_
- Package Metadata Standards
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/mods.xml_
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/premis.xml_
- Package Table of Contents
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/toc.html_
- Today’s Table of Contents
- _http://www.gpo.gov/fdsys/html/FR/todays_toc.html_

## Granule-Level URLs

- HTML and PDF Files
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/html/E6-1423.html_
- _http://www.gpo.gov/fdsys/pkg/FR-2006-01-01/pdf/E6-1423.pdf_
- Granule Content Detail
- _http://www.gpo.gov/fdsys/granule/FR-2006-01-01/__E6-1423/content-detail.html_
- Granule Metadata Standards
- _http://www.gpo.gov/fdsys/granule/FR-2006-01-01/__E6-1423/mods.xml_

## MODS Example

## Repository Design

- Johnny Gee

## Repository Key Requirements

- Basic Requirements
- Submission
- Ingestion
- Access Processing
- Metadata Management
- Preservation
- Publishing
- Integration Interfaces
- Content Parser
- Virus Check
- Adobe LiveCycle
- AIP Reconstruct Utility
- ILS/ESB
- Publishing to ACP Cache

- Use Cases:
- Login
- Upload
- Submit Package
- Upload ACP Rendition
- Update Content
- Update Metadata
- Delete ACP Package/Rendition
- Delete AIP Package
- Validate Package
- Search
- Browse
- Reporting
- Etc.

## Enterprise Content Management Platform

## FDsys Repository Package

## Roles in FDsys

- Roles

| Role Name | Description |
| --- | --- |
| ext_search_and_access | External Search and Access |
| submission_search_and_access | Submission Search and Access |
| standard_search_and_access | Standard Search and Access |
| content_submission | Content Submission |
| content_mgmt | Content Management |
| reporting_manifests | Manifest Reporting |
| reporting_metrics | Metrics Reporting |
| workflow_admin | Workflow Administrator |

**Notes:**
Webtop actions can be configured to use any role.  These actions can be available only to certain roles or cannot be performed by certain roles.  
The use of role based actions in conjunction with group based security on objects provides a flexible model to support application level security.  
Users can belong to multiple roles, yet unless they have group level access to content objects, they will not be able to perform actions on those objects.  
For example, a user who belongs to Content Management role will be able to see File > Edit menu item.  However, if the user only belongs to ACP_DELETE group, he/she will not be able to view/edit AIP content, since the ACP_DELETE group has NONE access to AIP cabinet and AIP content.

## FDsys Security

- Used on Work In Progress (SIP) and Submitted (ACP) cabinets.  Document level security will be handled by alias sets and template ACLs.
- Used only in the Archived (AIP) cabinet and the contained folders and archive documents.  A server method will perform deletions in order to support group approval for package deletions.

|  | fdsys_sip_ cabinet_acl | fdsys_acp_ cabinet_acl |
| --- | --- | --- |
| dm_owner | Delete+LOP | Delete+LOP |
| dm_world | None | None |
| sip_delete | Read | None |
| acp_read | None | Read |
| acp_delete | None | Read |
| aip_write | None | Read |
| fdsys_admins | Delete+LOP | Delete+LOP |

|  | fdsys_aip_cabinet_acl | fdsys_aip_folder_acl | fdsys_aip_archive_doc_acl |
| --- | --- | --- | --- |
| dm_owner | None | None | None |
| dm_world | None | None | None |
| sip_delete | None | None | None |
| acp_read | None | None | None |
| acp_delete | None | None | None |
| aip_write | Read | Write | Write |
| fdsys_admins | Write+LP | Write+LP | Write+LP |

## Repository Process Overview

## Submission to FDsys

- FDsys will support uploading content files into FDsys via two methods:
- Upload Package (interactive)
- Hot Folder Job (folder-based)
- Hot Folder Job will perform the upload automatically, instead of manually by a user.
- Once uploaded, a user can select multiple packages to submit at one time.  The submit action will launch the Ingestion Workflow.

## Ingestion Workflow

- Create and Validate AIP Package
- Generate METS, MODS, and PREMIS xml files
- Create CryptoTimeStamp (Fixity)
- Generate two separate message digests using unique FDsys ID and ingestion date-time
- Create Preservation Rendition (XML)
- Generate xml rendition from pdf content files using Adobe LiveCycle
- Invoke Access Processing Workflow

## ACP and AIP Packages

- ACP packages are stored in ACP filestore
- AIP packages are stored in AIP filestore, which is physically different than the ACP filestore
- ACP is designed to support publishing to ACP Cache and daily updates to packages
- AIP is designed to be self-contained, self-describing, and independent of DCTM to support long-term preservation

## METS, MODS, & PREMIS

- METS, MODS, & PREMIS are generated and updated when AIP package is created, and when any metadata and/or content files are updated
- MODS file will be versioned when there is a change in the metadata
- This will facilitate tracking of metadata updates
- PREMIS file will have a accumulative provenance information.
- METS file will have complete information about metadata and content files for a package, independent of DCTM
- AIP Reconstruct Utility can be used to reassemble AIP package from AIP filestore, independent and outside of Documentum

**Notes:**
METS file will have complete information about metadata and content files for a package, independent of DCTM
METS list file will have pointers to all of the METS file located in the AIP filestore

## Access Processing Workflow

- Call Appropriate Parser
- Load parsed data into Documentum for ease of metadata updating
- Parser determines quality of metadata based on heuristic rules
- Validate Quality of Data
- If quality errors exist or metadata is missing, notify metadata correction group to review and edit metadata
- Users can search for package with low quality metadata and update metadata values appropriately
- Create PDF granules
- Generate pdf granules from submitted PDF files using Adobe LiveCycle

## Access Processing Workflow (cont’d)

- Create HTML rendition
- Extract images from PDF files using Adobe LiveCycle
- Perform OCR on images to extract indexable text
- Include text as part of ALT tag for the image in HTML
- Convert URLs and E-mail addresses to clickable links
- Digitally Sign PDF (if required)
- Apply GPO Digital Signature and Certificate
- Invoke ILS Workflow
- Publish Package for Access

## FDsys.xml Metadata Mapping to CMS

## Metadata Management for Authorized Users

- For submission_search
- _and_access role

## Metadata Management for Authorized Users (cont’d)

- For standard_search
- _and_access role

## Advanced Search

## Architecture & Design

- Q & A