Federal Geographic Data Committee

Building a 

Geospatial Data Clearinghouse

for

Discovery and Access

Douglas Nebert

ddnebert@usgs.gov

Mark Reichardt

International Activities

10 - 13 April 2000

mreichardt@usgs.gov

Good morning.  My name is Mark Reichardt, and I represent the United States Federal Geographic Data Committee. My role at the FGDC is to coordinate international SDI activities.  Today, I will be providing you with an overview of the progress and issues regarding the US National Spatial Data Infrastructure.  

I encourage you to ask questions at any time during the presentation.

Agenda
        - U.S. National Spatial Data Infrastructure
        - Clearinghouse Defined
        - Clearinghouse Growth Nationally
        - Towards a Global Implementation 
        - Issues and Opportunities
        - Next Steps

The NSDI

Established by Presidential Executive Order on 11 April 1994

NSDI Vision:  

Current and accurate geospatial data are readily available locally, nationally, and globally to contribute to economic growth, environmental quality, and social progress

Here is the NSDI vision.

Federal Geographic Data Committee
        - Created by the US Office of Management and Budget in 1990 to coordinate geospatial activities across government
        - Representation:

17 cabinet and executive level agencies

State, local, tribal, academic, industry stakeholders

Chaired by Secretary Babbitt

FGDC staff supports operations, administered through the U.S. Geological Survey

The Federal Geographic Data Committee, or FGDC for short, was formed in the early 1990’s after many years of examining the mapping, surveying, and related geospatial activities across government.  Based on concerns about duplication of effort and insufficient coordination, the FGDC was formed with the objective of eventually developing a “national spatial information resource”  to enhance geographic data sharing and use nation-wide.  The FGDC’s role would be to formalize the coordination of geospatial programs between federal agencies, and to foster collaboration with state, local, tribal governments, academia, and the private sector.  The major goal was to coordinate the generation, use, distribution and sharing of surveying, mapping and related geospatial information.

Although the FGDC started with only 12 Federal members, that membership has grown to 17 federal organizations.  With the advancement of GIS and geographic data in management and decision process, some agencies, considered previously not to be engaged in geospatial activities, have been added:  The Department of Health and Human Services and the Department of Justice are the latest entries to the FGDC.     Federal agencies are members of a Steering Committee, chaired personally by Secretary Babbitt (he has missed only one Steering Group Meeting since the inception of the FGDC).  Other state, local, tribal, private sector, and academic stakeholder organizations participate in FGDC activities.  I’ll detail those activities and organizations on the next few slides

Core Components of the NSDI

Partnerships

Metadata

GEOdata

Clearinghouse (catalog)

Framework

Standards

The FGDC identified some basic components of the US NSDI,  they are Standards; geographic data; Metadata; Clearinghouses, and partnerships.   Given time constraints, I will cover the catalog and metadata portions of the NSDI in detail. I invite you to review the additional slides in my package after the session.

Who Is Involved?

National League of Cities 	National Association of Counties

Intertribal GIS Council		University Consortium on GI Sciences

Open GIS Consortium		FGDC

National States Geographic Information Council  

International City / County Managers Association

VI

The NSDI has a network of stakeholders: federal, state, local, and tribal governments and associations.  The NSDI also has stakeholders from the academic community, and has a relationship with the Open GIS Consortium, which presently represents over 200 members of government, industry and academia in advancing interoperability of geographic processing. 

FGDC Standards...
        - Created by thematic subcommittees, representing consensus view of data theme
        - Submitted for 90-day public review 
        - Reviewed across disciplines for uniformity
        - Published as US Federal Standards
        - Standards by ISO and other national standards are used FIRST!
        - Standardization makes the Clearinghouse work

Framework supports...
        - Community development of sets of spatial primitives, feature representation, and attribution to a lowest common denominator
        - Participant collecting or converting information to common Framework feature specifications
        - Multiple representations of real-world features at different scales and times by feature identifier and generalization

Clearinghouse Defined
        - A distributed network that includes:

a registry of servers

several WWW-to-Z39.50 gateways

many Z39.50 (GEO) profile servers
        - Allows a single query of all or a portion of world-wide servers in a single session

The Clearinghouse concept is essentially a distributed set of servers registered through a catalog.  These servers are accessible through a www gateway and allow for comprehensive searches through the use of common search protocols and consistently defined metadata.  Adequate and consistently defined metadata is the key for successful clearinghouse searches.  

The goal here is to enable detailed searches for geographic information across the entire community of servers.

Clearinghouse Defined
        - Links through to full data access where desired / available 
        - Will not operate without Metadata

ANSI Z39.50/ISO 23950
        - Application protocol layered upon OSI or TCP/IP (now HTTP)
        - Developed by the library community initially, adopted by museums, chemical abstracts, information locator, now geomatics 
        - Provides Search and Retrieval services in a platform-independent fashion

Z39.50 Details
        - Stateful session-based connection (usually)
        - Query expressed using registered integer codes to reference terms, operators, and formats
        - Query usually expressed using RPN-style syntax
        - Request can return well-known groups of fields Element Sets (what) 
        - Request also packaged as text, XML, GRS-1, MARC, etc. (how)

Init

Init

Response

Search

Search

Response

Present

Request

Present

Response

Client

Server

Document

Display

Z39.50 Functions

What is GEO?
        - Implementing guidelines for defining the content and operations for FGDC geospatial metadata in a compliant Z39.50 software system
        - Extended set of bibligraphic fields to exactly search geospatial holdings

Geo coordinates

Temporal conditions

Free text (i.e. anywhere in metadata)

Z39.50 Profiles are registered
        - Identify queryable fields or constructs, operators, structures for re-use
        - Specify query syntax to be used
        - Define standard groupings of elements to be returned (e.g. Brief, Summary, Full)
        - Define expected delivery formats (preferred syntax)

Metadata

Metadata
        - FGDC Metadata Standard in wide use at national, state, local levels
        - Metadata publishing tools available
        - ISO TC-211  International Metadata Standard  (second draft is in review)

FGDC endorses ISO metadata standard

FGDC will develop a profile of the ISO standard, and will support nationwide implementation

In the US, the FGDC Metadata standard is in wide use to support clearinghouse operations.  There is an international (ISO) metadata standard presently under review by the U.S. and other nations.  The FGDC endorses this international standard, and will develop an implementation strategy and training program to support transition from the FGDC metadata specification.

Clearinghouse Method

Metadata

preparation

Metadata

validation/

staging

Metadata

publication

User

access

The Clearinghouse Method is a publication method for documented spatial data. At its root, it requires the preparation of adequate metadata, or data descriptions, to put into a searchable and potentially browseable catalog. This distributed catalog of data and metadata is known as “Clearinghouse” and employs metadata and catalog service standards for interoperability. The producer of spatial data or designee prepares or extracts public metadata values. These values are sometimes numeric (dates, times, coordinate “footprints”) and sometimes text or narrative fields providing story-telling about the data set and its history. This metadata is often brought to or stored in a central location within an organization. This staging area, where QC checks may be run on the data and metadata, may be on a LAN, on each publisher’s desktop, or in a central agency repository. Metadata publication, then, is the placement of public metadata with links to data in a searchable database service to provide user access.

Discovery in Clearinghouse

Web

Client

Clearinghouse

Nodes

Gateway(s)

User

HTTP protocol

Service

Registry

Web

Server

Z39.50 protocol

Spatial data (via metadata) are discovered typically in this configuration. A user query is sent from a web client to a special web server (via a form) that forwards the search request to many servers through a distributed search client installed in the web server. This client can connect to multiple (over 200) Clearinghouse Nodes (aka Catalog Servers) at the same time. Results are collated by the gateway and sent to the user as HTML. MapInfo has also written a Java-based client and gateway-in-one to bring all this distributed search functionality to the desktop.

Abstraction
        - Z39.50 allows us to separate the implementation of a local database and its data element name from an external or public set of fields and format
        - Abstraction allows metadata providers to select from various search engines or databases
        - Supports semantic translation between discipline, language, and other communities through server-side “mappings” of public to local fields

Abstraction of Query

Client

zserver

zserver

API

API

Oracle

Isite

GUI

Water in 

the title

This set of slides depicts the abstraction features of Z39.50 to search one or more servers with metadata. The forest green parts are the Z39.50 protocol. A Graphical User Interface collects the user’s search criteria by one of many methods.

Abstraction of Query

Client

zserver

zserver

API

API

Oracle

Isite

GUI

“water” in

Attribute 4 ?

This query is translated or packaged into a special query for transmission across the Internet in a form understood by all Z39.50 servers. This query is transmitted to one or more Z39.50 servers. Each client to server connection is a session.

Abstraction of Query

Client

zserver

zserver

API

API

Oracle

Isite

GUI

“water” in

Attribute 4 ?

“water” in

Attribute 4 ?

The queries are received by the Z39.50 servers.

Abstraction of Query

Client

zserver

zserver

API

API

RDBMS

Isite

GUI

IDINFO.CITEINFO.

CITATION.TITLE eq ‘water’

Select * from main  where 

name like %water%;

Behind the Z39.50 servers exists some translation software to direct the general purpose query against local fields that match the search criteria. One special field in most Z39.50 implementations is called Anywhere, which stands as a surrogate for a full-text search. Multiple term searches may be composed and passed, joined with ANDs or ORs. The power of this abstraction is that a client does not need to know the details of the database or schema or query language used by the remote server, and in so doing can easily approach many servers within a community with the same query without special modification.

Clearinghouse Interfaces
        - Access to Clearinghouse may be via:

formal Gateway search forms hosted at FGDC and participating organizations

Custom “local” access web forms

Custom applications that support standard search using ISO 23950/Z39.50 
        - Next slides show one customized search

How do servers cooperate?
        - Global registry of Clearinghouse servers is maintained by FGDC on behalf of the Global Spatial Data Infrastructure activity
        - Each gateway communicates with the server registry to update search forms and parameters
        - Service Registry will provide for intelligent selection of servers based on server metadata (Q2 FY 2000)

NSDI Clearinghouse Growth 1994 – 1999

January

Growth of searchable clearinghouse sites has been accelerating, not only nationally, but internationally as well.  Via growth in Z39.50 and FGDC metadata compliant sites, the number of spatial data catalog sites has grown to approximately 20o, with aboutr 40% of the sites international in scope or location.

Towards a Global Catalogof Geospatial Data and Services
        - Many Z39.50 Gateway and Server resources available on the Internet today:
        - FGDC National Clearinghouse
        - CEONet Canada
        - Australian Spatial Data Directory
        - South African National Spatial  Information  Framework

Z39.50 Geo / Metadata Compliant Sites

Gateway Design
        - Gateway acts as a WWW (HTTP) to Z39.50 translator or protocol bridge
        - Enables search in parallel to multiple servers 
        - What’s Out There?

Isite zgate (UNIX) is reference gateway functionality (serial search only)

Blue Angel MetaStar Gateway is commercial solution

Blue Angel MetaStar(tm) Gateway

Web Server

BER Client

Z39.50

Server

(Isite)

Web Client

CORBA Client

Z39.50

Server

(Compusult)

CORBA

Server

(GEODAN)

Open Source Version being commissioned. Due to the street in July 2000

Server (Node) in More Detail

Metadata

Index/DB

Z39.50

server

Internet

Data

Available Server Solutions
        - Isite is XML-database solution using open source as a reference implementation of a searchable metadata service
        - YAZ Toolkit (www.indexdata.dk) is a development system for building service solutions

Commercial Clearinghouse Server Solutions
        - Compusult MetaManager Toolkit to build an FGDC-compliant server on top of a RDBMS (Oracle, MS SQL-Server, Access) http://www.fgdctoolkit.com
        - Blue Angel Technologies has a MetaStar Repository for support of many types of metadata (GILS, DC, MARC, GEO, etc.) http://www.blueangeltech.com
        - Isite OpenSource: free use and download from http://clearinghouse4.fgdc.gov/ftp

Isite solution
        - Isite software suite includes Z39.50 server and XML-based (1.0) document database
        - Runs on Windows 9X/NT and UNIX
        - Efficient indexes for thousands to hundreds of thousands of entries
        - Includes “script” API to interface to DB
        - Version 2.07 to have mySQL/Oracle backend examples
        - Isite OpenSource: free use and download from http://clearinghouse4.fgdc.gov/ftp

Contents of Isite Installation

Isite
        -  Iindex
        -  Isearch
        -  zclient
        -  Izclient
        -  Iutil
        -  zserver
        -  zserver.ini
        -  sapi.ini
        -  fgdc.fields
        -  bib1.fgdcmap
        -  gils.fgdcmap
        -  geo.fgdcmap

metadata

index

Place XML, HTML, and TEXT here

Direct Iindex to place the index files here

Iindex -d  \index\oregon  -t fgdc 

-o fieldtype=fgdc.fields  \metadata\*.sgml

Isite Architecture

Z39.50

server

XML

index

zserver.ini:
        - port number
        - location of sapi.ini
        - debugging info
        - Index name(s)

sapi.ini:
        - index names
        - location of index
        - search method
        - field maps

Iindex command

Index on XML, return 

alt form in HTML, text

Metadata

“Entry”

Metadata

“Entry”

Metadata

“Entry”

Metadata

“Entries”

Database Option

Internet

Relational Database

(Metadata)

Z39.50

server

Data

Interface

Compusult 

MetaManager Toolkit

Compusult MetaManager
        - Allows FGDC metadata to be stored in many tables (ISO within the year)
        - Preserves hierarchy/collection metadata
        - Manages multiple collections at a site
        - Allows re-usable fields as static value, function, field value(s), or combination
        - Includes web search interface
        - Available for Access ($1995) and Oracle ($5000)

MetaManager(tm) Design

Z39.50

server

Metadata Schema(s)
        - field names
        - structures
        - join information
        - configuration

MetaManager

Toolkit

Relational

Database

Metadata

“Entry”

Metadata

“Entry”

Metadata

“Entry”

Metadata

“Entries”

The MetaManager Toolkit is an administrative interface to building an FGDC-compatible service interface to an existing RDBMS. This is done by creating a set of small tables and storing some constants there. The toolkit runs a transformation of the output metadata, like a report function, delivering formatted metadata like every other server.

Federal Geographic Data Committee

Clearinghouse methods and protocol usage were offered as the “WWW Profile” of OpenGIS Catalog Services Request for Proposals approved in August 1999.

Catalog Services and the OpenGIS Consortium
        - OpenGIS Consortium enlisted member organizations to develop an open Catalog Services Standard
        - A team comprised of international government, industry 
        - OGC Standard recently approved
        - Publicly available at www.opengis.org

Web-based MappingDefined
        - Users easily and rapidly search, retrieve, access & exploit geospatial information from multiple locations on the Internet

Open Technology specifications that enable Internet-wide exploitation of geodata and sharing of processing services

Better address the vast opportunities to utilize NSDI resources via the Internet

DISTRIBUTED

MULTI-

VENDOR

Web Mapping was a term created several years ago by OGC members in their efforts to define how the World Wide Web could be used as an infrastructure to support the search, discovery, access, and exploitation of geographic data from sites across the web.

Web-based mapping assumes the use of standardized protocols that allow servers and clients to interact for data exchange, and to share processing - regardless of the product being used.  Vendors designing their products simply have to use the open technology specifications to gain  open access.

Examples of web based mapping include the identification of the existence of data to meet your need (Catalog Services), the ability to easily access and retrieve spatial data in the form needed for your application (geodata translation services, data retrieval services)

Web Mapping Testbed

OpenGIS Consortium (OGC) members conducted Web Mapping Testbed Phase I over 4 months to:

define interfaces for web mapping services based on existing mature technology

demonstrate integration of multiple OGC services together with different vendor products

Publish open technology standards

Redefine the environment for web based spatial applications

3. LandSAT with Roads & Hospitals

Ionic Software Viewer Client

MIT LandSAT image

Laser-Scan Roads

Object/FXHospitals

This is just one of many vendor products which have been developed or modified to follow the Open GIS Web Mapping Specifications.  Ionic, a company in Europe designed this JAVA based client to provide detailed and very high resolution views of data from multiple sources.  

Other industry members that employed Open GIS specifications for web mapping include ESRI, Lockheed Martin, Cubewerx, and many others.  For a full listing of the Web Mapping Testbed sponsors and participants, please consult the full version of the Web Mapping Testbed briefing located under Web Mapping Testbed at www.opengis.org.

3. Road Restrictions to EmergencyVehicles

Ionic Software Viewer Client

MIT LandSAT image

Laser-Scan RoadsHurricane Flooding

Object/FXHospitals

Future Directions 
        - Global Service Catalog of geospatial services under the GSDI initiative (2000)
        - Metadata management tools integrated with vendor products – metadata as part of the information management process
        - Semantic mapping – consistent thesaurus, multilingual searches across global data bases.
        - Customized web-based geospatial applications reduce cost, increase access, and better support decision-making

Tomorrow:  A Global Infrastructure Enabled 

Through Partnerships, Standards, Technology

Businesses

s

Citizens

s

Government

s

Users

Registered

Data 

Servers

Imagery

Web-Enabled 

Services

Map

Services

Catalog

Services

Service

Registry

Internet

Elevation

Hydrography

Demographics

Health

Geodetic

Control

Transportation

Internet

 Clearinghouse

Economic
        - E-Commerce
        - Authentication
        - Other

Metadata

Metadata

Metadata

Metadata

Metadata

Metadata

Metadata

Crime

Environment

Combine the successes to date with open technology protocols, catalog services, map services, and other registered WWW services, and we could see a near term future of greatly enhanced applications that can search, access, and exploit a vastly expanded world of geographic data.  

None of this could have happened without the focused efforts of national spatial data infrastructures working together toward a core set of common goals.  

	Standards

	Best Practices

	Clearinghouses

	Metadata

Additional Information
        - www.fgdc.gov		Federal Geographic 						Data Committee
        - www.opengis.org          OpenGIS Consortium
        - www.gsdi.org		Global Spatial Data 						Infrastructure

# UNKNOWN-0.xls