# Presentation: 664932

## Building a 

**Geospatial Data Clearinghouse**
**for**
**Discovery and Access**

**Douglas Nebert**
**ddnebert@usgs.gov**
**Mark Reichardt**
**International Activities**
**10 - 13 April 2000**
**mreichardt@usgs.gov**

**Notes:**
Good morning.  My name is Mark Reichardt, and I represent the United States Federal Geographic Data Committee. My role at the FGDC is to coordinate international SDI activities.  Today, I will be providing you with an overview of the progress and issues regarding the US National Spatial Data Infrastructure.  

I encourage you to ask questions at any time during the presentation.

## Agenda

- U.S. National Spatial Data Infrastructure
- Clearinghouse Defined
- Clearinghouse Growth Nationally
- Towards a Global Implementation
- Issues and Opportunities
- Next Steps

## The NSDI

- Established by Presidential Executive Order on 11 April 1994
- NSDI Vision:
- Current and accurate geospatial data are readily available locally, nationally, and globally to contribute to economic growth, environmental quality, and social progress

**Notes:**
Here is the NSDI vision.

## Federal Geographic Data Committee

- Created by the US Office of Management and Budget in 1990 to coordinate geospatial activities across government
- Representation:
- 17 cabinet and executive level agencies
- State, local, tribal, academic, industry stakeholders
- Chaired by Secretary Babbitt
- FGDC staff supports operations, administered through the U.S. Geological Survey

**Notes:**
The Federal Geographic Data Committee, or FGDC for short, was formed in the early 1990’s after many years of examining the mapping, surveying, and related geospatial activities across government.  Based on concerns about duplication of effort and insufficient coordination, the FGDC was formed with the objective of eventually developing a “national spatial information resource”  to enhance geographic data sharing and use nation-wide.  The FGDC’s role would be to formalize the coordination of geospatial programs between federal agencies, and to foster collaboration with state, local, tribal governments, academia, and the private sector.  The major goal was to coordinate the generation, use, distribution and sharing of surveying, mapping and related geospatial information.

Although the FGDC started with only 12 Federal members, that membership has grown to 17 federal organizations.  With the advancement of GIS and geographic data in management and decision process, some agencies, considered previously not to be engaged in geospatial activities, have been added:  The Department of Health and Human Services and the Department of Justice are the latest entries to the FGDC.     Federal agencies are members of a Steering Committee, chaired personally by Secretary Babbitt (he has missed only one Steering Group Meeting since the inception of the FGDC).  Other state, local, tribal, private sector, and academic stakeholder organizations participate in FGDC activities.  I’ll detail those activities and organizations on the next few slides

## Core Components of the NSDI

***Partnerships***

**Notes:**
The FGDC identified some basic components of the US NSDI,  they are Standards; geographic data; Metadata; Clearinghouses, and partnerships.   Given time constraints, I will cover the catalog and metadata portions of the NSDI in detail. I invite you to review the additional slides in my package after the session.

## Who Is Involved?

**National League of Cities ****	****National Association of Counties**
**Intertribal GIS Council****	****	****University Consortium on GI Sciences**
**Open GIS Consortium****	****	****FGDC**
**National States Geographic Information Council  **
**International City / County Managers Association**

**Notes:**
The NSDI has a network of stakeholders: federal, state, local, and tribal governments and associations.  The NSDI also has stakeholders from the academic community, and has a relationship with the Open GIS Consortium, which presently represents over 200 members of government, industry and academia in advancing interoperability of geographic processing. 

## FGDC Standards...

- Created by thematic subcommittees, representing consensus view of data theme
- Submitted for 90-day public review
- Reviewed across disciplines for uniformity
- Published as US Federal Standards
*Standards by ISO and other national standards are used FIRST!*
*Standardization makes the Clearinghouse work*

## Framework supports...

- Community development of sets of spatial primitives, feature representation, and attribution to a lowest common denominator
- Participant collecting or converting information to common Framework feature specifications
- Multiple representations of real-world features at different scales and times by feature identifier and generalization

## Clearinghouse Defined

- A distributed network that includes:
- a _registr_y of servers
- several WWW-to-Z39.50 _gateways_
- many Z39.50 (GEO) profile _servers_
- Allows a _single query_ of all or a portion of world-wide servers in a single session

**Notes:**
The Clearinghouse concept is essentially a distributed set of servers registered through a catalog.  These servers are accessible through a www gateway and allow for comprehensive searches through the use of common search protocols and consistently defined metadata.  Adequate and consistently defined metadata is the key for successful clearinghouse searches.  

The goal here is to enable detailed searches for geographic information across the entire community of servers.

## Clearinghouse Defined

- Links through to full data access where desired / available
- Will not operate without Metadata

## ANSI Z39.50/ISO 23950

- Application protocol layered upon OSI or TCP/IP (now HTTP)
- Developed by the library community initially, adopted by museums, chemical abstracts, information locator, now geomatics
- Provides Search and Retrieval services in a platform-independent fashion

## Z39.50 Details

- Stateful session-based connection (usually)
- Query expressed using registered integer codes to reference terms, operators, and formats
- Query usually expressed using RPN-style syntax
- Request can return well-known groups of fields Element Sets (what)
- Request also packaged as text, XML, GRS-1, MARC, etc. (how)

## Z39.50 Functions

- Init

- Init
- Response

- Search

- Search
- Response

- Present
- Request

- Present
- Response

*Client*

*Server*

- Document
- Display

## What is GEO?

- Implementing guidelines for defining the content and operations for FGDC geospatial metadata in a compliant Z39.50 software system
- Extended set of bibligraphic fields to exactly search geospatial holdings
- Geo coordinates
- Temporal conditions
- Free text (i.e. anywhere in metadata)

## Z39.50 Profiles are registered

- Identify queryable fields or constructs, operators, structures for re-use
- Specify query syntax to be used
- Define standard groupings of elements to be returned (e.g. Brief, Summary, Full)
- Define expected delivery formats (preferred syntax)

## Metadata

- Metadata

- FGDC Metadata Standard in wide use at national, state, local levels
- Metadata publishing tools available
- ISO TC-211  International Metadata Standard  (second draft is in review)
- FGDC endorses ISO metadata standard
- FGDC will develop a profile of the ISO standard, and will support nationwide implementation

**Notes:**
In the US, the FGDC Metadata standard is in wide use to support clearinghouse operations.  There is an international (ISO) metadata standard presently under review by the U.S. and other nations.  The FGDC endorses this international standard, and will develop an implementation strategy and training program to support transition from the FGDC metadata specification.

## Clearinghouse Method

- Metadata
- preparation

**Notes:**
The Clearinghouse Method is a publication method for documented spatial data. At its root, it requires the preparation of adequate metadata, or data descriptions, to put into a searchable and potentially browseable catalog. This distributed catalog of data and metadata is known as “Clearinghouse” and employs metadata and catalog service standards for interoperability. The producer of spatial data or designee prepares or extracts public metadata values. These values are sometimes numeric (dates, times, coordinate “footprints”) and sometimes text or narrative fields providing story-telling about the data set and its history. This metadata is often brought to or stored in a central location within an organization. This staging area, where QC checks may be run on the data and metadata, may be on a LAN, on each publisher’s desktop, or in a central agency repository. Metadata publication, then, is the placement of public metadata with links to data in a searchable database service to provide user access.

## Discovery in Clearinghouse

**Web**
**Client**

***Clearinghouse***
***Nodes***

***Gateway(s)***

***User***

- HTTP protocol

- Service
- Registry

**Web**
**Server**

**Notes:**
Spatial data (via metadata) are discovered typically in this configuration. A user query is sent from a web client to a special web server (via a form) that forwards the search request to many servers through a distributed search client installed in the web server. This client can connect to multiple (over 200) Clearinghouse Nodes (aka Catalog Servers) at the same time. Results are collated by the gateway and sent to the user as HTML. MapInfo has also written a Java-based client and gateway-in-one to bring all this distributed search functionality to the desktop.

## Abstraction

- Z39.50 allows us to separate the implementation of a local database and its data element name from an external or public set of fields and format
- Abstraction allows metadata providers to select from various search engines or databases
- Supports semantic translation between discipline, language, and other communities through server-side “mappings” of public to local fields

## Abstraction of Query

- Client

- zserver

- zserver

- API

- API

- Oracle

- Isite

- GUI

- Water in
- the title

**Notes:**
This set of slides depicts the abstraction features of Z39.50 to search one or more servers with metadata. The forest green parts are the Z39.50 protocol. A Graphical User Interface collects the user’s search criteria by one of many methods.

## Abstraction of Query

- Client

- zserver

- zserver

- API

- API

- Oracle

- Isite

- GUI

- “water” in
- Attribute 4 ?

**Notes:**
This query is translated or packaged into a special query for transmission across the Internet in a form understood by all Z39.50 servers. This query is transmitted to one or more Z39.50 servers. Each client to server connection is a session.

## Abstraction of Query

- Client

- zserver

- zserver

- API

- API

- Oracle

- Isite

- GUI

- “water” in
- Attribute 4 ?

**Notes:**
The queries are received by the Z39.50 servers.

## Abstraction of Query

- Client

- zserver

- zserver

- API

- API

- RDBMS

- Isite

- GUI

- IDINFO.CITEINFO.
- CITATION.TITLE eq ‘water’

**Notes:**
Behind the Z39.50 servers exists some translation software to direct the general purpose query against local fields that match the search criteria. One special field in most Z39.50 implementations is called Anywhere, which stands as a surrogate for a full-text search. Multiple term searches may be composed and passed, joined with ANDs or ORs. The power of this abstraction is that a client does not need to know the details of the database or schema or query language used by the remote server, and in so doing can easily approach many servers within a community with the same query without special modification.

## Clearinghouse Interfaces

- Access to Clearinghouse may be via:
- formal Gateway search forms hosted at FGDC and participating organizations
- Custom “local” access web forms
- Custom applications that support standard search using ISO 23950/Z39.50
- Next slides show one customized search

## Slide 25

## Slide 26

## Slide 27

## Slide 28

## Slide 29

## Slide 30

## Slide 31

## How do servers cooperate?

- Global registry of Clearinghouse servers is maintained by FGDC on behalf of the Global Spatial Data Infrastructure activity
- Each gateway communicates with the server registry to update search forms and parameters
- Service Registry will provide for intelligent selection of servers based on server metadata (Q2 FY 2000)

## NSDI Clearinghouse Growth 1994 – 1999

- January

**Notes:**
Growth of searchable clearinghouse sites has been accelerating, not only nationally, but internationally as well.  Via growth in Z39.50 and FGDC metadata compliant sites, the number of spatial data catalog sites has grown to approximately 20o, with aboutr 40% of the sites international in scope or location.

## Towards a Global Catalogof Geospatial Data and Services

- Many Z39.50 Gateway and Server resources available on the Internet today:
- FGDC National Clearinghouse
- CEONet Canada
- Australian Spatial Data Directory
- South African National Spatial  Information  Framework

## Z39.50 Geo / Metadata Compliant Sites

## Gateway Design

- Gateway acts as a WWW (HTTP) to Z39.50 translator or protocol bridge
- Enables search in parallel to multiple servers
- What’s Out There?
- Isite zgate (UNIX) is reference gateway functionality (serial search only)
- Blue Angel MetaStar Gateway is commercial solution

## Blue Angel MetaStar(tm) Gateway

- Web Server

- BER Client

- Z39.50
- Server
- (Isite)

- Web Client

- CORBA Client

- Z39.50
- Server
- (Compusult)

**Notes:**
Open Source Version being commissioned. Due to the street in July 2000

## Server (Node) in More Detail

- Metadata

- Index/DB

## Available Server Solutions

- Isite is XML-database solution using open source as a reference implementation of a searchable metadata service
- YAZ Toolkit (www.indexdata.dk) is a development system for building service solutions

## Commercial Clearinghouse Server Solutions

- Compusult MetaManager Toolkit to build an FGDC-compliant server on top of a RDBMS (Oracle, MS SQL-Server, Access) http://www.fgdctoolkit.com
- Blue Angel Technologies has a MetaStar Repository for support of many types of metadata (GILS, DC, MARC, GEO, etc.) _[http://www.blueangeltech.com](http://www.blueangeltech.com/)_
- Isite OpenSource: free use and download from _[http://clearinghouse4.fgdc.gov/ftp](http://clearinghouse4.fgdc.gov/ftp)_

## Isite solution

- Isite software suite includes Z39.50 server and XML-based (1.0) document database
- Runs on Windows 9X/NT and UNIX
- Efficient indexes for thousands to hundreds of thousands of entries
- Includes “script” API to interface to DB
- Version 2.07 to have mySQL/Oracle backend examples
- Isite OpenSource: free use and download from http://clearinghouse4.fgdc.gov/ftp

## Contents of Isite Installation

## Isite Architecture

- Z39.50
- server

- XML
- index

**zserver.ini:**
- port number
- location of sapi.ini
- debugging info
- Index name(s)

**sapi.ini:**
- index names
- location of index
- search method
- field maps

**Iindex** command

*Index on XML, return *
*alt form in HTML, text*

## Database Option

- Internet

## Compusult MetaManager

- Allows FGDC metadata to be stored in many tables (ISO within the year)
- Preserves hierarchy/collection metadata
- Manages multiple collections at a site
- Allows re-usable fields as static value, function, field value(s), or combination
- Includes web search interface
- Available for Access ($1995) and Oracle ($5000)

## MetaManager(tm) Design

- MetaManager(tm) Design

- Z39.50
- server

**Metadata Schema(s)**
- field names
- structures
- join information
- configuration

- MetaManager
- Toolkit

- Relational
- Database

**Notes:**
The MetaManager Toolkit is an administrative interface to building an FGDC-compatible service interface to an existing RDBMS. This is done by creating a set of small tables and storing some constants there. The toolkit runs a transformation of the output metadata, like a report function, delivering formatted metadata like every other server.

## Clearinghouse methods and protocol usage were offered as the “WWW Profile” of OpenGIS Catalog Services Request for Proposals approved in August 1999.

## Catalog Services and the OpenGIS Consortium

- OpenGIS Consortium enlisted member organizations to develop an open Catalog Services Standard
- A team comprised of international government, industry
- OGC Standard recently approved
- Publicly available at _[www.opengis.org](http://www.opengis.org/)_

## Web-based MappingDefined

**Users easily and rapidly search, retrieve, access & exploit geospatial information from multiple locations on the Internet**

**Open Technology specifications that enable Internet-wide exploitation of geodata and sharing of processing services**

**Notes:**
Web Mapping was a term created several years ago by OGC members in their efforts to define how the World Wide Web could be used as an infrastructure to support the search, discovery, access, and exploitation of geographic data from sites across the web.

Web-based mapping assumes the use of standardized protocols that allow servers and clients to interact for data exchange, and to share processing - regardless of the product being used.  Vendors designing their products simply have to use the open technology specifications to gain  open access.

Examples of web based mapping include the identification of the existence of data to meet your need (Catalog Services), the ability to easily access and retrieve spatial data in the form needed for your application (geodata translation services, data retrieval services)

## Web Mapping Testbed

- OpenGIS Consortium (OGC) members conducted Web Mapping Testbed Phase I over 4 months to:
- define interfaces for web mapping services based on existing mature technology
- demonstrate integration of multiple OGC services together with different vendor products
- Publish open technology standards
- Redefine the environment for web based spatial applications

## 3. LandSAT with Roads & Hospitals

*3. LandSAT with Roads **& Hospitals*

*Ionic Software* Viewer Client
*MIT* LandSAT image
*Laser-Scan* Roads
*Object/FX*Hospitals

**Notes:**
This is just one of many vendor products which have been developed or modified to follow the Open GIS Web Mapping Specifications.  Ionic, a company in Europe designed this JAVA based client to provide detailed and very high resolution views of data from multiple sources.  

Other industry members that employed Open GIS specifications for web mapping include ESRI, Lockheed Martin, Cubewerx, and many others.  For a full listing of the Web Mapping Testbed sponsors and participants, please consult the full version of the Web Mapping Testbed briefing located under Web Mapping Testbed at www.opengis.org.

## 3. Road Restrictions to EmergencyVehicles

*Ionic Software* Viewer Client
*MIT* LandSAT image
*Laser-Scan* RoadsHurricane Flooding
*Object/FX*Hospitals

## Future Directions 

- Global Service Catalog of geospatial services under the GSDI initiative (2000)
- Metadata management tools integrated with vendor products – metadata as part of the information management process
- Semantic mapping – consistent thesaurus, multilingual searches across global data bases.
- Customized web-based geospatial applications reduce cost, increase access, and better support decision-making

## Tomorrow:  A Global Infrastructure Enabled 

- Tomorrow:  A Global Infrastructure Enabled
- Through Partnerships, Standards, Technology

**Businesses**

**s**

**Citizens**

**s**

**Government**

**s**

**Users**

**Registered**
**Data **
**Servers**

**Imagery**

**Web-Enabled **
**Services**

**Map**

**Services**

**Catalog**

**Services**

**Service**

**Registry**

**Internet**

**Elevation**

**Hydrography**

**Demographics**

**Health**

**Geodetic**
**Control**

**Transportation**

**Internet**

** ****Clearinghouse**

**Economic**

**E-Commerce**
**Authentication**
**Other**

**Metadata**

**Metadata**

**Metadata**

**Metadata**

**Metadata**

**Metadata**

**Metadata**

**Crime**

**Environment**

**Notes:**
Combine the successes to date with open technology protocols, catalog services, map services, and other registered WWW services, and we could see a near term future of greatly enhanced applications that can search, access, and exploit a vastly expanded world of geographic data.  

None of this could have happened without the focused efforts of national spatial data infrastructures working together toward a core set of common goals.  

	Standards
	Best Practices
	Clearinghouses
	Metadata

## Additional Information

**www.fgdc.gov****	****	****Federal Geographic ****	****	****	****	****	****	****Data Committee**
**www.opengis.org****          OpenGIS Consortium**
**www.gsdi.org****	****	****Global Spatial Data ****	****	****	****	****	****	****Infrastructure**