# Presentation: 496313

## NCDC Station Metadata System

**Jeff Arnfield**
**Active Archive Branch**
**National Climatic Data Center**
**Asheville, NC**

**Notes:**
Hi.  I’m Jeff Arnfield.  I work in the Active Archive Branch at NOAA’s National Climatic Data Center -- we call it NCDC -- in Asheville, North Carolina.  

## NCDC’s Role

**Nation’s focal point and scorekeeper for information about weather and climate variations and changes**
**Currently ingest and archive 150 terabytes (150,000 gigabytes) of data every year**
**Maintain the metadata necessary to interpret these data**

**Notes:**
NCDC is the nation's focal point and scorekeeper for information about weather and climate variability.  

This role involves management of a considerable volume of data.  Besides ingesting and archiving 150 terabytes of data every year, we must maintain the metadata needed to interpret them.

Today I’m going to tell you a bit about a project we’ve undertaken to more efficiently manage that metadata.

## Metadata: Data About Data

**Information necessary to describe and interpret collections of data and the observing systems that report them   **
**Station location **
**Station configuration **
**Observing standards **
**Reporting protocols **
**Data inventories**
**Data product documentation**
**History of changes over time**

**Notes:**
The stock definition of metadata is “data about data.” In this case, we define “metadata” to be the information necessary to describe and interpret our collections of data and the observing systems that report them.   

Station location, configuration, observing standards, reporting protocols, data inventories and data product documentation are all examples of this metadata. Any of this information can change over time, so tracking these historical changes is critical.

## Current Observing Systems

**NWS cooperative observing system**
**National surface observing system**
**Upper air observing system**
**Climate Reference Network (CRN)**
**Marine observing system**
**Profiler**
**Precipitation observing network**
**Radar**
**Global Climate Observing System**
**Satellite systems**
**Special national/international experiments**

**Notes:**
Our data come from a variety of sources and systems, both domestic and international, fixed and mobile, in situ and remote sensing.  These are some of the sources from which we receive data.

Besides observation data, each observing system provides a variety of related metadata.  

## Why Is Metadata Important?

**Critical to NCDC's ingest, archive and access systems  **
**Gives data users perspective on reported data values**
**Station moves & changes to surroundings**
**Sensor changes**
**Quality control algorithms**
**Helps in selection of stations, data products for study**

**Notes:**
Why is metadata important?  It plays a critical role in NCDC's ingest, archive and access systems.

It also gives users perspective on the observed values they see.  If a station has moved, or if a sensor has changed, that may explain a subtle difference in temperature over time.  A change in a quality control algorithm may affect values as well.

Metadata also helps in selection of stations and data products for further study.

## The Rest Is Just Metadata

**  ****42**

**Notes:**
To put it more simply: the data are hard to interpret without the necessary metadata

## 2000: Need and opportunity meet

**Reviewed metadata holdings and systems **
**Strengths **
**Shortcomings**
**Opportunities for improvement**
**Climate Database Modernization Program (CDMP)**
**Partnership with private industry**
**Increase digital data holdings**
**Improve database quality**
**Improve access to and utilization of data**

**Notes:**
Early in 2000, NCDC reviewed its metadata holdings and management systems.  We found both strengths and shortcomings, and identified opportunities for improvement.

At about the same time, NCDC began its Climate Database Modernization Program (CDMP).  This is a partnership with private industry. The basic goals are to increase our digital data holdings, improve the quality of our database and to improve access to and utilization of our data. CDMP provided a wonderful opportunity to address some of our metadata needs. 

## Reality Can Be Ugly

**Notes:**
When we examined how all the metadata for the various observing systems was managed and accessed, we came to a conclusion: 

Reality can be ugly!

This diagram looks a little like a squashed bug, but it actually documents some of our metadata systems and the data flows between them. 

## Initial Situation

**Metadata distributed in a combination of formal and ad hoc systems**
**Different systems may contain information about the same station**
**Multiple sources and procedures for the same metadata result in discrepancies**
**Data freshness and accuracy vary**
**Updates may not affect all similar data**

**Notes:**
Metadata is distributed around the center in a combination of formal and ad hoc systems.

Different systems may contain information about the same thing, often collected from different sources.  Multiple sources and procedures for the same data often result in discrepancies, as well as duplication of effort.

Data freshness and accuracy vary from system to system, and updates to one system may not affect all similar data. 

## The Problem of Scope

- The human mind is so complex and things are so tangled up with each other that, to explain a blade of straw, one would have to take to pieces an entire universe. . . . A definition is a sack of flour compressed into a thimble.
**Rémy de Gourmont** (1858–1915)

**Notes:**
A problem faced by all developers is scope:  where do you draw the line? Metadata is a network of interrelated nodes, and it’s tough to focus on only one or two and still have an accurate representation of the picture.  

We aren’t the first ones to notice this.  In reference to definitions, a 19th century French novelist said that
The human mind is so complex and things are so tangled up with each other that, to explain a blade of straw, one would have to take to pieces an entire universe. . . . A definition is a sack of flour compressed into a thimble.

Our long term goal was to provide for management and integrated access to all our metadata, but a scope that large seemed a recipe for guaranteed failure.  We decided our most pressing need was in the station history arena and focused our initial effort there. 

## Existing Station History Systems

**SHIPS**** – Station History Information Production System**
**Highly detailed, with heavy quality control**
**Coop, ASOS and some surface weather observing sites**
**UNIX on old Sun workstation**
**Empress database**
**Other ad hoc and project-specific systems**
**Database**
**Flat text files, word processing files**
**Many paper records**
**Some are essentially static**
**Access via lists, Cliserv and Web CliServ**

**Notes:**
Station history at NCDC is currently maintained in a variety of systems.

SHIPS – the Station History Information Production System -- is our most detailed and closely controlled system.  It contains Coop stations, as well as ASOS and some surface weather observation (SWO) sites..  It runs under UNIX on old Sun workstation, and is implemented using an Empress database

There are various other ad hoc and project-specific systems in use around the center.  Some are in database form, others are maintained using flat text files or a word processor.  Some metadata is still on paper.

Some of the project-specific systems are essentially static, getting few or no updates. 

## Shortcomings of SHIPS system

**Developed to meet Coop data ingest and publication needs**
**Database design not normalized**
**New networks may require structure changes**
**Lack of keys, data integrity checks**
**Cumbersome interface with limited queries**
**No “query only” option, no outside access **
**Ad hoc queries are complex**

**Notes:**
SHIPS is our most detailed and controlled system, but it isn’t without its flaws.

It was initially developed to support our cooperative data ingest and publication needs, and is centered around Coop stations.  The design is not well normalized, and adding stations from different observing systems can involve database structure and user interface changes.

The lack of normalization and integrity enforcement results in data duplication across tables and overlapping time series dates, and makes it easy to enter invalid data.

The old system had a cumbersome interface with limited query capabilities.  There was no easy way to set up “query only” users, and access was only available from inside firewall.  While interface limitations made frequent ad hoc queries necessary, database design made queries complex.

To get a "simple" list of all equipment installed a station at a given time you must query 11 columns in six separate tables, each requiring an outer join and a correlated subquery.  If you want descriptions, not just codes, add another 11 tables.  If you're a database layman this probably sounds cumbersome or complex; if you're a structured query language buff you know that it is. The old system had a cumbersome interface with limited query capabilities.  There was no easy way to set up “query only” users, and access was only available from inside firewall.

## Challenges

**Technical, cultural and logistical**
**Metadata conflicts and inconsistencies**
**Complex table key, versioning, attribution**
**Security and audit requirements**
**Informal knowledge base, imprecise terms**
**Loose system documentation**
**Geographically dispersed team**
**Resource competition**

## Metadata Project Goals

**Strategic architecture to manage metadata**
**Leverage CDMP project tasks and resources**
**Accommodate imperfect, real world metadata**
**Accept new information without modification**
**Flexible queries for dispersed users**
**Modular for multi-organization participation**
**Deliver useful releases within a year**

**Notes:**
The ultimate goal was to end up with a strategic architecture to effectively manage and expand our metadatabase.  This was a natural fit with the CDMP initiative to increase, improve and provide access to our data holdings.

This system needed to accept the imperfect, real world metadata we had, and give us the tools to find and fix problems.

We couldn’t afford to have system redevelopment be a biannual exercise, so the system had to accommodate new instances of information -- like a new station network -- without changes to database structure or user interface.

Access to metadata has been a real weakness for us, and the new system had to provide fast and extensive query capabilities to geographically dispersed users.

We also wanted a modular development approach so that other projects and organizational units could work on other pieces of the metadata puzzle, tieing into the work we'd done without duplicating it.

Lastly, because people are notoriously impatient with lengthy projects, we knew we’d need to make the system do something useful within a year to ensure continued support 

## Technological Foundation

**Normalized relational database**
**Oracle 8i database, CASE design tools**
**Model entire subject, not one instance**
**Surrogate keys minimize dependencies**
**Enforce business rules in database**
**Declarative, triggers, stored procedures**
**Accept flawed data, identify and correct**
**Separate database, application servers**

**Notes:**
To ensure data integrity, extensibility and flexible access we wanted to base the new system on a normalized relational database.  Oracle is becoming a standard at NCDC, as it is in many other organizations, and we selected that as our platform.

The system needed to handle a variety of station groups, so we developed the data model by considering many different station networks, not just Coop stations.  Tight timelines and limited resources forced us to restricted our model to fixed-position stations, but we think the design can readily be extended to accommodate stations like boats, buoys and satellites.

We used surrogate, non-information bearing keys for all tables to avoid basing our key structure on values we had no control over, like an ID assigned to a station by an outside organization.

We wanted our business rules enforced in the database.  A combination of declarative constraints, stored procedures and triggers help insure integrity of the database.  They also give us the flexibility to accept flawed data but later identify and correct it

Separate database and application servers improve performance and make the system more scaleable and secure. We used multi-processor Dell servers with lots of RAM and multiple RAID arrays to balance performance with cost. 

## Technological Foundation (Cont’d)

**Similar query needs for research and maintenance**
**Web-based solution**
**Distributed access**
**Easy administration, maintenance**
**Standard interface minimizes training**
**ColdFusion web-based environment**
**Crystal Reports for flexible output**

**Notes:**
Experience taught us that research & maintenance require similar query functionality, so we could meet both needs with one system.

We decided on a web-based interface to provide distributed access and easy administration.  The standard web interface, with its drop down menus, pick lists, buttons and hyperlinks, also minimizes training

The Coldfusion development environment lets us develop screens quickly, lets us reuse code and permits us to implement complex business logic both in the interface and with database-level stored procedures.

The convenience of the web did involve some trade-offs.  We give up some speed and interface power compared with traditional client/server approach.  There are the additional complexities of handling transactions in a sessionless environment, and the browser differences complicate coding. 

## Station History Subject Areas

**Identity**
**Names**
**IDs**
**Period of record**
**Location**
**Lat/Lon, elevation**
**Geographic descriptors**
**Exposure, topography**
**Classification**
**Observers**
**Equipment **

**Observing Practices**
**Phenomena**
**Schedule**
**Reporting protocols**
**Data programs**
**Administration**
**Supporting documents**
**Forms**
**Photos**

**Notes:**
Our initial, top-level design exercise considered metadata well beyond station history.  However, even limiting our intensive focus to station history was still too general.  To keep with our “manageable chunks” philosophy, we broke station history into a number of interrelated subject areas, each focusing on a specific aspect of a station.  There is some overlap between the topics, but they are useful from both a development and user interface organization standpoint.

The rough subject areas are :
Identity 
Location
Classification
Observers
Installed Equipment 
Observing Practices (what is observed, how often, using what equipment, and how is it reported)
Data program participation (cooperative summary of day, hourly precipitation and so on)
Administrative details
Supporting documents  
The initial supporting documents are images of paper forms, but will eventually include digital photos.  National Weather Service has discussed providing digital photos of Coop stations, and NCDC is currently using digital images to help document its new Climate Reference Network stations 

## Current Status

**Modeling and requirements workshops held**
**Contractors familiarized with subject**
**Hardware, development software installed**
**SHIPS ported to Oracle, Web-accessible**
**Anomalies being identified and corrected**
**First cut database design completed**
**Interface prototyping in progress**
**Testing NWS Coop metadata acquisition**

**Notes:**
So when does the data meet the display device?  Last summer I planned to have an initial release that I could demonstrate here today.  A combination of factors, mostly resource-related, caused us to revise our schedule and delay the initial release.  Despite this, there has been tangible progress in the project.
We held a series of formal and ad hoc data modeling workshops with a diverse group of participants to ensure we had end-user perspective in determining requirements.  This step is often omitted, and I believe it is critical for success.
The contractors became familiar with the subject matter -- no small task, given the complexity of station metadata!
The hardware and development software have been installed.
I ported the current station history database to Oracle and developed some basic Web-based query tools for NCDC personnel
Some of the data anomalies I referred to earlier are being resolved in preparation for data migration to the new system.
The initial database design for the new system has been completed, and user interface prototyping is in progress.  We’ve also defined some of the “behind the scenes” functionality required to replace existing systems.
A critical component of the new system is automation.  We are currently testing our data acquisition interface with the National Weather Service’s new Cooperative Station Service Accountability system. CSSA is being developed by NWS to help them gather and report higher quality Coop station information in more timely fashion, and we will benefit from this work. 

## 2001: A Metadata Odyssey

**Design document complete – 1****st**** Qtr**
**Physical DB design complete – 1****st**** Qtr**
**Initial release of new system – 2****nd**** Qtr**
**Automated Coop QC, ingest – 3****rd**** Qtr**
**Access to document images – 3****rd**** Qtr**
**Intensive manual QC, updates – 4****th**** Qtr**
**Merge Pre-1948, Pre-1890 coop– 4****th**** Qtr**

**Notes:**
Now, you knew you wouldn’t get through a whole conference in 2001 without some oblique reference to Kubrick’s film.  An odyssey can be defined as an extended adventurous trip, or an intellectual or spiritual quest.  I’m sure many of us have been involved in projects that leave us thinking "what a long, strange trip it's been".  To track our progress we’ve established some target milestones for this project for the coming year.

We plan to complete the formal design document and Physical database design in the 1st qtr
An initial release of the new system will occur in the 2nd qtr, with subsequent releases to follow throughout the year.
In the 3rd qtr we plan to automate our quality control and ingest of coop data from the National Weather Service’s CSSA system, as well as providing on-line access to form images.
We will use our new tools to in the 4th quarter to improve the quality and scope of the database, including bringing some of our “disembodied” station lists into the fold

I’d like to thank you for your interest in our project.  If anyone has questions, I’ll be happy to address them now.