# Presentation: 271682

## Comprehensive Large Array-data Stewardship System(CLASS)

- Project and Technical Overview
- December 11, 2003

## Agenda

- Project Background/Overview (Alexander Kidd)
- Technical Overview (Tino Cremidis)
- Issues and Open Discussion (Alexander Kidd)

## Project Background

## Agenda

- Project Background
- CLASS Team Organization
- Suitland team responsibilities
- Overview of the Archive, Access and Distribution System (AADS)
- Project Overview
- Project Organization
- Activities
- Documentation
- CMM Activities

## Background information (1of 2)From the Satellite Active Archive (SAA) to CLASS AADS

- The SAA started as demonstration project in 1994. The SAA is considered operational since 1995
- The SAA archives and distributes: Polar data and some products data. (Sea Surface Temperature, Coast Watch, etc)
- CLASS study - Phase 1, was tasked to survey existing NESDIS’ data archive and distribution systems and to define CLASS requirements
- The SAA was considered one of the most mature systems
- The Goes Active Archive project was initiated as an effort to provide access and distribution for GOES data and was based on the SAA application software

## Background Information (2 of 2)(From the SAA to CLASS)

- In  November 2001, the NESDIS CIO combined  these efforts into a single project
- The NESDIS CIO declared that the SAA will be the baseline for Data Archive, Access and Distribution component of CLASS
- The CLASS Project Management Team (CPMT) was formed to manage the project

## Functional Areas of SupportSuitland Team

- Project Team Management
- Systems Administration (Hardware/Systems Software)
- Application Software Development & Maintenance
- Systems Engineering/Integration
- Database Administration
- Central Application Software Repository Maintenance
- Quality Assurance (CMM)
- Documentation Control (Requirements, Software Library)
- Earned Value (All Teams)
- Operations (Data Monitoring/Tape Robotics/Help Desk)

## AADS Overview: Operation within CLASS

- AADS software will run on suite of UNIX processors at two sites: OSDPD (Suitland, MD) and NCDC (Asheville, NC) -  01/04
- The two sites will use one common database
- The sites will maintain duplicate archives
- The two sites will have identical versions of all software
- In normal operational mode, ingest and data distribution activities will be divided between the two sites
- Each site ingests certain types of data
- Each order is assigned to a site based on types of data in order and processing loads at the two sites
- In failover mode, each site will be capable of performing all functions.
- Each processor will host at least one runtime environment consisting of a root directory with subdirectories that contain all files and storage areas needed to perform CLASS/AADS functions:
- Executable modules, parameter files and work space
- Each process is assigned to run in one environment.

## CLASS/AADS Suitland, MD StatisticsAverage Last 12 Months

- Ingest - 40GB/Day
- Ingest – 4,000 Data Sets/Day
- Distribution (On-line & Subscriptions) – 3.3TB/Month
- Distribution (On-line & Subscriptions) - 265,000 Data Sets/Month

## Project Overview

## Project Organization

- Data Archive Board

- Richard Reynolds

- CLASS Project Management Team
- (CPMT)

- NGDC Development Teams
- (Boulder, CO)

- NCDC
- Development Team
- (Fairmont, WV)

- OSDPD Development Team
- (Suitland, MD)

- System Integration & Test Team
- (Suitland, MD)

- Alex Kidd and Tino Cremidis represent OSDPD on CPMT

- System Integration and Test, and OSDPD Operations, are managed and performed by OSDPD Team

- OSDPD Operations
- (Suitland, MD)

- NCDC
- Operations
- (Asheville, NC)

- Archive Requirements
- Council

## Development Activities

- Release development
- Release planning:  CCRs assigned for implementation
- Design and Implementation
- System Integration and Test
- Deployment Test phase at secondary site after System Test in Suitland and before promotion to operations
- Promotion to Operations
- Requirements definition and analysis for new campaigns
- Support for other development teams
- Process improvement

## Coordination Across Teams (1)

- CLASS Project Management Team (CPMT)
- Management oversight and coordination
- Risk management
- Project tracking
- Overall decision-making body for CLASS
- System Engineering Team (SET)
- Technical oversight and coordination
- Provides recommendations to CPMT on technical direction
- Provides technical input to CCB
- Software Engineering Process Group (SEPG)
- Process definition and improvement
- Provides recommendations to CPMT on process definition

## Coordination Across Teams (2)

- Configuration Control Board (CCB)
- Change review and control
- Includes CPMT members and SET members
- Chaired by Alex Kidd
- CLASS Operations Team (COT)
- Operations at Suitland and Asheville sites
- Includes operations support personnel, system administrators, database administrators and archive managers

## Coordination Across Teams – Tools (1)

- Collaboration and online library
- CLASS Project Development Team – repository for approved documents, weekly reports, etc.
- CLASS Project Management Team – working area for CPMT
- CLASS Process Management Team – working area for CCB and SEPG, repository for peer review records
- Other projects as needed
- Library.saa.noaa.gov – online design documentation
- CVS – source code control

## Coordination Across Teams – Tools (2)

- Remedy – CCR and PR tracking
- CCR – Configuration Change Request: request for change to operational system or requirements
- PR – Problem Report: problem identified in test environment during System Integration and Test
- DOORS – requirements management
- ComMIT – Earned Value Measurement

## Documentation: Process Baseline

- CLASS            Master Project Management Plan

- OSDPD            Activity Plan

- CLASS            Software Development Guide

- CLASS            Configuration Management Plan

- Other             Activity Plans

- Procedures

- CLASS            Quality Management Plan

## Documentation: Procedures (1)

- Project Management Procedures
- Risk Management
- Issues and Action Tracking
- Process Baseline Document Management
- Cost and Schedule Tracking
- Estimation Cycle
- Quality Management Procedures
- QMO Process Audits
- Corrective and Preventive Action
- Process Assurance Cycle
- QMO Product Review
- Peer Review
- Waiver Requests

## Documentation: Procedures (2)

- Configuration Management Procedures
- CCR
- Source Code Control
- Software Change Promotion
- Requirements Baseline Management
- Document Management
- Configuration Audits
- CI Identification
- Database Configuration Management

## CMM Activities

- Primary focus on CMM Level 2
- Project Planning
- Project Tracking and Oversight
- Requirements Management
- Configuration Management
- Quality Assurance
- Also addressing CMM Level 3
- Defined process baseline
- Practice SCE held August 19-21 for Levels 2 and 3
- Formal SCE held Sept. 22-26
- Result: CLASS Suitland team certified @ CMM Level 3

## CMM Activities: The five levels of software process maturity

**2. Repeatable**

**1. Initial**

**3. Defined**

**4. Managed**

- Disciplined Process

- Standard, Consistent Process

- Predictable Process

- Continuously Improving Process

- Unpredictable and poorly controlled

- Can repeat previously mastered tasks

- Process characterized, fairly well understood

- Process measured and controlled

- Focus on process improvement

**5.Optimizing**

- Project Management

- Integrated Engineering Process

- Product and Process Quality

- Managing Change

## Technical Overview

- Figure Here

## Agenda

- System Functions
- System Overview
- Dual Site Architecture
- Hardware Architecture

## System Overview

***System Overview***

**Ingest and **
**Store Data**

**Visualization**
**Data**

**Data Set**
**Inventory**

**Data**
**Caches**

**Orders**

**Maintain,**
**Monitor,**
**Control**

**Process**
**Orders**

**Access**
**Data**

**Visualize**
**Data**

**Interface**
**with **
**Users**

**Data Products**
**And Metadata**

**Data**
**Providers**

**Customers**

**CLASS**
**Operators**

**Archive**

**CLASS Internet/Intranet**

## System Functions (Ingest and Store Data)

- Ingest POES, GOES  and derived product data
- Create browse image data files for AVHRR and GVAR data sets; store on-line
- Create netCDF files from IPD product data; store on-line for visualization
- Store some files in permanent cache, others in temporary cache
- Archive all files in robotic system

## System Functions (Interface with Users)

- Log in, set up user profile
- Initiate catalog search
- View search results:  catalog data, browse images, dataset coverage maps
- Order data
- View order status
- Visualize and download product data netCDF files

## System Functions (Visualize Data)

- Invoked by the User Interface
- Interfaces with tool for creating visualization images
- Ferret for data analysis
- Browse Image Generator for AVHRR and GVAR images
- Coast Watch Image Generator for Coast Watch images
- Returns URL of file containing image to user interface

## System Functions (Access Data)

- Invoked by the User Interface and Order Processing
- Search catalog for data sets that meet user-specified criteria:
- Data type
- Time range
- Geographic coverage
- Other criteria appropriate for each data type
- Locate files on-line or retrieve files from robotic storage
- After ingest or retrieval, keep files on-line for several days for quick access

## System Functions (Process Orders)

- Fill orders submitted through user interface:
- Put ordered files in FTP area for users to pull or put on physical media
- Notify users when files are available
- Provide subscription order service:
- Place orders automatically for newly ingested data sets that meet subscription criteria
- Push files to subscribers or make files available for pull
- Provide bulk order service:
- Create orders for large amounts of data that cannot be ordered conveniently through user interface
- Retrieve files in time-ordered blocks and place in FTP area for user to pull or put on physical media
- Notify user when each block is available

## System Functions (Maintain, Monitor, Control )

- Maintenance
- Automatically create new log files each day
- Automatically create new FTP area directories as needed
- Automatically clean up temporary caches, work directories, FTP area, database tables
- Monitoring and Control
- Record all activity and error messages in standard log files
- Send error and warning messages from log files to operators via e-mail
- Provide tools and operator interface for monitoring and controlling functions
- View order status; cancel, restart, prioritize orders
- View ingest statistics; initiate re-ingest
- View ingest and order activities; stop and restart activities
- Modify allocation of processes to platforms
- Modify processing parameters

## System Overview (Subsystem Design - 1)

- Functions are performed by several subsystems that fall into 4 design categories:
- Data Storage and Distribution Subsystems:  *Ingest, Subscription and Bulk Order Generation, Data Recall, Delivery*
- Each subsystem consists of several independent processes
- Data are transferred between processes via the database
- A workflow engine, the Activity Controller, starts processes as needed, routes work items (data sets, orders, recall requests) to processes in predefined sequences, monitors processes and work item progress
-  Object oriented design implemented in C++
-  Servers: *Inventory, Visualization, Order *
- Each server is a resident process designed for quick response
- Requests and responses are transmitted via socket messages in XML format
- Object oriented design implemented in C++

## System Overview (Subsystem Design - 2)

- Functions are performed by several subsystems that fall into 4 design categories (Cont.)
- User Interface
- Java/XML-based web interface uses off-the-shelf components:
**Apache** – Web Server; **Cocoon** - publishing framework; ***Avalon/Excalibur*** - database connection pooling; ***Tomcat***** **- servlet engine; ***LogKit***** ** - message logging;  *Informix Java Database Connectivity (****JDBC****) Driver*
- Java servlets perform special functions
- Communication with Servers via socket messages in XML format
- Monitoring and Maintenance Tools:  *Log Monitor, Cache Cleanup, Work Space Cleanup, Independent Monitoring, Operator Interface*
- Stand-alone tools
- Various invocation modes:  resident (*Log Monitor*); run periodically by cron (*Independent Monitoring*);  run as needed by operators or Background Subsystems (*Cache Cleanup, Work Space Cleanup*).
-  Various implementations: C++ (*Cache Cleanup*), Perl (*Log Monitor, Work Space Cleanup, Independent Monitoring, Operator Interface*)

## System Overview (Ingest and Store Data)

## System Overview (Interface with Users, Visualize Data, Access Data, Order Process) 

## System Overview (Maintenance, Monitoring, Control)

## System Overview (Distributed Redundant Archive)

- Ingestprocess

- Operationalinventory

- Archiver

- Archiveinterchange

- Roboticstorage

- Supplier

- Supplier

- Supplier

- Supplier

- Supplier

- Supplier

- Ingestprocess

- Operationalinventory

- Archiver

- Archiveinterchange

- Roboticstorage

**Suitland**

**Asheville **

- Operationaldatastore

- Operationaldatastore

## System Overview (Hardware)

## Issues and Open Discussion

## Backup Slides