# Presentation: 489842

## Aura Science MeetingData Systems Working Group HIRDLS SIPS StatusOctober 1, 2007

**Vince Dean, Brendan Torpy, Greg Young**
**Univ. of Colorado, Boulder**
**Cheryl Craig**
**NCAR**

## Overview

**Status of processing—science and data users**
**Software versions**
**Delivery to AVDC and to DISC**
**Next steps**
**SIPS system—developers and data managers**
**Lessons learned**
**Support for multiple processor versions and experimental runs**
**File and processor versioning**

## Since September 2006

**Delivered v2.02.07 HIRDLS Level 2 product to Goddard DISC (314 days of interest)—our first public release.**
**Installed 106 versions of science processors in HIRDLS SIPS for experimental and production processing.**
**Processed 897 days of data (full mission) with v2.04.08 software.**
**Processing full mission with v2.04.09 software; 679 days completed so far.**
**Supported ESDIS evolution and mini-MOSS tests:**
**S4PA mini-MOSS – December 2006**
**TADS mini-MOSS – April 2007**
**GEOS-5 v1.0 mini-MOSS – September 2007**
**Hardware acquisitions:**
**Installed 32-processor SGI Itanium system**
**Replaced SIPS PC cluster with new rack-mount systems.**

## Obstruction

**An obstruction partially obscures the light path within the instrument.**
**Algorithms continue to be developed to correct for the effects of the obstruction.**
**This highly experimental software development has impact on:**
**Plans for processing and data release.**
**SIPS operations and software systems.**

## Recent Data Releases

**v2.04.08**
**Processed 350 selected days of interest.**
**Basis for several publications.**
**Bug found during QA of v2.04.08.**
**Review shows no impact on conclusions in publications, but warranted reprocessing.**
**v2.04.09**
**Fixes the bug.**
**Basis for other publications.**
**Plan to process entire mission: 679 days so far.**
**Plan to release to AVDC and DISC for public distribution in October.**
**DISC now supports arbitrary strings for version numbers, like**
**“****2.04.09”**

## Next Release

**Driven primarily by advances in algorithm development.**
**Will incorporate:**
**Refinements to correction for obstruction**
**Possible scientific enhancements**
**GEOS-5 1.0**

## GEOS-5

**Our recent processor versions use GEOS-5 v0.1 for temperature line-of-sight gradients:**
**2.04.08**
**2.04.09**
**GEOS-5 1.0 can have a significant impact on HIRDLS data product; this new release will necessitate a new version of HIRDLS products.**
**Will wrap reprocessing using GEOS-5.1 along with further data processing refinements**
**When GEOS-5 v0.1 stops production, we will discontinue daily processing with v2.04.09 until our next major software release.**

## ESDIS Evolution

**Significant, but bearable, impact on SIPS operations.**
**HIRDLS appreciates quick response of Goddard DISC as S4PA and our needs evolve.**
**Enhancements to Machine Request Interface will provide valuable tool for data reconciliation and routine operations.**

## SIPS System—Software and Operations

**Embrace change—embrace experimentation.**
**Supported by some fortunate design decisions.**
**Some examples:**
**Issues—conflicting forces**
**Resolution**
**Consequences**

## Processing History

- Processor

- Processing history is a complex graph of data files and processing steps.
- Metadata is preserved from every run.
- Every product is traceable back to raw inputs.
- Supports experimentation, but complex to manage.

- Processor

- Input Files

- Output Files

- Processor

## Experiments in a Production System

**Forces:**
**Many experimental processor versions**
**Manual bookkeeping strained by complexity**
**Resolution:**
**Embrace the complexity**
**Accept the many experimental processors into the production system.**
**Run experiments in production system**
**Consequences:**
**Reliable audit trail.**
**Efficient bulk re-processing.**
**Version tracking is more complex. **
**More complex work for SIPS operator.**

## Specifying Processing Tasks

**Forces:**
**We frequently run tests where only a few processors have changed.**
**We don’t want to rerun previous steps.**
**It is inconvenient to customize the processing specification according to the state of processing on each day.**
**Resolution:**
**Operator specifies desired outcome—required steps.**
**System identifies which steps have already been run and does not repeat them.**
**Like a Unix “make file”.**
**Make it so!**

## Sharing Experimental Versions

**Forces:**
**HIRDLS SIPS experimental processing history is complex**
**Users, especially external users, need easy access to the right data**
**Risk of using wrong data files**
**Prefer a simple, hierarchical file structure**
**Resolution:**
**HIRDLS “data pool”**
**Web server with selected files in a simple hierarchy.**
**External program periodically copies selected data products to the web server. **
**Creates a directory for each data version**
**External users download new products automatically with ****wget.**

## Identifying File Versions

**Forces:**
**We release one component at a time.**
**Want to have unique, informative labels for all products.**
**Want to reuse old processing products in new sequences.**
**Work in progress ...**

- L1X
- 7.4.1

- L1C
- 4.6.1

- L1C
- 4.5.1

- HIRDLS1
- Data
- Product

- Reusing this file saves time
- and makes experiments faster

- Which version is this?
- A, B or both?

- Version B

- Version A

## Monitoring Many Systems

**Forces**
**System has many components**
**Failures may not be immediately obvious**
**Each has a different mechanism for reporting problems**
**Email notifications can be overwhelming**
**Resolution**
**External monitoring tool **
**Nagios:  **_**[www.nagios.org](http://www.nagios.org/)**_
**Email notification and web interface**
**Consequences**
**State of system is readily visible**
**We are aware of problems before the users are**
**Some outages cause multiple notifications**

## Slide 16