# Presentation: 427977

## ADPS Science Software Development

**Bryan Franz**
- NASA Ocean Biology Processing Group

- Aquarius Data Processing Workshop, NASA/GSFC, 21-22 March 2007

## aql0gen 	- L0 generation from downlink
aql1agen 	- L0 to per orbit L1A 
aql1amerge 	- L1A duplicate orbit reduction

aql1bgen 	- radiometer L1A to L1B (TA)
aql2gen 	- radiometer L1B to L2 (SSS)

aql1bscat 	- scatterometer L1A to L1B (TA)
aql2scat 	- scatterometer L1B to L2 (SSR)

aql2bin 	- daily space binning
aql3bin 	- temporal averaging
aql3map 	- projected images

- aql0gen 	- L0 generation from downlink
- aql1agen 	- L0 to per orbit L1A
- aql1amerge 	- L1A duplicate orbit reduction
- aql1bgen 	- radiometer L1A to L1B (TA)
- aql2gen 	- radiometer L1B to L2 (SSS)
- aql1bscat 	- scatterometer L1A to L1B (TA)
- aql2scat 	- scatterometer L1B to L2 (SSR)
- aql2bin 	- daily space binning
- aql3bin 	- temporal averaging
- aql3map 	- projected images

## Science Processing Flow

- L1A File_x0003_

- Ancillary Data

**aql1bgen**

- Radiometer L1B File

- Radiometer L2 File

- Level-3 Bin File

**aql2gen**

**aql2bin**

- One Orbit

## Science Processing Flow Cont.

## aql0gen 	- L0 generation from downlink
aql1agen 	- L0 to per orbit L1A 
aql1amerge 	- L1A duplicate orbit reduction

aql1bgen 	- radiometer L1A to L1B (TA)

aql2gen 	- radiometer L1B to L2 (SSS)

aql1bscat 	- scatterometer L1A to L1B (TA)
aql2scat 	- scatterometer L1B to L2 (SSR)

aql2bin 	- daily space binning
aql3bin 	- temporal averaging
aql3map 	- projected images

- aql0gen 	- L0 generation from downlink
- aql1agen 	- L0 to per orbit L1A
- aql1amerge 	- L1A duplicate orbit reduction
- aql1bgen 	- radiometer L1A to L1B (TA)
- aql2gen 	- radiometer L1B to L2 (SSS)
- aql1bscat 	- scatterometer L1A to L1B (TA)
- aql2scat 	- scatterometer L1B to L2 (SSR)
- aql2bin 	- daily space binning
- aql3bin 	- temporal averaging
- aql3map 	- projected images

- }

- }

- }

**}**

**}**

- ADPS Team

- Wentz

- Yueh

- Lagerloef, Lilly, ADPS Team

- Levine, Wentz

## Science Software Integration

- working code (C, Fortran)

- published algorithms

- in-house development

- written specifications (memos, emails)

- prototype code (IDL, MATLAB?)

- Science Software Development and Algorithm Integration

- Aquarius Team

- concepts on a napkin

- algorithms

- issues and test results

- ADPS Team

## SUBVERSION configuration control system

trunk   (stable development version)
src
aql1agen
aql1amerge
etc.
bin
lib
data    (static data and tables)
test    (standard test files)

tags    (snapshots of trunk, tested and approved for production)
version 1
version 2
etc.

- SUBVERSION configuration control system
- trunk   (*stable development version*)
- src
- aql1agen
- aql1amerge
- etc.
- bin
- lib
- data    (*static data and tables*)
- test    (*standard test files*)
- tags    (*snapshots of trunk, tested and approved for production*)
- version 1
- version 2
- etc.

- the development tree can be made directly accessible to the Science Team or others
- e.g., the ocean color software is exported nightly to public ftp

## ADPS

- Science Software Integration

- working code (C, Fortran)

- published algorithms

- in-house development

- written specifications (memos, emails)

- prototype code (IDL, MATLAB?)

- Science Software Development and Algorithm Integration

- Aquarius Team

- concepts on a napkin

- algorithms

- issues and test results

- Software Repository
- Version Controlled Code and Tables

- software  updates

- production  code

- ADPS Team

## Processing inputs and controls are best handled through a parameter file with a list of parameter=value pairs, e.g.: 

- Common Software Attributes

*% aql2gen par=**parameter-file*
*parameter-file:*
*ifile=**l1b-filename*
*ofile=**l2-filename*
*proc_opt=**3*
*etc.*

- the level-2 code for ocean color and SST supports over 100 parameters to specify various ancillary input files, alternative algorithm switches, flag thresholds, and the list of **products to be included in output**.

## Processing inputs and controls are best handled through a parameter file with a list of parameter=value pairs, e.g.: 

Default values (standard production settings) for each input parameter are defined within the static data tables.

Programs easily executed from unix commandline
no elaborate environments  or GUIs required

Given no inputs, programs provide version information and usage
l3bin 2.17 (Feb 16 2007 23:00:05)

Programs return exit status (e.g., 0=success, 1=failure)

- Default values (standard production settings) for each input parameter are defined within the static data tables*.*
- Programs easily executed from unix commandline
- no elaborate environments  or GUIs required
- Given no inputs, programs provide version information and usage
*l3bin 2.17 (Feb 16 2007 23:00:05)*
- Programs return exit status (e.g., 0=success, 1=failure)

- Common Software Attributes

*% aql2gen par=**parameter-file*
*parameter-file:*
*ifile=**l1b-filename*
*ofile=**l2-filename*
*proc_opt=**3*
*etc.*

- the level-2 code for ocean color and SST supports over 100 parameters to specify various ancillary input files, alternative algorithm switches, flag thresholds, and the list of **products to be included in output**.

## The ADPS Team will work closely with the Aquarius algorithm developers to implement all processing codes into the production system.

This will include support for testing changes at the granule (orbit) level as well as facilitating global, life-of-mission analyses prior to adoption.

It is anticipated that the ADPS team will develop the driver code for some or all processing levels (e.g., aql2gen)
the driver will read the input files, pass arguments to the processing function(s), and write the output files in HDF5 format

It is also understood that the ADPS Team will have primary development responsibilities for L0 through L1A as well as implementing L3 binning & mapping (with detailed requirement input from the Science Team).

Finally, it is expected that the ADPS team will develop procedures to acquire, archive, and interface with ancillary data sources (sst, winds, etc.)

- The ADPS Team will work closely with the Aquarius algorithm developers to implement all processing codes into the production system.
- This will include support for testing changes at the granule (orbit) level as well as facilitating global, life-of-mission analyses prior to adoption.
- It is anticipated that the ADPS team will develop the driver code for some or all processing levels (e.g., aql2gen)
- the driver will read the input files, pass arguments to the processing function(s), and write the output files in HDF5 format
- It is also understood that the ADPS Team will have primary development responsibilities for L0 through L1A as well as implementing L3 binning & mapping (with detailed requirement input from the Science Team).
- Finally, it is expected that the ADPS team will develop procedures to acquire, archive, and interface with ancillary data sources (sst, winds, etc.)

## Level-2 Format and Processing Options

**Bryan Franz**
- NASA Ocean Biology Processing Group

- Aquarius Data Processing Workshop, NASA/GSFC, 21-22 March 2007

## Separate files for radiometer and scatterometer.

Formatted using HDF5

Product-level metadata similar to Level-1A, Level-1B.

Temporal resolution per subcycle (120 or 180ms) or common multiple (360, 720, 1440ms).

Geolocation (lon/lat) per time and per beam. 

Retrieved geophysical parameters (SSS or SSR) and associated quality flags (l2_flags), per time per beam.

Additional products TBD.

- Separate files for radiometer and scatterometer.
- Formatted using HDF5
- Product-level metadata similar to Level-1A, Level-1B.
- Temporal resolution per subcycle (120 or 180ms) or common multiple (360, 720, 1440ms).
- Geolocation (lon/lat) per time and per beam.
- Retrieved geophysical parameters (SSS or SSR) and associated quality flags (l2_flags), per time per beam.
- Additional products TBD.

## code and formats (HDF5) can be developed to allow for run-time specification of the output content. 

aql2gen l2prod=“SSS,l2_flags”

- code and formats (HDF5) can be developed to allow for run-time specification of the output content.
- aql2gen l2prod=“SSS,l2_flags”

- software would then have the ability to produce alternate product suites for specialized studies, diagnostic analyses, and algorithm development.
- aql2gen l2prod=“SSS,TA_meas,TA_space,TA_earth,l2_flags”

- consider defining a minimal set of derived products to be included in the “standard” Level 2 file (e.g., SSS, l2_flags).
- then, consider what additional products may be useful as optional output  (e.g., brightness temperatures, correction terms, co-located ancillary, intermediate computations).
- code should be developed to allow the standard product suite and optional product list to grow and evolve.

## flags are additional bits of information carried along with each measurement that may be used at a later stage for masking.

masks are conditions for which particular measurements are excluded from subsequent processing (e.g., flag i was set).

masking is generally applied during spatial or temporal aggregation (e.g., suspect retrievals flagged at L2 and masked at L3).

- flags are additional bits of information carried along with each measurement that may be used at a later stage for masking.
- masks are conditions for which particular measurements are excluded from subsequent processing (e.g., flag i was set).
- masking is generally applied during spatial or temporal aggregation (e.g., suspect retrievals flagged at L2 and masked at L3).

## Possible Radiometer L2 Flags and Masking Scenario16-bit integer per L2 sample (e.g., per horn, per sub-cycle)

| Bit | Name | Description | L2 Mask | L3 Mask |
| --- | --- | --- | --- | --- |
| 1 | TAFAIL | Ant. Temperature Not Available |  |  |
| 2 | TAWARN | Ant. Temperature Suspect |  |  |
| 3 | RFIHI | RFI Contamination High |  |  |
| 4 | RFIMD | RFI Contamination Moderate |  |  |
| 5 | RFILO | RFI Contamination Low |  |  |
| 6 | LANDHI | Land Contamination High |  |  |
| 7 | LANDMD | Land Contamination Moderate |  |  |
| 8 | LANDLO | Land Contamination Low |  |  |
| 9 | RAINHI | Rain Contamination High |  |  |
| 10 | RAINMD | Rain Contamination Moderate |  |  |
| 11 | RAINLO | Rain Contamination Low |  |  |
| 12 | WINDHI | High Wind Speeds |  |  |
| 13 | SFLARE | Solar Flare Predicted |  |  |
| 14 |  |  |  |  |
| 15 |  |  |  |  |
| 16 |  |  |  |  |

## flags are additional bits of information carried along with each measurement that may be used at a later stage for masking.

masks are conditions for which particular measurements are excluded from subsequent processing (e.g., flag i was set).

masking is generally applied during spatial or temporal aggregation (e.g., suspect retrievals flagged at L2 and masked at L3).

aggregation and associated masking is best applied within a new processing level 
e.g., any temporal averaging between L1B and L2 is done in aql2gen
allows same input file to be processed with multiple rejection criteria
facilitates diagnostic analysis of retrieved geophysical parameters and associated quality flags via standard output files

- flags are additional bits of information carried along with each measurement that may be used at a later stage for masking.
- masks are conditions for which particular measurements are excluded from subsequent processing (e.g., flag i was set).
- masking is generally applied during spatial or temporal aggregation (e.g., suspect retrievals flagged at L2 and masked at L3).
- aggregation and associated masking is best applied within a new processing level
- e.g., any temporal averaging between L1B and L2 is done in aql2gen
- allows same input file to be processed with multiple rejection criteria
- facilitates diagnostic analysis of retrieved geophysical parameters and associated quality flags via standard output files

## Back-up Slides

## Science Processing Flow

- L1A File_x0003_

- Ancillary Data

- Radiometer L1B File

- Radiometer L2 File

- Level-3 Bin File

**aqrl2gen**

**aql2bin**

- L2  = 5.76 sec

- L1B = 0.10 sec

## L2GEN

- read parameters

- write L1B

- write L2

- compute L1B

- read L1A

- compute L2

**if (want L1B)**

**if (want L2)**

**endif**

**endif**

- read ancillary

- per radiometer step or
- scatterometer subcycle

- per frame (5.76 sec)

## reading entire orbit into memory?

binning swath to monthly or daily to monthly?

faraday rotation: radiometer feeding scatterometer processing?

- reading entire orbit into memory?
- binning swath to monthly or daily to monthly?
- faraday rotation: radiometer feeding scatterometer processing?