# Presentation: 490085

## Scientific Data StewardshipPilot Studies – Engineering and Science

- John Bates, NOAA
- Chief, Remote Sensing Applications Division
- NCDC, NESDIS

## SDS – Climate data records, data sets, data provenance

## The Future CLASS

- SDS Interface

- SDS
- Interface

## SDS Reprocessing

*** ******Facility at NCDC***

- Polarjet
- 2.5GHz
- 440GB RAID (38% used)

- Haze
- 4x2.8GHz
- 150 GB RAID (0% used)

- Suomi
- 2.5GHz
- 660GB RAID (20% used)

- New Linux Box 1
- 2x3.1GHz
- 2048GB RAID

- New Linux Box 2
- 2x3.1GHz
- 2048GB RAID

- New Linux Box 3
- 2x3.1GHz
- 2048GB RAID

- New Linux Cluster
- 2x3.1GHz
- 2x3.1GHz
- 2x3.1GHz

- Network Switch

- DataServer
- 2x3.1GHz

- RAID
- 2048 GB
- 2048 GB
- 2048 GB
- 2048 GB

- ~20 Desktop
- PCs

- Server

- Firewall

- Router

- NCDC FTP Servers

- NCDC Web Servers

- C3P0

- IG88

- R2D2

- OU812

- Rime
- Access to HDSS

- Drizzle
- Access to B2 Data

- 1Gb/s

- T-3 Suitland CLASS

- T-3 BellSouth Internet

- T-3 UNCA/Abilene

- T-1 Suitland #1

- T-1 Suitland #2

- 1Gb/s

- New RSAD Linux Computers

- Existing RSAD
- Computers

- NCDC LAN
- 100 Mb/s

- Tape Robot Systems

- Server
- Fast access to HDSS

## Scientific Data Stewardship Reprocessing Facility at NCDC

- Need to scope data rates, CPU cycles, software engineering, depositing reprocessed climate data records into CLASS
- This requires a set of pilot studies both at NCDC and between ORA/CLASS Suitland
- Pilot study results will be incorporated into concept of operations for NOAA’s climate data records

## Pilot Study using Sampled Geo and Leo data - ISCCP-B1 Processing

- sites ...

- ISCCP
- B1
- Data

- ISCCP
- B1
- Access

- NCDC

- GEWEX

- Other
- Collaborators

- Products ...

- Albedo

- Surface Radiation
- Budget Parameters

- Reprocessed
- Cloud
- Climatology

- Other Parameters

- _NCDC will archive_
*Data products*
*Algorithms*
*Metadata*
*...*

- Current ability
- Developing capability

- Sampled
- data ...

- B1 Metadata

- 1.2Tb
*61 Gb/yr*

- IDL & Fortran
-   Read,
-   navigation and
-   calibration
-        routines
**Data Characterization**

- New data quality and updated calibration information

- GPCP, GACP, ...

- e.g., from ISCCP

- Example of internal processing

- Example of external processing

- Government (e.g., NASA/Langley),
- Universities, etc.

- RSAD Research

## ISCCP B1 Data

- Total Volume ~1.2 TB (reduced from full res.)
- Temporal Data Coverage
- 1983 to present
- Spatial Data Coverage
- Global
- Satellites
- GOES, GMS, Meteosat, Insat (B2 Data)
- NOAA afternoon and morning polar orbiters (B2 data)

## Current data rates

- 					Full Resolution	ISCCP B1
- Europe/Africa	Meteosat-8	XX GB/day	72? MB/day
- 			Meteosat-7	     XX		36
- Asia		Meteosat-5	XX		36
- Western Pac.	GOES-9	14		72
- Eastern Pac.	GOES-10	14		72
- Americas	GOES-12	14		72
- Total Global Rate per day	XX		288MB/day
- Total Global Rate per year			102Gb/yr
- ISCCP is reduced temporal (3hr) and spatial (10km) resolution
- All values are approximate

## ISCCP Data Access

- Data Access
- Decoders
- 18 satellites from 5 Satellite Processing Centers (SPCs) in 9 different formats
- Navigation
- ~ 5 navigation routines needed
- Calibration
- ISCCP collected a special dataset designed for satellite calibration
- The result: BT calibration data (which is specific for each image)
- Data Characterization
- Data Quality Assurance tests will quantify noise levels, calibration, satellite movements, etc.

## Data Characterization

- Apply a set of data analysis tools to determine data characteristics.
- For example:
- Means and standard deviations on varying temporal and spatial scales
- Hovmoller diagrams
- First and second harmonics of annual cycle
- Empirical Orthogonal Function Analysis
- One-point Correlation Diagrams and Lag Correlation
- Frequency Histogram
- Power Spectra
- Cross Correlation and Cross Spectra
- Wavelet Analysis
- Wavenumber-Frequency

## B1 Metadata

- Quantify information about the data that will facilitate reprocessing ...
- Missing lines
- Data gaps
- Noise levels
- Satellite subpoints (and changes therein)
- Calibration
- Navigation
- Available channels
- Etc.

## Generic R.T.

- Other a priori data

- Understanding
- Satellite
- Data

- Process studies
- of climate
- applications

- Improved
- Climate
- Prediction

- Observed
- Geophysical
- Variables

- Observed
- Radiances

- Instrument error characteristics

- Spectral response functions

- Simulated
- Radiances

- Geophysical
- retrievals

- Inverse Model
- F-1(I) = q, T, ...

- Forward Model
- F(q, T, ...) = I

- Forecast
- First Guess

- Radiance bias
- Corrections

- Satellite
- Observed
- Radiances

- Geophysical
- Variables

*Inverse Problem*

*Forward Problem*

## Albedo Retrieval

- NCEP Gaseous
- Absorber Amounts (H2O,O3)

- Understanding
- Satellite
- Data

- Process studies
- of climate
- applications

- Improved
- Climate
- Prediction

- Observed
- Albedo

- Observed
- Radiances

- Instrument error characteristics

- Spectral response functions

- Simulated
- Radiances

- Albedo
- retrievals

- LUT retrieval
- F-1(I) = 

- SBDART
- F(, U(H2O), ...) = I

- ISCCP Calibration Information

- B1 Visible & IR
- Radiances

- Geophysical
- Variables

*Inverse Problem*

*Forward Problem*

- Generic RT

- Look Up
- Table

## Possible Collaborators

- When the B1 dataset is available, many studies will be able to make use of the advantages of this dataset:
- Global observations
- 3 hour temporal resolution
- VIS and IR observations (and sometimes more)
- Long term observations (1983-present)
- Also this work can be extended to earlier satellite observations. For example:
- 1979-1983 TIROS-N, NOAA-6, SMS, GOES
- Early Meteosats
- Possible collaborators include:
- Other government institutions
- Academia

## Satellite Coverage

-  *Missing NOAA-16/17*
* **Need to Color code GEOs by location*

## Polar Imager and Sounder 1b Data Rates (per satellite-day)

| Sounder | Full
Resolution | Sampled
Resolution | Imager | Full
Resolution | Sampled
Resolution |
| --- | --- | --- | --- | --- | --- |
| VTPR | 7 MB |  | VHRR | 800 MB |  |
| TOVS | 70 MB |  | AVHRR | 1 GB |  |
| ATOVS | 300 MB |  | AVHRR | 1 GB |  |
| AIRS | 30 GB |  | MODIS | 600 GB |  |
| IASI | 50 GB |  | AVHRR | 1 GB |  |
| CrIS | 50 GB |  | VIIRS | 600 GB |  |

## Polar Imager and Sounder History

## Inverse Problem

- Other a priori data

- Understanding
- HIRS
- Data

- studies of Atmospheric
- Temperature and
- Humidity Variability

- QC/QA Analysis

- Observed
- T(p), q(p), ...

- Observed
- HIRS Channel
- Radiances

- Instrument error characteristics

- HIRS Channel spectra

- Simulated Radiances
- NN Training

- Retrievals of
- T(p), q(p), ...

- Inverse Model
- Neural network
- F-1(I) = q, T, ...

- Forward Model
- MODTRAN
- F(q, T, ...) = I

- HIRS
- Radiance Bias
- Corrections

- HIRS l1b

- Sampled ERA40 T(p), q(p), and ozone

*Inverse Problem*

*Forward Problem*

- HIRS Processing
- (also applies to MSU/AMSU)

- Other Retrieval
- methods

- Cloud Cleared
- Limb corrected
- Calibrated

- Archived with Corresponding Metadata

## WSR88D

- NCDC

**Level II**
- _[(~70 TB/year)](slide4.xml)_

**Level III**
- _[(~1.5 TB/year)](slide4.xml)_

**QPE**

- QC/QA

- QC/QA

**Radar Rainfall**
**Climatology**

- Data Resolution

- Processing Time

## QC/QA

- Suppression

- Bright Band
- Contamination

- AP Identification

- Advection Effect
- Correction

- Range Effect
- Correction

- Processing Time/
- Complexity

- _[QC/QA](slide1.xml)_

## Radar-Gauge 

- Reflectivity Estimate

- Inverse Model
- Z=ARb

- In Situ
- Data

- Parameter Estimation
- Radar-Gauge

- Regridding
- Polar to Cartesian

- Temporal Integration
- 5-min, 15-min,Hourly

- _[Quantitative Precipitation Estimation](slide1.xml)_

- Supplemental Information
- Satellite Rainfall Estimates

## Radar Rainfall Climatology

- Mosaic

- _[Radar Rainfall Climatology](slide1.xml)_

- Large-Scale
- Long-Term
- Accumulations

- Regional
- Accumulations

## Level II

- Full Volume Scan Data
- Variables – Reflectivity, Radial Velocity, Spectral Width
- 4–19 Elevation Angles
- 5, 6, 10 Minute Resolution
- 1o x 1km Spatial Resolution
- Radar Reflectivity           0.5 dBZ precision

- WSR-88D Site Processing
- 20 + Products -Reflectivity and Precipitation Products (QPE)
- 4 Elevation Angles
- 5,6,10 Minute Resolution
- 1o x 1km, 2 x 2km, 4 x 4km Spatial Resolution
- Data Resolution Degraded (4 or 5 dBZ precision)

- _[Level III](slide1.xml)_

## Level II 500 GB/radar/year
Level III 10 GB/radar/year
140 + Radars in CONUS
Possible - 70 TB/year Level II
Possible - 1.5 TB/year level III

- Level II 500 GB/radar/year
- Level III 10 GB/radar/year
- 140 + Radars in CONUS
- _[Possible - 70 TB/year Level II](slide26.xml)_
- _[Possible - 1.5 TB/year level III](slide26.xml)_

- _[Data Size](slide1.xml)_

## QC/QA

- Quality Issue

- Reason

- Algorithm  -  Complexity

- Ground
- Return

- AP/Clutter

- Large Reflectivity
- Value

- Under
- Estimation

- Bright Band

- Radar Sampling

- Threshold

- Hail

- Neural
- Network

- Advection
- Effect
- Correction

- Vertical
- Profile
- Adjustment

- Image
- Processing

## NEXRAD Data Volume (NCDC)