![](media/image1.pct){width="1.2in" height="0.99375in"}

Verification and Validation of NASA-Supported Enhancements to PECAD's
Decision Support Tools

Kenton Ross, Science Systems and Applications, Inc.

Rodney McKellip, NASA

John C. Stennis Space Center, Mississippi

Acknowledgments

This report has been authored by Science Systems and Applications, Inc.,
under Task Order NNS04AB54T with the National Aeronautics and Space
Administration.

The authors appreciate the contributions of Roxzana Moore, Science
Systems and Applications, Inc., and

Debbie Fendley, Computer Sciences Corporation, to the preparation of
this report.

# Table of Contents {#table-of-contents .Heading-NoTOC}

[Executive Summary v](#executive-summary)

[1.0 Introduction 1](#introduction)

[1.1 Systems Engineering Considerations
1](#systems-engineering-considerations)

[2.0 V&V of Application-Tailored MODIS Products
3](#vv-of-application-tailored-modis-products)

[2.1 Calibration and Validation of MODIS Sensor and Standard Products
4](#calibration-and-validation-of-modis-sensor-and-standard-products)

[2.2 Characterization of the MODIS Application-Tailored Products
5](#characterization-of-the-modis-application-tailored-products)

[2.2.1 MODIS Product Accuracy 5](#modis-product-accuracy)

[2.2.2 MODIS Product Delivery 6](#modis-product-delivery)

[2.2.3 MODIS Product Coverage 8](#modis-product-coverage)

[3.0 V&V of Global Reservoir and Lake Monitor Products
9](#vv-of-global-reservoir-and-lake-monitor-products)

[3.1 Calibration and Validation of TOPEX/Poseidon and Jason-1 Radar
Altimeters
9](#calibration-and-validation-of-topexposeidon-and-jason-1-radar-altimeters)

[3.2 Characterization of the Global Reservoir and Lake Monitor
11](#characterization-of-the-global-reservoir-and-lake-monitor)

[3.2.1 GRLM Accuracy 11](#grlm-accuracy)

[3.2.2 GRLM Delivery 13](#grlm-delivery)

[3.2.3 GRLM Coverage 15](#grlm-coverage)

[4.0 Conclusions and Recommendations
15](#conclusions-and-recommendations)

[5.0 References 16](#references)

[6.0 Acronyms 18](#acronyms)

# Tables {#tables .Heading-NoTOC}

[Table 1. PECAD accuracy, delivery, and coverage requirements for
near-real-time imaging. 3](#__RefHeading___Toc124078776)

[Table 2. PECAD accuracy, delivery, and coverage requirements for an
NDVI database. 4](#__RefHeading___Toc124078777)

[Table 3. PECAD accuracy, delivery, and coverage requirements for
surface water level variation. 9](#__RefHeading___Toc124078778)

[Table 4. Total error budget for lake height.
11](#__RefHeading___Toc124078779)

[Table 5. Verified GRLM performance over select North American surface
water bodies.\* 12](#__RefHeading___Toc124078780)

# Figures {#figures .Heading-NoTOC}

[Figure 1. Systems engineering approach.
2](#__RefHeading___Toc124078781)

[Figure 2. Vegetation index time series for the Maize Triangle region of
South Africa and for the Victoria wheat-growing region of eastern
Australia. 5](#__RefHeading___Toc124078782)

[Figure 3. July 2005 data interruption as reported on the MODIS Rapid
Response System Status page. 6](#__RefHeading___Toc124078783)

[Figure 4. Distribution of Crop Explorer MODIS Image Gallery delivery
time lag from October 1, 2005, through October 15, 2005.
7](#__RefHeading___Toc124078784)

[Figure 5. Variation of Crop Explorer MODIS Image Gallery delivery time
lag by time of day (October 2005 observation period).
7](#__RefHeading___Toc124078785)

[Figure 6. Geographic distribution of Crop Explorer MODIS Image Gallery
delivery and lag times for Terra and Aqua (October 2005 observation
period). 8](#__RefHeading___Toc124078786)

[Figure 7. Comparison of Crop Explorer MODIS Image Gallery and MODIS
NDVI Database geographical footprints as of October 2005. The footprints
are overlaid on a global map of croplands as estimated in the 2001 MODIS
Land Cover product (MOD12). 9](#__RefHeading___Toc124078787)

[Figure 8. Gauge stations in the general vicinity of the Great Lakes.
13](#__RefHeading___Toc124078788)

[Figure 9. Histogram of time to last estimated water level checked at
the end of May 2004 and again at the end of May 2005.
14](#__RefHeading___Toc124078789)

[Figure 10. Geographic distribution of time to last estimated water
level at the end of May 2005. 15](#__RefHeading___Toc124078790)

# Executive Summary {#executive-summary .Heading-TOC}

The NASA Applied Sciences Directorate, part of the Earth-Sun System
Division of NASA's Science Mission Directorate, has partnered with the
U.S. Department of Agriculture (USDA) to enhance decision support in the
area of agricultural efficiency---an application of national importance.
Key components of national agricultural decision making are to
understand the global context of crop conditions, to understand how
those conditions affect food security, and to predict global crop
production of key commodities. The Production Estimates and Crop
Assessment Division (PECAD) of the USDA Foreign Agricultural Service
(FAS) accepted the mandate to support national decision making with
timely global crop intelligence. PECAD accomplishes its mission with a
proven set of decision support tools that facilitates both its own
internal analyses and its dissemination of relevant information to
policy makers and other stakeholders.

This Verification and Validation (V&V) report is part of series that
tracks the NASA/PECAD partnership through the systems engineering
process. An ***Evaluation Report*** for PECAD was published in 2004 to
document the decision support environment and to identify relevant NASA
science assets that might improve the PECAD decision system. A
***Benchmarking Report*** was produced concurrent with this V&V report
to measure the value of the initial NASA products that have been
integrated into the PECAD system. This V&V report summarizes the
***Verification and Validation*** of the same enhancements covered in
the Benchmarking Report:

-   Near-real-time Moderate Resolution Imaging Spectroradiometer (MODIS)
    products through PECAD's [MODIS Image Gallery]{.underline}

-   MODIS Normalized Difference Vegetation Index (NDVI) time series data
    through the [USDA-FAS MODIS NDVI Database]{.underline}

-   Jason-1 and TOPEX/Poseidon lake level estimates through PECAD's
    [Global Reservoir and Lake Monitor]{.underline} (GRLM)

Where possible, each enhanced product was characterized in terms of
accuracy, timeliness, and coverage, and the characterized performance
was compared to PECAD operational requirements (*the operational
requirements do not necessarily match performance requirements for
projects as funded*).

The accuracy of the MODIS Image Gallery NDVI product is being studied,
but results are not yet available. The timeliness of the MODIS Image
Gallery as measured by product latency was estimated to be 4.4 hours.
Availability of MODIS Image Gallery products within the PECAD
operational requirement of 6 hours was measured at 87 percent. In terms
of coverage, the MODIS Image Gallery includes the great majority of crop
production areas of economic importance but still lacks coverage in many
areas where food security is a concern.

PECAD's requires that MODIS NDVI time series be consistent with the
legacy of the Advanced Very High Resolution Radiometer and SPOT
VEGETATION measurements. Studies indicate that such consistency has been
achieved, and ongoing work shows good agreement among products as well.
The latency of the USDA-FAS MODIS NDVI Database appears to be several
days longer than the 72-hour PECAD requirement. USDA-FAS MODIS NDVI
Database coverage of crop production areas of economic importance was
all but exhaustive.

The surface water level relative accuracy for the GRLM was estimated to
be 5--7 cm over the Great Lakes, exceeding the 10‑cm requirement
established by PECAD. Some more challenging lakes (smaller lakes,
reservoir in rugged terrain) did not meet the 10‑cm requirement but
still arguably provided valuable information. In 2005, GRLM latency was
estimated to be 6 days, which was a 10‑day improvement over 2004
performance. GRLM coverage was expanded, but because of challenges with
current standard Jason-1 data processing, most of the new water bodies
in the GRLM have few, if any, valid height estimates in the Jason-1 era.
This leaves South America, south and east Asia, and Australia with very
few near-real-time data points.

As of summer 2005, NASA-supported enhancements to the PECAD decision
support tools have had varying degrees of success in meeting PECAD's
requirements. The MODIS Image Gallery and the GRLM are more mature and
have achieved a semi-operational status, whereas the USDA-FAS MODIS NDVI
Database is still evolving and should presently be considered
developmental.

One overall recommendation resulting from the V&V process is that as
NASA-supported enhancements become operational, regularly generated
metrics should be introduced to track certain types of performance, such
as delivery and coverage. While certain facets of decision support will
require more careful analysis, the more routine aspects that relate to
satisfaction of customer requirements should be amenable to simple
self-monitoring that could be publicly available as product metadata.

# Introduction

NASA partners with the U.S. Department of Agriculture (USDA) to bring
about innovative uses of NASA Earth-Sun system science to inform and
improve decision making for the sake of U.S. agricultural efficiency. In
this partnership, NASA and the USDA seek to identify assessments, tools,
and systems that impact decision making at a national scale and that are
amenable to enhancement with NASA observations or predictions. One
decision support system (DSS) that has been identified belongs to the
Production Estimates and Crop Assessment Division (PECAD).

PECAD, a component of the USDA Foreign Agricultural Service (USDA-FAS),
monitors global crop conditions, assesses how those conditions affect
food security, and predicts crop yields for grains, oilseeds, and
cotton. PECAD essentially performs an intelligence mission that affects
U.S. government actions and global commodity markets. PECAD has several
analysts who make crop assessments for certain regions of the globe.
These analysts must make informed decisions about crop area, yield, and
production. These decisions are supported by a framework of tools that
bring together multiple, independent information sources in a spatial
and temporal context. These tools include the Crop Assessment Data
Retrieval and Evaluation (CADRE) geospatial data management system,
which provides internal access to a wealth of remote sensing and weather
information and to the Crop Explorer Web interface ([USDA,
2005](#USDA2005)) which gives both internal and external users a portal
to view remote sensing observations and model outputs for fixed
geospatial subsets. Many of the Crop Explorer data streams are provided
with near-real-time access.

More detailed information about PECAD's DSS can be found in *Decision
Support Tools Evaluation Report for FAS/PECAD, Version 2.0*
([NASA, 2004a](#NASA2004a)). This report was part of a systems
engineering approach to the USDA/NASA partnership supporting the
enhancement of PECAD's DSS. Three points of documentation are planned in
this process: (1) DSS evaluation, (2) verification and validation of
specific enhancement performance, and (3) benchmarking to show net
effects of enhancements on the DSS performance as a whole. The systems
engineering approach in its entirety is represented in Figure 1. This
report documents the verification and validation (V&V) of implemented
enhancements to PECAD's DSS that are based on NASA Earth-Sun system
science.

The implemented enhancements include application-tailored Moderate
Resolution Imaging Spectroradiometer (MODIS) products and Global
Reservoir and Lake Monitor (GRLM) products. The MODIS products are
currently made available through two separate decision support tools:
the MODIS Image Gallery and the USDA-FAS MODIS Normalized Difference
Vegetation Index (NDVI) Database. Both the GRLM and the MODIS Image
Gallery provide near-real-time products through PECAD's Crop Explorer.
This discussion will address two areas:

-   Assessments of existing or previous NASA products on which these
    enhancements are based, and

-   Characterizations of the performance of the newly implemented
    products.

## Systems Engineering Considerations

While there is an overall thematic direction to the systems engineering
process as defined in Figure 1, the process is not a simple sequence.
For example, V&V and benchmarking for PECAD's DSS have proceeded
concurrently with the V&V effort coordinated by NASA Stennis Space
Center's Applied Sciences Directorate and benchmarking carried out by a
joint team of scientists from the University of Arizona and the
University of Missouri ([Haithcoat et al., 2005](#Haithcoat2005); [van
Leeuwen et al., 2005](#vanLeeuwen2005)).

+-----------------------------------------------------------------------+
| Source: Adapted from [Bahill and Gissing, 1998](#Bahill1998)          |
|                                                                       |
| []{#__RefHeading___Toc124078781 .anchor}Figure 1. Systems engineering |
| approach.                                                             |
+-----------------------------------------------------------------------+

V&V and benchmarking milestones' being reached does not imply that the
systems engineering process is complete. This certainly applies to the
design and implementation stage. The projects currently delivering both
MODIS and GRLM products are at various levels of maturity and are only
meant to attain semi-operational or perhaps even demonstration status.
Since the PECAD DSS certainly requires fully operational solutions, a
transition to operational status is inferred. The manner and schedule of
the operational transition is a matter of ongoing dialogue among DSS
stakeholders.

The distinction between pre-operational and operational status has
important consequences in the understanding of DSS requirements. This
V&V report and its predecessor evaluation report ([NASA,
2004a](#NASA2004a)) consider the *operational* requirements of PECAD's
DSS. The projects implementing enhanced products are properly focused on
their *project performance* requirements as defined in agreements with
NASA and the USDA. Where a clear separation between operational and
project performance requirements is known to exist, this separation is
noted in the report.

Simply defining PECAD's operational requirements has been a difficult
process, and the process is ongoing. The challenges faced by PECAD's
analysts are diverse and resistant to simplification. Spatial scale
ranges from small subsistence plots to the entire crop-producing area of
the globe. Temporally, analysts must track multiyear means and must also
be able to predict agronomic damage only hours after an event occurs. As
a result, some requirements are a matter of continuing discussion
between PECAD and NASA. Specifications for the spatial characteristics
of Earth science inputs fall into this category. PECAD uses remote
sensing data with ground sample distances (GSDs) ranging from less than
a meter up to multiple kilometers. Teasing out spatial requirements for
PECAD's DSS would necessitate a careful breakdown of analysts' typical
tasks followed by a detailed sensitivity analysis for each task. Even
then, trade studies would be required to balance spatial considerations
with information technology and resource constraints and with temporal
drivers.

The operational requirements suitable for attention in this report fall
into three areas:

-   Accuracy: measured uncertainty of a particular product's parameter
    of interest

-   Delivery:

    -   Data time step, or frequency of product delivery

    -   Latency, or mean lag time between original data acquisition and
        final product delivery

-   Coverage: geographical area covered or spanned by the product

The requirements as specified for each product are taken from the
earlier evaluation report ([NASA, 2004a](#NASA2004a)) unless otherwise
noted.

# V&V of Application-Tailored MODIS Products

As mentioned above, PECAD's overall set of decision support tools are
enhanced by the routine provision of two application-tailored MODIS
products: the Crop Explorer MODIS Image Gallery and the USDA-FAS MODIS
NDVI Database. Both products have been developed and implemented by the
Global Agricultural Monitoring (GLAM) project, whose collaborators
include NASA's Goddard Space Flight Center (GSFC); USDA-FAS; Science
Systems and Applications, Inc.; and the University of Maryland
Department of Geography.

**MODIS Image Gallery**: The Crop Explorer MODIS Image Gallery is
accessed through the Crop Explorer interface at the following USDA-FAS
Web site: <http://www.pecad.fas.usda.gov/cropexplorer/modis_summary/>

The Crop Explorer MODIS Image Gallery provides near-real-time access to
processed MODIS image subsets. This interface is actually a thin client
linked to a selected set of the imagery available through GSFC's MODIS
Rapid Response System. For each image subset, the following images are
provided in JPEG format: a natural color combination (Red, Green, Blue
(RGB): MODIS bands 1, 4, and 3); a shortwave infrared/near infrared/red
combination (RGB: MODIS bands 7, 2, and 1); and an NDVI. GeoTIFF format
is provided at many locations and is planned for all image subsets as
hardware improvements permit (Descloitres, personal communication,
2005). Each image type may be downloaded in 250 m, 500 m, and 1 km
ground sample distances. Accuracy, delivery, and coverage requirements
related to near-real-time MODIS images are detailed in Table 1.

  --------------------------------- ----------------------------------------------------
  []{#__RefHeading___Toc124078776   
  .anchor}Table 1. PECAD accuracy,  
  delivery, and coverage            
  requirements for near-real-time   
  imaging.                          

  Category                          Requirement

  Product accuracy                  Undetermined

  Data time step                    1 day

  Latency                           6 hours or less

  Coverage                          All land regions important for crop production and
                                    food security

  Source: [NASA, 2004a](#NASA2004a) 
  --------------------------------- ----------------------------------------------------

Further comment is called for in the case of product accuracy and
coverage requirements. For product accuracy, a need for consistency with
multiday composite products (such as the USDA-FAS MODIS NDVI Database)
is implied, but no requirements or specifications have been established
in discussions with PECAD. The MODIS Rapid Response team is studying the
accuracy of the Rapid Response NDVI relative to other estimates, but
results are not yet available (Descloitres, personal communication,
2005).

The coverage requirement is a case where operational and project
performance requirements diverge. That PECAD needs imaging over "all
land regions important for crop production and food security" (NASA,
2004a) is self-evident, but the MODIS Rapid Response contribution as
originally proposed did not specify coverage. The project focus was on
generating prototyping products ([Justice et al.,
2002](#JusticeEtAl2002)). In early 2005, PECAD and the MODIS Rapid
Response team agreed to bring the project performance requirement into
alignment with PECAD's quasi-global operational requirement. Contingent
on the procurement and integration of new hardware to support this
effort, the MODIS Rapid Response team expects to complete the increase
in coverage by the end of the 2005 calendar year (Kendall and Schmalz,
personal communication, 2005).

**USDA-FAS MODIS NDVI Database**: While the MODIS Image Gallery meets
near-real-time needs, PECAD also requires a quasi-global NDVI image and
time series database to observe crop trends. Multiday compositing is
necessary for this product to reduce the affect of clouds, bidirectional
reflectance distribution function effects, and other confounding
factors. Through the USDA-FAS MODIS NDVI Database, users may display or
download the archived NDVI data as single-date images or as chart time
series of NDVI variation for selected points or areas. This database is
currently maintained for PECAD at the following GSFC Web site:
<http://pekko.geog.umd.edu/usda/ndvi/>

This product is still a beta version. The NDVI observations in this
product are not produced by the GLAM project; they are NDVI estimates
computed from standard Level 3, 16-day surface reflectance composites
(MODIS MOD09 product) generated by the MODIS Data Production System
(MODAPS). The USDA-FAS MODIS NDVI Database is developing tools for MODIS
NDVI products that should play a similar role to Advanced Very High
Resolution Radiometer (AVHRR) vegetation index information currently
available to PECAD analysts through CADRE. The PECAD accuracy, delivery,
and cover requirements for an NDVI database are summarized in Table 2.

  --------------------------------- --------------------------------------------------------
  []{#__RefHeading___Toc124078777   
  .anchor}Table 2. PECAD accuracy,  
  delivery, and coverage            
  requirements for an NDVI          
  database.                         

  Category                          Requirement

  Product accuracy                  Must be consistent with existing NDVI database going
                                    back to 1981

  Data time step                    Maximum of 10 days ([Doorn, 2004](#Doorn2004))

  Latency                           24 hours or less measured from the end of the
                                    compositing period ([Doorn, 2004](#Doorn2004))

  Coverage                          All land regions important for crop production and food
                                    security

  Source: [NASA, 2004a](#NASA2004a) 
  --------------------------------- --------------------------------------------------------

The USDA-FAS MODIS NDVI Database contains points of difference between
operational and project performance requirements. For the data time
step, the product as proposed is limited to 16-day compositing periods,
which are the minimum available through MODAPS. Similarly, the product
is also limited in latency by the performance of MODAPS. The project's
original proposal suggested a nominal latency of 7 days.

## Calibration and Validation of MODIS Sensor and Standard Products

The on-orbit calibration and validation of the MODIS sensor and standard
MODIS products was dealt with comprehensively (for Terra MODIS) in a
special issue of *Remote Sensing of Environment* in 2002, which
characterized several intrinsic properties of interest to operational
users. [Wolfe et al. (2002)](#Wolfe2002) reported a geolocational
accuracy of 50 m (1 σ) at nadir. For MODIS Bands 1 and 2 (the red and
near-infrared bands used for PECAD tailored products), [Rojas et al.
(2002)](#Rojas2002) found that the spatial response was consistent with
pre-launch performance in the cross-track direction but was a bit
degraded in the along-track direction. [Huete et al.
(2002)](#HueteEtAl2002) demonstrated that MODIS NDVI performed at higher
fidelity than AVHRR NDVI. In a discussion of atmospheric correction for
standard MODIS products, [Vermote et al. (2002)](#Vermote2002) showed a
clear example of how not correcting for aerosol can dramatically affect
the distribution of NDVI values throughout a scene.

Further work has been done regarding the cross validation of MODIS NDVI
with other sensors. [Gallo et al. (2004)](#Gallo2004) found a linear
relationship between AVHRR and MODIS 16-day composite time series with
overall coefficient of variation (*r*^2^) above 0.9 (considering nine
diverse land cover types). Working directly in support of the NASA/FAS
partnership, the Global Inventory Monitoring and Modeling Studies
(GIMMS) group at NASA GSFC has found similar results studying AVHRR,
MODIS (MOD13 standard products), and SPOT VEGETATION ([Justice,
2005](#Justice2005)). In Figure 2, NDVI time series are displayed from
GIMMS estimates for one crop production region in South Africa and for
another region in eastern Australia (the green line in each plot is
MODIS Enhanced Vegetation Index, which is not used in current MODIS
products tailored for PECAD). All of the NDVI time series have
correlation coefficients above 0.9 relative to each other.

  ----------------------------------- -----------------------------------
                                      

  []{#__RefHeading___Toc124078782     
  .anchor}Figure 2. Vegetation index  
  time series for the Maize Triangle  
  region of South Africa and for the  
  Victoria wheat-growing region of    
  eastern Australia.                  
  ----------------------------------- -----------------------------------

## Characterization of the MODIS Application-Tailored Products

The product characterizations in this report summarize quantifiable
product performance in the areas of accuracy, delivery, and coverage as
outlined in the systems engineering discussion (Section 1.1).

### MODIS Product Accuracy

The only parameter of interest that has been clearly established for
MODIS application-tailored products is NDVI. Given that PECAD's accuracy
requirement has been stated only in qualitative terms (observations
should be "consistent" with the multiyear database), only a
cross-comparison of the application-tailored products to standard
products is needed.

A cross-comparison for the MODIS Rapid Response NDVI is not yet
complete. The MODIS Rapid Response System team is currently undertaking
a comparison with standard MODIS products (Descloitres, personal
correspondence, 2005).

Because NDVI in the USDA-FAS MODIS NDVI Database is actually computed by
MODAPS, it is assumed to be validated for the purposes of this report.
It should be noted, however, that the USDA-FAS MODIS NDVI Database
product is not the standard MOD13 NDVI; this NDVI product is computed in
the MOD44C vegetation intermediate composite product. Although MOD44C is
primarily an intermediate step in production of the MOD44A vegetation
cover conversion product, its origins at the University of Maryland give
the GLAM team greater creative control in support of PECAD. Both MOD13
and MOD44C take daily MOD09 surface reflectance as input and both use
the standard equation (
$\left( \rho_{\text{NIR}} - \rho_{\text{red}} \right)/\left( \rho_{\text{NIR}} + \rho_{\text{red}} \right)$
) to compute NDVI, but the two slightly differ in their compositing
schemes. MOD44C seeks the most cloud free, highest resolution data over
the compositing period ([USGS, 2005](#USGS2005)), and MOD13 has a
similar approach called "constraint view angle maximum value
compositing" which diverges in some details ([Gao et al.,
2003](#Gao2003)). Given the product similarities, previous studies
regarding consistency of MODIS with legacy systems should be considered
extensible to USDA-FAS MODIS NDVI Database estimates.

### MODIS Product Delivery

The MODIS Rapid Response System product delivery was observed for two
15-day periods. The first period ran from June 23, 2005, (Julian day
173) through July 6, 2005 (Julian day 187). However, on July 2, 2005, an
interruption of data lasted for more than 2 days. This problem affected
all the data that was passed through the MODIS Rapid Response System to
the Crop Explorer MODIS Image Gallery and was reported on the MODIS
Rapid Response System Status page as shown in Figure 3.

+-----------------------------------------------------------------------+
|   ------------ -----------------------------------------------------  |
|   **Date**     **Description**                                        |
|                                                                       |
|   2 July 2005\ Some current data is not available due to hardware     |
|   2005/183     issues.                                                |
|   ------------ -----------------------------------------------------  |
|                                                                       |
| Source: <http://rapidfire.sci.gsfc.nasa.gov/status/>                  |
|                                                                       |
| []{#__RefHeading___Toc124078783 .anchor}Figure 3. July 2005 data      |
| interruption as reported on the MODIS Rapid Response System Status    |
| page.                                                                 |
+-----------------------------------------------------------------------+

Even 5 days after the July 2, 2005, interruption, the MODIS Rapid
Response Delivery System delivery times were notably out of step with
pre-interruption system performance. As a result, only the first 10 days
of observation were used to estimate system performance. Based on the
June-July observations, the estimated system latency for 59 tiles with
an Aqua and a Terra delivery opportunity for each of 10 days (1180
distinct delivery opportunities) was 4.7 hours. The latency was computed
as the mean lag time. All lag times greater than 24 hours were excluded
on the assumption that these times represented anomalous problems that
were not typical of system throughput. Just over 65 percent of the MODIS
Rapid Response System information was available within the 6-hour
requirement stated by PECAD and 80 percent of the information was
available within 24 hours.

The second observation period ran from October 1, 2005, (Julian day 274)
through October 15, 2005 (Julian day 288). Although 12 tiles were added
to cover North Africa before the October observation period, these tiles
were left out of this analysis to allow valid comparisons between the
two periods. Based on the October observations, the estimated system
latency for 59 tiles with an Aqua and a Terra delivery opportunity for
each of 15 days (1770 distinct delivery opportunities) was 4.4 hours.
About 87 percent of the MODIS Rapid Response System information was
available within the 6-hour requirement stated by PECAD and 100 percent
of the information was available within 24 hours.

Given the July data interruption, the October period should be
considered more representative of expected operational system
performance. Several distinct trends can be discerned in the October
period. First, it should be noted that the lag times are heavy-tailed on
the long side (Figure 4). The observed lag time data bears some
resemblance to heavy-tailed distributions such as the inverse Gaussian
and lognormal distributions. The implication is that for mature
requirements it will be important to develop thresholds for both mean
delivery and for some higher threshold with 90 percent delivery.

A systematic relationship existed between lag time and the time of day
in which a particular subset-tile was acquired and processed. Figure 5
shows that the lag time trend reaches a minimum of about 3 hours near
6:00 AM -- 12:00 AM (UTC) and then rises slowly through the course of a
day, reaching a maximum of more than 4 hours between 6:00 PM and
midnight (UTC). The geographic distributions of delivery and lag times
for Terra and Aqua are shown in Figure
6.![](media/image2.png){width="3.9256944444444444in"
height="2.686111111111111in"}

+-----------------------------------------------------------------------+
| ![](media/image3.wmf){width="4.038888888888889in"                     |
| height="3.0409722222222224in"}                                        |
|                                                                       |
| []{#__RefHeading___Toc124078785 .anchor}Figure 5. Variation of Crop   |
| Explorer MODIS Image Gallery delivery time lag by time of day         |
| (October 2005 observation period).                                    |
+-----------------------------------------------------------------------+

Taken as a whole, the Crop Explorer MODIS Image Gallery latency is
meeting the stated PECAD requirement of 6 hours. Nominally, the data
time step is being met, but lapses in availability (both identified and
unidentified in the MODIS Rapid Response System Status page) are a cause
for concern.

The USDA-FAS MODIS NDVI Database was not monitored in as careful a
manner because it is more developmental in nature. However, it was
observed that through June and July 2005, the shortest time lapse
between end of compositing window and USDA-FAS MODIS NDVI Database
timestamp was about 9.5 days. This latency has not yet met the project
goal of 7 days nor the PECAD goal of 72 hours as stated in the DSS
evaluation (NASA, 2004a). Given the 16-day time step and a latency of
about 10 days (or more), the chance of PECAD analysts having more than
one NDVI Database reading for the month leading to lock-up is small.

### MODIS Product Coverage![](media/image4.png){width="4.596527777777778in" height="1.9152777777777779in"}![](media/image5.png){width="4.586805555555555in" height="1.91875in"}

The global footprints of the Crop Explorer MODIS Image Gallery and the
USDA-FAS MODIS NDVI Database are shown in Figure 7. Curiously, the
Database omits Madagascar, even though Madagascar is available through
the Crop Explorer MODIS Image Gallery. However, the USDA-FAS MODIS NDVI
Database coverage limitations are strictly a result of the current
system configuration since the MODAPS source data is essentially global
in coverage.

The Crop Explorer MODIS Image Gallery omits a few crop production areas
of some economic importance including Central America, parts of South
America, the northernmost productive region of the Great Plains of North
America, and parts of central Asia. Note the presence of the 12 northern
Africa subset-tiles for the Crop Explorer MODIS Image gallery that were
left out of the delivery time analysis.

Overall, the MODIS application-tailored products have nearly met the
PECAD coverage requirements for economically important crop production,
but important coverage gaps exist in the Crop Explorer MODIS Image
Gallery in terms of PECAD's food security mandate: notably Central
America, northern South America, Indonesia, and the
Philippines.![](media/image6.png){width="6.000694444444444in"
height="2.5083333333333333in"}

# V&V of Global Reservoir and Lake Monitor Products

PECAD's decision support requirements related to surface water level
variation are detailed in Table 3.

  --------------------------------- --------------------------------------
  []{#__RefHeading___Toc124078778   
  .anchor}Table 3. PECAD accuracy,  
  delivery, and coverage            
  requirements for surface water    
  level variation.                  

  Category                          Requirement

  Surface water level relative      10 cm
  accuracy                          

  Data time step                    10 days

  Latency                           7--14 days

  Coverage                          Surface waters in all land regions
                                    important for crop production and food
                                    security

  Source: [NASA, 2004a](#NASA2004a) 
  --------------------------------- --------------------------------------

## Calibration and Validation of TOPEX/Poseidon and Jason-1 Radar Altimeters

Global Reservoir and Lake Monitor products are based on measurements
from radar altimeters onboard the Jason-1 and TOPEX/Poseidon satellites
([USDA, 2004](#USDA2004)), which are joint missions between NASA and the
French space agency, the Centre National d'Etudes Spatials (CNES). The
primary purpose of these altimeters is to increase understanding of
ocean surface topography. Their principal measurement is sea surface
height ([Ménard et al., 2003](#Menard2003)). The ranging approach for
determining sea surface height differs in some particulars from the
approach for determining surface water levels, but the two measurements
are based on many common principles. Given that Jason-1 and
TOPEX/Poseidon were designed for open sea measurements, the accuracy
achieved by these instruments in measuring sea surface height sets the
optimistic bound on accuracy that can be expected in measuring surface
water levels.

After the launch of Jason-1 on December 7, 2001, about 10 months were
dedicated to calibration of the new instrument and assessment of its
performance. For much of that period, the Jason-1 and TOPEX/Poseidon
satellites were operated in a common orbit with about one minute of
separation between the two platforms, allowing an updated assessment of
the TOPEX/Poseidon performance as well as a cross-validation between the
two systems.

Three primary levels of distributed products were evaluated:

1.  Geophysical Data Record (GDR) -- final product based on ground
    processing, available in 30 days;

2.  Interim Geophysical Data Record (IGDR) -- intermediate products
    based on a preliminary orbit determination, available in three days;
    and

3.  Operational Sensor Data Record -- near-real­-time products based on
    onboard processing, available in 3--5 hours.

Of these products, the GDRs and IGDRs are relevant to the Global
Reservoir and Lake Monitor. The GRLM used GDRs for TOPEX/Poseidon data
to establish historical data going back to 1992 and used IGDRs for
Jason-1 data to provide a near-real­-time product with a latency period
on the order of 10 days.

[Ménard et al. (2003)](#Menard2003) summarized the Root-Sum-of-Squares
estimates of Jason-1 sea surface height errors as 3.3 cm for the GDR
product and as 3.9 cm for the IGDR products. Working in diverse
locations, including the Harvest platform near California ([Haines et
al., 2003](#Haines2003)), Corsica ([Bonnefond et al.,
2003](#Bonnefond2003)), and the Bass Strait on the north side of
Tasmania ([Watson et al., 2003](#Watson2003)), the individual
calibration/validation teams commissioned by CNES and NASA all achieved
sea-based results consistent with the overall estimates. Each of these
studies also measured TOPEX/Poseidon errors and found them to be in the
same 3-4 cm range. [Chambers et al. (2003)](#Chambers2003)
cross-validated Jason-1 against TOPEX/Poseidon. The estimates of
individual sensor error were similar among the analyses cited. The
magnitude of the global bias between the sensors was found to be
14-15 cm. Residuals showed dependency on significant wave height, with
local biases near coastlines. More placid waters had somewhat lower
biases, and local biases in rougher waters were somewhat higher. This
dependency on significant wave height accounts for some differences in
bias estimates at the fixed sites, and it has implications for surface
water level estimates since surface waters frequently have smaller wave
heights than the open ocean.

Further highlighting the challenge of altimetry over surface waters, one
Jason-1 calibration/validation effort studied sea surface height as
measured over the Great Lakes with a special focus on a specific
calibration site on the Ohio shore of Lake Erie ([Shum et al.,
2003](#Shum2003)). After accounting for instrument bias, the Jason-1
residuals with Great Lake tidal gauges ran between 3 and 5 cm, which is
consistent with other results. However, the relative bias between
Jason-1 and TOPEX/Poseidon was estimated to be about 8 cm, which is 6 cm
less than the global bias. Only a small portion of the difference could
be attributed to wave height effects. Ultimately, the study could not
explain the difference.

An extensive evaluation of the capability of TOPEX/Poseidon to monitor
lake/reservoir water level variation was carried out by [Birkett
(1995)](#Birkett1995). Birkett's study included an analysis of 68 lakes
and developed an error budget for surface water level variation as
detailed in Table 4. In this budget, two factors stood out as creating
differences between large and small lakes: range precision and wet
tropospheric correction.

Range precision was significantly degraded by freezing of the lake
surface. Also, range precision varied according to the number of data
points collected across a lake. As the number of data points per pass
($N$) over a lake was increased, the impact of range precision error was
reduced by a factor of$\frac{1}{\sqrt{N}}$. Birkett noted that the range
precision error contribution was in the sub-centimeter range for large
water bodies.

The wet tropospheric correction was based on the onboard microwave
radiometer for open ocean readings. However, because of the large
footprint of the radiometer (20--40 km), the corrections based on
microwave radiometer readings near shore could not be used. Thus, larger
lakes benefited from a better correction than the smaller lakes.

Aside from considerations related to the generalized error budget,
Birkett also drew attention to several other conditions that could lead
to error in estimation of surface water level variation. These
conditions included unusually stormy or calm lake surfaces,
unidirectional wind setup, and geoid surface slope error.

Birkett's study performed a validation versus gauge data for Lake
Superior, Lake Michigan, and Lake Ontario and, consistent with [Shum et
al. (2003)](#Shum2003), found accuracy in the 3--5 cm range.

## Characterization of the Global Reservoir and Lake Monitor

An initial verification and validation effort for the GRLM decision
support tool was carried out in 2004. Though somewhat early in the
implementation phase for the tool, this effort touched on all the
requirements mentioned in Table 3. The tool's delivery and coverage have
matured since this initial V&V effort ended, so requirements related to
these properties were re-evaluated through the month of May 2005. As for
the product accuracy requirement, the fundamental algorithm is
essentially unchanged, so the results from the 2004 report are
summarized below. For further detail, refer to the online version of the
previous report, *PECAD's Global Reservoir and Lake Monitor: A Systems
Engineering Report* ([NASA 2004b](#NASA2004b)).

### GRLM Accuracy

The GRLM root mean square accuracy estimates over several North American
lakes are summarized in Table 5. These estimates are based on
comparisons with water level gauges. The Great Lakes gauges were chosen
from U.S. National Oceanic and Atmospheric Administration and Canadian
Department of Fisheries and Oceans gauges to frame the Jason-1 pass used
for GRLM estimates (Figure 8). Lake Winnebago and Lake of the Woods were
each referenced to a single U.S. Geological Survey gauge. Lake Powell
was monitored by the U.S. Bureau of Reclamation using a well-gauge built
directly into the face of Glen Canyon Dam. Essentially, the Great Lakes
can be considered representative of the larger water bodies in the GRLM.
Great Lakes errors ran between 5 and 7 cm. Lake Winnebago and Lake of
the Woods may be somewhat representative of smaller water bodies. Each
estimate represents only a handful of potential readings per pass. Lake
Winnebago error was calculated to be 27 cm and Lake of the Woods error
was computed to be 26 cm. Lake Powell represents the most challenging
type of lake or reservoir. The lake has very few potential readings per
pass, and it is surrounded by mountainous terrain. Lake Powell error
over the period assessed was about 140 cm.

Note the Great Lakes errors computed for the GRLM are somewhat larger
than those reported by [Shum et al. (2003)](#Shum2003). This discrepancy
may be explained by the earlier study's focus on the verification of
sensor performance and by corrections for the sea surface height
estimates being carefully tailored for the Great Lakes. The 2004 GRLM
verification effort made no special corrections to the data because the
focus of that study was the GRLM operational product accuracy.

+-------------+--------+--------+--------+--------+--------+--------+
| []{#__Ref   |        |        |        |        |        |        |
| Heading___T |        |        |        |        |        |        |
| oc124078780 |        |        |        |        |        |        |
| .a          |        |        |        |        |        |        |
| nchor}Table |        |        |        |        |        |        |
| 5. Verified |        |        |        |        |        |        |
| GRLM        |        |        |        |        |        |        |
| performance |        |        |        |        |        |        |
| over select |        |        |        |        |        |        |
| North       |        |        |        |        |        |        |
| American    |        |        |        |        |        |        |
| surface     |        |        |        |        |        |        |
| water       |        |        |        |        |        |        |
| bodies.\*   |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Water Body  | S      | σ\     | Number | Ou     | **     | **Mean |
|             | urface | (ou    | Passes | tlying | Number | P      |
|             | Ar     | tliers |        | Passes | Ga     | ass-to |
|             | ea^†^\ | rem    |        | R      | uges** | -Gauge |
|             | (      | oved)\ |        | emoved |        | Dist   |
|             | km^2^) | (cm)   |        |        |        | ance** |
|             |        |        |        |        |        |        |
|             |        |        |        |        |        | **     |
|             |        |        |        |        |        | (km)** |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake        | 82,100 | 7.3    | 81     | 0      | 3      | 72     |
| Superior    |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake        | 57,800 | 5.3    | 75     | 0      | 3      | 51     |
| Michigan    |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake Huron  | 59,600 | 6.2    | 80     | 2      | 4      | 55     |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake Erie   | 25,700 | 6.5    | 73     | 1      | 4      | 29     |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake        | 18,960 | 4.5    | 79     | 3      | 3      | 52     |
| Ontario     |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake        | 557    | 27.0   | 24     | 0      | 1      | 7      |
| Winnebago   |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake of the | 1,900  | 26.1   | 42     | 1      | 1      | 31     |
| Woods       |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+
| Lake Powell | 500    | 140.6  | 244    | 6      | 1      | 84     |
+-------------+--------+--------+--------+--------+--------+--------+
| \* All      |        |        |        |        |        |        |
| lakes and   |        |        |        |        |        |        |
| reservoirs  |        |        |        |        |        |        |
| were        |        |        |        |        |        |        |
| ch          |        |        |        |        |        |        |
| aracterized |        |        |        |        |        |        |
| using       |        |        |        |        |        |        |
| Jason‑1     |        |        |        |        |        |        |
| IGDR-based  |        |        |        |        |        |        |
| estimates   |        |        |        |        |        |        |
| with the    |        |        |        |        |        |        |
| exception   |        |        |        |        |        |        |
| of Lake     |        |        |        |        |        |        |
| Powell. No  |        |        |        |        |        |        |
| estimates   |        |        |        |        |        |        |
| based on    |        |        |        |        |        |        |
| Jason‑1data |        |        |        |        |        |        |
| existed for |        |        |        |        |        |        |
| Lake Powell |        |        |        |        |        |        |
| as of May   |        |        |        |        |        |        |
| 2004, so    |        |        |        |        |        |        |
| TOP         |        |        |        |        |        |        |
| EX/Poseidon |        |        |        |        |        |        |
| GDR-based   |        |        |        |        |        |        |
| estimates   |        |        |        |        |        |        |
| were used.  |        |        |        |        |        |        |
|             |        |        |        |        |        |        |
| ^†^ Surface |        |        |        |        |        |        |
| areas from  |        |        |        |        |        |        |
| [www.worl   |        |        |        |        |        |        |
| dlake.org]( |        |        |        |        |        |        |
| ../www.worl |        |        |        |        |        |        |
| dlake.org). |        |        |        |        |        |        |
+-------------+--------+--------+--------+--------+--------+--------+

Note the Great Lakes errors computed for the GRLM are somewhat larger
than those reported by [Birkett (1995)](#Birkett1995) or [Shum et al.
(2003)](#Shum2003). Much of the increased Great Lakes error is likely
due to ice contamination, storms, or wind setup. The 2004 GRLM
validation effort attempted to screen data only by removing a small
number of statistical outliers and, in the case of Lake Winnebago and
Lake of the Woods, by excluding completely frozen conditions. The goal
was to represent the unvarnished operational product accuracy.

More problematic, perhaps, are the larger error figures for the three
smaller water bodies: Lake Winnebago, Lake of the Woods, and Lake
Powell. These error figures can be partially explained by less ideal
reference data. As mentioned above, each of these three water bodies was
referenced to only a single gauge. Lake Winnebago and Lake of the Woods
estimates were based on fewer passes (24 and 42, respectively). The Lake
Powell gauge would be expected to be markedly influenced by local
draw-down associated with release events. Beyond reference issues, each
of the three lakes had much less surface area than even the smallest of
the Great Lakes. Consequently, the three lakes had fewer data points per
pass and reduced range precision relative to the Great Lakes. The three
lakes suffered from poorer estimation of the wet tropospheric
correction. The steep terrain surrounding Lake Powell made it difficult
for the altimeter sensors to "lock" on the water surface as the
satellite crosses from land to water. "Loss of lock" can cause a loss of
some or all data points for surface water bodies. No valid data points
have been returned for Lake Powell since the transition from
TOPEX/Poseidon to Jason-1.

+-----------------------------------------------------------------------+
| ![](media/image7.png){width="6.083333333333333in"                     |
| height="4.406944444444444in"}                                         |
|                                                                       |
| []{#__RefHeading___Toc124078788 .anchor}Figure 8. Gauge stations in   |
| the general vicinity of the Great Lakes.                              |
+-----------------------------------------------------------------------+

While not all lakes appeared to meet the stated FAS accuracy
requirement, perhaps the accuracy requirement should be revisited. For
deep, narrow reservoirs, fairly large height uncertainty may be
acceptable because the information regarding water volume is still
readily discernable. For shallow, closed lakes (e.g., Lake Chad), the
current standards are appropriate. Furthermore, FAS lake water height
requirements should be developed in more detail to include a
comprehensive listing of water bodies that FAS believes should be
monitored.

### GRLM Delivery

The latency of the GRLM products decreased from 10.3 days to 5.8 days
between May 2004 and May 2005. Latency was defined as the mean lag time
between data acquisition and GRLM product upload. The maximum lag time
in May 2004 was 19.2 days, but in May 2005 the maximum lag time was
reduced to 10.4 days. This dramatic improvement in delivery time was
accomplished by reducing the average time between GRLM updates from 16
days to 6 days. This improvement clearly meets the stated FAS 10-day
latency requirement for surface water level variation.

Once every 9.9 days (the Jason-1 orbital period), the GRLM adds a new
data record for each monitored body of water, so the FAS 10-day time
step requirement is nominally met. However, the Jason-1 standard product
algorithms yield no height information for some passes, perhaps because
of issues like "loss of lock" or other conditions that are screened in
the standard algorithms to improve sea surface height estimates. For
many instances in both 2004 and 2005, months to years passed between
data acquisitions in which water levels were actually estimated. These
dated sites have archival value, but they no longer provide information
about current regional water availability. A breakdown of the time to
last estimated water level is shown in Figure 9.

+-----------------------------------------------------------------------+
| ![](media/image8.wmf){width="4.510416666666667in"                     |
| height="3.170138888888889in"}                                         |
|                                                                       |
| []{#__RefHeading___Toc124078789 .anchor}Figure 9. Histogram of time   |
| to last estimated water level checked at the end of May 2004 and      |
| again at the end of May 2005.                                         |
+-----------------------------------------------------------------------+

From May 2004 to May 2005, the number of lakes and reservoirs monitored
increased from 52 to 84 -- more than 60 percent. An incremental increase
occurred in the number of water bodies with near-real-time monitoring,
but most of the increase consisted of lakes and reservoirs that have had
little or no valid data since Jason-1 became operational. The dearth of
data for near-land sites results from the algorithms implemented for
standard Jason-1 data processing. These algorithms filter out the
near-land data. This issue was noted in the 2004 report ([NASA,
2004b](#NASA2004b)) with the indication that it was being addressed,
but, as of this writing, no substantial improvement in processing
near-land data was noted. The geographic distribution of more current
and more out--of-date sites is shown in Figure 10.

+-----------------------------------------------------------------------+
| ![](media/image9.png){width="5.995833333333334in"                     |
| height="2.484027777777778in"}                                         |
|                                                                       |
| []{#__RefHeading___Toc124078790 .anchor}Figure 10. Geographic         |
| distribution of time to last estimated water level at the end of May  |
| 2005.                                                                 |
+-----------------------------------------------------------------------+

### GRLM Coverage

In 2004, little coverage existed in South America, western North
America, or eastern Asia. Australia had no monitored sites. As can be
seen in Figure 10, through May 2005 sites have been added in all of
these areas. Unfortunately, the overwhelming majority of the new sites
are badly dated, so the near-real-time coverage has changed little.

# Conclusions and Recommendations

As of summer 2005, NASA-supported enhancements to the decision support
tools of PECAD have had varying degrees of success in meeting PECAD's
requirements. The MODIS Image Gallery and the Global Reservoir and Lake
Monitor are more mature and have achieved a semi-operational status,
whereas the USDA-FAS MODIS NDVI Database is still evolving and should
presently be considered developmental.

Considering each decision support enhancement individually, their
performances can be summarized according to product accuracy, delivery,
and coverage. The MODIS Image Gallery has met delivery requirements with
a latency of less than 5 hours and a time step of 1 day. Down time is a
concern. Coverage is nearing sufficiency for economically important crop
production regions, but some significant gaps remain in Central America,
in parts of South America, in the northernmost productive region of the
Great Plains of North America, and in parts of central Asia. Additional
large coverage gaps for PECAD's food security mission exist, notably in
the Congo basin, in Indonesia, and in the Philippines.

Although the USDA FAS MODIS NDVI Database is less finished than the
MODIS Image Gallery, the Database has already largely met its coverage
requirement with fewer gaps than the MODIS Image Gallery. The principal
exception is the Congo basin, where the Database does not seem to be
meeting the 72-hour latency requirement. Cursory observations indicated
latency of 9 days or more. The Database is being delivered on a 16-day
time step, which is at the upper end of the range currently considered
acceptable by PECAD. PECAD's final time step requirement is a matter of
ongoing discussion. The consistency of standard MODIS NDVI products with
the AVHRR NDVI products has been established.

The Global Reservoir and Lake Monitor exceeded PECAD delivery
requirement with a latency of less than 6 days and a time step of 10
days. Validation with water level gauges showed that the 10 cm PECAD
surface water level relative accuracy requirement was being met in the
Great Lakes. Results were not as good over smaller water bodies, which
can be partially explained by less ideal reference data. While not all
lakes appeared to meet the stated FAS accuracy requirement, perhaps the
accuracy requirement should be revisited given that even "noisier" data
often seemed to satisfy underlying information needs.

The amount of data that is not being processed by the standard Jason-1
algorithms was established as a matter of concern in 2004 and continues
to be a problem. Coverage improved from 2004 to 2005, so that lakes and
reservoirs being monitored are approaching quasi-global status, but the
overwhelming majority of the added lakes have had no valid heights
estimated because of the problems with the standard Jason-1 algorithms.
Data dropouts should be reduced. At present, the source of science data
for the GRLM is the "Archivage, Validation et Interprétation des données
des Satellites Océanographiques" (AVISO) data center for CNES. If AVISO
is unable to improve the data stream, perhaps a separate channel for
providing appropriate sea surface height data should be engaged at the
Jet Propulsion Laboratory's Physical Oceanography Distributed Active
Archive Center. If no action is forthcoming, then the need should be
elevated to higher levels within NASA.

One overall recommendation resulting from the verification and
validation process is that as NASA-supported enhancements become
operational, regularly generated metrics should be introduced to track
certain types of performance, such as delivery and coverage. While
certain facets of decision support will require more careful analysis,
the more routine aspects that relate to satisfaction of customer
requirements should be amenable to simple self-monitoring that could be
publicly available as product metadata.

# References

[]{#Bahill1998 .anchor}Bahill, A.T., and B. Gissing, 1998. Re-evaluating
systems engineering concepts using systems thinking. *IEEE Transactions
on Systems, Man and Cybernetics, Part C: Applications and Reviews,*
28(4):516--527, doi:10.1109/5326.725338.

[]{#Birkett1995 .anchor}Birkett, C.M., 1995. The contribution of
TOPEX/POSEIDON to the global monitoring of climatically sensitive lakes.
*Journal of Geophysical Research,* 100(C12):25,179--25,204.

[]{#Bonnefond2003 .anchor}Bonnefond, P., P. Exertier, O. Laurain, Y.
Ménard, A. Orsoni, G. Jan, and E. Jeansou, 2003. Absolute calibration of
Jason‑1 and TOPEX/Poseidon altimeters in Corsica. *Marine Geodesy,*
Special Issue on Jason‑1 Calibration/Validation, Part 1,
26(3-4):261--284.

[]{#Chambers2003 .anchor}Chambers, D.P., J.C. Ries, and T.J. Urban,
2003. Calibration and verification of Jason‑1 using global along-track
residuals with TOPEX. *Marine Geodesy,* Special Issue on Jason‑1
Calibration/Validation, Part 1, 26(3-4):305--317.

[]{#Doorn2004 .anchor}Doorn, B., 2004. *FAS Evaluation of Products and
Expectations for FY05*. Presentation for NASA/USDA-FAS project review at
NASA Stennis Space Center, 30 June 2004.

[]{#Gallo2004 .anchor}Gallo, K., L. Ji, B. Reed, J. Dwyer, and J.
Eidenshink, 2004. Comparison of MODIS and AVHRR 16-day normalized
difference vegetation index composite data. *Geophysical Research
Letters*, 31: L07502, doi:10.1029/2003GL019385.

[]{#Gao2003 .anchor}Gao, X., A.R. Huete, and K. Didan, 2003. Multisensor
comparisons and validation of MODIS vegetation indices at the semiarid
Jornada experimental range. *IEEE Transactions on Geoscience and Remote
Sensing*, 41(10):2368--2381, doi:0.1109/TGRS.2003.813840.

[]{#Haines2003 .anchor}Haines, B.J., D. Dong, G.H. Born, and S.K. Gill,
2003. The Harvest experiment: Monitoring Jason-1 and TOPEX/POSEIDON from
a California offshore platform. *Marine Geodesy,* Special Issue on
Jason‑1 Calibration/Validation, Part 1, 26(3-4):239--259.

[]{#Haithcoat2005 .anchor}Haithcoat, T., V. Likholetov, V. Kaupp, B.
Doorn, D. Tralli, W. van Leeuwen, S. Drake, and C. Hutchinson, 2005.
Benchmarking the performance of a decision support system. In
*Proceedings of* *31^st^ International Symposium on Remote Sensing of
Environment: Global Monitoring for Sustainability and Security*,
International Society for Photogrammetry and Remote Sensing et al., St.
Petersburg, Russian Federation, June 20--24, 2005; available online at
<http://www.isprs.org/publications/related/ISRSE/html/papers/1031.pdf>
(accessed January 1, 2006).

[]{#HueteEtAl2002 .anchor}Huete, A., K. Didan, T. Miura, E.P. Rodriguez,
X. Gao, and L.G. Ferreira, 2002. Overview of the radiometric and
biophysical performance of the MODIS vegetation indices. In "The
Moderate Resolution Imaging Spectroradiometer (MODIS): A new generation
of land surface monitoring," ed. C. Justice and J. Townshend, special
issue, *Remote Sensing of Environment*, 83(1-2):195--213.

[]{#Justice2005 .anchor}Justice, C., 2005. *Support of Application of
NASA EOS MODIS Data by USDA*, annual report of the Global Agriculture
Monitoring Project (a joint project between NASA, USDA/FAS, and
University of Maryland) for the period from April 2004 through March
2005, 17 p.

[]{#JusticeEtAl2002 .anchor}Justice, C., A. Anyamba, C. Reynolds, J.
Owens, S. Liang, and C.J. Tucker, 2002. *Support of Application of NASA
EOS MODIS Data by USDA Foreign Agriculture Service*. Proposal submitted
in response to CAN-02-OES-01: The Earth Science REASoN -- Research,
Education and Applications Solutions Network, 47 pp.

[]{#Menard2003 .anchor}Ménard, Y., L.‑L. Fu, P. Escudier, F. Parisot, J.
Perbos, P. Vincent, S. Desai, B. Haines, and G. Kunstmann, 2003. The
Jason‑1 mission. *Marine Geodesy,* Special Issue on Jason‑1
Calibration/Validation, Part 1, 26(3-4):131--146.

[]{#NASA2004a .anchor}NASA, 2004a. *Decision Support Tools Evaluation
Report for FAS/PECAD, Version 2.0*. Report commissioned by NASA/SSC
Earth Science Applications Directorate, Agricultural Efficiency
Integrated Product Team, 76 p. Available online at
[http://aiwg.gsfc.nasa.gov/esappdocs/evalreports/
pecad_dss_eval_rpt.doc](http://aiwg.gsfc.nasa.gov/esappdocs/evalreports/pecad_dss_eval_rpt.doc)
(accessed July 24, 2005).

[]{#NASA2004b .anchor}NASA, 2004b. *PECAD's Global Reservoir and Lake
Monitor: A Systems Engineering Report*. Report commissioned by NASA/SSC
Applied Science Directorate, Agricultural Efficiency Integrated Product
Team, 64 p. Available online at
[http://aiwg.gsfc.nasa.gov/esappdocs/benchmarks/
PECADGRLM.pdf](http://aiwg.gsfc.nasa.gov/esappdocs/benchmarks/PECADGRLM.pdf)
(accessed July 24, 2005).

[]{#Rojas2002 .anchor}Rojas, F., R.A. Schowengerdt, and S.F. Biggar,
2002. Early results on the characterization of the Terra MODIS spatial
response. In "The Moderate Resolution Imaging Spectroradiometer (MODIS):
A new generation of land surface monitoring," ed. C. Justice and
J. Townshend, special issue, *Remote Sensing of Environment*,
83(1-2):50--61.

[]{#Shum2003 .anchor}Shum, C., Y. Yi, K. Cheng, C. Kuo, A. Braun, S.
Calmant, and D. Chambers, 2003. Calibration of JASON-1 altimeter over
Lake Erie. *Marine Geodesy,* Special Issue on Jason‑1
Calibration/Validation, Part 1, 26(3-4):335--354.

[]{#USDA2005 .anchor}USDA, 2005. Crop Explorer. Production Estimates and
Crop Assessment Division, Foreign Agricultural Service.
<http://www.pecad.fas.usda.gov/cropexplorer/> (accessed January 1,
2006).

[]{#USDA2004 .anchor}USDA, 2004. *Global Reservoir and Lake Monitor.*
Foreign Agricultural Service, Production Estimates and Crop Assessment
Division, <http://www.pecad.fas.usda.gov/cropexplorer/global_reservoir/>
(accessed August 9, 2005).

[]{#USGS2005 .anchor}USGS, 2005. MODIS/Terra Vegetation Cover Conversion
32-Day L3 Global 250m ISIN Grid (MOD44A). Land Processes Distributed
Active Archive Center. National Center for Earth Resources Observation
and Science. <http://edcdaac.usgs.gov/modis/mod44a.asp>, accessed
January 1, 2006.

[]{#vanLeeuwen2005 .anchor}van Leeuwen, W., S. Drake, C. Hutchinson, B.
Doorn, D. Tralli, V. Kaupp, T. Haithcoat, and V. Likholetov, 2005.
Assimilating NASA data into a crop production estimation system: Risk
management. In *Proceedings of* *31^st^ International Symposium on
Remote Sensing of Environment: Global Monitoring for Sustainability and
Security*, International Society for Photogrammetry and Remote Sensing
et al., St. Petersburg, Russian Federation, June 20--24, 2005; available
online at
<http://www.isprs.org/publications/related/ISRSE/html/papers/229.pdf>
(accessed January 1, 2006).

[]{#Vermote2002 .anchor}Vermote, E.F., N.Z. El Saleous, and C.O.
Justice, 2002. Atmospheric correction of MODIS data in the visible to
middle infrared: First results. In "The Moderate Resolution Imaging
Spectroradiometer (MODIS): A new generation of land surface monitoring,"
ed. C. Justice and J. Townshend, special issue, *Remote Sensing of
Environment*, 83(1-2):97--111.

[]{#Watson2003 .anchor}Watson, C., R. Coleman, N. White, J. Church, and
R. Govind, 2003. Absolute calibration of TOPEX/Poseidon and Jason‑1
using GPS buoys in Bass Strait, Australia. *Marine Geodesy,* Special
Issue on Jason‑1 Calibration/Validation, Part 1, 26(3-4):285--304.

[]{#Wolfe2002 .anchor}Wolfe, R.E., M. Nishihama, A.J. Fleig, J.A.
Kuyper, D.P. Roy, J.C. Storey, and F.S. Patt, 2002. Achieving sub-pixel
geolocation accuracy in support of MODIS land science. In "The Moderate
Resolution Imaging Spectroradiometer (MODIS): A new generation of land
surface monitoring," ed. C. Justice and J. Townshend, special issue,
*Remote Sensing of Environment*, 83(1-2):31--49.

# Acronyms

AVHRR Advanced Very High Resolution Radiometer

AVISO Archivage, Validation et Interprétation des données des Satellites
Océanographiques (French data archive)

CADRE Crop Assessment Data Retrieval and Evaluation

CNES Centre National d'Etudes Spatials (the French space agency)

DORIS Doppler Orbitography and Radiopositioning Integrated by Satellite

DSS Decision Support System

ECMWF European Centre for Medium-Range Weather Forecasts

FAS Foreign Agricultural Service

GDR Geophysical Data Record

GIMMS Global Inventory Monitoring and Modeling Studies

GLAM Global Agricultural Monitoring

GRLM Global Reservoir and Lake Monitor

GSD ground sample distance

GSFC Goddard Space Flight Center

IGDR Interim Geophysical Data Record

MODAPS MODIS Data Production System

MODIS Moderate Resolution Imaging Spectroradiometer

NDVI Normalized Difference Vegetation Index

PECAD Production Estimates and Crop Assessment Division

RGB Red Green Blue

TMR TOPEX Microwave Radiometer

USDA U.S. Department of Agriculture

USDA-FAS USDA Foreign Agricultural Service

USGS U.S. Geological Survey

V&V Verification and Validation