# Presentation: 186507

## Benchmarking USDA/FAS/PECAD DSS National Applications DSS Program Support

**Benchmarking ****USDA/FAS/PECAD DSS ****National Applications DSS Program Support**

**Chuck Hutchinson**
**Willem van Leeuwen**
**Sam Drake**
**University of Arizona **
**	****NAG5 – 12484**

**   ****David Tralli (JPL)**

**Verne Kaupp**
**Tim Haithcoat**
**Vlad Likholetov**
**University of Missouri NAG5 – 12482**

## Slide 2

**Notes:**
USDA’s idea of clearly explaining FAS DSS

## Overview of PECADDecision Making Environment

**Overview of PECAD****Decision Making Environment**

## Benchmarking Intent

**Provide consistent information about the current characteristics of PECAD’s Decision Support System**
**Demonstrate the value of DSS enhancements **
*The outcome of benchmarking is intended to show how investments from USDA and NASA have changed PECAD’s DSS.  *
*The benchmarking process measures the delta between state 1 and state 2 by using metrics that indicate performance improvements or changes in the DSS after infusion of NASA science, data and technology (e.g. MODIS products). *

## Benchmarking Objectives

**DSS characterization**
**Interview DSS users/developers to obtain candidate metrics of performance and usability **
**Standardize assessment of DSS requirements, risks and mitigations by using the DDP tool**
**Create Benchmarking Team to establish performance criteria, and measure delta between State 1 and State 2**
**Benchmarking report**

## Benchmarking Procedure for Assimilation of Data, Science or Technology into a DSS

**Involve users and experts in all stages of the enhancement process.  **
**	****“****As is”******** “To be”    OR    State 1 ********  State 2 **
**Characterize state or baseline of the DSS.**
**Select performance and usability indicators to analyze and compare the “as is” and “to be” systems (questionnaire and DDP approach).**
**Evaluate “as is” DSS performance. **
**Plan assimilation process (transition), and formulate enhancements to the DSS. **
**Develop enhanced prototype with benchmarking partners. **
**Optimize prototype based on user and expert input and other constraints. **
**Evaluate performance of prototype (“to be”).**
**Compare performance of “as is” and “to be.”**** **

## DSS Performance and Usability Metrics: an Example

**Data **(Sources, Latency, Consistency, Frequency of use, Cost effectiveness, Enhancements, Site specific data, User tasks, Data access and interoperability)
**Analysis and Modeling Tools** (Usability, Network bottlenecks, Usefulness, Prioritization, Frequency of use, Forecast/trend accuracy)
**DSS Functionality** (Usability, Learning curve, Usefulness, Prioritization, Frequency of use, Forecast/trend accuracy, Organizational infrastructure)
**                        ****Country-,  Commodity-  and User-Specific**

## Activities to Date (PECAD DSS)

**Zeroth-order assessment (UA-UM report, 2003)**
- Interviews – Analysts, management, IT team
- Review current state
- Define requirements
- Consider upgrade plans
- Characterize DSS and document
**Benchmarking meetings (July, Oct. 2004; Feb., May 2005)**
**Development  and deployment of State 1 surveys (3 questionnaires; 2004)**
**DDP implementation (questionnaire input; 2004)**
**Analysis of results (survey and DDP) ******** Measures of improvement and change in the DSS (performance metrics; 2005)**
**Refinement and deployment of State 2 surveys and DDP (2005)**
**Benchmarking report (outline)**

## Some Questionnaire Findings

**Some Questionnaire Findings**

## Slide 10

## Slide 11

## Defect Detection and Prevention
(DDP)
as a Benchmarking Tool

**(DDP)**
**as a Benchmarking Tool**

**The DDP process aims to determine:**
-   Where we want to be (Objectives/Goals/Requirements)
-   What could get in the way (Risks/Obstacles)
-   How we will get there (Mitigations/Solutions)
**The DDP process is intended to facilitate benchmarking and risk management over the entire DSS project life cycle, beginning with architectural and advanced technology decisions all the way through operation.**

## PECAD DDP Process Implementation – Interviews, Meetings, Surveys

**  ****Capturing**
**Objectives/Requirements**
**Risks/Obstacles**
**Mitigations/Solutions**
**Risk Impacts**
**Mitigation Effectiveness**
**Delta**

## Key Questions

**Identify all goals, requirements and objectives of PECAD for State 1  (... State 2).**
**What are the risk factors that make these goals less attainable or prevent a successful DSS (e.g. lack of data, user-friendliness, usefulness)?**
**What mitigation factors reduce the risks and fulfill PECAD requirements? **
**Rate the importance of the PECAD requirements.**
**What are the impacts of the risks on the requirements? **
**How effective are the mitigating factors in reducing or preventing the risks?**

## PECAD DDP Objectives

**Provide accurate monthly production numbers by commodity and country**
**Crop area estimates**
**Early in the crop growing season**
**Late in the crop growing season**
**Crop yield estimates**
**Pre-planting stage**
**Planting stage**
**Vegetative growth stage**
**Reproductive stage**
**Grain filling stage**
**Provide the supporting evidence at lock-up**
**Extend the external Crop Explorer; provide automated analysis products to external customers         **
**Provide an ad hoc analysis capability, including US crop condition assessments**

## PECAD DDP Baseline Risks

**Lack of continuous access to field-level multispectral observations (e.g. Landsat-like)**
**Lack of continuous access to national level multispectral observations (e.g. AVHRR-like)**
**Untimely receipt of remote sensing data**
**Inadequate accuracy/reliability of weather data**
**Significant obstacles (i.e. Lack of facilities and IT resources, training, experience, etc ) to operational use of RS within PECAD**
**Lack of baseline or historical information for comparative RS analysis **
**Inadequate crop models**
**Lack of integration of tools and data (Landsat-like)**
**Lack of Landsat-like data**
**Lack of tools**
**Lack of integration of tools and data (AVHRR-like)**
**Lack of AVHRR-like data**
**Lack of tools **
**Lack of ancillary data (in-country data)**

## PECAD DDP Mitigations

**Land remote sensing**
**MODIS products**
**MODIS Rapid Response Products**
**MODIS Vegetation index product (NDVI and EVI)**
**Surface reflectance**
**Crop Mask**
**ASTER**
**SPOT-VEGETATION product**
**AVHRR products**
**AVHRR 1 km (daily)**
**AVHRR 8 km NDVI composite **
**Reservoir heights (Topex/Jason)**
**NOAA/SSMI (surface wetness)**
**Multiple sources of Atmospheric remote sensing and weather data**
**TRMM**
**FAS attaché ground truth**
**External data sources (incl. Reuters, economic, internet, etc.)**
**PECAD-obtained ground truth in field**
**Having a trained analyst for each region**
**Availability of a DBMS (continuously being maintained and extended)**
**CADRE DBMS as an aid in decision making**
**Crop Explorer as a tool in aiding decision making**
**Current suite of tools for early warning of crop conditions**
**Improved crop yield and soil moisture models**
**Crop stage models**
**Crop yield models**
**Soil moisture model**
**Adopting new technology**

** ****Models     Tools   Resources                  RS   DATA**

## DDP GUI

**DDP GUI**

**Notes:**
Generate report capability
Etc

## DDP Online Survey

**DDP Online Survey**

**What is the impact.../effectiveness... ?**
***None***
***Low***
***Medium***
***High***
***Unknown***

## Quantification of Survey Responsesto provide input numbers for the DDP Tool

*None = 0.1*
*Low = 1*
*Medium = 5*
*High = 9*
*Unknown – disregard -**
- Geometric Mean (GEOMEAN)

## Impact Matrix

**Impact Matrix**

## Effectiveness Matrix

**Effectiveness Matrix**

## Impact Matrix – State 2

**Impact Matrix – State 2**

## Effectiveness Matrix – State 2

##        Preliminary DDP Resultsfor PECAD DSS

**State 0**
**State 1**
**State 2**

## State 0

**State 0 Risk Balance**
**Objectives Attainment = 14.06**

**State 0**

**Notes:**
State 0 Risk Balance

## State 1

**State 1 Risk Balance**
**Objectives Attainment = 21.40**

**State 1**

**Notes:**
State 1 Risk Balance

## State 2

**(Pre) State 2 Risk Balance**
**Objectives Attainment = 23.83 **

**State 2**

**Notes:**
State 2 Risk Balance

## Attainment of Objectives forDifferent Mitigation Categories

**Mitigation categories**

**Notes:**
Total attainment is less than the sum of the categorical attainments.
Diminishing return/benefit for each added mitigations.

## PECAD DSS Risk Balance Scenarios

**Notes:**
Reduced risks by having trained analysts for each region.
The analysts mitigate several risks (1 thru 7)

## Risk balance without any mitigations

**Risk balance without any mitigations.**
**Attainment of Objectives = 0.37 out of 25**

**Notes:**
Risk balance without any mitigations

## Reduced risks by having trained analysts for each region.

**Risks reduced by having a trained analyst for each region.**
**The analysts mitigate several risks (1 thru 7).**
**Attainment of Objectives = 0.89 out of 25**

**Notes:**
Reduced risks by having trained analysts for each region.
The analysts mitigate several risks (1 thru 7)

## Reduced risks (i.e. 3, 10, 13) by having MODIS Rapid Response Products

**Risks (3, 10, 13) reduced by having MODIS Rapid Response Products.**
**Risk 3 (Timely RS data) is “completely” mitigated by MODIS Rapid Response. **

**Attainment of Objectives = 2.86 out of 25**

**Notes:**
Reduced risks (i.e. 3, 10, 13) by having MODIS Rapid Response Products
Risk 3 (Timely RS data) is mitigated completely by MODIS Rapid Response

## Risk Balance (Synergistic Mitigations)

- Mitigation 3  Attainment = 0.47
- Mitigation 4  Attainment = 0.89
- Mitigations 3 & 4  Attainment = 2.86

**Risk balance after Mitigations (3)“MODIS Rapid Response Products” and (4)“Having a trained analyst for each region” were applied. **
**Impacted risks were reduced by the yellow and green amounts respectively.**

**Notes:**
Mitigations can also work synergistically as the benefit (attainment of objectives) of two single mitigations is less than the benefit of the pair of mitigations used simultaneously. For example the attainment of objectives by mitigating risk with trained analysts and MODIS rapid response data is 2.86 out of 25, while the attainment for the single mitigations are 0.89 and 0.47 respectively. The above Figure shows the corresponding risk balance after applying the two mitigation factors. Risks 1 through 7 are reduced by mitigating them with having the trained analysts. Risks 3, 10, 13 are significantly reduced by implementing the MODIS rapid response data.
Risk balance after Mitigations “MODIS Rapid Response Products” and “Having a trained analyst for each region” were applied to reduce the risk with the yellow and green amounts respectively. 

## Performance Metrics

- Attainment of objectives for State 1 and State 2
- Residual risk profiles as a result of mitigations
- Benefit = Objectives Attainment or Risk Reduction?
- Objectives can be more than 100% at-risk
- Objectives attainment may be the preferred measure when evaluating an entire set of Mitigations
*Cost?*
*Optimization given relative weightings *
*Search for greatest benefit, least cost, or a hybrid of both?*
*Sensitivity? *
*Not a performance metric*
*Guideline for assimilation and transition*

**Derived from the DDP Process**

**Derived from the survey Questionnaires**

- Usage statistics
- Usefulness, Satisfaction

## Delta Analysis for Performance Metrics

- 	  Δ = State 1 – State 2
- Change in performance metrics shows
- the value of enhancements to the DSS

## PECAD-DDP Lessons Learned

**Prevent ambiguity and overlap in risks and mitigation factors**
- Inadequate crop models
- Adopting new technology
**Dynamic DSS environment and state are “captured” with DDP’s “when” function, but remember that responses are representing different situations at different times **
**Information Redundancy and Convergence of Evidence approach are weakly handled by DDP**
- Each additional mitigation shows only a marginal benefit in the DDP tool
- Increasing access to redundant data sources is of higher benefit to the analysts (providing confidence) than the DDP tool would indicate
**Integration as a Risk**
- Lack of Data    “OR”
- Lack of a Tool
**Consistency across respondents (variability, all high or low scores) through better framing of questions **
**Facilitates discussions and learning among all involved**

## Benchmarking Report - Outline

**Executive summary**
**Introduction (from guideline document)**
**Scope, goals, Objectives**
**FAS/PECAD DSS description (Characterization report AZ/UM; state 0)**
**FAS/PECAD mission**
**DSS framework structured, unstructured etc**
**NASA products (data, science, technology etc)**
**Approach **
**Introduction (Interviews, survey, DDP)**
**Interviews**
**Questionnaires (IT, analysts, managers, DDP)**
**DDP tool**
***Performance metrics***** (Qs, DDP, webstats, after action reports)**
**Results**
**State 1 (define)**
**Survey results**
**DDP**
**State 2 (define)**
**Survey results**
**DDP**
**Delta State 1 – State 2**
**Supporting measures (Webstats, After action reports, Interviews)**
**Lessons learned**
**Future Potential activities**
**Conclusions**
**Appendices**
**Survey Questionnaires (Analysts, IT, Managers)**
**DDP Questionnaires (Analysts)**
**Overall Summary DDP report**

## PECAD Benchmarking

**Enhancements**

**State 0  (n-1)**

**State 1 (n)**

**Metrics**
**Delta**

**PECAD Benchmarking**

**Development plans**