# Presentation: 403538

## Evaluation and Enhancement of Decision Support Tools – FY2003 Report

**Vicki Zanoni**
*Earth Science Applications Directorate*

## Initial Systems Engineering Process

**Define **
**Requirements**** & ****Specifications**

**DSS **
**Selection**

**Investigate **
**Alternative **
**NASA Inputs**

**Design**
**& **
**Implement**

**Benchmark**
**(Baseline**
**&**
**Assess **
**Performance)**

**Enhanced**
**DSS**

**Refine**

**Refine**

**Refine**

**Refine**

**Evaluation**

**V&V**

**Benchmarking**

**Verify**
**&**
**Validate**

**Refine**

**Use of *****Systems Engineering***** principles leads to scalable, systemic, and sustainable solutions and processes, which in turn contribute to the success of the mission, goals, and objectives of each National Application.**
***Evaluation***** phase involves understanding the requirements for and technical feasibility of Earth science and remote sensing tools and methods for addressing DSS needs.**

***Verification and Validation***** (*****V&V*****)**** phase includes measuring the performance characteristics of data, information, technologies, and/or methods, and assessing the ability of these tools to meet the requirements of the DSS.**
**In the *****Benchmarking *****phase, the adoption of NASA inputs within an operational DSS and the resulting impacts and outcomes are documented.**

## DSS Preliminary Evaluation Steps

_**First look Evaluation**_
- National Application Relevance
- DSS / National Application Synergy
- DSS Inputs & Outputs (technical requirements)
- Users
- Possible NASA Contributions (further define roles of NASA Centers, etc.)
- Define Next Steps (Federal Agency Visit)

_**DSS Baseline – One Pager **_
_**Development for all DSSs**_
- Short Description of Function
- Federal Agency Owner
- National Application
- POC Information
- Operational Date
- Potential Use of NASA Data (cursory look)

_**DSS Identification**_
- NASA HQ
- Input to NASA HQ by others

*Web Searches*
*Literature Searches*

*Individual Contacts*
*Phone & Email*

_**ESE Inventory**_
-  Missions
-  Sensors
-  Products
-  Models

*Additional Phone Follow-up Conversations*

_**DSS In-Depth Evaluation**_
- Meetings with DSS owner
- Perform detailed requirements definition with DSS owner

## In-Depth Evaluation of DSSs

_**Reverse Engineering Process**_
-  Develop specifications that can improve DSS solution
-  Identify important measurements & models
-  Assess existing assets
-  What missions will provide measurements?
-  What models or computational technologies are available?
-  What information systems are necessary?
-  Identify gaps for ESTO potential investment?

_**DSS Mission**_
-  Operational Requirements
-  NASA Partnership Criteria
-  Stake Holders
-  Funding Profile
-  Related National Application
-  Relevant NASA Capability

- Performed in partnership with DSS owner

**Each complete evaluation process is documented with a report that defines the baseline DSS, describes requirements for improvements, and identifies NASA products and models to be incorporated into DSS:**
** ****Background description of DSS: What is it? How does it work? Who uses it?**
** ****Detailed description of DSS: Architecture, modules, inputs, outputs, technical details, etc.**
** ****Possible NASA inputs and description of how inputs would be used:**
** ****Discussion of all the possible measurements and predictions, **
**including those for which specific implementation plans exist**
** ****Planned data flow charts when available**
** ****Identified NASA technology gaps in meeting DSS needs**
** ****Recommendations / Next Steps:**
** ****Preliminary (conceptual) plans for implementation, V&V, and benchmarking of the DSS**

## Reverse Engineering Paradigm

- 120 cities

- Flood maps

- FEMA

- USGS

- Ecosystems

- Models

- EPA

- CDC

- Leverage agency research

**Decision**
**Support Tools**

**Assessments**

**Workforce Development**
**Outreach**

- Develop specifications that can improve socioeconomic benefits.

- What measurements are important and why?

- What model outputs are important and why?

- Reverse engr

- Reverse engr

- Translate specs into results

**What missions or ground systems will provide these measurements?**
**What models or computational technologies are available?**
**What information systems are necessary?**
**If not available, specify need to ESTO.**

- Assess against existing assets

- Process flow

## Design & Implementation, V&V, and Benchmarking

_**Benchmark**_
-  Measure / Quantify impact of result of NASA input
-  Cost to operate / cost savings
-  Time to produce results
-  Accuracy, quality, and reproducibility of results
-  Socioeconomic impact
-  Fills a previously unmet need

_**V&V**_
-  Verify Results – compare to technical specifications
-  In situ measurements
-  Cross – comparisons to other DSSs, models, other data sources
-  Validate – Does it meet functional desires?
-  Interaction with end users

_**Design & Implementaton Phase**_
-  Integrate NASA capabilities into DSS
-  Generate DSS products, solutions, outputs

## V&V Hierarchy

**Intrinsic system and data performance characterization**
**Interface with EOS cal/val community**
**Modification of NASA products**
**Accelerated product validation**

**Verify DSS technical requirements**
**Validate DSS performance**

**Interface with modelers**
**Scaling and sensitivity analyses**
**Modification of models**
**Model verification and validation**

## Lessons Learned

**Successful use of NASA observations and predictions within DSSs often evolves from existing science research performed in collaboration with agency partners.**
***Recommendation*****: Identify and continue supporting existing NASA projects that produce valuable results; Mine successes from organizations already using NASA data.**
**The greatest opportunities for integrating NASA data into DSSs are often found with DSSs that are in early development stages.**
***Recommendation*****: Seek opportunities to work with DSSs in their early stages. **

## Lessons Learned (cont’d.)

**3.  Many of the currently identified DSSs do not fit the Applications Program definition of DSS.**
***Recommendation*****: In addition to the term “DSS,” use terms such as “decision support tools,” “decision support environment,” “decision making processes,” and others to describe more accurately the tool being evaluated. Ensure that definitions of the new descriptive terms are documented for use with similar procedures in the future. Additionally, when the evaluation team encounters a tool that does not support decision-making or fit the definition of a DSS, it should seek guidance on NASA’s interest in participation.**

## Lessons Learned (cont.)

**It is often difficult to gain access to the appropriate persons to understand DSS operation and technical requirements.**
***Recommendation*****: Empower the NASA project teams to establish personal contacts for accessing detailed DSS information****.**
**Leveraging opportunities exist between many DSSs and applications.**
***Recommendation*****: Continue to have an overall systems engineering team that provides support to all applications so that NASA can identify opportunities to exploit synergy as a third-party broker between related DSS and application partners. **
**The process to bring R&D technology and products to operational use requires significant investments of time and funds.**
***Recommendation*****: Create rapid prototype product capability that can develop new application-specific products to be generated by direct broadcast receiving stations or the current Distributed Active Archive Centers (DAACs).**
***Recommendation*****: Provide seed funding for applications research to move NASA research outputs into operational environments.**

## Lessons Learned (cont.)

**DSS technical requirements and specifications are difficult to identify.**
***Recommendation*****: NASA must take the lead in working with other agencies to develop specifications for improvements / enhancements or for development of new DSSs.**
**8.  NASA scientists in residence at operational agencies (or other agency scientists resident at a NASA center) can foster the incorporation of NASA inputs into operational environments**
***Recommendation*****: NASA should leverage from and seek opportunities for detail assignments with partner agencies to further accelerate applications research and operational use of NASA observations and model predictions.**

## Lessons Learned (cont.)

**9. ****DSS owners and DSS users are not always the same entity. Frequently the user / analyst involvement will impact the DSS's operation in ways the owner / developer will not have anticipated.**
***Recommendation*****: Establish direct links with users of the DSS so effective and truly representative benchmarking can be done. **
**10. NASA ESE Applications Program definitions for key terms, such as systems engineering, evaluation, assimilation, and benchmarking, are not always consistent with commonly accepted use of these terms.**
***Recommendation*****: NASA should develop a simplified set of key terms and definitions that are consistent with commonly accepted (i.e., outside of NASA) uses of such terms.**

## Technology Gaps

- _System Redundancy_: Lack of redundancy in crucial data sources can be a critical issue for operational agencies using NASA data streams.
- _System and Product Continuity_: Incorporation of NASA inputs into an operational environment will also depend on the long-term availability of that input.
- _Data Size and Format_: The HDF-EOS data format is not easily incorporated into many operational environments.
- _Thermal IR_: Many operational applications use thermal infrared (TIR) or mid-wave infrared (MWIR) data; however, there is currently no plan to continue TIR measurements from space.
- _System Specifications_: Application needs should be taken into account when specifications for a new mission are developed.
- _Product Timescales_: Many NASA data products are produced on timescales that are not consistent with the needs of a given DSS.

## Proposed Systematic DSS Selection Process

**Stop**

**Stop**

**4. Meet **
**PURSUE Criteria?**

**1. Identify Potential **
**DSS Assimilation **
**Opportunities from the **
**Opportunity Pool**

**2. Meet **
**INTEREST****Criteria?**

**YES**

**NO**

**9. Begin DSS**
**Assimilation**
**Process**

**5. Develop NASA **
**Pre-Partnering/Assimilation **
**Plan***

**6. Meet **
**with DSS Owners/**
**Stakeholders**

**NO**

**YES**

**Start**

**8. Develop Joint**
**Partnering/Assimilation **
**Plan and “Contract”***

**YES**

**NO**

**Stop**

**3. Identify DSS **
**Owner  Agency POC;**
**Collect Additional Data**

**7.  Partnership?**

- *not necessarily formal documents

## DSS Opportunity Identification

**Potential Opportunity**
**Pool**

**Other**
**Government**
**Agencies**

**NASA**
**Headquarters**
**and**
**NASA ESA**

**Other Sources**
**Internal and/or**
**External**

**Industry **
**Associations**

**Private**
**Sector**

**Scientific**
**Studies **
**& Analyses**

**Identify Best DSS**
**Opportunities to**
**Pursue**

**Selection**
**Process**

## Draft Selection Criteria

- Interest Criteria
- Fit DSS / DST definition
- Opportunity for partnership
- DSS Characteristics
- Current State (planning / prototype, operational, mature)
- Mission, Goals, Objectives (clear)
- Operational concept (feasible)
- Required outputs (known and related to mission, goals, objectives)
- Program plan (achievable)
- Appropriate for NASA
- Compatible with NASA Mission
- Fits NASA investment portfolio
- National applications-related
- NASA technology meets DSS requirements (DSS enhancement potential - Includes ARL)
- NASA has science, data, technology and/or models that match DSS output requirements
- NASA application maturity / readiness levels meet DSS timelines
- Cost of Participation to NASA
- Time
- Dollars
- Offers Measurable Outcomes
- Barriers

- Pursue Criteria
- Fits NASA program (amplifies and refines earlier analyses, re: Appropriateness and program considerations)
- Funding/Fiscal feasibility
- NASA Budget (Code YO)
- DSS Budget (DSS Owner)
- Stakeholders (Who is involved in DSS decisions beyond owners? What are the politics involved?)
- NASA capability to meet DSS requirements
- Probability of success
- DSS
- NASA science, data, technology and/or models for assimilation
- NASA ARL meets DSS program plan
- Partnering strategy is feasible
- Dual-use potential (Is value for the proposed NASA science, data, technology and/or models in more than one DSS?)
- Potential socio-economic value

## Application Readiness Levels

| DSS maturity and NASA product assimilation matrix with notional time frames. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | NASA Product Readiness for Assimilation |  |  |  |
|  |  | A.
Not Planned | B.
Planned | C.
Current;
Not Easily Assimilated | D.
Current;
Easily Assimilated |
| DSS Maturity Level | 1. Conceptual | >6 Years | 3-6 Years | 3-6 Years | 3-6 Years |
|  | 2. Design | >6 Years | 3-6 Years | 3-6 Years | 1-3 Years |
|  | 3. Development | >6 Years | 3-6 Years | 1-3 Years | 0-1 Years |
|  | 4. Test/Demonstrate | >6 Years | 3-6 Years | 1-3 Years | 0-1 Years |
|  | 5. Operational | >6 Years | 1-3 Years | 1-3 Years | 0-1 Years |

## Modified Systems Engineering Process

- Selection

- Investigate Alternatives

**DSS Opportunity Pool **
**(one-pagers)**

**First-Look **
**Assessment & ID Possible NASA Inputs**

**Meet w/Partner to Explore Partnership Concept**

**Go /**
**No Go**

**Define Baseline DSS**

**Define Requirements & Specifications for Enhancement**

**Design**

**Implement**

**V&V**

**Benchmark**

**Design**

**Implement**

**V&V**

**Benchmark**

- Heavy NASA effort / Little to no Partner effort

- Heavy Partner effort / Little to no NASA effort

- Comparable NASA and Partner efforts

- Color Legend:

**Go /**
**No Go**

## Back up

## Processes Defined by NASA HQ

_**Evaluation**_ – Identify decision support tools (assessments and DSSs) that have been developed by Federal agencies and other partners with a priority to citizens of our nation and can be enhanced by NASA ESE results. Develop the specifications for how the candidate DSS can be augmented by assimilating NASA ESE observations and predictions.
_**Verification**_ – A life cycle process to ensure the products being developed meet the stated specifications (functional, design, and performance).
_**Validation**_ – A process to ensure the completed products (software, algorithm, model) effectively serve the functional requirements.
_**Benchmark**_ – A standard by which a product can be measured or judged (i.e., How did the DSS that assimilated NASA measurements compare in its operation, function, and performance to the earlier version?). The benchmarking process is required to support adoption of innovative solutions into operational environments that affect life and property.

## Other NASA HQ Definitions

_**Decision Support System**_ – a computer based information-processing system for scenario optimization through multi-parametric analysis. A DSS utilizes a knowledge base of information with a problem solving strategy that may routinely assimilate measurements and/or model predictions in support of the decision making process. The DSS provides an interface to facilitate human inputs and to convey outputs. Outputs from a DSS would typically be used for making decisions at the local level and outputs from multiple DSSs may be used in establishing policy.** **
_**Decision Support Tools**_** **– a suite of solutions owned by NASA partners that are used in a variety of problem domains for decision and policy making. These solutions could include assessments, decision support systems, decision support calendars, etc.
_**Reverse Engineering**_ – the process through which NASA works with a partner to develop performance specifications for enhancing the partner’s decision support tool by assimilating NASA ESE observations and predictions. These specifications are then translated into specific observations, predictions, and technologies and matched against existing NASA ESE results.