# Presentation: 075425

## Proposed Activity to support Management of National ApplicationsPART performance improvementCompetitive Sourcing

**1 April 2004**
**Prepared by**
**DEVELOP Team**
**&**
**Greg Stover**

## Today’s Discussion

**Executive Summary**
**Issue Identification**
**Background Research**
**Go Ahead Recommendations**
**Discussion**

## Executive Summary

**PART highlighted several areas where OMB perceived management of National Applications needs improvement **
**Review of other R&D program PART assessments indicate OMB accepted evidence of performance in areas we were not rated high.**
**AIWG raised issue program evaluation and entry to program**
**Indicators suggest the need for an open, well understood, and accepted management process.**
**Not a one size fits all approach but an understood and consistently applied approach.**

## PART Areas and Scores

**Program Purpose & Design****	****	****100**
**Strategic Planning****	****	****	****	****90**
**Program Management****	****	****	****64**
**Program Results****	****	****	****	****45**

## Issues from last AIWG Meeting

**1.  Center roles (and capabilities)**
**	****- Program vs. project**
**	****- Expertise and interfaces (who?)**
**	****	****- system engineering (SSC as a resource)**
**	****	****- technical interface**
**	****- Collaboration vs. competition**
**	****- Better communications cross-center, to & from HQ, and otherwise**
** **
**2.  Defining processes to be used**
**	****- Reports, other deliverables**
**	****	****- Outcomes, products should be recognizable, understandable, flexible**
**	****- Need a catalog system, a central retrieval repository**
**- Common processes for consistency, but flexible**
**- Define minimum required documents**
**- This discussion (about processes) includes implementation, not just documentation**
**- Concern expressed about imposing inappropriate standards (e.g. 7120.5b)**
**- Where do program planning & analysis activities now go?**

## Executive Summary cont.

**Propose to develop, capture, and implement the process(es) that**
**Address OMB perceptions of areas needing improvement**
**Strengthen management of National Applications**
**Allow for decisions regarding program element efficiencies and effectiveness**
**Provide tools to accommodate varied portfolio funding mechanisms**
**Provide transparency into management processes**
**Allow for accountability and public trust**

## Executive Summary cont.

**Approach**
**Identify each perceived area of deficiency in the PART and issues raised internally by community of practice**
**From the PART, perform analysis of successful R&D programs with YES answers where we were deficient and examine best practices**
**From AIWG, address program management concerns**
**Examine our YES answers to continue practices**
**Create the management processes**
**Implement**
**Continuously improve process**

## Executive Summary cont.

***Scope of activity***
***Capture process used for selection of activities (Competitive Sourcing, Peer Review)***
***Capture process used for assessing and measuring performance of activities (Program Review)***
***Capture process used for redirecting or modifying activities (Program Decisions)***
***Capture continuous improvement mechanisms (Improve Effectiveness) ***
***The captured processes will not define criteria for portfolio selection. ***
***Criteria for selection and assessment established by program managers and community of practice.  ***
***This establishes the common framework that will be followed as these activities are performed.***

## Executive Summary cont.

**Milestones**
**Present Idea to National Apps leadership**
**HQ**
**AIWG**
**Perform background PART research**
**Develop and capture process**
**Implement process**
**Want to make a difference in time for FY06 budget cycle.**

## Issue Identification (partial)

**Question 2.5  Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?**
**Answer: NO**
**Question ****3.4 Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?**
**Answer: NO **
**Question 3CO1 Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?**
**Answer: NO **
**Others found in PART and captured by AIWG**

## Go Ahead Recommendations

**Engage AIWG in addressing management issues**
**Capture or create processes for improvement of perceptions and capability to manage**
**Review processes with community of practice**
**Implement**
**Continuously improve**

## Approach in Implementing Applications: General Project Approach

- Evaluation Research

- Partnership

- Prototype Development

- Operations

- Evaluation with partner agency
- Req. Development

## Approach in Implementing Applications: General Project Approach

- Evaluation Research

- Partnership

- Prototype Development

- Operations

- Formalize partnership to support collaboration

## Approach in Implementing Applications: General Project Approach

- Evaluation Research

- Partnership

- Prototype Development

- Operations

- Dataset Development
- Evaluation

## Approach in Implementing Applications: General Project Approach

***Approach in Implementing Applications: General Project Approach***

- Evaluation Research

- Partnership

- Prototype Development

- Operations

- NASA Supports with scientific expertise

- Operations transferred to partner agency

## Process for Project Activities (provides outcomes and impacts for society)

**Identification of need for NASA data**
**What missions or ground systems will provide these measurements?**
**What models or computational technologies are available?**
**What information systems are necessary?**
**If not available, assess need & priority.**

- Evaluation Research

**Process flow **

**Make contact and build partnerships with prospective DSS developers and data users in government and industry**** **

- Develop specifications that can improve socioeconomic benefits.
- Translate specs into results
- Cultivate new users.
- Upgrade capabilities and parameters as new analysis and modeling techniques are developed and improved

- Partnership Development

- Prototype Datasets

- Transfer Outputs to Operations

**Identification of proposed effort**
** ****Scope**
**Partners**
**Outcome**
**Milestones**
**Obs./Models**
**Budget**

- Proposed Effort

**Program Review**

## Discussion

**Scope Sufficient?**
**Other similar activities?**
**Approach Acceptable?**
**Community Involvement?**

## REFERENCE CHARTS

## Earth Science Applications PART Assessment 

## “NO” Evaluation Questions

**2.5-- “Do all partners ( including grantees, sub-grantees, contractors cost-sharing partners, and other govt. partners) commit to and work toward the annual and/or long term goals of the program?”**
**3.4 --“Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?”**
**3.CO1 --“Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?”**

## 3.CO3-- “Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?”
3.RD1-- “For R & D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?”
4.3-- “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year”

**3.RD1-- “For R & D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?”**
**4.3-- “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year”**

***“******NO” Evaluation Questions***

## 1.1  Is the program purpose clear?

**Explanation:  The Earth Science Applications program has a well-defined and focused purpose that ties directly to the NASA vision and Mission, and the Earth Science Enterprise (ESE) strategic plan.**
**Evidence:  The Earth Science Applications mission is “to expand and accelerate the realization of societal and economic benefits from Earth science information and technology.”  ESE works jointly with its national and international partners to develop this scientific understanding by employing space-based, air-borne and in-situ data.  The core of these data sets consists of products from ESE’s 18 orbital missions containing approximately 80 instruments to develop solutions to applications of national importance.**

## 1.2  Does the program address a specific and existing problem, interest or need?

**Explanation:  The Earth Science Applications program bridges the gap between Earth science research results and the use of observations and prediction capabilities in national and International decision support tools associated with weather, climate and natural hazards.  The program also addresses the need for Earth science education.**
**Evidence:  The following documents reinforce the benefits of using Earth system science results to serve society: (a) Review of NASA's Earth Science Enterprise Applications Program Plan (National Research Council); (b) The Science of Regional and Global Change: Putting Knowledge to Work (National Research Council); and (c) Blueprint for Change: Report from the National Conference on the Revolution in Earth and Space Science Education (National Science Foundation). National and international needs for Earth science education are outlined in the report "Revolution in Earth and Space Science Education".**

## 1.3  “Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?”

**Explanation:  The Earth Science Applications program is the Nation's only program designed to systematically benchmark uses of NASA's remote sensing data and research results into decision support systems designed to support operational agencies and organizations.**
**Evidence:  Memoranda of Agreement are executed with partners to ensure that duplication of efforts does not occur. Partnering organizations include federal agencies (FAA, USDA, USGS, DHS, FEMA, EPA, CDC, NIH, DOE, DOD, DOI), state organizations (NSGIC, ASA, AAGS) and other national and international organizations. A detailed listing and status of Earth Science Applications Program Memoranda of Agreement (MOAs) are maintained by ESE. (http://www.earth.nasa.gov/eseapps/).**

## 1.4— “Is the program design free of major flaws that would limit the program's effectiveness or efficiency?”

**Explanation:  The Earth Science Applications program architecture and plan have been vetted with a broad stakeholder community(including representatives of the public, private, policy academic, international, aerospace and science sectors) over the past two years.**
**Evidence:  The National Academy of Sciences and NASA ESE’s Earth System Science and Applications Advisory Committee reviewed and endorsed the program design presented in the Earth Science Applications Strategy.  Several external and NASA studies on Socio-economic benefits resulting from the Earth Science Applications projects estimate potential annual benefits totaling approximately $20 billion.  Specific studies include: (a) “An Estimate of NASA/ESE Power Program Benefits to the U.S. from 2002 through 2017; and (b)  The Socio-economic Benefits of Earth Science and Applications Research:Reducing the Risks and costs of Natural Disasters in the United States.**** **

## 1.5  Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program’s purpose directly?

**Explanation:  The Earth Science Applications architecture and plan have identified twelve national applications (each with a partner federal agency) that can benefit from NASA science and technology.**
**Evidence:  MOAs with partnering organizations effectively target the use of resources aimed at the benchmarking and improvement of the decision support tools owned by those organizations.  Partnering organizations that further the use of Earth Science Applications program products include federal agencies (FAA, USDA, USGS, DHS, FEMA, EPA, CDC, DOE, NIH, DO”D, DOI, state organizations (NSGIC, ASA, AAGS) and other national and international organizations.  Unintended subsidies are avoided by clarifying roles and responsibilities in Memoranda of Agreement with partners.  A detailed listing and status of Earth Science Applications of memoranda of Agreement (MOAs) are maintained by ESE. (http://www.earth.nasa.gov/eseapps/).**

## 2.1--Does the Program have a limited number of specific long-term performance measured that focus on outcomes and meaningfully reflect the purpose of the program?

**Explanation:  Outcomes reflect NASA’s responsibility for ensuring that NASA data and research is fully utilized and is translated into measurable improvement to existing products and programs.  Once NASA data has been incorporated in decision support systems of national priority, outcomes that demonstrate direct benefit to society (including measures such as lives saved due to improved hurricane forecasting and economic benefits associated with improved agricultural efficiency) are typically the responsibility of partner agencies.  As the program matures, NASA needs to continue to improve measures to address the value added of incorporating NASA data (i.e., measure quality of products versus quantity.  NASA also needs to develop metrics that better encompass outcomes resulting from the education and outreach portions of the program.**
**Evidence:  Three long-term performance measures in the Measures tab.**

## 2.2  Does the program have ambitious targets and timeframes for its long-term measures?

**Explanation:  Each of the Earth Science Applications program’s 12 National Applications (e.g., air quality, public health) has developed or is developing a specific roadmap that lay out plans through 2012 supporting the outcome supporting the outcome performance measures.  The “ambitiousness” of the targets should be more readily apparent as the program matures and can assess the time effort necessary to develop products.**
**Evidence:  Each roadmap has discrete, mid-term performance goals.  The goals include specific, quantifiable targets and timeframes.  For example, in aviation safety, the present WX Visualization System is a discrete, stand-alone weather product, with little satellite sounding data or imagery.  The 2012 performance measure is for a WX visualization System hat is fully integrated with SVS, WARP, and ITWS in-cockpit graphical WX displays featuring real-time weather information with global coverage.  Roadmaps have been completed for most of the applications and are available from the Applications Office.  Once finalized, they will be available through the program website.**

## 2.3  Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program’s long-term goals?

**Explanation:  Specific annual performance measures demonstrate progress towards achieving the long-term goals and measures.  As the program matures, NASA needs to develop measures to address efficiency and the value added of incorporating NASA data (I.e., measure quality of products versus quantity).  NASA also need to develop metrics that better encompass outcomes resulting from the education and outreach portions of the program.**
**Evidence:  Specific annual performance measures directly supporting the long-term goals are included in the Measures tab.**

## 2.4  Does the program have baselines and ambitious targets for its annual measures?

**Explanation:  Annual performance goals are designed to demonstrate progress against the baselines in the roadmaps by meeting specific targets. The "ambitiousness“ of the targets should be more readily apparent as the program matures and can assess the time and effort necessary to develop products.**
**Evidence:  Baselines and targets are specified in the program's roadmaps. The systems engineering approach being used in the evaluation of decision supportnsystems for each of the national applications contains a fundamental concept of decision support system (DSS) baselining from which to measure improvement. The baselines consist of the respective DSS State 1's (current) shown in the Roadmaps. The Roadmaps also show the necessary steps to progress from the State 1 to State 2 (benchmarked/improved) of the DSSs with the infusion of earth science and technology research results.**

## 2.5  Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

**Explanation:  The Earth Science Applications program partners (Federal agencies, NASA Centers, private sector, universities, international organizations, and others) are directly involved in planning and establishing the program's goals and objectives, and design and implement programs of work supporting our common objectives. However, the large percentage of earmarked funds (over 25 percent) indicate that NASA has little control over a substantial number of partners. In addition, while grants may be focused on the priority applications, it is not clear how grantee performance ties back into NASA's performance measures.**
**Evidence:  In NASA's FY 2003 Operating Plan, nearly $20M of ESA funds of a $78M total budget are Congressionally directed grants and transfers.**

## 2.6  Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?  

- Explanation:  Earth Science Applications program plans and activities are evaluated on a regular basis by the National Academy of Sciences and the Earth System Science and Applications Advisory Committee to ensure effectiveness and relevance to needs. Results of these reviews are the basis for revision and update of all Earth Science Applications program activities and plans.
- Evidence:  Scientific and programmatic progress and performance for the Earth Science Applications program is presented to the ESSAAC and the National Academy of Sciences on a periodic basis (Review of NASA's Earth Science Enterprise Applications Program Plan). An NRC review of NASA's Aviation Safety Program is underway assessing how NASA and the FAA collaborate to leverage R&D into operations. In addition, an integral part of the Program is the evaluation and measurement of DSS improvements by the partner agency. Following the completion of the benchmarking process, the enhanced/ upgraded DSS i.e., State 2 is evaluated against the State 1 (i.e., original state) of the same DSS to measure the enhanced operational status. This type of evaluation will be conducted in the context of each National Applications once benchmarking is completed.

## 2.7  Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

**Explanation:NASA's IBPD provides the budget request and performance targets on a full-cost basis. Detailed budgets are being developed for each application of national priority to identify critical elements leading to measurable success. Once these are completed, NASA should fully integrate these into the budget requests.**
**Evidence:  The IBPD can be found at **_**[www.nasa.gov/about](http://www.nasa.gov/about)**_**/budget/.**** Detailed budgets have been developed for most of the National Applications.**

## 2.8  Has the program taken meaningful steps to correct its strategic planning deficiencies?

**Explanation:  NASA's Earth Science Enterprise, which manages the Earth Science Applications program, updates its Strategic Plan every three years. This process is presently under way as the Program and Enterprise align with the new NASA Vision and Mission. The Earth Science Applications program reviews and updates its program strategic plan on an annual basis. Any strategic planning deficiencies are identified and corrected as part of the update process.**
**Evidence:  The National Academy of Sciences has reviewed Earth Science Applications program strategic planning, and identified areas for improvement (Review of NASA's Earth Science Enterprise Applications Program Plan). Management action was taken to improve the areas identified in the NAS Report. These can be seen through the development of the road maps and program plans.**

## 2.CA1  Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals and used the results to guide the resulting activity?

**Explanation:  The Earth Science Applications program does not fund acquisition of capital assets.**

## 2.RD1  If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

**Explanation:  The Earth Science Applications program continually evaluates its efforts with respect to the relative potential benefits of alternatives in identifying opportunities for enhancing partners operations.  **
**Evidence:  Alternative approaches to enhancement of partnering agencies systems are evaluated as part of the joint project design process in each National Application area. In addition, alternative approaches to accomplishing the partner's goals are normally evaluated during the conduct of socio-economic benefits studies such as the those referenced in the answer to 2.1 (NASA/ESE Power Program Benefits and Reducing the Risks and Costs of Natural Disasters in the United States). In addition, the joint National Applications Projects with partner agencies inherently compare alternative approaches to enhancing the DSS.**

## 2.RD 2 Does the program use a prioritization process to guide budget requests and funding decisions?

**Explanation:  The Earth Science Applications program uses a prioritization process that draws on internal and external reviews to guide budget requests and funding decisions.**
**Evidence:  Overall program priorities (e.g., selection of the 12 National Applications), criteria for selection, and programmatic direction are presented in the Earth Science Applications Strategy. The NAS endorsed this prioritization in their review of the program strategy. The more detailed priorities reflected in the roadmaps and program plans are determined jointly with the partner organization and become the basis for solicitations. There are three Strategies (Earth Science Applications, Education, and Outreach), 12 roadmaps for the national applications and 18 Program Plans.**

## 3.1  Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

**Explanation:The Earth Science Applications program routinely collects relevant technical and programmatic performance information. As the program matures, in order to demonstrate results through the process of baselining and benchmarking, NASA will rely on the collection of extensive performance data.**
**Evidence: The Earth Science Applications program continually monitors progress against the national applications roadmaps and annual performance goals.  Progress is also reviewed in Enterprise-level "Focus Area Reviews." In those areas where NASA has started to demonstrate results (e.g. hurricane forecasting and wildland fire management), extensive performance data has been collected (see http://www.esad.ssc.nasa.gov/background_documents.asp)**

## PART - Question 3.2

**Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- Federal managers and partners are required to meet annual performance standards.
**Evidence:**
- Performance standards are included in personnel performance evaluation criteria (example: Individual Performance Plans) and in contracts with outside sources. For example, Cooperative Agreements have defined milestones that must be met by the Principal Investigators (PI's) in order for the PI's to receive payment. These milestones are measured against performance metrics that are part of the Cooperative Agreements and approved by the NASA Program Managers in order to release further funding. Similarly, contracts have deliverables that must be approved by the Contracting Technical Officer (COTR) before payment is released to the organization.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.3 

**Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- The Earth Science Applications program obligates its funding in a timely manner and spends it for the purpose as appropriated by Congress.
**Evidence:**
- On average, the Earth Science Applications program obligates approximately 85% of its authorized annual budget within the fiscal year for which funds are provided. One hundred percent of funds are obligated over the two years available for obligation. Enterprise and Agency-wide controls ensure that funds are spent for the intended purpose.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.4

**Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?**
**Explanation:****	****	****	****	****	****	**** Answer**: NO
- The Earth Science Applications program does not have adequate efficiency measures that aptly demonstrate efficiencies and cost effectiveness in program execution. However, they are moving forward on emphasizing competitive sourcing to achieve program goals and are expected to demonstrate efficiency improvements as a result in future years. The Earth Science Applications program goal for competitive sourcing is 80%.
**Evidence:**
- nothing cited
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.5

**Does the program collaborate and coordinate effectively with related programs?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- The Earth Science Applications program strategy is based on engaging in partnerships to contribute systems solutions to national priorities.
**Evidence:**
- The Earth Science Applications program is an integral component of Administration and interagency collaborations including the Climate Change Science Program, the Climate Change Technology Program, National Blueprint for Aviation, National Agenda on Disaster Management, Geospatial One Stop and the Blueprint for Revolution in Earth and Space Science. The Program Plans, National Application Success Stories and 2002 Year in Review are evidence of success that have come out of collaboration with other organizations.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.6

**Does the program use strong financial management practices?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- The Earth Science Applications program uses the Agency's financial management practices in administering program funds, and is free from material internal control weaknesses.
**Evidence:**
- Sound financial performance is evidenced by the Agency's unqualified audit opinion on our FY 2002 financial statements. The integration of budget and performance is defined through the IBPD as the basis for planning. As previously stated in Block 3.4, the Business Division also reviews and presents the Program/Financial Status to ESE Management on a Monthly basis.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.7

**Has the program taken meaningful steps to address its management deficiencies?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- Program management is evaluated on the basis of the ability of the program to meet its performance objectives with the resources available. Deficiencies in performance are corrected through adjustments in management strategy.
**Evidence:**
- The Earth Science Applications program addressed management deficiencies through a restructuring in 2002 that led to the Science for Society architecture and the development of a structured budget aligned with Enterprise and Agency goals. That restructuring marked a shift in focus of the program that resulted in new top level management and a revised strategic plan.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.CA1

**Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?**
**Explanation:****	****	****	****	****	****	**** Answer**: N/A
- The Earth Science Applications program does not fund acquisition of capital assets.
**Evidence:**
- nothing cited
**Question Weight**: 0%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.CO1

**Are grants awarded based on a clear competitive process that includes a qualified**
**Explanation:****	****	****	****	****	****	**** Answer**: NO
- Substantially less than 95% of funds are awarded on a clear competitive process both due to Congressional direction and other sole source activity.
**Evidence:**
- As indicated previously, over 25% of ESA funding is Congressionally directed.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.CO2

**Does the program have oversight practices that provide sufficient knowledge of grantee activities?**
**Explanation:****	****	****	****	****	****	**** Answer**: YES
- Program managers have a sufficient understanding of grantees use of funds.
**Evidence:**
- Close contact with funding recipients is maintained through site visits, required reports, Principal Investigator meetings, and discipline-oriented meetings. Also, major meetings of professional societies provide the opportunity to hear results as presented to the community, and place them in the context of work done by others under a broad range of sponsorship. Each center conducts Project Level Reviews to ensure that performance by the grantee meet expectations.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.CO3

**Does the program have oversight practices that provide sufficient knowledge of grantee activities?**
**Explanation:****	****	****	****	****	****	**** Answer**: NO
**Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?**
**Evidence:**
- nothing cited
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 3.RD1

**For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?**
**Explanation:****	****	****	****	****	****	**** Answer**: NO
- The Earth Science Applications program allocates funding using a broadly competitive peer review process and employs sole-source procurements only on the basis of a demonstrated unique expertise or capability. However, over 25% of program funds are earmarked for specific purposes and not subject to any review.
**Evidence:**
- In NASA's FY 2003 Operating Plan, nearly $20M of ESA funds of a $78M total budget are Congressionally directed grants and transfers. Within the remaining funds, competitive solicitations include the Research, Education and Applications Solutions Network (REASoN) and GLOBE Cooperative Agreement Notices (CANs); the Solid Earth and Natural Hazards (SENH), New Investigators Program (NIP) and Graduate Fellowship Program (GFP) NASA Research Announcements (NRAs); and NASA Center Contractor Support.
**Question Weight**: 9%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.1

**Has the program demonstrated adequate progress in achieving its long-term performance goals?**
**Explanation:****	****	****	****	****	****Answer**: LARGE 							              EXTENT
- The restructured Earth Science Applications Program is on track to achieve its long-term goals, however, the program is only been in its current iteration since January 2002 and to date has focused primarily on establishing the framework necessary to achieve the long-term goals.
**Evidence:**
- The Earth Science Applications program has signed MOUs with key Federal partners that have defined steps toward assimilating NASA Earth science research results into decision support tools owned and operated by the partners. Successful projects in disaster management (wildfire tracking with the USFS and hurricane prediction with NOAA) and aviation safety (volcanic ash plume tracking with the FAA and NOAA) are representative contributions of the Earth Science Applications program. (http://www.earth.nasa.gov/eseapps/)
**Question Weight**: 34%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.2

**Does the program (including program partners) achieve its annual performance goals?**
**Explanation:****	****	****	****	****	****Answer**: LARGE 							              EXTENT
- The program is making progress on the annual targets listed. It should be noted, however, that the program has recently revised its strategic direction of the program as well as created more meaningful annual performance measures. The program did achieve most of their performance goals identified through GPRA.
**Evidence:**
- Successful projects in disaster management (wildfire tracking with the USFS and hurricane prediction with NOAA) and aviation safety (volcanic ash plume tracking with the FAA and NOAA) are representative contributions of the Earth Science Applications program. The Earth Science program (of which Earth Science Applications is a part) achieved 29 of 31 2002 performance goals for a 94% success rate (reference 2002 Performance and Accountability Report).
**Question Weight**: 33%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.3

**Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?**
**Explanation:****	****	****	****	****	****	****Answer**: NO
- There are no indicators to suggest that the program has demonstrated improved efficiency or cost-effectiveness over the prior year.
**Evidence:**
- nothing cited
**Question Weight**: 33%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.4

**Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?**
**Explanation:****	****	****	****	****	****	****Answer**: N/A
- There are no other programs that which provide a good basis for comparison. Programs including NOAA's National Centers of Environmental Prediction (NCEP); DoE Laboratories e.g., Pacific Northwest Research Center and Oakridge; EPA Office of Research; and on the international level, the Global Monitoring Environmental System (GMES) are all being looked at as a source of "lessons learned".
**Evidence:**
- nothing cited
**Question Weight**: 0%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.5

**Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?**
**Explanation:****	****	****	****	****	****	****Answer**: N/A
- An independent evaluation conducted by the National Academies of Science confirmed the effectiveness of the program strategy. However, due to the programs relatively recent origin, there have been limited independent evaluations documenting results.
**Evidence:**
- nothing cited
**Question Weight**: 0%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.

## PART - Question 4.CA1

**Were program goals achieved within budgeted costs and established schedules?**
**Explanation:****	****	****	****	****	****	****Answer**: N/A
- The Earth Science Applications program does not fund acquisition of capital assets.
**Evidence:**
- nothing cited
**Question Weight**: 0%

**Notes:**
Out of the 13 programs we reviewed only 10 met the criteria and 3 did not.
** NASA Earth Science Applications - Partners
DOE  Basic Energy Science – Advisory Committee
Interior--National Mapping—04 – National Academy of Science/NSF

The pie chart shows the freq. with which those who met the criteria used the different kinds of evidence to demonstrate that they deserved a yes.

Evidence sited included Peer/Merit/Tech. Review, Program Manager Review, Annual Review, Advisory Committee, Partners, National Academy of Science/National Science Foundation, Panel Review

As shown the program manager review technique was used most often.