IG-00-059

AUDIT

REPORT SOFTWARE ASSURANCE

September 28, 2000

National Aeronautics and

Space Administration

OFFICE OF INSPECTOR GENERAL
Additional Copies

To obtain additional copies of this report, contact the Assistant Inspector General for

Auditing at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General

for Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing

Code W

NASA Headquarters

Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800)

424-9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form;

or write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station,

Washington, DC 20026. The identity of each writer and caller can be kept confidential,

upon request, to the extent permitted by law.

Reader Survey

Please complete the reader survey at the end of this report or at

http://www.hq.nasa.gov/office/oig/hq/audits.html.

________________________________________________________________________

Acronyms

CIO Chief Information Officer

CMM Capability Maturity Model

EMC Engineering Management Council

FY Fiscal Year

ISO Internationa l Standards Organization

IV&V Independent Verification and Validation

NPD NASA Policy Directive

OSMA Office of Safety and Mission Assurance

PMC Program Management Council

SOFIA Stratospheric Observatory for Infrared Astronomy

SWG Software Working Group
W September 28, 2000

TO: A/Administrator

FROM: W/Inspector General

SUBJECT: INFORMATION: Audit of Software Assurance

Report Number IG-00-059

The NASA Office of Inspector General has completed an audit of Software Assurance.

The audit focused on determining whether the Agency had established adequate

guidelines for using independent verification and validation (IV&V) during the software

life cycle and whether program and project managers had implemented NASA’s IV&V

Facility recommendations to perform IV&V. We found that NASA lacked adequate

management controls for using IV&V in software development projects. In addition,

NASA lacked adequate controls for collecting, analyzing, and reporting software metrics.

Accordingly, NASA lacks assurance that it can effectively mitigate potential software

failures through the use of IV&V and monitoring of software assurance activities.

NASA recently issued interim guidelines that are intended to improve the software

development process. The guidelines include criteria for using IV&V during software

development.

Background

In 1993, NASA established the IV&V Facility in Fairmont, West Virginia,1 as part of an

Agencywide strategy to provide the highest achievable levels of safety and costeffectiveness for mission-critical software. The IV&V Facility currently supports 21

NASA programs and projects including the International Space Station, Space Shuttle,

Earth Observing System Data and Information System, Checkout and Launch Control

System at the Kennedy Space Center, and Advanced Air Transportation Technology.

The Facility’s budget for fiscal year (FY) 2000 is $26 million.

 

1

 NASA established the Facility as a result of recommendations made by the National Research Council

and the “Report of the Presidential Commission on the Space Shuttle Challenger Accident.”
2

Recommendations

We recommended that NASA establish procedures for evaluating the adequacy of program

and project managers’ actions in implementing the interim IV&V criteria and conduct

evaluations, as appropriate. Without procedures, NASA lacks assurance that the interim

criteria are effectively applied to software development projects. We also recommended that

NASA issue guidelines for the IV&V Facility review of programs and projects with

significant software applications; for implementing recommendations to perform IV&V; and

for collecting, analyzing, and reporting software metrics. Without such guidance, NASA

lacks assurance that the risk of potential software failures has been adequately reduced

through IV&V. Also, without software metrics guidance, NASA has an unmet requirement

for assessing the adequacy of its software policies and procedures.

Management’s Response and OIG Evaluation

Management concurred with the recommendations. NASA management issued interim

IV&V criteria for use by program and project managers in determining whether new or

existing projects should be subject to IV&V. Also, IV&V Facility personnel are assisting the

projects in determining the necessary level of IV&V and in developing IV&V implementation

plans. NASA management also issued a set of software metrics that it will evaluate over a 12-

month period. Upon conclusion of the evaluation period, management will determine whether

the metrics gathered satisfy the objectives of the metrics program. The approved metrics will

become part of a NASA policy guideline.

The actions planned or taken by management are responsive to the recommendations. Details

on the status of the recommendations are in the Recommendations section of the report.

[original signed by]

Roberta L. Gross

Enclosure

 Final Report on Audit of Software Assurance
FINAL REPORT

AUDIT OF SOFTWARE ASSURANCE
W September 28, 2000

TO: AE/Chief Engineer

AO/Chief Information Officer

Q/Associate Administrator for Safety and Mission Assurance

FROM: W/Assistant Inspector General for Auditing

SUBJECT: Final Report on Audit of Software Assurance

Assignment Number A9906600

Report Number IG-00-059

The subject final report is provided for your information and use. Our evaluation of your

response is incorporated into the body of the report. The corrective actions planned for

the recommendations are responsive. The recommendations will remain open for

reporting purposes until agreed to corrective actions are completed. Please notify us

when action has been completed on the recommendations, including the extent of testing

performed to ensure corrective actions are effective.

If you have questions concerning the report, pleas contact Mr. David Gandrud, Program

Director, Information Technology Program Audits, at (650) 604-2672, or Mr. Roger

Flann, Program Manager, at (818) 354-9755. We appreciate the courtesies extended to

the audit staff. The final report distribution is in Appendix D.

[original signed by]

Russell A. Rau

Enclosure
2

cc:

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

JM/Acting Director, Management Assessment Division

200-1/Director, Ames Research Center

100/Director, Goddard Space Flight Center

307/Acting Director, NASA Independent Verification and Validation Facility,

Goddard Space Flight Center
NASA Office of Inspector General

IG-00-059 September 28, 2000

A9906600

Software Assurance

Introduction

Software assurance is the planned and systematic set of activities for ensuring that

software processes and products conform to established requirements, standards, and

procedures. Verification and validation of the software processes and products are part of

software assurance. IV&V is a process used to ensure that software products of the

software development life-cycle phases are independently reviewed, verified, and

validated by an organization that is neither the developer nor the acquirer of the software.

IV&V is a vital part of a sound management process because it ensures that program

advocacy is balanced by substantive evidence. NASA’s planned information technology

investment for FY 2000 is $2.2 billion. 2

NASA Policy Directive (NPD) 2820.1, “NASA Software Policies,” May 29, 1998,

defines NASA policies regarding management, engineering, and assurance of software

created or purchased by the Agency. The NPD requires program and project managers to

employ verification and validation techniques for risk mitigation and requires the Agency

to collect, analyze, and report on software metrics.

The overall audit objective was to determine whether NASA has exercised effective

software assurance. Due to the importance of IV&V in the software assurance process,

we limited our review to determining whether the Agency had established adequate

guidelines for using IV&V during the software life cycle and whether program and

project managers had implemented recommendations to perform IV&V. Details on our

audit objectives, scope, and methodology are in Appendix A.

Results in Brief

NASA lacked adequate management controls for using IV&V in software development

projects. Additionally, management had not established metrics for evaluating software

policies and procedures and for reporting to the NASA Engineering Management

Council3 (EMC) as required by NPD 2820.1. As a result, NASA is not assured that it can

 

2

 Information technology investments include computers, ancillary equipment, software, firmware,

networks, services and support services, personnel, funds, and related information resources. Because

software costs are not separately tracked, that portion of the information technology investment cost is not

known.

3

 The Engineering Management Council is a forum for assessing and improving Agency engineering

practices, policies, training and certification standards, procedures, and capabilities. The Engineering

Management Council conducts or supports independent technical reviews of NASA programs and informs

the Chief Engineer about NASA-wide engineering activities.
2

effectively mitigate potential software failures through robust use of IV&V and effective

monitoring of software assurance activities. NASA recently issued interim guidelines

that are intended to improve the software development process.

Background

The IV&V Facility in Fairmont, West Virginia, is intended to be the NASA center of

excellence for applying software verification and validation technology. It manages

independent assessments,4 software and system engineering tasks, and IV&V for NASA

programs and projects.5

The NASA Office of Safety and Mission Assurance (OSMA) had initial management

responsibility for the Facility. In 1995, management responsibility transferred to Ames.

In July 2000, NASA transferred management responsibility to Goddard.6 The transfer

was intended to better integrate the IV&V Facility into the software development life

cycle of NASA’s programs and projects.

Management Controls

Finding. NASA lacked adequate management controls for determining whether to use

IV&V in its software development projects and for collecting, analyzing, and reporting

software metrics designed to monitor these projects. This condition occurred because

NASA had not issued guidelines to implement the controls. As a result, NASA has less

assurance that the risk of potential software failures has been adequately reduced through

IV&V and implementation of sound software assurance policies and procedures.

NASA Software Policies and Procedures

NPD 2820.1 requires program and project managers to employ verification and validation

techniques for risk mitigation, including IV&V as appropriate, based on project cost, size,

complexity, life span, risk, and consequences of failure.7 The NPD does not include

specific criteria for determining whether IV&V is appropriate for a software development

project.

 

4

 An independent assessment identifies the risks to the critical software elements that could jeopardize

mission safety and success. Program and project managers can use the results of an independent

assessment to determine whether a software development project should undergo IV&V. Software and

system engineering tasks identify potential issues based on software design analyses, software code

reviews, and peer reviews. These tasks do not constitute IV&V, which involves evaluating the software

development project throughout its life cycle. The IV&V Facility and the project managers determine

which tasks will be performed based on the status of the project.

5

 The IV&V Facility uses contractors to perform work supporting NASA programs and projects.

6

 OSMA has management responsibilities for the Agency’s software assurance and IV&V of critical flight

systems and conducts these responsibilities through Goddard.

7

 Consequences of failure include loss of life, serious injury, catastrophic mission failure, partial mission

failure, loss of equipment, waste of resource investment, adverse visibility, and impact on routine

operations.
3

NPD 2820.1 also requires the IV&V Facility to collect, analyze, and report on software

metrics. Software metrics include:

• Evidence of project compliance with the NPD.

• Agency trends on software cost and schedule baseline deviations and the

degree to which delivered software satisfies Agency requirements, including

safety, quality, and reliability measures.

• Assessments and audits of conformance to International Standards

Organization (ISO)8 9001 and the Capability Maturity Model (CMM)9 for

Software in NASA software creation and acquisition organizations.

• Other surveys relating to the implementation of the Directive.

• Improvements in software acquisition and creation of software projects,

resulting from the use of the CMM.

• Improvements in management of software creation and acquisition, resulting

from case studies and shared experiences.

The NASA Chief Information Officer (CIO), Chief Engineer, and Associate

Administrator for OSMA are responsible for jointly promoting software policies,

standards, best practices, and guidelines. The CIO has primary responsibility for

developing the Agency’s software policies. During a Senior Management Council10

meeting in June 1999, the NASA Administrator stated that only the IV&V Facility should

conduct IV&V for NASA projects. The Associate Administrator for OSMA reiterated

the Administrator’s decision in a November 10, 1999, memorandum. 11

 

8

 ISO 9000 is a series of standards and guidelines that define the minimum requirements for an effective

Quality System that is accepted internationally. The ISO 9001 standard requires NASA Headquarters to

document what it does, do what it documents, and provide objective evidence of accomplishment. The

standard also requires that NASA review its processes for improvement when necessary.

9

 The CMM for Software describes the principles and practices underlying software process maturity and is

intended to help software organizations improve the maturity of their software processes in terms of an

evolutionary path from informal, chaotic processes to mature, disciplined software processes.

10 NASA's Senior Management Council is chaired by the Administrator and consists of Associate

Administrators, Officials-in-Charge of Headquarters offices, and installation Directors. This council

advises the Administrator on the status of Agency programs and plans and serves as a forum for discussing

issues affecting Agency management.

11 The Associate Administrator for OSMA issued the November 10, 1999, memorandum on “Software

Assurance for Safety and Mission Assurance,” to Enterprise Associate Administrators, Center Directors,

and Center Safety and Mission Assurance Directors. (The Agency established four Strategic Enterprises to

function in primary business areas for implementing NASA’s mission and serving customers. The four

Enterprises are (1) Aerospace Technology, (2) Earth Science, (3) Human Exploration and Development of

Space, and (4) Space Science.
4

On May 10, 2000, NASA established an initiative to improve software quality and

safety. 12 As part of this initiative, the CIO, Chief Engineer, and Associate Administrator

for OSMA assigned the Software Working Group13 (SWG) the responsibility for

developing criteria for use in determining whether to use IV&V. The initiative also

included the requirement to collect meaningful software metrics.

On July 21, 2000, the Chief Engineer issued a memorandum that included the SWGdeveloped interim criteria for IV&V. 14 The interim criteria provide quantifiable

standards for determining whether IV&V should be applied to a software development

project. Program and project managers are to use the criteria to evaluate specific aspects

of a project for consequences of failure and probability of failure. The criteria also

identify factors for evaluating the projects and for rating the risks to software

development. The Deputy Associate Administrator for OSMA stated that, as an interim

measure, program and project managers must apply the criteria to selected existing and

all new software development projects.

Use of IV&V

NASA has not established guidelines to help users determine whether to use IV&V based

on a software project’s cost, size, complexity, life span, risk, and consequences of failure

as required by NPD 2820.1.

The benefits of performing IV&V have been well demonstrated. For example, through

the application of IV&V for the Space Shuttle Program, the IV&V Facility identified 15

software errors that could have resulted in loss of the Shuttle or crew. With the

correction of the software errors, the Space Shuttle Program increased mission safety and

reliability and reduced program cost. Notwithstanding such efforts, the IV&V Facility

had performed IV&V on only 915 of about 170 programs and projects, as of

December 22, 1999 managed by Program Management Councils (PMC’s).16

With issuance of the interim criteria, NASA has taken substantial steps toward improving

its software assurance program. For example, managers must now evaluate their

programs and projects during project formulation to determine whether software IV&V

 

12 The CIO, Chief Engineer, and Associate Administrator for OSMA issued the May 10, 2000,

memorandum on “NASA’s Initiative to Improve Quality and Safety of Software,” to Officials-in-Charge of

Headquarters Offices, Center Directors, and the Director of the Jet Propulsion Laboratory.

13 The SWG is the responsibility of the Chief Engineer. The SWG advises the Agency on software-related

matters and recommends software management, engineering, and assurance polices, standards, best

practices, and guidance.

14 The Chief Engineer issued the July 21, 2000, memorandum on “Interim NASA Software Independent

Verification and Validation (IV&V) Policy and IV&V Facility Planning Action to Project Managers,” to

Officials-in-Charge of Headquarters Offices, Center Directors, and the Director of the Jet Propulsion

Laboratory.

15 Appendix B identifies the NASA programs and projects for which the IV&V Facility performed an

independent assessment, software and system engineering tasks, or IV&V.

16 NASA has established a hierarchy of PMC’s that are responsible for assessing program and project

formulation and implementation and for providing oversight and direction. PMC’s exist at the Agency,

Lead Center, and Center levels.
5

or an independent assessment is required. Further, managers must coordinate their

evaluation results with the governing PMC’s and the Center Directors. To ensure that the

interim criteria are effectively applied to software development projects, responsible

NASA officials should evaluate the adequacy of program and project managers’ actions

in implementing the criteria.

Recommendations to Perform IV&V

The IV&V Facility performs independent assessments that may result in a

recommendation to perform IV&V on software development projects. However, NASA

had not established guidelines to help ensure that program and project managers

adequately address the recommendations.

As of August 1, 2000, the IV&V Facility performed independent assessments for 12

NASA programs and projects (see Appendix B) and recommended IV&V for 4 of them:

the Checkout and Launch Control System, 17 Mars Surveyor Program,18 Stratospheric

Observatory for Infrared Astronomy (SOFIA),19 and Boeing 757 New Generation

Display Simulation. 20 Managers for three of the four programs and projects chose to not

implement IV&V. Specifically:

• The project manager for the Mars Surveyor Program did not follow the

Facility’s recommendations to perform IV&V due to insufficient funding for

IV&V. The Mars Surveyor Program later failed, in part, due to reported

software problems or failures.

• The project manager for SOFIA initially determined that the recommendation

to perform IV&V would not be implemented because “an additional review

effort would be deleterious to the program’s already seriously challenged

schedule and cost status.” The IV&V Facility and program management have

since begun negotiations to perform IV&V.

• Project management for the Boeing 757 project determined that the

verification and validation processes for the project sufficiently mitigated

project risks and considered IV&V to be unnecessary.

 

17 The Checkout and Launch Control System processes Space Shuttle data at the Kennedy Space Center.

This includes providing multi-orbiter support from one control room; multi-system monitoring capability

from one console; and local monitoring, command, and control.

18 The Mars Surveyor Program included the Mars Climate Orbiter and Mars Polar Lander.

19 SOFIA will be the largest airborne telescope in the world, making observations that are impossible for

even the largest, highest ground-based infrared telescopes. NASA, the German Aerospace Center, and an

international contractor team are working together to create SOFIA—a 2.5-meter (98.5-inch)-diameter

reflecting telescope mounted in a modified Boeing 747SP. Ames manages the project.

20 The Boeing 757 is a flying laboratory for aeronautical research. NASA has modified the aircraft for a

broad range of flight research programs and uses the aircraft to conduct research to increase aircraft safety,

operating efficiency, and compatibility with future air traffic control systems. The Langley Research

Center maintains and flies the Boeing 757.
6

Improved management controls for addressing recommendations to perform IV&V,

based on independent assessments, are needed to ensure that program and project

managers adequately address software development risks through IV&V. Without such

controls, the value added by the independent assessment process may not be realized.

Use of Software Metrics

The Agency has not collected, analyzed, or reported software metrics as required by NPD

2820.1. The NPD requires the IV&V Facility to collect and analyze metrics and to

submit an annual report on Agency software policies and practices to the NASA EMC.

The Directive further requires the governing PMC’s to review software processes and

products and to provide the results to the IV&V Facility. The Directive references

unpublished NASA Policy Guidelines 282021 for specific responsibilities related to

collecting, analyzing, and reporting software metrics. Further, the Directive does not

identify the types of information that should be included in the annual report and does not

address how the IV&V Facility or NASA management should use the information.

Guidelines for collecting, analyzing, and reporting software metrics are needed to help

the Agency meet NPD 2820.1 software metric requirements.

In the May 10, 2000, memorandum on NASA’s initiative to improve the quality and

safety of software, the CIO, Chief Engineer, and Associate Administrator for OSMA

addressed software metrics as an important part of the initiative. The metrics currently

required are presented on page 3 of this report. Beginning with the second half of FY

2001, program and project managers will be required to collect and report software

metrics. The memorandum states the SWG will analyze the metrics. Subsequently, the

Deputy Chief Engineer stated that the IV&V Facility will collect and analyze the metrics

and provide the results to the SWG. The NASA official further stated that the SWG has

begun reevaluating the metrics to ensure that NASA has identified the appropriate

software metrics to be collected.

While NASA’s recent actions represent important steps to improving software policies

and procedures, NASA should ensure that it develops and issues policy guidelines for

collecting, analyzing, and reporting appropriate software metrics. Without the guidelines,

the requirement for collecting, analyzing, and reporting software metrics may remain

unmet and Agencywide software policies and procedures will lack adequate visibility.

 

21 Until recently, the CIO, Chief Engineer, and Associate Administrator for OSMA had not determined

whether NASA would issue NPG 2820 (title not known). As part of NASA’s actions to improve software

policies and procedures, the NASA managers have determined that the NPG will be issued.
7

Recommendations, Management’s Response and Evaluation of

Response

1. The NASA CIO, in collaboration with the NASA Chief Engineer and Associate

Administrator for OSMA, should establish procedures for evaluating the adequacy

of program and project managers’ actions in implementing the interim IV&V

criteria and conduct evaluations, as appropriate.

Management’s Response. Concur. The Goddard Office of System and Mission Assurance

issued an IV&V Interim Policy. The policy states that all projects meeting defined criteria

will document and implement a plan that addresses the performance of IV&V during software

development. The Goddard Office of Systems Safety and Mission Assurance is leading the

development of a NASA policy document for IV&V. The document is sponsored by the

NASA OSMA and will be in the initial stage of the NASA Directives Management review

cycle by October 31, 2000. The complete text of management’s response is in Appendix C.

Evaluation of Management’s Response. Management’s ongoing and planned actions are

partially responsive to Recommendation 1. However, management's actions discussed

regarding Recommendation 2 below describe additional actions that effectively meet the

intent of Recommendation 1. Recommendation 1 is resolved but will remain undispositioned

and open pending the final issuance and implementation of an NPD relating to the use of

IV&V in NASA’s software development projects.

2. The NASA CIO, in collaboration with the NASA Chief Engineer and Associate

Administrator for OSMA, should issue guidelines for IV&V Facility review of

programs and projects with significant software applications; for implementing

recommendations to perform IV&V; and for collecting, analyzing, and reporting

software metrics.

Management’s Response. Concur. On June 28, 2000, NASA’s SWG released the criteria

for assessing project software risk and for determining the necessity for IV&V. About 100

NASA projects under development and not currently implementing IV&V applied the criteria

during August 2000. The projects sent the results to the IV&V Facility for it to determine the

necessity to implement IV&V. The IV&V Facility used the results to develop an initial list of

projects that should have IV&V. Personnel from the IV&V Facility are in the process of

meeting with the projects identified for application of IV&V in order to determine the

necessary level of IV&V and to develop implementation plans. These initial meetings will be

completed by December 31, 2000. IV&V activities on identified projects will be initiated

during FY 2001 and fully under way on all projects by the end of FY 2001.

The draft NPD for IV&V specifies that, in their planning stages, all new software projects

shall determine the need for IV&V. The issuance of the NPD will formalize this requirement.

Regarding the need for software metrics guidelines, in July 2000, the SWG released a set of

metrics applicable to all NASA software development programs. The IV&V Facility will

analyze the metrics over a 12-month period. When this period concludes in October 2001, the
8

Facility will use these analyses to determine whether the metrics collected satisfy the

objectives of the metrics program. If the results indicate changes are needed in the metrics set,

the SWG will reevaluate the metrics and make necessary changes to the set of metrics to be

collected. Metrics collection will begin on all projects as specified in NPD 2820.1 starting

October 2001.

Evaluation of Management’s Response. The actions ongoing and planned are responsive to

the recommendation. The recommendation is resolved but will remain undispositioned and

open pending final issuance and implementation of the NPD for IV&V and implementation of

software metrics for all projects.
9

Appendix A. Objectives, Scope, and Methodology

Objectives

The overall audit objective was to determine whether NASA has exercised effective

software quality assurance. Specifically, we were to determine whether selected software

development projects had complied with applicable software quality assurance standards

and procedures related to project planning, acceptance-level testing, and reporting. Due

to the importance of independent verification and validation (IV&V) in the software

assurance process, we limited our review to determining whether:

• the Agency established adequate guidelines for using IV&V during the

software life cycle;

• program and project managers had implemented recommendations to perform

IV&V; and

• software metrics were being collected, analyzed, and reported in accordance

with NPD 2820.1.

A future audit will include the review of software quality assurance standards and

procedures.

Scope and Methodology

We limited our review to NASA’s use of IV&V when developing software for NASA

programs and projects. We performed the following:

• Reviewed applicable policy directives, guidelines, guidebooks, standards;

reports and presentations; and other documentation to gain an understanding

of NASA processes and procedures for software assurance.

• Identified NASA programs and projects for which IV&V and independent

assessments had been performed (by the IV&V Facility and Goddard) to

determine the extent of IV&V performed.

• Interviewed NASA officials at Headquarters, Ames Research Center (Ames),

Goddard Space Flight Center (Goddard), the Jet Propulsion Laboratory,

Johnson Space Center (Johnson), Langley Research Center (Langley), and the

IV&V Facility to determine their roles and responsibilities in the software

assurance area.
10

Appendix A

Management Controls Reviewed

We reviewed management controls for determining whether program and project

managers should incorporate IV&V into their software development projects. NASA had

not established the management controls for determining whether to use IV&V; for

implementing independent assessment recommendations; and for collecting, analyzing,

and reporting software metrics.

Audit Field Work

We performed audit field work from October 1999 through August 2000 at NASA

Headquarters, the IV&V Facility, Ames, Goddard, Johnson, the Jet Propulsion

Laboratory, and Langley. We conducted the audit in accordance with generally accepted

government auditing standards.
11

Appendix B. NASA Programs and Projects

Supported by the IV&V Facility

June 1994 – July 2000

Status of Work

by Independent Verification & Validation (IV&V) Facility

Program/Project Independent

Assessment

Software & System

Engineering Tasks

(SET)

IV&V

Advanced Air Transportation

Technology Not performed Work ongoing Not applicable

Blended Wing Body Low Speed Vehicle

Independent

assessment

completed

Agreement for SET

being negotiated Not applicable

Boeing 757 New Display Simulation

System software

assessment

completed

Not performed

Recommendations

for IV&V not

implemented 1

Cassini Not performed

Life-cycle process

assessment

completed

Not performed

Checkout Launch and Control System

Criticality

assessment

completed—lead

to IV&V

Not performed IV&V ongoing

Earth Observing System Data

Information System Not performed SET completed—leading to IV&V IV&V ongoing

Gravity Probe-B Not performed

SET ongoing with

negotiations to

expand tasks

Not applicable

Hyper-X

Completed

independent

assessment—

lead to SET

SET ongoing Not applicable

Integrated Asset Management Not performed Not performed IV&V ongoing

Integrated Financial Management

Program Not performed Not performed IV&V ongoing

1

 The project manager determined that IV&V activities were not needed and that verification and

validation activities adequately mitigated project risks.
12

Appendix B

Status of Work

by Independent Verification & Validation (IV&V) Facility

Program/Project Independent

Assessment

Software & System

Engineering Tasks

(SET)

IV&V

International Space Station Not performed Not performed IV&V ongoing

Mars Surveyor Program 1998

Criticality risk

assessment

completed

SET completed

Recommendations

for IV&V not

implemented2

Mars 2001/Genesis Not performed

Agreement to

perform SET being

negotiated

Not applicable

Mission Control Center

Independent

assessment

completed

Not performed Not performed

National Polar-orbiting Operational

Environmental Satellite System Not performed

SET ongoing, and

recommendations for

IV&V anticipated

Agreement for

IV&V to be

negotiated, as

appropriate

Picasso-Cena

Independent

assessment

completed

Not performed

Project identified

as a candidate for

future IV&V

Production Support Flight Control

Computers Not performed SET completed Not performed

Small Spacecraft Technology

Initiative/Clark

Independent

assessment

completed

Not performed

Not performed—

launch delayed,

and project later

cancelled

Space Shuttle Not performed Not performed IV&V ongoing

Space Shuttle Upgrade Not performed Not performed IV&V ongoing

Space InfraRed Telescope Facility

Independent

assessment is

ongoing

Not applicable Not applicable

2 The project manager determined that IV&V activities were not needed and that verification and validation

activities adequately mitigated project risks.
13

Appendix B

Status of Work

 by Independent Verification & Validation (IV&V) Facility

Program/Project Independent

Assessment

Software & System

Engineering Tasks

(SET)

IV&V

Stratospheric Aerosol and Gas

Experiment III

Independent

assessment is

ongoing

Not applicable Not applicable

Stratospheric Observatory for Infrared

Astronomy

Independent

assessment

completed

Not performed

Agreement for

IV&V being

negotiated 3

X-33 Not performed Not performed IV&V ongoing

X-34

Independent

assessment

completed—SET

recommended

SET ongoing Not applicable

X-37 Not performed Not performed

Negotiating

agreement for

IV&V

X-38 Not performed Not performed

Negotiating

agreement for

IV&V

3 The project initially determined that IV&V was not needed. Subsequently, the project determined that

the IV&V Facility would perform IV&V. The Facility and project are negotiating an agreement for the

IV&V.
14

Appendix C. Management’s Response
15

Appendix C
16

Appendix C
17

Appendix C
18

Appendix D. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters

A/Administrator

AE/Chief Engineer

AI/Associate Deputy Administrator

AO/Chief Information Officer

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

C/Associate Administrator for Headquarters Operations

G/General Counsel

H/Associate Administrator for Procurement

HK/Director, Contract Management Division

HS/Director, Program Operations Division

J/Associate Administrator for Management Systems

JM/Acting Director, Management Assessment Division

L/Associate Administrator for Legislative Affairs

M/Associate Administrator for Space Flight

Q/Associate Administrator for Safety and Mission Assurance

R/Associate Administrator for Aerospace Technology

S/Associate Administrator for Space Science

U/Associate Administrator for Life and Microgravity Sciences and Applications

Y/Associate Administrator for Earth Science

Z/Associate Administrator for Policy and Plans

NASA Centers

Director, Goddard Space Flight Center

 Acting Director, NASA Independent Verification and Validation Facility

Chief Counsel, John F. Kennedy Space Center

Director, Jet Propulsion Laboratory

Director, Lyndon B. Johnson Space Center

Director, Langley Research Center

Non-NASA Federal Organizations and Individuals

Assistant to the President for Science and Technology Policy

Director, Office of Management and Budget
19

Appendix D

Non-NASA Federal Organizations and Individuals (Cont.)

Deputy Director of Management, Office of Management and Budget

Deputy Associate Director, Energy and Science Division, Office of Management and

 Budget

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office

 of Management and Budget

Professional Assistant, Senate Subcommittee on Science, Technology, and Space

Chairman and Ranking Minority Member – Congressional Committees and

Subcommittees

Senate Committee on Appropriations

Senate Subcommittee on VA, HUD, and Independent Agencies

Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science, Technology, and Space

Senate Committee on Governmental Affairs

House Committee on Appropriations

House Subcommittee on VA, HUD, and Independent Agencies

House Committee on Government Reform

House Subcommittee on Government Management, Information, and Technology

House Subcommittee on National Security, Veterans Affairs, and International Relations

House Committee on Science

House Subcommittee on Space and Aeronautics, Committee on Science

Congressional Member

Honorable Pete Sessions, U.S. House of Representatives
NASA Assistant Inspector General for Auditing

Reader Survey

The NASA Office of Inspector General has a continuing interest in improving the

usefulness of our reports. We wish to make our reports responsive to our customers’

interests, consistent with our statutory responsibility. Could you help us by completing

our reader survey? For your convenience, the questionnaire can be completed

electronically through our homepage at http://www.hq.nasa.gov/office/oig/hq/audits.html

or can be mailed to the Assistant Inspector General for Auditing; NASA Headquarters,

Code W, Washington, DC 20546-0001.

Report Title: 

Report Number: Report Date: 

Circle the appropriate rating for the following statements.

Strongly

Agree Agree Neutral Disagree

Strongly

Disagree N/A

1. The report was clear, readable, and

logically organized.

5 4 3 2 1 N/A

2. The report was concise and to the

point.

5 4 3 2 1 N/A

3. We effectively communicated the

audit objectives, scope, and

methodology.

5 4 3 2 1 N/A

4. The report contained sufficient

information to support the finding(s)

in a balanced and objective manner.

5 4 3 2 1 N/A

Overall, how would you rate the report?

Excellent Fair

Very Good Poor

Good

If you have any additional comments or wish to elaborate on any of the above

responses, please write them here. Use additional paper if necessary. 

 

 

 
How did you use the report? 

 

 

 

 

 

 

How could we improve our report? 

 

 

 

 

 

 

How would you identify yourself? (Select one)

Congressional Staff Media

NASA Employee Public Interest

Private Citizen Other: 

Government: Federal: State: Local: 

May we contact you about your comments?

Yes: ______ No: ______

Name: ____________________________

Telephone: ________________________

Thank you for your cooperation in completing this survey.
Major Contributors to the Report

David L. Gandrud, Program Director, Information Technology Program Audits

Roger W. Flann, Program Manager

Bessie J. Cox, Auditor-in-Charge

Sandra L. Laccheo, Auditor

Nancy C. Cipolla, Report Process Manager

Barbara J. Smith, Program Assistant