AUDIT REPORT IG-03-011

Report Recipients: 

AE/Chief Engineer 

Q/Associate Administrator 

 for Safety and Mission 

 Assurance 

S/Associate Administrator 

 for Space Science

INDEPENDENT VERIFICATION AND VALIDATION 

OF SOFTWARE 

March 28, 2003 

cc: 

ADT/Associate Deputy

 Administrator for 

 Technical Programs

AE/Software Engineering 

Initiative Program

Executive 

AO/Acting Chief 

 Information Officer 

B/Deputy Chief Financial 

 Officer for Financial 

 Management 

B/Deputy Chief Financial 

 Officer for Resources 

 (Comptroller) 

G/General Counsel 

R/Associate Administrator 

 for Aerospace Technology

Y/Associate Administrator 

for Earth Science 

JM/Director, Management 

 Assessment Division 

GRC/0100/Director, Glenn 

 Research Center 

GSFC/100/Director, NASA 

 IV&V Facility

JPL/1000/Director, Jet 

 Propulsion Laboratory

National Aeronautics and 

Space Administration

OFFICE OF INSPECTOR GENERAL 

Released by: ___[Original Signed By]_______

Alan J. Lamoreaux, Assistant Inspector General for Audits
IG-03-011 March 28, 2003 

 A-02-005-00

Independent Verification and Validation of Software 

Independent verification and validation (IV&V) is a critical management control for 

minimizing the risk of software-related, catastrophic mission failure. The audit focused 

on the effectiveness of NASA’s procedures for ensuring that the appropriate level of 

IV&V is performed on its software development projects. Details regarding the audit 

objectives, background, scope, and methodology are in Appendix B. 

We found that NASA had not effectively ensured that all applicable software 

development projects were assessed to determine their appropriate level of IV&V. 

• NASA did not provide a complete list of all applicable software development 

projects to the IV&V Facility – the Agency’s center of expertise for IV&V 

processes and technology. Such a list would have enabled Facility personnel to 

identify projects that had not yet been assessed to determine the need for IV&V.

• NASA did not include IV&V requirements in the current Jet Propulsion 

Laboratory (JPL) contract that will expire on September 30, 2003. NASA 

management agreed to incorporate the requirements into the follow-on contract 

effective October 1, 2003.

As a result of this condition, the safety, quality, and reliability of some of the Agency’s 

programs and projects, including mission-critical programs and projects, could be 

compromised. For example, the Fluids and Combustion Facility (FCF) project, managed 

by Glenn Research Center, was a likely candidate for independent software reliability 

assurance. The FCF will be a permanent multi-rack research laboratory for conducting 

microgravity experiments onboard the International Space Station. The $72 million 

project consists of hardware and related software that is expected to host more than 100 

fluids and combustion experiments over its anticipated lifespan. After we brought our 

concerns about this project to management’s attention, the NASA IV&V Facility told the 

project manager to assess the FCF project for IV&V; project management officials 

subsequently concluded that the FCF required IV&V. Without IV&V, the microgravity 

research program faced an unnecessary risk that the FCF project would not perform as 

intended. 

Management Control Needed to Ensure Effective Application of IV&V Policy 

NASA Policy Directive (NPD) 8730.4 requires that the IV&V Facility maintain “explicit 

involvement” with project managers in determining the appropriate level of IV&V for 

their software development projects. To ensure effective implementation of this 

requirement, the Agency should provide the IV&V Facility a complete list of all 

applicable software development projects. Facility personnel could then identify 

software projects for which IV&V assessment criteria should be applied and, in turn, 
advise the cognizant project managers to apply the criteria to those projects. Details 

regarding NASA’s IV&V policy for software are in Appendix C. Absent a complete list 

of applicable software development projects, IV&V Facility personnel initiated their own 

software project identification process by contacting Center personnel and searching the 

Agency’s mission and project Web sites for information on software development 

activities. The Facility’s efforts were commendable but did not result in a complete and 

accurate list of software development projects. 

IV&V Requirements Not Included in Current JPL Contract 

NPD 8730.4 states that the Agency’s IV&V policy is applicable to JPL to the extent 

specified in the Agency’s contract with the California Institute of Technology. In this 

regard, NASA had not incorporated the NPD requirement into the current contract, and 

JPL’s internal policies did not require project managers to complete the Agency’s IV&V 

assessment criteria. Further information regarding the Agency’s IV&V policy relative to 

the California Institute of Technology contract is in Appendix D. 

We discussed the Agency’s IV&V policy with JPL procurement officials, NASA Office 

of Space Science officials, and officials in the NASA Offices of Safety and Mission 

Assurance and Chief Engineer. NASA management told us that it would not be prudent 

or cost-effective to incorporate software IV&V requirements into the current JPL 

contract; however, management agreed to incorporate the requirements into the follow-on 

contract effective October 1, 2003. The follow-on contract was awarded in November 

2002 and requires JPL to comply with NPD 8730.4. 

Conclusion 

NASA should apply effective management controls, on a recurring basis, to ensure the 

IV&V policy is fully implemented. Until needed corrective actions are implemented, 

NASA’s software management, engineering, and assurance processes will not be fully 

integrated with the Agency’s program and project management processes. Further, 

NASA has not ensured that it has developed risk analyses and risk management strategies 

at each stage of the software development life-cycle. 

Recommendations for Corrective Action 

1. The NASA Chief Engineer, in coordination with the Associate Administrator for 

Safety and Mission Assurance, should establish a process that provides the NASA 

IV&V Facility, on a recurring basis, a complete and accurate list of the Agency’s 

programs and projects governed by either NASA Procedures and Guidelines 7120.5A 

or NASA Technical Standard 8719.13A. 

2. The NASA Chief Engineer should verify that the NASA IV&V Facility initiates 

appropriate actions to ensure that the programs and projects identified in 

2
Recommendation 1 comply with the Agency's software IV&V policy. 

Management’s Response and our Evaluation of the Response 

NASA concurred with the intent of the recommendations and has initiated corrective 

actions. Although we consider management’s comments (Appendix F) to be responsive 

to the recommendations, the success of the alternative actions to be taken will depend on 

effective coordination between the Program Management Councils and the IV&V 

Facility. Effective coordination will help ensure that the Agency has adequately 

considered the need for IV&V and has performed IV&V where appropriate. Details 

related to disposition and closure of the recommendations are in Appendix A. 

3
List of Appendixes 

Appendix A – Status of Recommendations 

Appendix B – Objective, Background, Scope, and Methodology 

Appendix C – NASA’s IV&V Policy for Software 

Appendix D – IV&V Policy Relative to NASA’s Contract with the 

California Institute of Technology 

Appendix E – Summary of Prior Coverage 

Appendix F – Management’s Response 

Appendix G – Report Distribution 

Acronyms Used in the Report 

FCF Fluids and Combustion Facility 

IV&V Independent Verification and Validation 

JPL Jet Propulsion Laboratory 

MCO Mars Climate Orbiter 

MPIAT Mars Program Independent Assessment Team 

MPL Mars Polar Lander 

NPD NASA Policy Directive 

NPG NASA Procedures and Guidelines 

OIG Office of Inspector General 

4
Appendix A. Status of Recommendations 

Recommendation No. Resolved Unresolved Open/ECD* Closed

1 X 9/30/2003

2 X 9/30/2003

* Estimated Completion Date 

 

5
Appendix B. Objective, Background, Scope, and Methodology 

Objective

The overall objective of this audit was to determine whether NASA had established and 

implemented effective management controls over the Agency’s software development 

process. Specifically, we determined whether effective controls existed to ensure the 

following: 

• safety, quality, and reliability of software developed by or for NASA; 

• integration of NASA software management, engineering, and assurance processes 

with the Agency’s program/project management processes; and 

• development of risk analyses and risk management strategies at each stage of the 

software development life-cycle.

Background 

Independent verification and validation (IV&V) is a critical management control that 

helps ensure the safety, quality, and reliability of NASA’s software. Software 

verification consists of assessing objective evidence at different phases of the 

development life cycle to ensure that the software product complies with specifications. 

Software validation takes place at the end of the software development process and 

consists of performing tests and assessing other objective evidence to determine whether 

a software system will accomplish its intended purpose. Validation ensures that a 

software system will perform as expected under operational conditions. 

Independence is a key component to successful software verification and validation. 

Software developers and project managers may have vested interests and may not be 

objective in their self-assessments. Performing software verification and validation 

independently of the development and management functions helps to ensure that 

verification and validation activities are unbiased and based on objective evidence. 

Congress established the IV&V Facility in Fairmont, West Virginia, in October 1991 to 

address recommendations made by the National Research Council and the Presidential 

Commission on the Space Shuttle Challenger Accident. The Facility is technically 

independent but is managed by the Goddard Space Flight Center (Goddard) and relies on 

Goddard for support functions such as personnel, training, and procurement. 

The NASA IV&V Facility provides two levels of software assurance: independent 

assessment (IA) and full IV&V. An IA is a one-time review of existing products and 

plans, whereas IV&V is applied over the entire life cycle of a software system. Full 

IV&V provides more rigorous software assurance. 

The importance of software testing is underscored by the failures of the Mars Climate 

Orbiter (MCO) and Mars Polar Lander (MPL) in 1999. NASA developed and launched 

6
Appendix B 

the two missions (both part of the JPL Mars '98 Development Project) at a total cost of 

more than $327 million. To investigate the failures, NASA established the Mars Program 

Independent Assessment Team (MPIAT). The MPIAT summary report states:

. . . one mistake can be mission catastrophic. Mistakes are prevented by

oversight, test, and independent analysis, which were deficient for MCO. 

Specifically, software testing was inadequate. 

Regarding the MPL, the assessment team’s report states: 

As with MCO, the most probable failure of the Mars Polar Lander resulted from

inadequate checks and balances that permitted an incomplete systems test and 

allowed a significant software design flaw to go undetected. 

NASA has made progress toward establishing and implementing effective management 

controls over the Agency’s software development process. For example, the Agency 

established policy (NPD 8730.4) to conduct IV&V of software based on the cost, size, 

complexity, life span, risk, and consequences of failure. In addition, the Agency 

established quantifiable criteria for assessing whether IV&V should be applied to a 

software development project and for performing the appropriate level of IV&V. 

However, as stated in this report, NASA had not ensured that needed IV&V was 

performed. 

Scope and Methodology 

NASA employs many processes, techniques, tools, and services to ensure the safety, 

quality, and reliability of its software. Because of the importance of IV&V as a 

management control in the software development process, our audit focused on the 

effectiveness of Agency procedures for ensuring that IV&V was performed. We did not 

assess the adequacy of detailed IV&V test procedures for ensuring that NASA software 

products will comply with specifications or that software systems will perform as 

expected under operational conditions.

To accomplish our objectives, we completed these steps: 

 

• Reviewed NASA policies, procedures, and guidelines pertaining to the Agency’s 

management of software development and use of the NASA IV&V Facility in 

Fairmont, West Virginia. 

• Interviewed officials in the NASA Office of Safety and Mission Assurance and 

NASA Office of Chief Engineer regarding the Agency’s IV&V policies and the 

applicability of IV&V policy to the Jet Propulsion Laboratory.

7
Appendix B 

• Interviewed officials from the NASA IV&V Facility regarding procedures for 

assessing applicable software development projects for IV&V. 

• Interviewed officials at selected Centers regarding procedures for complying with 

NASA’s IV&V policy. Center officials included project managers, software 

quality assurance managers, personnel in systems management offices, and 

procurement personnel. 

• Compared Center-provided lists of software development projects to IV&V 

Facility records of projects for which the Agency’s IV&V assessment criteria had 

been adequately applied. 

We did not assess the reliability of computer-processed data because we did not rely on 

such data to achieve our objectives. 

Management Controls Reviewed 

We interviewed officials at NASA Headquarters, the NASA IV&V Facility, and selected 

Centers to identify and assess management controls relating to the Agency’s management 

of the software development process. We considered the management controls to be 

adequate except that NASA had not fully implemented effective controls to ensure that 

all applicable software development projects were adequately assessed to determine the 

appropriate level of IV&V. 

Audit Field Work 

We performed the audit field work from October 2001 through August 2002 at NASA 

Headquarters, the IV&V Facility, Glenn Research Center, the Jet Propulsion Laboratory, 

Johnson Space Center, Langley Research Center, and Marshall Space Flight Center. We

conducted the audit in accordance with generally accepted government auditing 

standards. 

8
Appendix C. NASA’s IV&V Policy for Software

Memorandum Establishing Interim IV&V Policy. On July 21, 2000, the NASA Chief 

Engineer issued a memorandum on the Agency’s policy for software IV&V. The Chief 

Engineer identified assessment criteria that Agency personnel should use in deciding 

whether IV&V is needed for a software development project. The criteria included steps 

for assessing the following consequences of software failure: 

 

• loss of life, 

• catastrophic mission failure, 

• loss of equipment, 

• waste of software resource investment, 

• negative political and public image stemming from the failure of a system, and 

• effect on routine operations. 

The criteria also included steps for assessing the probability of software failure. Certain 

variables could affect the probability of failure: 

• amount of contractor support, 

• schedule pressure, 

• degree of innovation, and 

• number of software lines of code. 

NASA Policy Directive (NPD) 8730.4, "Software Independent Verification and 

Validation (IV&V) Policy," August 1, 2001. The NPD establishes Agency policy to 

conduct IV&V based on the cost, size, complexity, life span, risk, and consequences of 

software failure. Section 1 of the Directive states that NASA will do the following: 

a. Establish and apply criteria, tools, and methodology to evaluate and assess software 

risk for the purpose of identifying the appropriate level of IV&V. 

b. For programs and projects governed by NPG [NASA Procedures and Guidelines]

7120.5A, task the NASA IV&V Facility in Fairmont, WV, to manage the performance 

of all IV&V [emphasis added] for software identified per the established criteria, and for 

any other safety critical software (as defined in NASA-STD-8719.13A). [NASA 

Technical Standard NASA-STD-8719.13A, “Software Safety,” September 15, 1997, 

describes the activities necessary to ensure that safety is designed into software that is 

acquired or developed by NASA and that safety is maintained throughout the software 

life cycle.] 

c. Require programs and projects governed by NPG 7120.5A to determine the level of 

IV&V to be performed with the explicit involvement [emphasis added] of the NASA 

IV&V Facility. 

9
Appendix C 

Section 2 of the NPD states that the Agency’s IV&V policy is applicable to “... NASA 

Headquarters and NASA Centers, ... and to the Jet Propulsion Laboratory to the extent 

specified in the contract.” The NASA Office of Safety and Mission Assurance is 

responsible for NPD 8730.4 and the Agency’s overall policy regarding software IV&V. 

The NASA Chief Engineer is responsible for ensuring that all programs and projects 

follow the Agency’s IV&V policy. 

10
Appendix D. IV&V Policy Relative to NASA’s Contract With the 

California Institute of Technology

NASA issued contract NAS7-1407 to the California Institute of Technology to provide 

for operation of the Jet Propulsion Laboratory (JPL) from October 1998 through 

September 2003. Contract Section G-14, “NASA Issuance System,” states: 

The parties hereto agree that NASA Management Directives System publications 

("NASA Issuances") are not in and of themselves applicable to the Contractor, and that 

the Contractor therefore is not obligated merely by virtue of their issuance to implement 

their intent or to observe the policies and procedures set forth therein, irrespective of the 

fact that certain NASA Issuances may state that they apply to JPL. NASA Issuances 

become contractually binding and obligatory upon the Contractor only when and to the 

extent made so by appropriate contractual means.

The NASA Office of Space Science has cognizance over the JPL contract. 

During the audit, we found that JPL’s internal policies did not require project managers 

to complete the Agency’s IV&V assessment criteria. We brought this matter to the 

attention of cognizant JPL officials who subsequently established local procedures that 

required project managers to complete the Agency's IV&V assessment criteria for 

determining the need for IV&V. JPL issued the official requirement in Section 7.4 of 

"Flight Project Practices, Rev. 2" (Document Identification 58032), effective June 28, 

2002. Although JPL officials did not issue the official requirement until June 2002, 

JPL’s software development projects had been in compliance with the Agency’s software 

IV&V policy since July 2000. 

11
Appendix E. Summary of Prior Coverage

The NASA Office of Inspector General (OIG) issued one report on physical access 

controls at the NASA Independent Verification and Validation (IV&V) Facility and one 

report on the management of software development. The reports are summarized below, 

and copies are available at http://www.hq.nasa.gov/office/oig/hq/reports.html. 

“NASA's Badging Program and Physical Access Controls at the Goddard Space 

Flight Center Independent Verification & Validation Facility,” Report Number 

G-01-026, September 25, 2001. The OIG completed an inspection of NASA's badging 

program and physical access controls at the Goddard Space Flight Center IV&V Facility. 

The objective was to focus on whether policies and procedures were in place to 

adequately control access to facilities including mission-critical locations and facilities 

containing sensitive or controlled information or materials. NASA concurred with the 

report's four recommendations for improving security controls and operational 

effectiveness.

“Software Assurance,” Report Number IG-00-059, September 28, 2000. NASA 

lacked adequate management controls for determining whether to use IV&V in its 

software development projects and for collecting, analyzing, and reporting software 

metrics designed to monitor these projects. This condition occurred because NASA had 

not issued guidelines to implement the controls. As a result, NASA had less assurance 

that the risks of potential software failures were adequately reduced through IV&V and 

the implementation of sound software assurance policies and procedures. NASA 

concurred with each of the report’s two recommendations and issued interim IV&V 

criteria for use by program and project managers in determining whether new or existing 

projects should be subject to IV&V. Also, the Agency issued its IV&V policy in NASA 

Policy Directive (NPD) 8730.4, "Software Independent Verification and Validation 

(IV&V) Policy.” 

12
Appendix F. Management’s Response

13
Appendix F

14
Appendix F

15
Appendix F 

16
Appendix G. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters 

A/Administrator 

AA/Chief of Staff

AE/Chief Engineer 

AO/Chief Information Officer

ADT/Associate Deputy Administrator for Technical Programs 

B/Deputy Chief Financial Officer for Financial Management 

B/Deputy Chief Financial Officer for Resources (Comptroller) 

BF/Director, Financial Management Division 

G/General Counsel 

H/Assistant Administrator for Procurement 

HK/Director, Contract Management Division 

HS/Director, Program Operations Division 

J/Assistant Administrator for Management Systems 

JM/Director, Management Assessment Division 

L/Assistant Administrator for Legislative Affairs 

M/Associate Administrator for Space Flight 

Q/Associate Administrator for Safety and Mission Assurance 

R/Associate Administrator for Aerospace Technology 

S/Associate Administrator for Space Science 

Y/Associate Administrator for Earth Science 

NASA Centers 

GRC/0100/Director, Glenn Research Center 

GSFC/100/Director, Goddard Space Flight Center 

JPL/1000/Director, Jet Propulsion Laboratory 

KSC/AF/Chief Counsel, John F. Kennedy Space Center 

Non-NASA Federal Organizations and Individuals 

Assistant to the President for Science and Technology Policy 

Deputy Associate Director, Energy and Science Division, Office of Management and 

 Budget 

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office 

 of Management and Budget 

17
Appendix G 

Non-NASA Federal Organizations and Individuals (Cont.)

Managing Director, Acquisition and Sourcing Management Team, General Accounting 

 Office 

Senior Professional Assistant, Senate Subcommittee on Science, Technology, and Space 

Chairman and Ranking Minority Member – Congressional Committees and 

Subcommittees 

Senate Committee on Appropriations 

Senate Subcommittee on VA, HUD, and Independent Agencies 

Senate Committee on Commerce, Science, and Transportation 

Senate Subcommittee on Science, Technology, and Space 

Senate Committee on Governmental Affairs 

House Committee on Appropriations 

House Subcommittee on VA, HUD, and Independent Agencies 

House Committee on Government Reform

House Subcommittee on Government Efficiency, Financial Management and 

 Intergovernmental Relations 

House Subcommittee on Technology and Procurement Policy 

House Committee on Science 

House Subcommittee on Space and Aeronautics 

Congressional Member 

The Honorable Pete Sessions, U.S. House of Representatives 

18
NASA Assistant Inspector General for Auditing 

Reader Survey 

The NASA Office of Inspector General has a continuing interest in improving the 

usefulness of our reports. We wish to make our reports responsive to our customers’ 

interests, consistent with our statutory responsibility. Could you help us by completing 

our reader survey? For your convenience, the questionnaire can be completed 

electronically through our homepage at http://www.hq.nasa.gov/office/oig/hq/audits.html

or can be mailed to the Assistant Inspector General for Auditing; NASA Headquarters, 

Code W, Washington, DC 20546-0001. 

Report Title: Independent Verification and Validation of Software 

 

Report Number: Report Date: 

Circle the appropriate rating for the following statements. 

Strongly 

Agree Agree Neutral Disagree 

Strongly 

Disagree N/A 

1. The report was clear, readable, and logically 

organized. 5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A 

3. We effectively communicated the audit 

objectives, scope, and methodology. 5 4 3 2 1 N/A 

4. The report contained sufficient information 

to support the finding(s) in a balanced and 

objective manner. 

5 4 3 2 1 N/A 

Overall, how would you rate the report? 

# Excellent # Very Good # Good # Fair # Poor

If you have any additional comments or wish to elaborate on any of the above 

responses, please write them here. Use additional paper if necessary. 
How did you use the report?

How could we improve our report?

How would you identify yourself? (Select one) 

 # Congressional Staff # Media 

# NASA Employee # Public Interest 

# Private Citizen # Other:

# Government: Federal: State: Local:

May we contact you about your comments? 

Yes: ________ No: ______ 

Name: ____________________________ 

Telephone: ________________________ 

Thank you for your cooperation in completing this survey. 
Additional Copies 

To obtain additional copies of this report, contact the Assistant Inspector General for 

Audits at (202) 358-1232. 

Suggestions for Future Audits 

To suggest ideas for or to request future audits, contact the Assistant Inspector General 

for Audits. Ideas and requests can also be mailed to: 

Assistant Inspector General for Audits 

Code W

NASA Headquarters 

Washington, DC 20546-0001 

NASA Hotline 

To report fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at (800) 

424-9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; 

or write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, 

Washington, DC 20026. The identity of each writer and caller can be kept confidential, 

upon request, to the extent permitted by law. 

Major Contributors to the Report 

David L. Gandrud, Program Director, Aeronautics and Infrastructure Audits 

Tony A. Lawson, Audit Manager, Aeronautics and Infrastructure Audits 

Roger W. Flann, Audit Manager, Aeronautics and Infrastructure Audits 

James H. Pearce, Auditor 

Bessie J. Cox, Auditor 

Barbara J. Smith, Program Assistant