IG-00-053

AUDIT

REPORT NASA’S AVIATION SAFETY PROGRAM

September 26, 2000

National Aeronautics and

Space Administration

OFFICE OF INSPECTOR GENERAL
Additional Copies

To obtain additional copies of this report, contact the Assistant Inspector General for Auditing

at (202) 358-1232, or visit www.hq.nasa.gov/office/oig/hq/issuedaudits.html.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General for

Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing

Code W

NASA Headquarters

Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement contact the NASA Hotline at (800)

424-9183, (800) 535-8134 (TDD), or at www.hq.nasa.gov/office/oig/hq/hotline.html#form; or

write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, Washington,

DC 20026. The identity of each writer and caller can be kept confidential, upon request, to the

extent permitted by law.

Reader Survey

Please complete the reader survey at the end of this report or at

http://www.hq.nasa.gov/office/oig/hq/audits.html.

Acronyms

ASIST Aviation Safety Investment Strategy Team

AvSP Aviation Safety Program

FAA Federal Aviation Administration

GAO General Accounting Office

GPRA Government Performance and Results Act

NTSB National Transportation Safety Board
W September 26, 2000

TO: A/Administrator

FROM: W/Inspector General

SUBJECT: INFORMATION: NASA's Aviation Safety Program

Report Number IG-00-053

The NASA Office of Inspector General has completed an audit of NASA's Aviation Safety

Program (AvSP). We found that coordination between NASA, the Federal Aviation

Administration (FAA), and other partners is adequate. However, NASA has not consistently

portrayed the intended performance of its Aviation Safety Initiative. The Aviation Safety

Initiative is a combination of redirected Research and Technology Base1 activities and the

creation of the focused2 AvSP. Specifically, the Agency has been inconsistent in stating its goal

for the Initiative3, identifying all baselines4 that will measure progress towards meeting the goal,

emphasizing the risks5 involved with the development and implementation of the aviation safety

technologies and the effects of those risks on achieving the goal, and integrating the goal and

baseline with the FAA. As a result, NASA may not be able to measure its contribution towards

meeting the National Aviation Safety Goal. In addition, NASA may not fulfill the expectations

of the Congress, the aviation community,6 and the public relating to NASA's aviation safety

efforts.

 

1 Research and Technology Base programs enhance safety activities with tools and insights to fundamental principles

and support the development of technologies that will address accidents involving hazardous weather, controlled flight

into terrain, human error, and mechanical or software malfunctions.

2 NASA selects specific technologies based on national needs for further development in focused programs, such as the

AvSP, with a specified class of research for potential applications.

3

 The goal of the Aviation Safety Initiative, to contribute to reductions in the fatal aviation accident rate, is

inconsistently stated in Agency documents. These inconsistencies are identified in Appendix C.

4

 The Aviation Safety Initiative established a baselines to measure performance for the aviation fatal accident rates.

However, performance measures did not include baselines for aviation accident rates (nonfatal) or fatality rates. The

baselines are identified in Appendix D.

5

 The success of the AvSP is dependent on overcoming several risks or factors including the challenge of technical

development, user implementation, and availability of resources.

6 As used in this report, the aviation community includes Government agencies, airlines, airline maintenance, and

aviation-related manufacturers and suppliers.
2

Background

The commercial aviation accident rate is very low and has remained fairly constant. Air travel is

currently one of the safest modes of transportation. However, the flat accident rate and the

projected growth in air travel will inevitably produce a higher number of accidents in future

years, unless the aviation community acts to reduce the rate of accidents. NASA is developing

technology to make air travel safer through the Aviation Safety Initiative. Without the

development of new technology, there could be a fatal airliner accident somewhere in the world

almost weekly by the year 2015.7 The 1997 report, "White House Commission on Aviation

Safety and Security," recommended a national goal to reduce the aviation fatal accident rate by

a factor 5 (80 percent) within 10 years (2007). The AvSP consists of six projects that will

provide research and technology needed to help the FAA and the aviation community achieve

the national goal. The AvSP is an ambitious program that includes a partnership between

NASA, FAA, other Government agencies, and the aviation industry. For fiscal years 2000

through 2004, NASA budgeted about $500 million for the NASA Aviation Safety Initiative, of

which about $377 million is for AvSP.

Recommendations

We recommended that the Associate Administrator for Aerospace Technology (1) clarify the

Agency's contribution toward the National Aviation Safety Goal to ensure a consistent

representation of NASA's intended performance; (2) identify all baselines necessary to measure

the Agency's performance in meeting the established goals; (3) revise program documentation

available to stakeholders to adequately reflect the risks of development and implementation; and

(4) coordinate with the FAA to resolve differences in baselines and metrics used to guide the

national efforts on aviation safety. These improvements will allow NASA to appropriately

measure NASA's contributions toward meeting the National Aviation Safety goal and provide a

more accurate portrayal of its efforts and the risks involved to Congress, the aviation

community, and the public.

Management's Response

Management concurred with all recommendations. The Associate Administrator for Aerospace

Technology will revise all documentation to ensure that it consistently reflects the Agency's

contribution toward the National Aviation Safety Goal. Management also identified the

baselines necessary to measure its performance; agreed to communicate and stress to partners,

customers, and stakeholders the facts concerning the risk of implementation; and adjusted its

baseline to match the FAA baseline years.

 

7 The 1997 report, "White House Commission on Aviation Safety and Security," contains the projection by the Boeing

Commercial Aircraft Group.
3

Details on the status of the recommendations are in the recommendations section of the report.

[Original signed by]

Roberta L. Gross

Enclosure

Final Report on Audit of NASA's Aviation Safety Program
FINAL REPORT

NASA’S AVIATION SAFETY PROGRAM
W September 26, 2000

TO: R/Associate Administrator for Aerospace Technology

FROM: Assistant Inspector General for Auditing

SUBJECT: Final Report on the Audit of NASA’s Aviation Safety Program

Assignment Number A0000100

Report Number IG-00-053

The subject final report is provided for your information and use. Please refer to the Executive

Summary for the overall audit results. Our evaluation of your response is incorporated into the

body of the report. Your comments on a draft of this report were responsive, and actions are

sufficient to close recommendations 2 and 4. Recommendations 1 and 3 will remain open for

reporting purposes until corrective action is completed. Please notify us when action has been

completed on these recommendations, including the extent of testing performed to ensure

corrective actions are effective.

If you have questions concerning the report, please contact Ms. Karen E. VanSant, Program

Director, Aerospace Technology Audits, at (256) 544-1149,

Ms. Carol A. St. Armand, Program Manager, at (301) 286-7269, or

Ms. Sandra K. Leibold, Auditor-in-Charge, at (256) 544-0970. We appreciate the courtesies

extended to the audit staff. The final report distribution is in Appendix F.

[Original signed by]

Russell A. Rau

Enclosure
2

cc:

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

JM/Acting Director, Management Assessment Division

RP/Director, Programs Division

LaRC/Director, Langley Research Center
Contents

Executive Summary, i

Introduction, 1

Finding and Recommendations, 2

NASA’s Aviation Safety Goal, 2

Appendix A - Objectives, Scope, and Methodology, 10

Appendix B - AvSP Contributions toward the National Aviation Safety

 Goal, 14

Appendix C - Examples of Inconsistent Aviation Safety Goals, 15

Appendix D - Comparison of NASA and FAA Methodologies, 16

Appendix E - Management’s Response, 17

Appendix F - Report Distribution, 21
NASA Office of Inspector General

IG-00-053 September 26, 2000

 A0000100

NASA’s

 Aviation Safety Program

Executive Summary

Background. The 1997 report, “White House Commission on Aviation Safety and Security,”

recommended a national goal to reduce the aviation fatal accident rate by a factor of five (80

percent) within 10 years (2007). NASA responded to the report by initiating a major program

planning effort to define the research the Agency will conduct. NASA initiated this effort with

the Aviation Safety Investment Strategy Team (ASIST) and involved industry, Government, and

academic organizations. The recommendations of the ASIST provided the foundation of the

NASA Aviation Safety Initiative. The Initiative is a combination of redirected Research and

Technology Base activities and the creation of the focused AvSP. The AvSP consists of six

projects that will provide research and technology needed to help the FAA and the aerospace

industry achieve the national goal.

For fiscal years (FY’s) 2000 through 2004, NASA budgeted about $500 million for the NASA

Aviation Safety Initiative, of which $377 million is for the AvSP. The AvSP is an ambitious

program that includes a partnership between NASA, FAA, other Government agencies, and the

aviation industry. The Langley Research Center (Langley) is leading the AvSP and works with

personnel at Ames Research Center, Glenn Research Center, and Dryden Flight Research

Center. Additionally, the AvSP works in concert with the full spectrum of commercial8 and

general aviation9 industry manufacturers, suppliers, and operators in implementing this effort.

Objectives. The audit objectives were to assess the adequacy of NASA’s coordination with

FAA and other partners and the Agency’s overall success in achieving program goals and

objectives. Appendix A contains details on the objectives, scope, and methodology used for

this audit.

Results of Audit. The coordination between NASA, FAA, and other partners is adequate.

NASA has a memorandum of understanding with FAA for the overall AvSP. Also, NASA and

 

8

 Commercial aviation includes both large air carrier operations and smaller commuter operations.

9

 General aviation includes a wide variety of aircraft, ranging from corporate jets to small piston-engine aircraft as well

as helicopters, gliders, and aircraft used in operations such as firefighting and agricultural spraying.
ii

FAA established a Joint Working Group to ensure implementation and monitoring of the

agreement and are preparing agreements for each of the six projects within the program.

Further, FAA detailed a representative to NASA at Langley. NASA also has an agreement in

process with the National Imagery and Mapping Agency to arrange for additional staff.

However, NASA has not consistently portrayed its goal and identified all the measurement

baselines for its Aviation Safety Initiative. Further, NASA has not adequately emphasized the

risks involved with developing and implementing various aviation safety technologies and how

those risks affect the achievement of program success. The Agency is also inconsistent in

integrating its goal and baseline with FAA. As a result, NASA may not be able to measure its

contribution towards meeting the National Aviation Safety Goal. Also, NASA may not fulfill the

expectations of Congress, the aviation community, and the public relating to NASA's aviation

safety efforts.

Recommendations. NASA should clarify its contribution toward the national aviation safety

goal and revise plans, including those with FAA, and goals accordingly to ensure various

Agency documents and Web sites are consistent with NASA’s intended performance. The

Agency should establish baselines to measure its performance relative to its established goals.

Also, NASA should place more emphasis on informing stakeholders about the development

and implementation risks that could adversely affect program success.

Management’s Response. Management concurred with all the recommendations. NASA

management will update documentation to consistently state the AvSP goal. Management has

also adjusted its baseline to match the FAA baseline and will ensure that the Agency’s partners,

customers, and stakeholders understand the development and implementation risks. The

complete text of the response is in Appendix E. Management’s comments are responsive to the

recommendations. 
Introduction

The commercial aviation accident rate is very low, but it has shown virtually no improvement

over the last 20 years. The flat accident rate and the projected growth in air travel will

inevitably produce a higher number of accidents as years pass, unless the aviation community

takes action to reduce the rate of accidents. If left unchecked, there could be a fatal airliner

accident somewhere in the world almost weekly by the year 2015. This provides a strong

motivation for aggressive efforts to lower the accident rate. Given the visible, damaging, and

tragic effects of even a single major accident, approaching this number of accidents would

clearly have an unacceptable effect on the public’s confidence in the aviation system and impede

the anticipated growth of the commercial air travel market.

General aviation accounts for the largest number of accidents and fatalities, but the focus of the

national goal is fatal accidents in commercial operations. The commercial airlines are the

primary mode of air transportation for most Americans, and airline accidents have the potential

for significant loss of life. The principal causes of general aviation accidents are similar to

commercial aviation accidents, therefore, the White House Commission recommendations will

also help address the safety of general aviation.

NASA has contributed to aviation safety for 40 years through its research and development

efforts in aeronautics. The achievement of the safety goal requires the continuing efforts of

Research and Technology Base programs10 and the focused AvSP. The AvSP is NASA's

most recent effort in aviation safety. The AvSP is structured around eight core technologies that

address the three investment areas identified by the ASIST. Appendix B contains details on the

core technologies under development and additional information on AvSP’s contributions

toward the National Aviation Safety Goal. The three investment areas and their respective

percentage of AvSP resources are Accident Prevention (65 percent); Accident Mitigation (10

percent); and Aviation System-Wide Monitoring, Modeling, and Simulation (25 percent). Full

system-wide implementation of the technologies under development will not be completed by

AvSP program end in 200411 and may not be completed by the targeted goal of 2007.

Success of the program will be based on delivering the planned program milestones, typically

with technology that has been demonstrated through models or prototypes.

 

10 Research and Technology Base programs involved in work supporting aviation safety include Information

Technology, Aerospace Operations Systems, Rotorcraft, Aerospace Propulsion and Power, Aerospace Vehicle

Systems Technologies, and Flight Research.

11 The AvSP runs from FY’s 2000-2004.
2

Finding and Recommendations

NASA’s Aviation Safety Goal

NASA has not clearly defined the intended performance of its Aviation Safety Initiative.

Specifically, the Agency has been inconsistent in: (1) stating its goal for the Initiative, including

its role in the accomplishment of the National Aviation Safety Goal and the accident rates it is

trying to affect; (2) identifying all baselines it will use to measure progress towards meeting its

goal; (3) emphasizing the risks involved with the development and implementation of its aviation

safety technologies and explaining how those risks affect achieving its aviation safety goal; and

(4) integrating its goal and baseline with FAA. These conditions exist because NASA

management publishes a variety of documents on this subject and has not consistently stated in

the various documents what the Agencywide Aviation Safety Initiative will accomplish. As a

result, NASA may not be able to measure its contribution toward meeting the National Aviation

Safety Goal. Also, Congress, the aviation community, and the public may draw inaccurate

conclusions about NASA’s aviation safety efforts, and the President’s overall national aviation

safety goal may not be achieved.

Laws and Guidelines

The Government Performance and Results Act (GPRA) was enacted in 1993 to improve public

confidence in the Federal Government by holding agencies accountable through setting program

goals, measuring performance against those goals, and reporting publicly on progress. NASA

Procedures and Guidelines 1000.2, “NASA Strategic Management Handbook,” dated

February 2000, documents the Agency’s policies, processes, guidelines, and responsibilities for

strategic management as required by GPRA. The Handbook requires that all Agency strategic

plans, the NASA Strategic Plan, Enterprise12 Strategic Plans, and lower level strategic planning

documents include specific elements to ensure consistency of Agency plans and meet GPRA

requirements.

NASA Procedures and Guidelines 7120.5, “NASA Program and Project Management

Processes and Requirements,” dated April 3, 1998, defines the requirements for formulating,

approving, implementing, and evaluating programs and projects. It is intended to support

accomplishment of the NASA programs and projects, consistent with established Agency

strategic planning while satisfying the requirements of multiple stakeholders and customers. The

guidelines require that programs establish metrics related to the program commitments and

 

12 NASA established four Strategic Enterprises to function in primary business areas for implementing NASA’s mission

and serving customers. The four enterprises are Human Exploration and Development of Space, Earth Science,

Aerospace Technology, and Space Science. AvSP is a part of the Aerospace Technology Enterprise.
3

ensure metric data are collected and reported as required by GPRA, the Program Commitment

Agreement13 and other strategic planning

requirements. The guidelines further stipulate that program/project documentation should be

consistent with and provide details for implementing Agency and Enterprise strategic plans.

Aviation Safety Initiative Goals

Although Aerospace Technology Enterprise officials understand that one of the Enterprise goals

is to contribute to reductions in the fatal aviation accident rate, that goal is inconsistently stated in

Agency documents that are available to the aviation community, Congress, and the public.

NASA Strategic plans, Performance Plans and Reports, other program documents, and

Agency Web sites inconsistently state the Agency’s goal regarding aviation safety. The

documents differ in what NASA plans to do and which accident rates are the targets of its

technologies.

Various Agency documents state that NASA’s role is to “enable” reductions in, “contribute” to

reductions in, or “reduce” aviation accident rates. There is a significant difference in meaning

between enabling a reduction in the aviation accident and fatality rates (as stated in the AvSP

draft Program Commitment Agreement, dated November 10, 199914), contributing to a

reduction in aviation accident and fatality rates (as stated in the AvSP program plan, dated

August 1, 1999), and reducing the fatal accident rate by 80 percent (as stated on the AvSP

Internet page, dated August 14, 2000). By using the term “reduce,” NASA does not make

clear that it is only one of several partners that have activities under way to lower the fatal

accident rate and improve aviation safety. The Agency’s Web site implies that the AvSP itself

will reduce the aviation fatal accidents by 80 percent. The Agency does not clarify that the

reduction is a national goal or that AvSP is a contributor toward achieving the goal. While the

terms “contribute” and “enable” are more appropriate in describing NASA’s role in meeting a

national goal, “contribute” most aptly recognizes that NASA is working in partnership with

others on a common goal. Although not as direct as the term "reduce," the use of "enable"

suggests that NASA can make the action possible, but does not clearly reflect the involvement

of others. NASA management agrees that they are one part of a team that together is

contributing toward the national goal. Appendix C provides details on the variations of the goal.

NASA has not consistently identified the accident rates that it wants to lower by the

technologies being developed. Variations range from whether NASA plans to affect the fatal

aircraft accident rate,15 the aircraft accident rate,16 or fatality rates,17 or a combination of all the

 

13 The agreement requires clearly defined objectives and public benefits stated in a way that can be understood by the

average citizen.

14 During our audit, the term “enable” changed to “contribute” in draft versions of the Program Commitment

Agreement.

15 The fatal accident rate can be calculated as the number of accidents with one or more fatalities divided by a measure of

aviation activity, such as the number of aircraft miles flown, aircraft hours flown, or departures.
4

rates. Most of the Agency documents include reductions to the aviation accident rate. Other

documents state the goal is to reduce the fatal accident rate. Reductions to the fatality rate are

declared primarily within the AvSP definition documents. The National Transportation Safety

Board (NTSB)18 separately calculates and individually tracks fatal aircraft accident rates,

aviation accident rates, and fatality rates.

AvSP documentation should be consistent with and provide details for implementing Agency

and Enterprise Strategic Plans. Goals should be expressed in a manner that allows a future

assessment of whether they are being achieved. Agency documentation should consistently

portray the goal because each of the rates is measurable and could be used as a measure of

success. The Independent Annual Review19 of the AvSP conducted in April 2000, also noted

these inconsistencies. The review briefing package states, “Without a clear distinction between

the program goal and the program objectives, and specific definitions of the program’s minimum

success criteria, it will be difficult to determine whether or not the AvSP is successful.”

Performance Measurement Baselines

While NASA has been inconsistent in stating the goal of the NASA Aviation Safety Initiative in

various Agency documents, the baselines used to measure performance consistently refer to

only the aviation fatal accident rates. Performance measures do not include a baseline for

aviation accident rates (nonfatal) or fatality rates. Aviation accidents, for example, outnumber

fatal aircraft accidents 170 to 24,20 respectively, for commercial carriers.21 Although

technologies developed within the AvSP are expected to have some effects on the nonfatal

accident rates and fatality rates, the program office is not measuring program success by their

effects on the various rates. According to program officials, the fatal accident rate is a subset of

the aviation accident rate and, therefore, a reduction in the fatal accident rate would also reduce

the accident rate. However, the AvSP has separately identified and equally stated in the

program’s goals, plans to also contribute to reductions in aviation accident rates and fatality

rates. Unless management revises the AvSP goal to reflect only the fatal accident rate, the

 

16 An accident is an occurrence associated with the operation of an aircraft that takes place between the time a person

boards the aircraft with the intention of flight and until such time as all such persons have disembarked and (1) in which

any person suffers death or serious injury or (2) in which the aircraft receives substantial damage.

17 Fatality rates include deaths and fatal injuries that result in death within 30 days of the accident.

18 The NTSB is an independent Federal agency charged by Congress with investigating every civil aviation accident in

the United States and is responsible for maintaining the Government's database on civil aviation accidents.

19 The Independent Program Assessment Office performs an Independent Annual Review of NASA programs to assess

progress and milestone achievement against the program baselines and to evaluate cost, schedule and technical content

of the program over its entire life cycle.

20 The number of fatal accidents and nonfatal accidents is for the AvSP baseline timeframe of 1990-1996, and is based

on data published by the NTSB.

21 Commercial carriers are those conducting operations under Federal Aviation Regulation Part 121. This applies to air

carriers, such as major airlines and cargo haulers that fly large transport aircraft. For the purposes of this report, future

reference to this type of aircraft will be commercial carriers.
5

Agency should establish baselines to also measure its performance relative to aviation accident

rates and fatality rates.

Factors Affecting Program Success

The six project areas in the AvSP have identified 40 products that contribute to an overall

capability expected to affect the AvSP goal. The products under development are projected to

reduce the fatal accident rate by about 52 percent for commercial carriers and by about 81

percent for general aviation. However, the success of the AvSP is dependent on overcoming

several risks or factors including:

• challenge of technical development;

• user implementation of the resulting technology; and

• availability of resources (staffing, time, and dollars).

Challenge of Technical Development. The AvSP project managers participated in a

preliminary assessment to estimate the effect of the risks of technology under development and

user implementation to AvSP’s successful achievement of its goal. AvSP managers recognize

that technology development for the 40 products is moderate to hard but agree it is an

appropriate application of NASA resources. The project managers’ assessment of technology

development for the 40 products showed that for 29 (72 percent), there was a high probability

that they would not be developed, as indicated below:

• 2 (5 percent) products required few technical developments and should be readily

accomplished;

• 9 (23 percent) products were a moderate technical challenge and should be

accomplished;

• 23 (57 percent) products were considered large technical challenges and may not be

developed; and

• 6 (15 percent) products required technical breakthroughs and it is likely that they cannot

be achieved.

User Implementation. Implementation of the AvSP’s 40 products is a challenge. The project

managers' assessment of user implementation showed that potential users of 21 (52 percent) of

the 40 products have a cautious attitude about implementation.

• 19 (48 percent) product users are supportive of the new technology.

• 6 (15 percent) product users are waiting to see how the technology develops.

• 6 (15 percent) product users are initially skeptical but could be convinced of product

value.

• 9 (22 percent) product users are unlikely to agree to field this capability without

intensive discussion and debate.
6

In addition, the return on investment for 11 (27 percent) of the 40 products is negative, and

NASA will need to convince the user community of the products' effectiveness to implement the

product technology. Further, for 7 (17 percent) of the 40 products, the industry partners have

little or no interest in developing and implementing the particular product technology. Of the 40

products, 3 have numerous or strong critics within NASA or FAA.

Availability of Resources. Adequate resources are key to program success. However,

three of the six AvSP projects have staffing shortfalls. The shortfalls range from 3 to as many as

11 personnel for a particular project. If program management is unsuccessful in increasing the

staff by October 2000, actions will be taken to descope planned activities, which may affect

meeting AvSP goals.

As a result of the dependence on technical development, implementation, and resources, the

success of the AvSP can only be projected. The actual effect of the AvSP technologies on the

fatal accident rate cannot be directly measured. The success of the technologies and

achievement of the goal is dependent on implementation by the aviation community. NASA

defined minimum success criteria as demonstrating technology that will have at least a 5022

percent impact on the national goal, and that impact will occur only if the aviation community

chooses to implement the technologies produced by the AvSP. While the factors affecting

program success are recognized in the AvSP Program Plan and draft Program Commitment

Agreement, they are not given adequate emphasis on how closely they are linked to the success

of AvSP. NASA needs to emphasize the risks involved with the development and

implementation of its aviation safety technologies and explain how these risks affect the

achievement of its aviation safety goal. Without an understanding of the risks involved, the

aviation community, Congress, and the public will not have a realistic concept of the difficulties

in developing aviation safety products that contribute to reductions in aviation fatal accidents

rates.

FAA and NASA Aviation Safety Activities

In response to the White House Commission’s recommendation on aviation safety, NASA and

FAA coordinated their research activities on aviation safety. In October 1998, the

Administrators of FAA and NASA signed a formal agreement establishing a partnership

between their agencies with the objective of articulating and achieving specific goals in aviation.

One of those goals is aviation safety.

 

22 During our audit, the minimum reduction rate changed from 70 to 50 percent in draft versions of the Program

Commitment Agreement.
7

NASA and the FAA are preparing the “FAA/NASA Integrated Safety Research Plan”23 in

recognition of the importance of cooperative efforts to optimize the Government research

needed to attain critical aviation safety goals. The plan will serve as a guide for FAA and

NASA cooperation in aviation safety research. The plan states that FAA and NASA must

strive to use common methodologies to assess the success of their research efforts toward

achieving the National Aviation Safety Goal, which is stated in this plan as “an 80 percent

reduction in the commercial fatal accident rate by 2007, as compared to a 1994-1996

baseline.” The plan further states this common goal has been acknowledged and agreed upon

by NASA and FAA. NASA’s aviation safety activities, however, are not in agreement with this

draft plan because Agency documents do not consistently state what the Aviation Safety

Initiative will accomplish. Further, NASA and the FAA are not using common methodologies in

their efforts to affect the National Aviation Safety Goal. For example:

• NASA is using fatal accident statistics for 1990-1996 and not the FAA’s 1994-1996

baseline;

• NASA is using “number of departures” as a measure of accidents and FAA is using

“flight hours;” and

• NASA includes general aviation in its performance measure, while FAA includes Part

135.24

See Appendix D for details on the comparison between NASA and FAA methodologies.

Also, the draft plan does not show the individual effects of NASA and FAA aviation safety

research activities on the goal. The plan should clearly state the individual contributions at these

Agencies so Congress, the aviation community, and the public will know what the respective

agencies are contributing toward reducing the fatal accident rate.

Conclusion

NASA's goal to reduce the fatal aviation accident rate is an important national priority.

Nevertheless, some of the goals, as stated, are optimistic and could lead to unfulfilled

expectations by Congress, the aviation community and the public. In addition, without

appropriate baselines to measure the program’s performance, NASA may not achieve its full

contribution to the overall national aviation safety goal, and that goal may not be achieved.

 

23 This draft plan is dated July 2000.

24 Part 135 applies to commercial air carriers commonly referred to as commuter airlines and to air taxis.
8

Recommendations, Management's Response, and Evaluation of

Response

The Associate Administrator for Aerospace Technology should:

1. Clarify the Agency’s contribution toward the National Aviation Safety Goal

and revise plans and goals in various Agency documents to ensure a consistent

representation of NASA’s intended performance.

Management's Response. Concur. The Office of Aerospace Technology clarified its

Aviation Safety Program goal as “develop and demonstrate technologies that contribute to a

reduction in the aviation fatal accident rate by a factor of 5 by year 2007 and a factor of 10 by

year 2022” and will update all documentation (strategic plan, Program Commitment Agreement,

Program Plan, Web page, etc.) accordingly. The complete text of management's response is in

Appendix E.

Evaluation of Response. Management’s actions are responsive to the recommendation.

After the receipt of the comments, management indicated that it would have actions completed

by December 31, 2000. The recommendation is considered resolved but will remain

undispositioned and open until the agreed corrective action is completed.

2. Identify all baselines necessary to measure the Agency’s performance in

meeting the established goals.

Management's Response. Concur. The Office of Aerospace Technology will adjust its

baseline to match the FAA baseline years of 1994–1996 (see Appendix E).

Evaluation of Response. Management's actions are responsive to the recommendation.

Management's actions are sufficient to close the recommendation for reporting purposes.

3. Revise program documentation available to stakeholders to adequately

reflect the risks of development and implementation of various technologies.

Management's Response. Concur. Management agrees that program documentation should

adequately reflect the risk of development and implementation, and will communicate and stress

these risks to their partners, customers and stakeholders (see Appendix E).

Evaluation of Response. Management's actions to ensure stakeholders appropriately

understand the risk are responsive to the intent of the recommendation. Our draft

recommendation requested that management revise goals and program documentation.

Management did not believe it appropriate to revise the goals, although the phrase “when

implemented” has been used in statements of the goal in AvSP’s budget and FAA
9

Memorandum of Understanding documentation to identify the development and implementation

risk. We defer to management's judgment regarding changes to the goal to reflect risk.

Management recognized that program documentation should reflect the risk of development and

implementation of the technologies. In their comments, management cited examples of program

documentation that does reflect the risks. However, in discussions with the OIG, management

acknowledged that information available to the public, such as the Aviation Safety Website,

could be improved. After the receipt of the comments, management indicated that it would

have actions completed by December 31, 2000. This recommendation is considered resolved

but will remain undispositioned and open until the publicly available documentation is updated.

4. Coordinate with the FAA to revise the draft “FAA/NASA Integrated Safety

Research Plan” to resolve differences in baselines and metrics used to guide

the national efforts on aviation safety.

Management's Response. Concur. The Office of Aerospace Technology will conform to

the baseline years established by the FAA, and will reflect them in the FAA/NASA Integrated

Safety Research Plan (see Appendix E).

Evaluation of Response. Management's actions are responsive to the recommendation.

Management's actions are sufficient to close the recommendation for reporting purposes.
10

Appendix A. Objectives, Scope, and Methodology

Objectives

Our overall objective was to determine whether overall program management of AvSP is

effective. Specifically, our audit work included an assessment of:

• the adequacy of coordination with FAA and other partners, and

• overall success in achieving program goals and objectives including metrics used to

measure program performance.

Scope and Methodology

To accomplish our objectives, we obtained an overall understanding of the National Aviation

Safety Goal by reviewing the following prior studies and assessments.

• National Research Council Report, “Aviation Safety and Pilot Control,” 1997.

• RAND Institute of Civil Justice Report, “Safety in the Skies,” 1999.

• National Academy of Public Administration, “A Review of the Aviation Safety

Reporting System,” 1994.

• Department of Transportation, Office of Inspector General Report, “Top Ten

Management Issues,” (Issue 1-Aviation Safety), 1998.

• Federal Aviation Administration FY 2000 Annual Performance Plan.

• National Transportation Safety Board, “Annual Review of Aircraft Accident Data,

Calendar Year 1996-U.S. Air Carrier Operations,” 1999.

• Aerospace Safety Advisory Panel Reports for 1998 and 1999.

• National Science and Technology Council, “National Research and Development Plan

for Aviation Safety, Security, Efficiency, and Environment Compatibility,” 1999.

• Final Report to President Clinton from the White House Commission on Aviation Safety

and Security, 1997.

• An Aviation Safety Research Investment Strategy: Report of the ASIST Team, 1997.

• ir Safety Foundation, “1999 Nall Report: Accident Trends and Factors for 1998.”

We focused our review on the NASA AvSP and its six projects. We reviewed the program to

determine whether reasonable milestones and measures had been established by which to

assess performance and ensure success. We also reviewed the performance and business

management plans. Specifically, we:
11

Appendix A

• Obtained various program planning documents such as the Program Commitment

Agreement (draft), Program Plan, Non-Advocate Review, FAA/NASA Integrated

Safety Research Plan (draft), FAA/NASA Memoranda of Agreement/Understanding,

Joint Working Group minutes, Monthly and Quarterly Program Reviews, and

Independent Annual Review.

• Interviewed management personnel in both the AvSP office and Aerospace Systems

Analysis branch at Langley; the NASA Headquarters Aviation Safety Goal Manager;

the Office of Aerospace Technology Deputy Director, Programs Division; and a

member of the Non-Advocate Review and the Independent Annual Review teams.

Management Controls Reviewed

We reviewed relevant Federal and NASA regulations on program management and aviation.

Specifically, we reviewed the GPRA; NASA Procedures and Guidelines 1000.2, “NASA

Strategic Management Handbook”; NASA Procedures and Guidelines 7120.5A, “NASA

Program and Project Management Processes and Requirements”; NASA Web Policy: Policy

for NASA Information Published over the Internet; and 49 Code of Federal Regulations Part

830, "Notification and Reporting of Aircraft Accidents or Incidents."

With respect to NASA’s Aviation Safety Initiative, we also reviewed the following documents

that implement the management controls identified.

• 1998 NASA Strategic Plan with 1999 Interim Adjustments

• Aerospace Technology Strategic Plan, 1995-2000

• Langley Implementation Plan for FY 2000

• NASA FY 2001 Performance Plan

• NASA FY 1999 Performance Report

• NASA FY 1999 Accountability Report

• AvSP Non-Advocate Review and Independent Annual Review

• Budget requests for FY 2000 and FY 2001

Management controls regarding program documentation should be improved as discussed in the

finding.

Prior Audit Coverage

There has been no prior audit coverage on the NASA Aviation Safety Program. However, we

issued the following report relating to Aviation Safety in 1998.
12

Appendix A

“Report on FAA/NASA Research and Development Coordination Efforts,” Report

Number P&A-98-005, October 8, 1998. This audit was a joint effort with the Department of

Transportation, Office of Inspector General. The audit focused on aviation safety and air traffic

management research because they are the two major joint activities supporting the National

Airspace System and receive most of the funding (projected $1.3 billion) through FY 2002.

The audit identified areas where the FAA and NASA can take action to enhance the

effectiveness of their coordination efforts and help ensure agency resources are used in the most

cost-effective manner.

The General Accounting Office (GAO) has issued several reports on the subject of Aviation

Safety; however, the GAO audits are of NASA’s aviation safety partner, the FAA. The

following reports specifically address the National Aviation Safety Goal and FAA’s history

regarding implementation of safety recommendations.

“Aviation Safety: Safer Skies Initiative Has Taken Initial Steps to Reduce Accident

Rates by 2007,” GAO/RCED-00-111, dated June 2000. The initiative should help improve

aviation safety, but has not challenged all sectors of the aviation community to push aggressively

for safety improvements. There are no challenging goals established for general aviation. The

Safer Skies25 initiative has made progress in selecting and implementing interventions for

identified safety problems, but in the past, FAA did not consistently implement the interventions

successfully. Performance measures need to be developed to evaluate the effectiveness of the

implemented interventions. Coordination has been extensive but needs improvement for the

initiative to succeed. Additional steps need to be taken to ensure that those safety interventions

most critical to reducing the nation’s fatal accident rate are given top priority and funding.

“Aviation Safety: FAA Has Not Fully Implemented Weather-Related

Recommendations,” GAO/RCED-98-130, dated June 2, 1998 and “Aviation Safety:

FAA Generally Agrees with but Is Slow in Implementing Safety Recommendations,”

GAO/RCED-96-193, dated September 23, 1996. Both of these reports noted FAA’s

limited progress in implementing safety recommendations. The FAA generally agreed with the

recommendations but did not consistently follow through on implementing them.

“Aviation Safety and Security: Challenges to Implementing the Recommendations of

the White House Commission on Aviation Safety and Security,” GAO/T-RCED-97-90,

dated March 5, 1997. GAO testified that the White House recommendations were a good

start toward ensuring greater safety for passengers, restructuring the relationships between

Government and private industry, and maintaining America’s global

 

25 Safer Skies is an FAA focused initiative addressing safety problems that have contributed to fatal accidents in the

past and is designed to bring about a five-fold reduction in fatal accidents.
13

Appendix A

leadership in aviation. Yet, key questions remain about how and when the recommendations

will be implemented, how much they will cost, and who will pay the cost.

Computer-Processed Data

We used computer-processed reports from NASA Headquarters, Langley, the FAA, and

NTSB to understand aviation fatal accident rates. We reviewed and tested selected data but

did not verify the overall validity of the reports. The lack of verification did not affect our audit

results.

Audit Field Work

During January through August 2000, we conducted field work at NASA Headquarters and

Langley. We performed the audit in accordance with generally accepted government auditing

standards.
14

Appendix B. AvSP Contributions toward the

National Aviation Safety Goal

Investment Area Project Technology Impact Number Of

Products

Accident

Prevention

System-Wide

Accident

Prevention

(SWAP)

Human error

assessment

methodologies

Improve human/machine

integration in design,

operations, and maintenance

8

Accident

Prevention

Single Aircraft

Accident

Prevention

(SAAP)

Health and usage

monitoring

technologies

Continuously track,

diagnose, and restore the

health of on-board system,

enabling self-healing designs

and “refuse to crash aircraft”

3

Accident

Prevention

SAAP Not reflected in core

technologies

Provide upset prevention and

recovery and identify future

group accident precursors

4

Accident

Prevention

Weather

Accident

Prevention

(WxAP)

Affordable

technologies and

systems to obtain

critical weather

information

Bring intelligent weather

decision making based on

worldwide, real-time hazard

awareness to every cockpit

3

Accident

Prevention

WxAP Turbulence modeling

and detection

technologies

Eliminate severe turbulence

as an aviation hazard

3

Accident

Prevention

Synthetic

Vision

(SV)

Synthetic vision

technologies

Make every flight the

equivalent of clear-day

operations

5

Accident

Mitigation

Accident

Mitigation

(AM)

Advanced structural

and material designs

Increase survivability when

accidents do occur

8

Aviation Systemwide Monitoring

Modeling and

Simulation

Aviation

System

Modeling and

Monitoring

System

(ASMM)

Integrated aviation

system monitoring

tools

Monitor and assess all data

from every flight for both

known and unknown issues

6

Total Products 40

Appendix C. Examples of Inconsistent Aviation Safety Goals

Document Time Period Highlighted Inconsistencies

White House Commission on Feb. 1997 Government and industry establish a national goal to reduce the
15

Aviation Safety and Security aviation fatal accident rate by a factor of 5 within 10 years and

conduct safety research to support that goal.

NASA Strategic Plan* 1998-2002

2003-2009

2010-2023

Develop technologies to reduce the aviation fatal accident

rate.

Contribute to a reduction in the accident rate by a factor of 5.

Contribute to a reduction in the aircraft accident rate by a

factor of 10.

Performance Report FY 1999 Contribute to aviation safety by reducing the aircraft

accident rate.

Aerospace Technology

Enterprise Strategic Goal

FY 2000 Reduce the aircraft accident rate by a factor of 5 within 10

years and by a factor of 10 within 25 years.

Accountability Report FY 1999 Same as Aerospace Technology Enterprise Strategic Goal.

Performance Plan FY 2001 Same as Aerospace Technology Enterprise Strategic Goal.

Budget Requests FY 2000/01 Develop and demonstrate technologies that

contribute to a reduction in aviation accident and fatality

rates by a factor of 5 by the year 2007 compared to the 1994-

1996 average.

AvSP Program Plan Aug. 1999 By 2004, develop and demonstrate technologies that

contribute to a reduction in aviation accident and fatality

rates by a factor of 5 by year 2007 and by a factor of 10 by year

2022.

AvSP Program Commitment

Agreement-Draft

Nov. 1999 Develop and demonstrate technologies that enable a

reduction in aviation accident and fatality rates by a factor of 5

by year 2007.

AvSP Internet page Aug. 2000 Reduce the fatal aircraft accident rate by 80

percent in 10 years and by 90 percent in 25 years.

“Turning Goals Into Reality”

Conference

May 2000 Reduce the accident rate by a factor of 5 within 10 years and

by a factor of 10 within 25 years.

*The “NASA Strategic Plan,” 1998 with 1999 interim adjustments, page 30, describes the Agency objectives for the

Aviation Safety Initiative.
16

Appendix D. Comparison of NASA and FAA Methodologies

The White House established the overall National Aviation Safety Goal; however, the White

House did not set forth specific recommendations as to categories of operation or specific

baseline criteria. NASA and the FAA determined their own specific criteria. The FAA-NASA

integrated plan, compared to the individual NASA and FAA stated goals and the variances in

metrics to measure achievement of the National Aviation Safety Goal are noted below.

FAA-NASA

Integrated

Safety Plan1 NASA FAA2

Category of

Operation

Commercial Part 1213 General Aviation Part 121 Part 1354

Measure Fatal accident

rate

Number of fatal

accidents per

100,000

departures

Number of fatal

accidents per

100,000 flight

hours

Number of fatal accidents per 100,000

flight hours5

Scope Not identified Scheduled

service

Not applicable Scheduled and

nonscheduled

services

Scheduled service

Source Not identified Flight data is submitted to Bureau

of Transportation Statistics.

Accident data is provided by

NTSB

Flight data is submitted to Bureau of

Transportation Statistics. Accident data

is provided by NTSB

Baseline Not identified The average of

Part 121

scheduled, for

fatal accidents

for the 7 years

from 1990-

1996

The baseline is

the sum of all

fatal accidents

from 1990-1996

(2,891) divided

by the sum of all

the estimated

hours flown over

that period

(171,120,000)

The average of all Parts 121 and 135 fatal

accidents for the 3 years from 1994-1996

Calculated

rate

Not identified 0.044 per

100,000

departures

1.690 per

100,000 flight

hours

0.037 per 100,000 flight hours

1This draft plan is dated July 2000.

2 FAA methodologies are taken from its Fiscal Year 2000 Annual Performance Plan, pages 14-15. FAA applied the 80-

percent reduction in the nation’s fatal accident rate only to commercial aviation and adopted a less aggressive accident

reduction goal for general aviation.

3Part 121 refers to large transport aircraft such as major airlines and cargo haulers.

4Part 135 applies to aircraft commonly referred to as commuter airlines and air taxis.

5The FAA Administrator’s Fact Book dated July 2000, page 45, refers to a measure of 100,000 departures instead of

the flight hours noted in the FAA Performance Plan for Fiscal Year 2000.
17

Appendix E. Management’s Response
18

Appendix E

Change made,

Page 1

Change made,

Page 1

Clarification

made, Page 5

Change made, Page 6

No change

required
19

Appendix E

Footnote 5

added,

Page 16
20

Appendix E
21

Appendix F. Report Distribution

National Aeronautics and Space Administration (NASA) Headquarters

A/Administrator

AI/Associate Deputy Administrator

B/Chief Financial Officer

B/Comptroller

BF/Director, Financial Management Division

G/General Counsel

J/Associate Administrator for Management Systems

JM/Acting Director, Management Assessment Division

L/Associate Administrator for Legislative Affairs

R/Associate Administrator for Aerospace Technology

RP/Director, Programs Division

NASA Advisory Officials

Chair, Aerospace Technology Advisory Committee

NASA Centers

Director, Langley Research Center

Chief Counsel, John F. Kennedy Space Center

Non-NASA Federal Organizations and Individuals

Assistant to the President for Science and Technology Policy

Deputy Associate Director, Energy and Science Division, Office of Management and

 Budget

Branch Chief, Science and Space Programs Branch, Energy and Science Division, Office

 of Management and Budget

Associate Director, National Security and International Affairs Division, Defense

 Acquisitions Issues, General Accounting Office

Professional Assistant, Senate Subcommittee on Science, Technology, and Space

Inspector General, Department of Transportation

Chairman and Ranking Minority Member – Congressional Committees and

Subcommittees

Senate Committee on Appropriations

Senate Subcommittee on VA, HUD, and Independent Agencies
22

Appendix F

Chairman and Ranking Minority Member – Congressional Committees and

Subcommittees (Cont.)

Senate Committee on Commerce, Science, and Transportation

Senate Subcommittee on Science, Technology, and Space

Senate Subcommittee on Aviation

Senate Committee on Governmental Affairs

House Committee on Appropriations

House Subcommittee on VA, HUD, and Independent Agencies

House Committee on Government Reform and Oversight

House Subcommittee on Government Management, Information, and Technology

House Subcommittee on National Security, Veterans Affairs, and International Relations

House Committee on Science

House Subcommittee on Space and Aeronautics, Committee on Science

Congressional Member

Honorable Pete Sessions, U.S. House of Representatives
NASA Assistant Inspector General for Auditing

Reader Survey

The NASA Office of Inspector General has a continuing interest in improving the usefulness of

our reports. We wish to make our reports responsive to our customers’ interests, consistent

with our statutory responsibility. Could you help us by completing our reader survey? For your

convenience, the questionnaire can be completed electronically through our homepage at

http://www.hq.nasa.gov/office/oig/hq/audits.html or can be mailed to the Assistant Inspector

General for Auditing; NASA Headquarters, Code W, Washington, DC 20546-0001.

Report Title: NASA's Aviation Safety Program

Report Number: Report Date: 

Circle the appropriate rating for the following statements.

Strongl

y

Agree

Agree Neutra

l

Disagre

e

Strongl

y

Disagre

e

N/A

1. The report was clear, readable, and logically

organized.

5 4 3 2 1 N/A

2. The report was concise and to the point. 5 4 3 2 1 N/A

3. We effectively communicated the audit objectives,

scope, and methodology.

5 4 3 2 1 N/A

4. The report contained sufficient information to

support the finding(s) in a balanced and objective

manner.

5 4 3 2 1 N/A

Overall, how would you rate the report?

Excellent Fair

Very Good Poor

Good

If you have any additional comments or wish to elaborate on any of the above

responses, please write them here. Use additional paper if necessary. 

 

 

 

 
 

How did you use the report? 

 

 

 

 

 

 

How could we improve our report? 

 

 

 

 

 

 

How would you identify yourself? (Select one)

Congressional Staff Media

NASA Employee Public Interest

Private Citizen Other: 

Government: Federal: State: Local: 

May we contact you about your comments?

Yes: ______ No: ______

Name:___________________________

Telephone: ________________________

Thank you for your cooperation in completing this survey.
Major Contributors to the Report

Karen E. VanSant, Program Director, Aerospace Technology Audits

Carol A. St. Armand, Program Manager, Aerospace Technology Audits

Sandra K. Leibold, Auditor-in-Charge

Betty G. Weber, Operations Research Manager

Nancy C. Cipolla, Report Process Manager

Patricia J. Atkinson, Program Assistant