Appendix B

30. George A. Slenski, “Wired for Success, Ensuring Aircraft Wiring Integrity Requires a
Proactive Systems Approach.” Volume 8, Number 3. http://amptiac.alionscience.com/quarterly.

31. “Aviation Safety, FAA and DOD Response to Similar Safety Concerns,” General Accounting
Office, Report Number GAO-02-77, January 2002. Appendix B.

32. “Wire Insulation Deterioration Analysis System, NASA Orbiter Sampling Test Results and

Analysis,” Lectromechanical Design Company for the Boeing Company, Dulles, VA., Report
# N224-RPT16SEO, September 16, 2000.

33. Cynthia Furse and Randy Haupt, “Down to the Wire,” Utah State University, IEEE Spectrum
Online, January 31, 2001, http://www.spectrum.ieee.org/WEBONLY/publicfeature.

34. “Review of Federal Programs for Wire System Safety,” National Sciences and Technology

Council Committee on Technology, Wire System Safety, Interagency Working Group,
November 2000.

35. C.G. Steven, “Thermal Evaluation and Life Testing of Solid Insulation,” Electrical Insulation
IEE Electrical and Electronics Materials and Devices, Series 2, London, 1973.

36. Space Shuttle Independent Assessment Team, Henry McDonald, Chair. Space Shuttle
Independent Assessment Team Report to Associate Administrator, Office of Space F light.
March 7, 2000. http://history.nasa.gov/sait.pdf.



Figure 2. Visual Inspection of Wiring in the Payload Bay of Orbiter Atlantis

The CAIB reported that 1,700 feet of Kapton wiring was inaccessible but did not state whether
the amount included wire bundles in inaccessible locations and/or wires inside wire bundles.
Space Shuttle Program staff at Kennedy calculated that roughly 86 percent’ of the orbiter’s
wiring is inaccessible and generally not examined during a visual inspection. The 86 percent
calculation conservatively estimated the amount of inaccessible wiring since it did not consider
wires inside wire bundles.

After NASA retires the Space Shuttle, any next generation space vehicle may use Kapton
wiring and will need a way of inspecting inaccessible and unseen wiring. Wiring officials with

the Space Shuttle Program confirmed that no good substitute has been developed to replace
Kapton wiring in space vehicles.

New Evaluation Technology Should Be Developed

In the longer term, only state-of-the-art technology might substantially improve an inspection
by evaluating the condition of unseen wiring. New nondestructive evaluation technology could
detect minute damage that cannot be identified through a visual inspection. For example, the
age-life tester could be used to predict when the Kapton insulation would begin to fail by
determining its degradation caused by time and the environment. Although testing inaccessible
wiring was not a requirement of the age-life tester, that approach was one of the methods the
Space Shuttle Program designed to determine when or if the orbiter would require rewiring.

Because new nondestructive evaluation technology could benefit the next generation of space
vehicles, NASA should take action to facilitate reestablishing plans for developing the
technology. The Space Operations Mission Directorate oversees the Space Shuttle Program
Office, which had identified and assessed new nondestructive evaluation technologies and
prepared development plans. The Exploration Systems Mission Directorate oversees the
programs for next generation space vehicles. The two Mission Directorates should establish a

“The calculation was made by engineering staff of the Orbiter Electrical, Power, Distribution, and Control Group
at Kennedy Space Center and is available on an internal Web site.

10


Appendix D

3

* Separation of wires performing criticality | functions (loss of which could lead
to loss of crew/vehicle) needed to attenuate potential impacts to critical
systems that might be affected during an arc tracking incident.

* Additional wire protection modifications to minimize potential wire damage.

As was stated in the NASA Implementation Plan for Space Shuttle Return to Flight and
Beyond, the SSP has approved, after careful review, the implementation of option 1 (inspect
and protect) and option 3 (perform destructive evaluations), as related to the initial four prong
approach that was provided in response to the CAIB’s recommendation 4.2-2. The decision
nol to go forward with option 2(develop NDE technology), does not suggest we have given up
‘on a dedicated NDE tool. Throughout the life of the SSP, vendors have provided NASA with
NDE type equipment for evaluation. We are currently evaluating a system now. [four
engineers find that an NDE tool has value, we will pursue it. In regard to option 4 (evaluate
Orbiter wire replacement), given the new Shuttle retirement date, it now becomes unrealistic
to consider the replacement of aif the Orbiter wiring, This time and effort is better spent on
inspection and prevention. However, wire segments will be replaced where practical, This
philosophy has been routinely implemented throughout the life of the program and is
performed on a case by case basis where wire segment damage is considered to be extensive.

It should also be noted that a type of NDE is being performed on the wiring every time power
is applied to the Orbiter. In addition, SSP regularly conducts numerous NDE type electrical
system checks during scheduled Orbiter processing to validate the electrical paths provided by
the wiring. These tests are performed by all onboard subsystems that use electrical power.

As an added authentication, every time an Orbiter connector is demated / mated an electrical
test is performed on cach of its pins to ensure circuit integrity. This occurs hundreds of times
during a standard Orbiter processing flow. In addition to these electrical validations,
technicians, engineers and quality control personne] routinely perform visual wire inspections.
As an added precaution, before an arca within the Orbiter can be closed out for flight, an
additional visual inspection is performed.

Periodically, each Orbiter is subjected to extensive maintenance down-time periods or Orbiter
Major Modifications (OMM) in which modifications and more in-depth visual inspections are
performed. These include inspecting areas that arc normally inaccessible during a standard
Processing flow as well as re-inspecting arcas that were done during normal processing, thus
Providing additional insight to the health of the wiring systems. Also, during an OMM period
additional electrical validation tests are performed above and beyond those tests that are
performed during a standard flow. We have completed the OV-103 OMM wiring inspections
prior to STS-114. We are completing the OV-105 inspections during its current OMM, and
we will follow up with the inspections on OV-104 in 2007,

Certain types of routinely performed wire repairs provide NDE type insight into wiring
health, These repairs necessitate a circuit integrity test to verify that the repair was donc
correetly, This retest not only verilies the repair, it also provides additional insight to the

20



SPACE SHUTTLE ORBITER WIRING INSPECTION

Objectives

The NASA Office of Inspector General initiated an audit with an overall objective of
determining whether Agency actions regarding inspections of orbiter wiring were responsive to
Columbia Accident Investigation Board (CAIB) Recommendation 4.2-2. The specific
objectives were to determine whether (1) planned operational changes would make visual
inspection more effective and (2) plans would enable the Agency to identify and test
state-of-the-art evaluation technology and make tested technology available for the orbiter.

Background

A significant amount of orbiter wiring (140 to 157 miles) is insulated with Kapton, a polyimide
film used as electrical insulation. Kapton has been widely used in aircraft and spacecraft for
more than 30 years because it is lightweight, nonflammable, has a wide operating temperature
range, and resists damage. However, Kapton insulation can break down or become damaged
with age, with mishandling, and from environmental stresses, particularly moisture.

Appendix B contains a bibliography related to Kapton-insulated wiring.

During the July 1999 launch of Columbia, a short-circuit 5 seconds after liftoff caused two of
the six main engine controller computers to lose power, which could have caused one or two of
the three main engines to shut down. The short-circuit incident was investigated and found to
be caused by damaged Kapton wiring. Subsequent to the incident, visual inspection procedures
were revised to limit wire damage from mishandling. Revised procedures state that the
inspection would examine the outer wires of wiring bundles and would not examine wires
inside wiring bundles unless insulation damage was noted on outer wires. If wire damage was
noted on outer wires, the wire harness would be opened up to perform further inspections to
ensure no other damage existed. To minimize damage of wiring in high traffic areas, wire
protection made from convoluted tubing was installed over the wiring. As an added control to
prevent damage, the Space Shuttle Program issued a new requirement that personnel who
require entry into orbiter areas must complete wire damage awareness training.

In addition to revised visual inspection criteria, additional insight into the health of orbiter
wiring is obtained through standardized electrical functional tests. Such tests are routinely
performed during visual inspection and expanded during orbiter maintenance periods. An
expanded test example is an electrical connector test. A functional electrical test is performed
every time an orbiter connector is demated (unplugged) to ensure that the electrical circuits
contained within the connector are operational.

The CAIB initially targeted Kapton-insulated wiring as a possible cause of the February 2003
Columbia accident because of previous problems with its use in the Space Shuttle and its
implications in aviation accidents. Ultimately, the CAIB found no evidence that Kapton wiring
problems caused or contributed to the accident and made no related recommendation for action
before returning the Space Shuttle to flight. Nevertheless, the CAIB made Recommendation
4.2-2 in its August 2003 report because visual inspection could not detect wiring damage in


Appendix C. Funded Return-to-Flight Acti

ies

As of December 2004, NASA had funded the following return-to-flight activities (listed in the
Implementation Plan) that were associated with the 29 recommendations that the CAIB made:

Orbiter Reinforced Carbon-Carbon Inspections

On-orbit and Tile Repair

Orbiter Workforce

Orbiter Thermal Protection System Hardening

Orbiter Government-furnished Equipment

Orbiter Contingency

Orbiter Certification and Verification

External Tank Items (Camera and Bipod Ramp)

Solid Rocket Booster Items (Bolt Catcher, External Tank Assembly Ring, and Camera)
Ground Camera Ascent Imagery Upgrade

Kennedy Space Center Ground Operations Workforce

Other (Systems Integration, Full Cost Accounting, Additional Workforce)
Stafford-Covey Team (Return-to-Flight Task Group)


Additional Copies

To obtain additional copies of this report, visit

www.hq.nasa.gov/office/oig/hg/issuedaudits.htm! or contact the Assistant Inspector
General for Auditing at (202) 358-1232.

Comments on This Report

In order to help us improve the quality of our products, if you wish to comment on the
quality or usefulness of this report, please send your comments to Ms. Jacqueline White,

Director of the Quality Control Division, at Jacqueline. White@nasa.gov or call (202)
358-0203.

Suggestions for Future Audits

To suggest ideas for or to request future audits, contact the Assistant Inspector General
for Auditing. Ideas and requests can also be mailed to:

Assistant Inspector General for Auditing
NASA Headquarters
Washington, DC 20546-0001

NASA Hotline

To report fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at (800)
424-9183, (800) 535-8134 (TDD), or at www.hg.nasa.gov/office/oig/hq/hotline.html#form;
or write to the NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station,
Washington, DC 20026. The identity of each writer and caller can be kept confidential, upon
request, to the extent permitted by law.

Major Contributors to the Report
Nora E, Thompson, Program Manager
Stephen D. DeVillo, Aerospace Technologist

Sandra Leibold, Auditor


Appendix D

ESMD is in contact with the Shuttle STWH, Electrical Power Distribution and Contral
DTWH, Nondestructive Evaluation DTWH, and NASA Technical Standards Program to:

© Ensure we establish a dialog on this issue,

* Document specific lessons learned, and

* Determine the need for and scope of complex electrical system integrity and
wire system NDE research,

This information will be shared with the Crew Exploration Vehicle Team and subsequent
ESMD programs/projects as well as with SOMD. Within the ESMD risk management
community. they will ensure that this effort 1s coordinated between the Exploration Systems
Research and Technology Division and Constellation Systems organizations,

With the announcement of the Vision for Space Exploration in January 2004, the goals and
objectives of the Space Shuttle Program (SSP) have dramatically changed. This change is felt
particularly in the operational life of the Orbiter. Prior to the announcement, the SSP was
planning on flying until at least 2020; now we are preparing for an Orbiter cetirement in 2010,
This is a significant vanation and, as would be expected, has necessitated an alteration in our
operations strategy

The mitial responses to the CAIB recommendation 4.2-2 were based on operating the Orbiter
to 2020 and beyond. To adjust to the 2010 date, we have modified our approach and have

reflected this in our most recent update to NASA's Implementation Plan for Space Shuttle
Ren to Flight and Beyond.

In response to the lack of a comprehensive risk assessment conducted conceming Orbiter
wiring, the SSP team has developed specific criteria for inspecting Orbiter wiring (i.e,
baseline wire inspections). Also, since we have implemented Critical 1 hardware wire
separation and are tracking moxlifications on all vetiicles, the relative risk for loss of
crew/vehicle associated with a wiring event has been significantly reduced. In addition, the
Orbiter wiring teum maintains a comprehensive database of all wire damage that is identified
across the entire fleet 1s tool assists 1 the management of potential areas of wiring risk
and drives future criteria for damage mitigation. We are also continuing our destructive
testing and analysis from which we will glean all available information to further advance our
understanding of the lifetime characteristics of Kapton wing.

tis noteworthy that significant wire health assessments and improvements have been
incorporated throughout the life of the SSP. Most notable were those following the wiring
concerns sternming from the STS.93 flight including:

* Refinement of wiring cnmping techniques and inspection criteria.



Appendix B. Bibliography Related to Kapton-Insulated Wiring

1, Kenneth Blemel and Cynthia Furse, PhD, “Applications of Microsystems and Signal
Processing for Wiring Integrity Monitoring,” IEEE Aerospace Symposium, March 2001.

2. “Discrepancies Discovered in Visual Inspections of Aging Aircraft Electrical Systems,” Air
Safety Week, January 24, 2000, Phillips Publishing International, Incorporated.

3. “Wire System Task Force Expands Scope of Activity,” Air Safety Week, March 19, 2001,
Phillips Publishing International, Incorporated.

4. “Wire Safety — A Slipping Priority,” Air Safety Week, July 21, 2003, Phillips Publishing
International, Incorporated.

5. Captain Edward R. Hanson Jr., “Product Focus: Aircraft Wiring,” Aviation Today, March 16,
2004.

6. Frank Hoertz and Dieter Koenig, “Studies of the Early and Late Degradation Phase of Wire
Insulation for Aircraft Applications,” Darmstadt University of Technology.

7. “Degradation of Kapton Electrical Insulation,” United States Nuclear Regulatory
Commission, Office of Nuclear Reactor Regulation, November 21, 1988.

8. Captain Paul McCarthy, “Aircraft Wiring,” Testimony, Committee on Transportation and
Infrastructure Subcommittee on Oversight, Investigations and Emergency Management, U.S.
House of Representatives, September 15, 1999.

9. David H. Johnson, “Wiring System Diagnostic Techniques for Legacy Aircraft.”

10. Dupont, “Kapton Polyimide Film, General Specifications,” Bulletin GS-97-6, September
2000. http:/Awww.Dupont.com.

11. The Gund Company, Incorporated, “Material Data Sheet.”

12. Brent Waddoups, Dr. Cynthia Furse, and Mark Schmidt; “Analysis of Reflectometry for
Detection of Chafed Aircraft Wiring Insulation,” Department of Electrical and Computer
Engineering, September 2001.

13. Paul Smith; “Using Inert Gas to Enhance Electrical Wiring Inspection,” 6" Joint
FAA/DoD/NASA Aging Aircraft Conference, September 19, 2002.

14, Dupont®, Dupont Films, Kapton® Polyimide Film, Corona Resistant Kapton® CR Takes
Electrical Insulation Design and Reliability to New Levels.

15. “Intrusive Inspections of Aircraft Wiring Show Deterioration Over Time,” Air Safety Week,
July 24, 2000.

16. D.S. Forsyth, J.P. Komorowski, R.W. Gould, and A. Marincak; “Automation of Enhanced
Visual NDT Techniques;” National Research Council Canada.
http://www.ndt.net/article/pacndt98/18/18.htm.



Risk Should Be Assessed

In the near term, the Space Shuttle Program should formally assess and articulate the risks of
aging and damaged orbiter wiring. A formal risk assessment ensures that wiring risk is not
only identified and analyzed but effectively tracked, controlled, and communicated with the
program team. A formal risk assessment also initiates a process that objectively documents and
communicates risk management steps to the program’s stakeholders. The Space Shuttle
Program has implemented destructive evaluation of wiring removed during orbiter rework to
determine the extent of wire aging. The destructive evaluation should provide data on wire
aging that will serve as a first step in performing a formal wiring risk assessment.

Recommendation for Corrective Action

1. The Associate Administrator for the Space Operations Mission Directorate should
formally assess the risk of aging and damaged orbiter wiring in accordance with NPR
8000.4 and develop a risk mitigation plan based on the risk assessment.

Management’s Response. Management concurred and agreed to initiate the process of
formalizing wiring risk assessments to comply with NPR 8000.4. In response to the lack of a
comprehensive risk assessment conducted concerning orbiter wiring, the Associate
Administrator for the Space Operations Mission Directorate stated that the Space Shuttle
Program team has developed specific criteria for inspecting orbiter wiring and has refined
wiring crimping techniques and specific criteria for inspecting orbiter wiring, to include
baseline wire inspections. The team has implemented Critical 1' hardware wire separation and
arc tracking modifications on all vehicles, significantly reducing the relative risk for loss of a
crew or vehicle associated with a wiring event. The Associate Administrator noted that
significant wire health assessments and improvements had been incorporated throughout the
life of the Space Shuttle Program, to include refinement of wiring crimping techniques and
inspection criteria, separation of wires performing criticality 1 functions, and additional wire
protection modifications to minimize damage. He also stated that the orbiter wiring team
maintains a comprehensive database of all wire damage identified across the fleet to assist in
the management of potential wiring risk and to drive criteria for damage migration.

Evaluation of Management’s Response. Management’s actions are responsive to the
recommendation. We consider the recommendation resolved, but the recommendation will

remain open for reporting purposes pending completion of necessary corrective action. The
complete text of management’s response is in Appendix D.

Ttems specified as Critical 1 are items that could cause the loss of crew, vehicle, or ground personnel, or could
endanger the public, in the event of failure.


Finding A: Formal Assessment of Wiring Risk

The Space Shuttle Program had not performed a formal risk assessment for aging and damaged
wiring in conformance with NASA requirements or developed a risk mitigation plan based on
such an assessment. The Space Shuttle Program did not perform a formal risk assessment
because the program obtained a general assessment of wiring risk from an analysis of historical
wiring damage and the visual inspection process. A formal risk assessment will initiate a
systematic process that ensures wiring risk will be effectively managed by the Space Shuttle
Program team and objectively communicated to the program’s stakeholders.

NASA Requirements

NASA Procedural Requirements (NPR) 8000.4 establishes risk management processes and
requirements that the program team must follow throughout the program life cycle. NPR
8000.4 defines the following six-step continuous process for managing risk and requires the
program team to complete specific procedures and documented outputs during each step:

1. Identification. The program team must identify program risks, describe the undesirable
event each risk presents, and state the consequences of the event to the program.

Outputs are a risk statement for each identified risk and a comprehensive list of the risk
statements.

2. Analysis. The program team must perform a risk analysis. Procedures include an
assessment of the consequences of each risk, an estimate of the likelihood that an
identified risk will occur, a prioritization of risks based on their consequences and
likelihood of occurrence, and a timeframe for preventive action. Outputs are clear

estimations of the risk consequences, likelihood, timeframe for preventive action, and
risk priority.

3. Planning. The program team must plan mitigation actions and assign responsibilities.
Procedures include establishing criteria for accepting risks and documenting the
rationale and acceptance of risk. Outputs are assignments of responsibility, individual
risk mitigation plans, and rationale for accepted risks.

4. Tracking. The program team must collect, update, and analyze risk data and report risk
trends. Procedures include determining whether risks are decreasing, staying the same,
or increasing over time. Outputs are risk status reports.

5. Control. The program team must reevaluate risk mitigation actions based on recent

tracking information. Outputs are documented decisions made by the appropriate
decision maker with respect to risk.

6. Documentation and Communication. The program team must maintain a formal
documentation trail within the program team and with stakeholders. Outputs include

the program plan, acquisition plan, risk management plan, statement of risk, and risk
list.


inaccessible areas. The CAIB recommended that “NASA should, as part of the Shuttle Service
Life Extension Program and potential 40-year life, develop a state-of-the-art means to inspect
all orbiter wiring, including that which is inaccessible.”


Appendix B

17, Christopher D. Smith and William J. Hughes, “FAA Aging Nonstructural Systems
Research,” Aging Transport Systems Rulemaking Advisory Committee (ATSRAC).
http://www. mitrecaasd.org/atsrac/index.html.

18. Edward B. Block, “Dissenting Opinion To The Training Working Group’s Final Report,
Attachment 1 to July 2002 ATSRAC Minutes,” March 4, 2002.

19. Tony Poole and Don Andersen, “Electrical Standard Wire Practices Manual,” Final Report
ATSRAC Harmonization Working Group 7. http://www.mitrecaasd.org/atsrac/index.html.

20. Paul Lapwood, “Aircraft Wiring Systems Training Curriculum and Lesson Plans,” Aging
Transport Systems Rule Making Advisory Committee.
http://www.mitrecaasd.org/atsrac/index.html.

21. Mark Brown and Fancois Gau, “Wire Integrity Programs and Aging Aircraft Sustainment,”
The 5" Joint NASA/FAA/DoD Conference on Aging Aircraft. July 27, 2001.

22. Robert Beremand, Chad Hanak, and Melissa Straubel, “Identification of Aging Aircraft
Electrical Wiring,” Woolrich Engineering Consulting Firm, May 5, 2003.

23. Julie Schonfeld, “Wire Integrity Research (WIRe) Pilot Study,” NASA Ames Research
Center, August 25, 2000.

24. Steven. J. Sullivan and George A. Slenski, “Managing Electrical Connection System and
Wire Integrity on Legacy Aerospace Vehicles.”

25. Kenneth G. Blemel and Peter A. Blemel, “Smart Wiring Prognostic Health Management,”
Management Sciences, Incorporated, June 27, 2000.

26. Kenneth Blemel, “Smart Connector and Wiring Systems for Enhanced Safety, Situation
Awareness and Reduced Costs of Ownership,” Management Sciences, Incorporated.

27. Erik C. Carlson, “A Smart Wire System For Non-Destructive Inspection of Aircraft Wire
Harnesses,” Aegis Devices Incorporated.

28. Yongxing Wang, Dieter Koenig, Frank Hoertz, and Thorsten Fugel; “Fault Arc Resistance

Tests of Wires for Aircraft Application;” High Voltage Laboratory; Darmstadt University of
Technology.

29. George A. Slenski and Joseph S. Kuzniar, “Aircraft Wiring System Integrity Initiatives, a
Government and Industry Partnership,” 6" Joint FAA/DOD/NASA Conference on Aging
Aircraft, September 2002.


orbiter wiring to increase the likelihood of flight safety. Aging has been identified as a
drawback of Kapton-insulated wiring and posés a potential risk for the Space Shuttle.

Aging and Environmental Effects on Kapton. Decades of use have revealed a significant
defect with Kapton-insulated wiring that was not apparent during its development and initial
use: Kapton insulation can break down or become damaged with age, with mishandling, and
from moisture. The insulation damage that occurs is a condition referred to as aging. Wire that
is aging is most susceptible to damage because the insulation becomes brittle and deteriorates
over time. Minute insulation breaks can gradually occur.

To illustrate the vulnerability of Kapton insulation, the typical wiring bundle of an orbiter will
experience extreme changes in heat, varying degrees of moisture, bending and contorting
during installation, chaffing, and heavy vibrations. The Kapton insulation will begin to lose its
elasticity because of the effects of time and harsh environmental elements—making it
susceptible to becoming brittle. Such brittleness can cause radial and longitudinal breaks in the
insulation. Insulation breaks or damage can potentially lead to a phenomenon known as arc
tracking. Arc tracking takes place after carbon collects at the damaged area. After carbon
deposits occur, Kapton becomes a conductor, leading to a “soft electrical short” that can cause
systems to gradually fail or operate in a degraded fashion. For example, a soft electrical short
in the wire system can cause a loss of critical functions in equipment, an unintended function of
the equipment, or the loss of information regarding its operation. An electrical short in the wire
system can lead to fire in an extreme case.

The useful life of a Kapton wire depends on the degree of deterioration of the polyimide
insulation. A useful life for Kapton wire has not been established by research. Research has
shown that the use of wiring over time is a factor in the deterioration of Kapton.

Age of Orbiter Wiring. Since the use of wiring over time has been associated with Kapton’s
deterioration, the age of wiring in the orbiters could indicate the potential for wiring
deterioration. In calculating the age of the orbiter wiring, it was determined that the main
wiring harnesses in the orbiters were manufactured 2 to 5 years before the orbiters were built.
The following table identifies the remaining orbiters and shows that the main wiring harnesses
have either reached or are nearing 25 years.

Age of Orbiter Wiring
Name of Date Construction Began Age of
Orbiter Vehicle on Crew Module Wiring in Years
Discovery August 1979 25+
Atlantis March 1980 25+
Endeavor February 1982 23+



Appendix D

health of the entire wire segment from source to load. And depending on the wire type and
function, additional circuit integrity tests are performed above and beyond the normal
electrical parameters that the wire segment would see during standard operation,

AS your report details, the complexity of evaluating the miles of wire flowing through each
Orbiter is very difficult to fully encompass. The need for diligent examination and testing of
the winng will continuously evolve within the framework of the Shuttle program, We are
committed to this undertaking now and through our last flight.

Sincerely.

Ui dliton flatts

William P, Ready

21



AUDIT REPORT 1G-05-023

SPACE SHUTTLE ORBITER
WIRING INSPECTION

July 14, 2005

OFFICE OF ee TOR Dt
Released by: LL a eo 4 ese”
National Aeronautics and

Space Administration Evelyn R. Klemstine
Assistant Inspector General for Auditing



Revised Shuttle Service Life. On January 14, 2004, the President announced a new vision for
the Nation’s space exploration program that would retire the Space Shuttle in 2010. The 2010
retirement would effectively reduce the Space Shuttle’s planned service life from 40 years to
less than 30 years. The PRCB concluded that the CAIB recommendation related to wiring was
no longer feasible because flights would not continue after 2010 and the remaining service life
of the Space Shuttle would be insufficient to develop and incorporate new evaluation
technology.

However, one CAIB investigator involved in the assessment of Kapton wiring issues stated that
the intent of Recommendation 4.2-2 was that NASA should develop and implement new
nondestructive evaluation technology for inspecting orbiter wiring regardless of the retirement
date. In addition, the investigator stated that NASA should still pursue research and
development of new evaluation technology because visual inspection alone would not detect
wiring and insulation problems that could cause systems to gradually fail or operate in a
degraded fashion.

Readiness of New Evaluation Technology. In deciding to cancel development of new
nondestructive evaluation technology, the PRCB concluded that a proof of concept would occur
during Fiscal Year (FY) 2005 but that the technology would not be operational until 2009 at the
earliest. The PRCB concluded that because new nondestructive evaluation technology was
unlikely to be made operational before 2009, the evaluation technology would benefit only
Space Shuttle flights planned from 2009 through the new Space Shuttle retirement date of
2010. However, an official from the Air Force Research Laboratory (AFRL) who conducts
research on aging aircraft wiring stated that a developmental test system might be available by
the end of 2006.

The same AFRL official stated that, although not yet suitable for the orbiter, diagnostic systems
for wiring are being used today on commercial and military aircraft to identify and locate
wiring problems. A study that two AFRL researchers presented at the Joint Conference of
Aging Aircraft in September 2002’ projected that new technologies capable of diagnosing
intermittent failures and damaged wiring might be available between 2004 and 2008.

Funding Constraints. Between January and July 2004, NASA estimated the cost to
implement the 29 recommendations’ that the CAIB made at $450 million to $760 million more
than the $1.082 billion the Agency requested in its FY 2005 budget (see Appendix C for details
of funded return-to-flight activities associated with the CAIB recommendations).

The Space Shuttle Program Office estimated that the Four-Prong Plan would cost
$12.6 million—$12.0 million to develop new nondestructive evaluation technology and

? “Aircraft Wiring System Integrity Initiatives - A Government and Industry Partnership,” presented by George A.
Slenski and Joseph S. Kuzniar of the Materials and Manufacturing Directorate, AFRL.

* The August 2003 report of the CAIB contained a total of 29 recommendations related to the physical and
organizational/cultural causes of the accident. The CAIB identified 15 recommendations mostly related to the
physical causes of the accident as “before return to flight” that must be addressed before the Space Shuttle returns
to flight. The remaining 14 recommendations were viewed by the CAIB as “continuing to fly” recommendations
that reflect the Board’s views on what’s needed to safely operate the Space Shuttle and future spacecraft in the
mid- to long-term.


Finding B: Cancellation of Plan for New Evaluation Technology

The Space Shuttle Program Requirements Control Board (PRCB) cancelled development and
testing of new technology for evaluating the condition of orbiter wiring. The PRCB decision
was based on a conclusion that new technology would not be ready before 2010, a conclusion
that developing new evaluation technology would not be cost effective because of the planned
2010 Shuttle retirement, and funding constraints. New evaluation technology could improve
NASA’s ability to detect damage to wiring not examined during visual inspections as well as
enhance the safety of the orbiter and any next generation space vehicle that uses Kapton-
insulated wiring.

CAIB Recommendation

In fall 2003, the Space Shuttle Program Office tasked the Wiring Working Group with
developing a plan to respond to CAIB Recommendation 4.2-2. That recommendation stated,
“NASA should, as part of the Shuttle Service Life Extension Program and potential 40-year
life, develop a state-of-the-art means to inspect all orbiter wiring, including that which is
inaccessible.” The intent of the recommendation was for NASA to develop new nondestructive
evaluation technology more advanced than technology used in visual inspection. The Working
Group’s plan, known as the Four-Prong Plan, contained four elements: (1) continue visual
inspection, (2) develop nondestructive technology to evaluate wiring in the orbiters, (3) conduct
destructive evaluation of damaged wiring removed from the orbiters, and (4) fund a study to
determine feasibility of replacing orbiter wiring.

Decision on the Four-Prong Plan

Prongs 1, 3, and 4 of the Four-Prong Plan (visual inspection, destructive evaluation, and a
feasibility study) did not relate to developing a state-of-the-art means of inspection. However,
Prong 2 of the Four-Prong Plan involved developing new nondestructive evaluation technology
that could automate wire damage detection as well as predict the remaining useful life of orbiter
wiring. Prong 2 recommended three new nondestructive evaluation technologies (technology
that could be used in the orbiter without removing wiring for testing): an insulation tester, age-
life tester, and ultrasonic crimp joint tool. An insulation tester allows for detection of subtle
defects in wiring insulation and shields. The tester would automate detection through software
signal processing. Age-life testers could determine insulation degradation as a result of aging
and the environment. The age-life testers considered several technologies, which included an
ultrasonic tester, wire indenter, and infrared spectroscopy. An ultrasonic crimp joint tool could

measure the integrity of a wire crimp when wires are spliced during maintenance or repair
work.

In June 2004, the Wiring Working Group presented the Four-Prong Plan to the PRCB. The
PRCB decided to continue visual inspection (Prong 1) and conduct destructive evaluation
(Prong 3) and not to pursue development of new nondestructive evaluation technology
(Prong 2) or the feasibility study on wiring replacement (Prong 4). The PRCB made the
decision after consideration of the revised Space Shuttle service life, the readiness of new
evaluation technology. and funding constraints.


Risk Assessment and Mitigation Documents

Space Shuttle Program officials provided two documents as evidence of the wiring risk
assessment and mitigation plan. The first document, the “Fleet Wire Inspection and Repair
Status Review,” dated October 1999, reported observations of wire damage from vibration,
wire routing, and physical impact. The second document, the “New Wire Insulation Study for
Potential Orbiter Use,” dated June 2000, evaluated characteristics of currently available wire
insulation to recommend a possible new wire insulation type for orbiter use.

The observations and evaluations in the documents would be useful inputs to identify and/or
analyze risk but do not comply with the procedures or outputs that are required in NPR 8000.4.
Specifically, the documents did not identify individual wiring risks, describe the undesirable
events that the risk presents to the Space Shuttle, state the consequences of the events, estimate
the likelihood that the consequences might occur, prioritize wiring risks based on their
consequences and likelihood of occurrence, project a timeframe for preventive action, and state
the actions that can be taken for each risk. The documents provided us showed no evidence of
tracking or controlling the wiring risks that had been identified by the Space Shuttle Program.
The documents were not outputs as described in NPR 8000.4.

Historical Damage and Visual Inspection

Space Shuttle Program officials stated that an analysis of wiring repairs on the orbiter showed
that mishandling has historically caused wiring damage. Based on that analysis, program
officials concluded that mishandling presented the greatest wiring risk. However, the officials

reached that conclusion without formally assessing the risks associated with aging and damaged
orbiter wiring.

Headquarters officials agreed that the Space Shuttle Program had not completed a formal risk
assessment according to NPR 8000.4. Officials further stated that the Space Shuttle Program
had a general assessment of wiring health and insight into wiring health through the visual
inspection process without strictly following NASA risk assessment requirements.

Those officials cited additional measures taken by the Space Shuttle Program after the CAIB
report to reduce wiring risk. For example, the Space Shuttle Program is separating redundant
wires that are in the same wire bundle to minimize the risk of losing a critical system to which
the wire bundle is attached. Redundant wires are designed to perform the same function and
serve as a backup to each other should one of the wires fail during flight. Separation is
completed on one orbiter and will be completed on the remaining two orbiters before the
orbiters are returned to flight. The Space Shuttle Program also implemented a database that
provides graphical images of wiring damage that show the severity and location of wiring
damage. The Space Shuttle Program intends to use the images to pinpoint high damage
locations so that a more thorough visual inspection can be conducted there.

Risks of Kapton-Insulated Wiring

Without following the systematic risk management approach prescribed by NPR 8000.4, the
Space Shuttle Program cannot ensure it has effectively managed the risks of aging and damaged


D. Management’s Response

Natana: Avronautes and

Space Administration
Headquarters
Washington, DC 20546-0001

June 28, 2005

Space Operations Mission Directorate

10: Inspector General

Associate Administrator, Space Operations Mission Directorate

Draft Audit Report titled “Space Shuttle Orbiter Wiring Inspection"
(Assignment number A-04-010-00)

Picuse find below the enclosed Space Operations Mission Directorate (SOMD) and
Exploration Systems Mission Directorate (ESMD) comments and responses to the above
NASA Office of Inspector General (OG) audit repor.

We acknowledge the hard work and professionalism of the NASA OIG in their recent audit of
the Space Shuttle program's Orbiter wiring inspections efforts as they relate to the Columbia
Accident Investigation Board's (CATB) recommendation 4.2-2. We are always looking for
‘opportunities to improve ourselves and our systems and uppreciate your insights.

We concur with both of your recommendations and will initiate the provess of formalizing our
Orbiter wiring risk assessments as to comply with NPR 8000.4. Your recommendation to

establish a formal exchange with the ESMD is already underway and valuable lessons leamed
are being transmitted.

As programs and projects evolve in ESMD, Systems Technical Warrant Holders (STWHs)
will be selected per the NASA Technical Authority guidelines. Discipline Technical Warrant

Holders (DTWHs) have already been involved in the review of ESMD. requirements and.
acquisition documents. The DTWHs are responsible for:

* Approval of all new or updated NASA Preferred Standards (including
nondestructive evaluation (NDE)) within their assigned disciplines,

Interfacing with other warrant holders, promoting communication throughout
the Agency's technical community to ensure that appropriate individuals and
organizations are aware of technical issues, and

Proactively communicating best practices, improving standards, providing
technical recommendations, and monitoring the general health of their
respective disciplines.



Table of Contents

Executive Summary
Objectives
Background
Findings
Finding A: Formal Assessment of Wiring Risk
Finding B: Cancellation of Plan for New Evaluation Technology
Appendixes

A. Scope and Methodology

ies]

. Bibliography Related to Kapton-Insulated Wiring
C. Funded Return-to-Flight Activities
D. Management’s Response

E. Report Distribution

Acronyms Used in the Report

AFRL Air Force Research Laboratory

CAIB Columbia Accident Investigation Board
FY Fiscal Year

NPR NASA Procedural Requirements

PRCB Program Requirements Control Board

12

14

17

18

22


formal procedure to share lessons learned by the Space Shuttle Program during the planning of
new technology development. Sharing of this information will facilitate the development of
nondestructive evaluation technology for wiring inspection in the Shuttle and/or next
generation space vehicles.

Recommendation for Corrective Action

2. The Associate Administrator for the Exploration Systems Mission Directorate and the
Associate Administrator for the Space Operations Mission Directorate should
establish a formal procedure that shares lessons learned on development of new

nondestructive evaluation technology for wiring inspection of the Space Shuttle
orbiter and/or next generation space vehicles.

Management’s Response. Management concurred and stated that implementation of a formal
procedure has begun. Implementation will include formal exchanges of information and
lessons learned between the Space Operations Mission Directorate and the Crew Exploration
Vehicle Team and subsequent Exploration Systems Mission Directorate programs and projects.
The Space Operations and Exploration Systems Mission Directorates will continue to document
and share lessons learned and determine the need for and scope of complex electrical system
integrity and wire system nondestructive evaluation research.

Evaluation of Management’s Response. Management’s actions are responsive to the
recommendation and we consider the recommendation closed for reporting purposes.
However, we will continue to monitor the technical recommendations resulting from the
exchange of information. The complete text of management’s response is in Appendix D.


Appendix A

Management Controls

Specific management controls we reviewed were wiring risk mitigation procedures, actions taken
in response to CAIB Recommendation 4.2-2, and procedures to communicate Space Shuttle
Program Office plans for the recommendation in the Implementation Plan.

Audit Work

We performed audit field work from January 2004 through June 2005 at Johnson Space Center
and Kennedy Space Center. We performed the audit in accordance with generally accepted
government auditing standards.


new evaluation techniques, any next generation space vehicle is likely to face challenges in
evaluating damage to inaccessible wires.

Summary of Recommendations. We recommend that the Associate Administrator for the
Space Operations Mission Directorate formally assess the risk of aging and damaged orbiter
wiring in accordance with NPR 8000.4 and develop a risk mitigation plan based on the risk
assessment. Additionally, the Associate Administrator for the Exploration Systems Mission
Directorate and the Associate Administrator for the Space Operations Mission Directorate
should establish a formal procedure that shares lessons learned on development of new
nondestructive evaluation technology for wiring inspection of the Space Shuttle orbiter and/or
next generation space vehicles.

Management Comments. Management concurred with our recommendations and is taking or
has taken appropriate corrective actions. In response to our first recommendation, NASA’s
Associate Administrator for the Space Operations Mission Directorate agreed to initiate the
process of formalizing wiring risk assessments to comply with NPR 8000.4. In response to our
second recommendation, NASA’s Associate Administrator for the Exploration Systems
Mission Directorate and the Associate Administrator for the Space Operations Mission
Directorate have begun to formally exchange information and lessons learned on new
nondestructive wiring evaluation technology.

The complete text of management’s response is in Appendix D.
'P! 8 Pp Pp


Appendix E. Report Distribution

NASA

Administrator
Deputy Administrator
Associate Administrator for the Space Operations Mission Directorate
Associate Administrator for the Exploration Systems Mission Directorate
Chief of Staff
Director, Management Systems Division, Office of Infrastructure and Administration,
Office of Institutions and Management
Audit Liaison Team Lead (HQ/OJD)

NASA Centers

Director, Johnson Space Center

Audit Liaison Representative (JSC/BD5)
Director, Kennedy Space Center

Audit Liaison Representative (KSC/QA-D)

Non-NASA Organizations and Individuals

Office of Management and Budget
Deputy Associate Director, Energy and Science Division
Branch Chief, Science and Space Programs Branch
Government Accountability Office
Director, NASA Issues, Office of Acquisition and Sourcing Management

Congressional Committees and Subcommittees, Chairman and Ranking Minority
Member

Senate Committee on Appropriations
Senate Subcommittee on Commerce, Justice, and Science
Senate Committee on Commerce, Science, and Transportation
Senate Subcommittee on Science and Space
Senate Committee on Homeland Security and Governmental Affairs
House Committee on Appropriations
House Subcommittee on Science, State, Justice, and Commerce
House Committee on Government Reform

House Subcommittee on Government Management, Finance, and Accountability
House Committee on Science

House Subcommittee on Space and Aeronautics

22


1G-05-023 July 14, 2005
A-04-010-00

SPACE SHUTTLE ORBITER WIRING INSPECTION

Executive Summary

The NASA Office of Inspector General initiated an audit with an overall objective of
determining whether Agency actions regarding inspections of orbiter wiring were responsive to
Columbia Accident Investigation Board (CAIB) Recommendation 4.2-2. The specific
objectives were to determine whether (1) planned operational changes would make visual
inspection more effective and (2) plans would enable the Agency to identify and test
state-of-the-art evaluation technology and make tested technology available for the orbiter.

Background. In its August 2003 report, the CAIB recommended that “NASA should, as part
of the Shuttle Service Life Extension Program and potential 40-year life, develop a
state-of-the-art means to inspect all orbiter wiring, including that which is inaccessible.” A.
significant amount of orbiter wiring (140 to 157 miles) is insulated with Kapton, a polyimide
film used as electrical insulation. Kapton has been widely used in aircraft and spacecraft for
more than 30 years because it is lightweight, nonflammable, has a wide operating temperature
range, and resists damage. However, Kapton insulation can break down or become damaged
with age, with mishandling, and from moisture. Since the CAIB recommendation, NASA has
adopted the President’s Vision for Space Exploration, which calls for the Space Shuttle
retirement in 2010.

Results. To address Kapton concerns, the Space Shuttle Program modified visual inspection
procedures for orbiter wiring. However, the Space Shuttle Program has not formally assessed
the risk of aging and damaged wiring in accordance with NASA Procedural Requirements
(NPR) 8000.4, “Risk Assessment Procedures,” or developed a risk mitigation plan based on
such an assessment. NASA has a well thought-out risk management policy. When NASA
elects to bypass its risk management requirements, it is turning away from the safety and
management principles built into the policy and is engaging in “decision-making processes that
operate outside the organization’s rules,” which was cited in the CAIB report’s executive
summary. Without following the systematic risk management approach prescribed by

NPR 8000.4, the Space Shuttle Program cannot ensure it has effectively managed the risks of
aging and damaged orbiter wiring to increase the likelihood of flight safety.

Additionally, the Space Shuttle Program cancelled plans to develop and test state-of-the-art
technology for evaluating orbiter wiring. As for whether CAIB Recommendation 4.2-2 has
been met, we conclude that the recommendation has been overtaken by new events. The
recommendation presumed that the Space Shuttle Program would continue to at least 2020;
the President’s Vision for Space Exploration reoriented NASA toward retiring the Shuttle in
2010, which removed the basis for the recommendation. However, it is important to note
from a risk standpoint that without the new evaluation technology that was at the heart of
Recommendation 4.2-2, the inability to detect problems with inaccessible wiring will continue
to be a safety risk for the orbiter. Additionally, since there is no commitment to develop the


$600,000 to conduct destructive evaluation. The Four-Prong Plan did not identify costs for
visual inspection and the wire replacement study in the $12.6-million estimate. Visual
inspections would be funded from Shuttle operations funds because they are an ongoing part of
operations, and a cost estimate had not been completed for the wire replacement study.

In August 2004, the Space Shuttle Program prepared closeout plans for CAIB
Recommendation 4.2-2, which showed no funding after FY 2004 for either new nondestructive
evaluation technology or destructive evaluation.

Wiring Not Examined During Visual Inspection

Visual inspections of orbiter wiring involve examining the outer wires of a bundle and not the
wires inside the bundle. Wires inside the bundle are inspected only when insulation damage is
detected on the outer wires. The inspection then includes a 360-degree check of loose wires
and the area of insulation damage. Figure 1 shows wiring bundles within the crew module of
the orbiter Atlantis.

Figure 1. Wiring Bundles Within the Crew Module of Orbiter Atlantis

Visual inspection is limited because inaccessible wiring may not be examined for damage.
Inaccessible wiring includes wire bundles in locations that are too constricted to permit the
inspector to reach the wiring with inspection tools, which consist of a small magnifying lens
and a flat metal spatula held behind the wiring. Also, wires inside a wire bundle are obscured
from view by other wiring and not examined unless damage is noted on the outer wires in the
bundle. More intrusive inspection of orbiter locations that are inaccessible to visual inspection
would require the inspector to pull apart, pull down, or otherwise disturb wire bundles.

Figure 2 shows a visual inspection of wiring in the payload bay of the orbiter Atlantis.


Appendix A. Scope and Methodology

Scope and Methodology

We interviewed personnel directly involved in CAIB Recommendation 4.2-2 and wiring visual
inspection, which included personnel at the Space Shuttle Program Office, the Kennedy Space
Center Quality Assurance Office, and two contractors. Specific personnel interviewed included
the Wiring Project Manager, quality assurance officials and inspectors, Space Shuttle Program
officials and engineers, and other management and contractor officials. We participated in
meetings of the orbiter wiring project team. An official from AFRL provided information on

Kapton and evaluation technologies. We also interviewed the leader of the CAIB team and a
CAIB evaluator for orbiter wiring issues.

The audit team observed visual inspection procedures and reviewed inspection policies for the
orbiter. The team reviewed technical studies on wire aging and inspection, including a report by
the Government Accountability Office, “Aviation Safety, FAA and DoD Response to Similar
Safety Concerns,” GAO-02-77, dated January 2002. We reviewed the CAIB report; Space
Shuttle Program plans for implementing CAIB Recommendation 4.2-2 (the Four-Prong Plan);
notes from the June 2004 meeting of the PRCB; the closeout plan for CAIB Recommendation
4.2-2, dated August 2004; and pertinent sections of the “NASA Implementation Plan for Return
to Flight and Beyond” (Implementation Plan), August and December 2004. Our review of those
plans included an evaluation of planned actions and funding.

The audit team received technical assistance from an electrical engineer and an aerospace
engineer technologist. Those employees researched properties of Kapton-insulated wiring used
in the orbiter, identified best practices for inspections and new technologies for evaluation,
reviewed studies of evaluation technologies, and evaluated the status of new technologies. The
aerospace engineer technologist also reviewed the estimated percentage of inaccessible orbiter
wiring that was calculated by Space Shuttle Program engineering staff at Kennedy Space Center.
To verify the estimated percentages, the aerospace engineer technologist reviewed the

methodology and supporting data and retraced calculations used to derive the estimated
percentage.

Use of Computer-Processed Data

We did not assess the reliability of computer-processed data because we did not rely on it to
achieve our objectives.