NOAA N-PRIME Mishap Investigation-Final Report 

NOAA N-PRIME Mishap Investigation 

Final Report 

September 13, 2004 

NASA

NATIONAL AERONAUTICS

 AND SPACE ADMINISTRATION

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

TABLE OF CONTENTS

VOLUME I

SECTION TITLE PAGE 

1 NOAA N-PRIME Mishap Investigation Board and Advisors 2 

2 NOAA N-PRIME Mishap Investigation Board Staff 4 

3 Executive Summary 5 

4 Program Description 11 

5 Description of Events Leading to Mishap 17 

6 Method of Investigation 54 

7 Findings 62 

8 Recommendations 78 

9 Observations 87

10 Acronyms List 88 

11 References List 90 

 

APPENDICES 

A NOAA N-PRIME Anomaly Report A-1 

B Letter- Appointment of Mishap Investigation Board B-1 

C NOAA N-PRIME Mishap Photographs C-1 

D NOAA N-PRIME Damage Assessment D-1 

E Root Cause Analysis E-1 

F LMC Accident Investigation Team (AIT) Findings Assessment (Proprietary) F-1 

G LMC AIT NOAA N-PRIME Accident Investigation Final Report (Proprietary) G-1 

H TIROS Anomaly Review-Final Presentation H-1 

I Mishaps and Near Misses Summary (Proprietary) I-1 

VOLUME II 

WITNESS INTERVIEWS/STATEMENTS* 

MISHAP INVESTIGATION BOARD ACTIVITIES SUMMARY* 

* UNDER SEPARATE COVER.

CONFIDENTIAL/PRIVILEGED INFORMATION - DO NOT DUPLICATE OR 

DISTRIBUTE

1

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 1 

NOAA N-PRIME MISHAP INVESTIGATION BOARD 

Christopher Scolese 

Deputy Associate Administrator 

for Space Science, NASA Headquarters 

Chairman 

Richard A. Foss 

Chief, Systems Integration and Test Branch 

Langley Research Center, NASA 

Malcolm Glenn 

Mission Assurance Engineer 

Shuttle Processing 

Kennedy Space Center, NASA 

Colonel James R. Horejsi 

Chief Engineer 

Space and Missile Systems Center 

Air Force Space Command 

Dr. Tina L. Panontin 

Chief Engineer 

Ames Research Center, NASA 

Abigail Harper 

Deputy Director, Office of Safety and 

Mission Assurance 

Goddard Space Flight Center, NASA 

Ex-Officio Member 

2

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

NOAA N-PRIME MISHAP INVESTIGATION BOARD ADVISORS 

Thomas Berndt 

Legal Advisor 

Deputy Chief Counsel 

NASA Ames Research Center 

3

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 2 

NOAA N-PRIME MISHAP INVESTIGATION BOARD STAFF 

Richard Ho Executive Secretary NASA Goddard Space Flight Center 

James Lumsden Consultant NASA Jet Propulsion Laboratory 

Carl Weber Advisor NASA Headquarters/Code H 

Lynn W. Bailets Advisor NASA Headquarters/Code H 

Merrylee Weber Advisor NASA Headquarters/Code Q 

Dr. Anngienetta R. Johnson Advisor NASA Headquarters/Code Y 

Robert Navarro Advisor NASA Ames Research Center 

Daniel Dittman Advisor NASA Ames Research Center 

Gary Davis Advisor National Oceanic and Atmospheric 

Administration 

Wilfred Mazur Advisor National Oceanic and Atmospheric 

Administration 

Dr. Scott Schappell Consultant Federal Aviation Administration 

Civil Aeromedical Institute 

4

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 3 

EXECUTIVE SUMMARY 

On Saturday, September 6, 2003 during an operation at Lockheed Martin Space Systems 

Company (LMSSC) Sunnyvale that required repositioning the Television Infrared Observational 

Satellites (TIROS) National Oceanic and Atmospheric Administration (NOAA) N-Prime satellite 

from a vertical to a horizontal position, the satellite slipped from the Turn-Over Cart (TOC) and 

fell to the floor (see Figure 3-1). The satellite sustained heavy damage (see Figure 3-2), although 

no injuries to personnel occurred. The exact extent of the hardware damage is still being 

assessed. 

The operation scheduled for that day was to shim the Microwave Humidity Sounder (MHS) 

instrument by removing and replacing the instrument. This operation required the spacecraft to 

be rotated and tilted to the horizontal position using the TOC. The spacecraft fell to the floor as 

it reached 13 degrees of tilt while being rotated. The reason was clear from inspection of the 

hardware: the satellite fell because the TOC adapter plate was not secured to the TOC with the 

required 24 bolts (see Figure 3-3). 

Three days after the mishap, on September 9, 2003, Dr. Ghassem Asrar, NASA Associate 

Administrator for Earth Science established the NOAA N-PRIME Mishap Investigation Board 

(MIB) in the public interest to gather information, conduct necessary analyses, and determine the 

facts of the mishap. To identify the root causes at work in the NOAA N-PRIME Mishap, the 

MIB undertook two approaches. The first was an extensive analysis of the sequence of events 

prior to and on the day of the mishap; the planned operational scenario vs. the actual execution; 

and the planning activities, including scheduling, crew assembly and test documentation 

preparation (see Section 5 – Description of Events Leading to Mishap). The second approach 

was to utilize the Human Factors Analysis and Classification System (HFACS) (2000) to provide 

a comprehensive framework for identifying and analyzing human error (see Section 6 – Method 

of Investigation and Section 7 – Findings). Evidence from a number of sources, including 

witness interviews, test and handling procedures, and project documents, were used to develop 

the accident scenarios and populate the HFACS model. 

5

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

FIGURE 3-1: NOAA N-PRIME FELL OFF THE TURNOVER CART

Figure 3-1: NOAA N-PRIME FELL OFF THE TURN OVER CART 

6

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Figure 3-2: NOAA N-PRIME DAMAGES

Figure 3-3: TURN OVER CART INTERFACES

7

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

The causes of the NOAA N-PRIME mishap are summarized below. More detailed findings are 

provided in Section 7. 

Proximate Cause: The NOAA N-PRIME satellite fell because the LMSSC operations team failed 

to follow procedures to properly configure the TOC, such that the 24 bolts that were needed to 

secure the TOC adapter plate to the TOC were not installed. 

The root causes are summarized below along the four levels of active or latent failures as 

ascribed by the HFACS framework. 

The TOC adapter plate was not secured to the TOC because the LMSSC operations team failed 

to execute their satellite handling procedures.

The Responsible Test Engineer (RTE) did not “assure” the turnover cart configuration through 

physical and visual verification as required by the procedures but rather through an examination 

of paperwork from a prior operation. Had he followed the procedures, the unbolted TOC adapter 

plate would have been discovered and the mishap averted. Errors were also made by other team

members, who were narrowly focused on their individual tasks and did not notice or consider the 

state of the hardware or the operation outside of those tasks. The Technician Supervisor even 

commented that there were empty bolt holes, the rest of the team and the RTE in particular 

dismissed the comment and did not pursue the issue further. Finally, the lead technician and the 

Product Assurance (PA) inspector committed violations in signing off the TOC verification 

procedure step without personally conducting or witnessing the operation. The MIB found such 

violations were routinely practiced. 

The LMSSC operations team’s lack of discipline in following procedures evolved from 

complacent attitudes toward routine spacecraft handling, poor communication and coordination 

among operations team, and poorly written or modified procedures. 

It is apparent to the MIB that complacency impaired the team directly performing the operation 

and those providing supervision or oversight to this team. The operation was consistently 

characterized as routine and low risk, even though it involved moving the spacecraft. Several 

other adverse mental states, including fatigue and external constraints that limited the availability 

of portions of the crew to a half day, also may have had roles in the mishap. Incomplete 

coordination concerning ground equipment use and status, and late notification of operation 

schedules exacerbated the lack of rigor in handling operations. Standard operating procedures 

contained ambiguous terminology (e.g., "assure") and can be significantly modified using 

redlines for unique (one time only) operations. These practices were the preconditions or latent 

failures that promoted the mishap occurrence. 

The preconditions within integration and test (I&T) operations described above existed because 

of unsafe supervision practices within the LMSSC project organization, including ad hoc 

planning of operations, inadequate oversight, failure to correct known problems, and 

supervisory violations. 

8

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

The RTE and I&T manager failed to provide adequate supervision and repeatedly violated 

procedures when directing and monitoring their operations crews. Waiving of safety presence, 

late notification of government inspectors, poor test documentation, and misuse of procedure 

redlines were routinely permitted. Further, the MIB believes that planning for the lift/turnover 

operation was hurried and resulted in a hastily formed operations team. Although all team

members were experienced and competent, this atypical mix of authority among the various roles 

created dynamics that were not conducive to open discussion and shared responsibility. The 

MIB concludes that the lack of enforcement and support by the supervisory chain concerning the 

roles and responsibilities of the operation team members and the hurried planning for this 

operation are factors in this mishap. 

The unsafe supervision practices within the TIROS program had their roots in the LMSSC 

organization: the inadequate resources and emphasis provided for safety and quality assurance 

functions; the unhealthy mix of a dynamic I&T climate with a well-established program and 

routine operations; and the lack of standard, effective process guidelines and safeguards for 

operations all negatively influenced the project team and activities. 

The MIB finds the LMSSC system safety program to be very ineffective. Few resources are 

allocated to system safety, few requirements for safety oversight exist and little programmatic 

supervision was provided for the safety representatives. The I&T environment within the TIROS 

program is engendered by routine operations for which schedules and specific activities are 

frequently optimized. Such an environment requires rigorous oversight and processes to prevent 

overconfidence and complacency. The MIB believes that LMSSC failed to provide the 

organizational safeguards to prevent this and other potential mishaps, especially in key areas that 

regulate operational tempo, operations planning, procedure development, use of redlines, and 

Ground Support Equipment (GSE) configurations. 

The in-plant government representation, Defense Contract Management Agency (DCMA), and 

the GSFC Quality Assurance (QA)/safety function failed to provide adequate oversight to 

identify and correct deficiencies in LMSSC operational processes, and thus failed to address or 

prevent the conditions that allowed the mishap to occur. 

The in-house Government Quality Assurance Representative (QAR) (acting as a DCMA agent) 

inappropriately waived a Mandatory Inspection Point during the Saturday morning operation. 

Although his presence may not have prevented the mishap, the MIB believes this waiver is 

indicative of a failed oversight process and barrier. The MIB finds that the government quality 

assurance and safety oversight at GSFC were also deficient, having become issue driven due to 

the maturity of the project. Once issues were brought to their attention, the QA/safety personnel 

worked their resolution but there was very little proactive oversight, audit, inspection, etc. of the 

LMSSC operations. The in-house Government QAR knew of some of the problems associated 

with procedure discipline and safety and program assurance oversight but did not communicate 

them to the NASA project. Given the prevalence of some of the contractor deficiencies 

identified in this investigation, however, it is the MIB’s assessment that the government in-plant 

representative, DCMA, and the GSFC QA/Safety function should have identified and demanded 

correction for these deficiencies. 

9

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

 The Government’s inability to identify and correct deficiencies in the TIROS operations and 

LMSSC oversight processes were due to inadequate resource management, an unhealthy 

organizational climate, and the lack of effective oversight processes. 

Relative to resource management, the GSFC project, in working to deal with a declining 

workload and resources, allowed and even encouraged trade-offs between the schedules, staffing 

and milestones for the two remaining satellites in the Polar Operational Environmental Satellite 

(POES)/(TIROS) project. These constant and rapid trade-offs exacerbated the already fast 

operational tempo of the LMSSC I&T team. Organizational climate was found to be an issue, 

primarily in the government on-site structure. There is no Project in-plant civil servant 

government presence. The Project in-plant government representatives (one in quality 

assurance, two in I&T) were past employees of LMSSC and were hired as outside contractors by 

the GSFC Project. The MIB believes that their past associations with the company might 

precipitate undue complacency due to familiarity. Although the POES Project and the contractor 

track and trend closure of contractor generated Non-Conformance Reports (NCRs) for 

timeliness, there is no process in place to analyze and trend NCRs for cause and to identify 

systemic problems. The MIB found no effective process in place to follow up on closure of 

Defense Contractor Management Agency (DCMA) generated Corrective Action Requests 

(CARs). Supplier Assurance Contract (SAC) generated audit deficiencies, and action items from

an external review (TIROS Anomaly Review). Likewise lacking is the government 

organizational oversight to monitor, verify, and audit the performance and effectiveness of the 

I&T processes and activities. 

The MIB found the DCMA CAR assessment and reporting process and other DCMA audit 

processes to be deficient in identifying troubling trends in the LMSSC facility. Review of CARs 

indicates repeated requirement violations and bypassing of Mandatory Inspection Points by the 

contractor. The DCMA Technical Assessment Group (TAG) facility audits, the DCMA annual 

safety audits, and the DCMA facility summary reports of CARs prior to the mishap, however, all 

indicated a healthy facility environment, with no noteworthy problems reported. 

MIB recommendations to correct the findings/deficiencies above are provided in section 8 - 

Recommendations. 

It is the MIB’s assessment that many of the findings uncovered in this mishap investigation are 

not specific to this mishap but are systemic in nature. A separate follow-up investigation should 

be conducted to further examine and characterize these systemic problems. 

10

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 4 

PROGRAM DESCRIPTION 

At the beginning of 1960, the United States had two credible space agencies – National 

Aeronautics and Space Administration (NASA) and Department of Defense (DOD). Each was 

pursuing very broad space programs, including manned space flight. In this same time frame, 

President Kennedy announced the development of a new operational weather satellite program. 

The Environmental Sciences Services Administration (ESSA), the predecessor agency of the 

National Oceanic and Atmospheric Administration (NOAA), by the authorization acts of 1962, 

was given management and operations responsibilities. The world’s first operational weather 

satellite, ESSA-1, was launched on February 3, 1966 and the second on February 28, 1966. 

NOAA’s operational weather satellite service, now an environmental satellite service, continues 

today. 

In 1973, a National Space Policy study, led by Office of Management and Budget, was 

undertaken to address the advantages of converging DOD and NOAA operational weather 

satellite programs. The 1973 review resulted in NOAA being directed to use DOD’s Defense 

Meteorological Satellite Program (DMSP) Block 5D spacecraft, which had recently been 

awarded to RCA by DOD. NASA retained their role as spacecraft system manager and funded 

the development and launch of the first satellite in this new series, called TIROS-N, which was 

launched in 1978. 

NOAA depends on NASA and DOD to procure and launch its spacecraft. The contracts for 

NOAA’s satellites are let and administered by NASA. NOAA’s polar orbiting satellites are 

launched from Vandenberg Air Force Base Facility. NOAA reimburses NASA and DOD for 

personnel and other costs incurred when helping NOAA meet its space mission objectives. 

NOAA is responsible for determining user requirements for satellite services, specifying the 

performance of the systems needed to satisfy these requirements, and obtaining the funds needed 

to build and launch the satellites and build and operate the ground segments of the systems. 

NOAA assigns a letter to the satellite before it is launched and a number once it has achieved 

orbit. For example, NOAA M, the newest in the series, was launched on June 24, 2002 and is 

now called NOAA-17. 

Technical Mission 

The NOAA polar satellite program is built and operated to support the needs of the National 

Weather Service’s global forecasting responsibility. Instruments have evolved over time. The 

most advanced of these satellite systems provide a suite of instruments for imaging and 

measuring the Earth’s atmosphere, its surface, and cloud cover, including earth radiation, 

atmospheric ozone, aerosol distribution, sea surface temperature, vertical temperature and water 

profiles in the troposphere and stratosphere; measurement of protons and electron flux at orbit 

altitude. The key instruments on the satellite are: the Advanced Very High Resolution 

Radiometer (AVHRR), which detects energy in the visible and infrared portions of the 

Electromagnetic spectrum; the High Resolution Infrared Radiation sounder (HIRS/3), which 

11

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

provides the atmosphere’s vertical temperature to about 40 km, ocean surface temperatures, total 

atmospheric ozone levels, precipitable water, cloud height and coverage, and surface radiance; 

and the Advanced Microwave Sounding Units-A/B, which measures scene radiance and there-by 

temperature and moisture. The satellites also support ozone monitoring and remote data 

collection and an international search and rescue program. Since 1982, this program is credited 

with saving more than 17,000 lives by detecting and locating emergency beacons from ships, 

aircraft, and people in distress. 

Project Management and Organization 

Contractor

The NOAA POES Program, based on TIROS-N design, has been flying since 1978. POES 

Satellite Launches are shown in Figure 4-1. Sixteen polar-orbiting, Earth observing satellites 

were built by RCA, and its successor organizations as it transitioned from RCA management to 

General Electric management, Martin Marietta management, and finally Lockheed Martin Space 

Systems Company (LMSSC) management using the DMSP spacecraft design. During these 

transitions in management, most of the spacecraft were built at the former RCA facility located 

at East Windsor, New Jersey. 

12

An orbit constellation 

consists of a pair of 

satellites, which 

ensures that every part 

of the Earth is 

regularly observed at 

least twice every 12 

hours. 

8

POES Spacecraft

On Orbit Life

As of: September 2003

3.3 Years

(Launch Failure)

(Power Sys Failure)

(Deactivated

02/13/98)

7.25 Years

5.0 Years

3.8 Years

4.9 Years

6.2 Years

7.3 Years

6.3 Years

4 Years

3 Years

(Deactivated

08/30/01)

1.4 Years

Planned

Launch 9/15/04

77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04

Baseline Launch Date

Expected 2yr Operational Life

Extended Operational Life (beyond 2 yrs)

Extended Backup Operational Life

Figure 4-1: POES Satellite Launches 

Both of Lockheed’s TIROS and DMSP Programs were relocated from East Windsor, New Jersey 

to Sunnyvale, California in 1998 as part of a planned consolidation of LMSSC Satellite 

integration and test (I&T) operations. Collocating TIROS and DMSP I&T operations with other 

satellite I&T operations to the LMSSC Sunnyvale facility was intended to create some efficiency 

in an era of dwindling Defense and commercial business. The last two spacecraft in the POES 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Program (NOAA N and N-PRIME) were largely built and integrated at the Sunnyvale, California 

facility. Many spacecraft subsystems were actually manufactured in the East Windsor facility 

and shipped with their associated support equipment to Sunnyvale, while other subsystems had 

their manufacturing and testing completed in Sunnyvale. Prior spacecraft in the TIROS series 

were manufactured, integrated, and tested in the East Windsor, New Jersey facility. DMSP 

finished production of the final five satellites in their series prior to closure of the East Windsor 

facility. Many of the personnel associated with the long history of the TIROS program did not 

relocate to the Sunnyvale operation. 

The last two of the Television Infrared Observational Satellites (TIROS) series spacecraft, 

NOAA N and N-PRIME are in final stages of testing at LMSSC under the Polar Operational 

Environmental Satellite (POES) program. LMSSC is also under contract to the Department of 

Defense (DOD) for the development, test and launch of the remaining Defense Meteorological 

Satellite Program (DMSP) series spacecraft, of which three are at various stages of testing at the 

Sunnyvale plant. The DMSP and NOAA spacecraft are located in adjacent integration and test 

facilities and the programs share a common ground support equipment (GSE) storage area. 

The TIROS Program organization at Lockheed is shown in Figure 4-2. TIROS is managed by 

the Program Manager who reports directly to the Vice President of Civil Space Programs. 

13

September 13, 2004 

Electrical Eng Electrical Eng

Mechanical Eng Mechanical Eng

Software S/C Core Team

Technicians

Project Eng Prod Assur GSE Business Ops Contracts Procurement

Software

Hardware

Technical

Program Manager

Systems Engineering Integration & Test

Figure 4-2: TIROS Program Organization at LMSSC
NOAA N-PRIME Mishap Investigation-Final Report 

The Program Manager is fully responsible for all activities on TIROS including interfacing with 

NASA, program planning, controlling program changes, providing resources, developing and 

allocating the budget, reviewing and reporting on program performance, and executing the terms 

and requirements of the cost plus award fee contract. The Program Manager is supported by a 

senior staff of technical and business managers who provide oversight for the daily activities of 

the program. The Systems Engineering Manager, supported by a staff of electrical, mechanical 

and software engineers, is responsible for spacecraft test and storage support, launch team

readiness, meeting support, verification engineering, and instrument interface management. The 

Integration and Test Manager, supported by a staff of electrical and mechanical engineers and 

technicians, is responsible for buildup of the satellite, environmental testing, and launch flow. 

The Ground Support Equipment Manager, supported by a staff of electrical and software 

engineers and technicians, is responsible for maintenance and repair of existing and new 

equipment required to support I&T and launch site operations. Other senior staff members 

reporting directly to the Program Manager are Project Engineering, Product Assurance, 

Procurement, Business Operations, and Contracts Administration. 

Government

The Polar Operational Environmental Satellites program falls under the NASA Headquarters 

Office of Earth Science (Code Y) which delegates the POES mission to GSFC. At GSFC, the 

POES Project is part of the Geostationary Operational Environmental Satellite (GOES)/POES 

Program in the Flight Programs and Projects Directorate. POES is a fully reimbursable project 

to NOAA. 

The scope of the POES Project at the time of the mishap was the completion and launch of 

NOAA N in September 2004; the completion, several year storage, and launch of NOAA NPrime in March 2008; the completion of instrument integration into the European MetOp-1 

satellite, storage and launch support of MetOp-1 in 2009; the completion of instrument 

integration into MetOp-2 and launch support in 2005 of MetOp-2; and the completion of 

instrument integration into MetOp-3 and storage until 2010 (launch in 2014 is beyond the POES 

completion date of 2010). The POES Project held the POES spacecraft contract with Lockheed 

Martin, instrument contracts with Ball Aerospace, Northrop Grumman and ITT. NASA KSC 

had the NOAA N and NOAA N-PRIME launch vehicle contracts with Boeing. NOAA provided 

foreign instruments and had an instrument contract with General Electric (GE) Panametrics. 

The estimated cost for the development and launch of 16 satellites (NOAA-A through NOAA NPrime) including instruments, launch vehicles, and three sets of instruments for MetOp through 

2010 is $2.2B. These costs do not include NOAA's costs to operate the satellites or its costs to 

provide the science processing. The total cost expended by NASA through fiscal year 2003 on 

the NOAA N-PRIME spacecraft, the one involved in the current mishap, is estimated at $223 

million dollars. 

14

POES PROJECT: The POES Project organization is shown in Figure 4-3. The POES Project is 

managed by the Project Manager who is fully responsible for all activities on the project 

including interfacing with NOAA, program planning, resources, developing and allocating the 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

budget, executing and overseeing the program plan, controlling program changes, and reviewing 

and reporting program performance. The Deputy Project Manager, a senior member of the 

management team, serves as the Project Manager when the Project Manager is unavailable. The 

Deputy Project Manager for Resources directs all the business aspects of the project including 

financial, budget, configuration management, scheduling, library, and Information Technology 

(IT). 

The Observatory Manager is responsible for spacecraft development and serves as the 

Contracting Officer’s Technical Representative for the Lockheed Martin contract. The 

Instrument Systems Manager is responsible for the development of the U.S. instruments and 

their delivery to Lockheed Martin and the MetOp program. The Flight Operations Manager is 

responsible for interfacing to the NOAA satellite operations and launch vehicle integration. The 

System Assurance Manager is from the GSFC Office of Systems Safety and Mission Assurance 

and is responsible for product assurance. The NOAA Liaison Office is staffed by NOAA 

engineers resident at GSFC providing continual interface to NOAA. Procurement support is 

provided by the GSFC Management Operations Directorate. Support staff includes financial 

managers, instrument managers, spacecraft engineers, and a part time Project Scientist. 

 

Code 480 (POES)

Project Manager

Secretary

DPMR DPM

Science Sys. Mgr.

Vacancies

NON-480 Personnel Co-located

NON-480 Personnel

PSM 

- Resident Manager

-SW Mgr/582

-ADACS/573

- Power/565

- Thermal/545

- Comm./567

- Mech & Electromech./544

- Propulsion/574

- C&DH/561

- Electrical & EGSE/565

Instrument

Systems Manager

Observatory Manager

NOAA-K,L,M,N,N’ Search & Rescue

Mission Manager

Search & Rescue

Mission Manager

Program Scientist

Code 200

Manager

Assistant

Contract Specialists

Code 200

Manager

Assistant

Contract Specialists

Code 300

SAM

NCR/CA Administrator

PAE

Code 300

SAM

NCR/CA Administrator

PAE

Systems Manager Systems Manager

- DCS, SARP, SARR,

SBUV

- 480

DTR/SSR/AVHRR

HIRS/SEM

- 480

AMSU-A

- 480

MHS, AMSU-B, MSU

FM

RA

RA

RA

Flight Ops

Manager

MetOp Mission

Manager

- 480

METOP I&T

February 24, 2004

Figure 4-3: POES Project Organization at NASA-GSFC 

15

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

NOAA: NOAA has over-all responsibility for the POES Program by providing the funding for 

the procurement of the satellites, as well as in operating, maintaining and processing the data 

from them once they are launched. Over-all program management and planning for the satellites 

in NOAA is led by the Office of Systems Development, which is part of the National 

Environmental Satellite Data and Information Service (NESDIS). To ensure effective 

communication of requirements, and to address issues relating to schedules, satellite technical 

performance and budgetary considerations, NOAA maintains a Liaison Office at Goddard Space 

Flight Center, which is co-located with the POES Project. The Polar Acquisition Manager, along 

with a small staff of engineers, monitor the progress of the satellite developments and address 

technical issues, such as potential performance waivers, and coordinates any necessary review of 

such issues within NOAA as information is required by the NASA POES Project. The Liaison 

Office personnel participate in most contractor reviews but do not have direct oversight of the 

NASA Contractors or over any other NASA personnel. They do participate as members of 

NASA’s Performance Evaluation Boards for POES instrument and satellite contracts for which 

award-fee evaluations are performed. Per the Memorandum of Understanding (MOU) between 

NASA and NOAA for the POES Program, a formal exchange of updated requirements and 

budgetary information is performed twice yearly. The NOAA Liaison Office is shown in the 

organization chart in Figure 4-4. 

Figure 4-4: NOAA POES Program Personnel at NASA/GSFC.

Director

Office of Systems Development

Chief,

Ground Systems

Division

Program Manager,

Geostationary

Satellites

Program Manager,

Polar 

Satellites

NOAA Liaison Office

Polar Satellite Acquisition 

Manager

(Silver Spring, Maryland)

(Suitland, Maryland) (Silver Spring, Maryland) (Suitland, Maryland)

(GSFC, Greenbelt, Maryland)

Satellite and Ground Systems

Program

Requirements, Planning, and 

Systems Integration Program

16

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 5 

DESCRIPTION OF EVENTS LEADING TO MISHAP 

On Saturday, September 6, 2003, during an operation that required repositioning (rotating) the 

TIROS NOAA N-PRIME spacecraft from a vertical to a horizontal position, the spacecraft 

slipped from the Turn-Over Cart (TOC) and fell to the floor. The spacecraft fell because the 

TIROS adapter plate to which it was mounted was not bolted to the TOC adapter plate with the 

required 24 bolts. The bolts were removed from the TOC by another project while the cart was 

in a common staging area, an activity which was not communicated to the NOAA project team. 

The operation scheduled for that day was to shim the Microwave Humidity Sounder (MHS) 

instrument by removing and replacing the instrument on the spacecraft. This operation required 

the spacecraft to be rotated and tilted to the horizontal position using the TOC. The operation 

involved preparing the spacecraft and TOC, installing the spacecraft on the TOC, and tilting the 

spacecraft to the horizontal position. After that, the MHS Instrument would have been removed 

and reinstalled. 

The plan to perform this activity on Saturday began on Wednesday, September 3. The document 

authorizing the MHS Shimming activity, Program Directive (PD) PD 03-58F-D2805, was 

initiated on Wednesday, and the Program Office at Goddard Space Flight Center (GSFC) was 

informed during a routine telecon on Friday, September 5. The paperwork providing detailed 

instructions for the operation was developed from the PD and included a single, hand-entered 

instruction in the Log of Operations (LOO) and four (4) hand-entered, redlined steps in the 

Instrument procedure, TI-MHS-3278200 “MHS Installation / Removal.” The four (4) redlined 

steps violated rules for the use of redlines, but were never-the-less approved by the LMSSC 

Responsible Test Engineer (RTE) and Quality Engineering (QE). 

The RTE, a lead technician (PQC), a technician, and the Technician Supervisor (acting as a 

technician equivalent) conducted the operation. The Product Assurance (PA) inspector, often 

referred to as “QA”, joined the operation in progress, thereby missing the key TOC preparation 

and configuration assurance activity. He stamped off the procedure without actually witnessing 

or visually inspecting the TOC configuration. The required Government Quality Assurance 

Representative (QAR or Customer Representative) was not present, nor was the required 

LMSSC safety representative. The TIROS (NOAA) acting Integration and Test (I&T) Manager 

was present as an observer during the actual spacecraft lift, but was not in the high-bay during 

preparations for the spacecraft lift, nor at the time of the mishap. 

17

The activity proceeded as planned through the installation of the spacecraft on the TOC, and was 

rotated about the vertical axis to align the instruments in the desired clock position. As the 

spacecraft reached 13 degrees of tilt from the vertical while being rotated to the horizontal 

position, it slipped off the TOC and fell approximately three (3) feet to the floor, tipping over in 

the process. Immediately following the mishap, the Technician Supervisor notified the I&T 

manager who then notified his management. The fallen spacecraft was “safed” to prevent further 

damage and to protect personnel and the co-located NOAA N spacecraft (the Nickel-Cadmium

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

batteries were fully charged, the propulsion system was pressurized, and the separation band was 

tensioned. (Figure 5-1: NOAA N-PRIME Safed) 

Figure 5-1: NOAA N-PRIME Safed 

This section describes the details of the actual incident, events leading up to the incident, and 

details describing the operating situation. Since the inter-relationship of many factors leading to 

this incident is complex, no single subsection or discussion can properly establish the specific 

cause. Missed Opportunities are identified throughout the discussion, and are summarized in 

Section 5.6, Summary of Missed Opportunities. Specific discussion of causal factor is included 

in Section 7. This section (Section 5) lays the groundwork for later more detailed analysis of 

each of the contributing factors:

Section 5.1 Planned Operation 

Section 5.2 Planning and Procedures (including the scheduling, crew, assembly and test 

documentation preparation activities.) 

Section 5.3 Crew Makeup 

Section 5.4 Turnover Cart (TOC) 

Section 5.5 Day of the Mishap. 

Section 5.6 Root Cause Analysis 

Section 5.7 Summary 

18

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

The primary mishap analysis was performed using the Human Factors Analysis and 

Classification system (HFACS) to identify, analyze, and classify the human errors. (See 

Sections 6 and 7) In addition, Root Cause Analysis diagrams were developed as a cross-check 

and are provided in Appendix E. 

5.1 Planned Operation 

The planned operation consisted of: 

1. preparing the TOC that had been utilized recently for a NOAA N spacecraft operation but 

subsequently stored in a support equipment storage area common to two programs; 

2. preparing the NOAA N-PRIME spacecraft for installation on the TOC; 

3. lifting the spacecraft onto the TOC and securing its conical payload (booster) adapter to

the TIROS adapter plate; 

4. and finally, rotating the TOC with the spacecraft attached from the vertical to the

horizontal position. 

The operation had been originally scheduled to begin the following Wednesday but was moved 

forward 3 working days to Saturday to take advantage of a schedule opportunity. The 

preparations required for the TOC and the spacecraft were likewise accelerated. This change in 

schedule played a role in the preparation activities. 

The operation on Saturday morning proceeded according to the following steps: 

1. Final preparations were made to the spacecraft, including removal of the work stands 

around the spacecraft and repositioning of the spacecraft within the high bay. 

2. The newly modified spacecraft vertical lifting sling was installed to the crane and then 

attached to the spacecraft. 

3. The spacecraft was detached from its handling cart and lifted high enough to permit 

moving the TOC under the suspended spacecraft. 

4. The spacecraft was lowered onto the TOC, 44 of 88 attachment bolts installed and 

torqued, and the lifting fixture detached from the spacecraft. 

5. The spacecraft was then to be rotated about its vertical axis to position the side-mounted 

instrument package, including MHS, such that it would be facing upward after the 

spacecraft was rotated to the horizontal position. 

6. Rotation of the spacecraft toward the horizontal position was begun. 

The planned final configuration is represented in the photograph of NOAA N shown in Figure 5-

2: NOAA N Spacecraft in Horizontal Position on TOC. 

A summary of the actual timing of the events is shown below in Figure 5-3: Timeline 

(Simplified)– Saturday Operation, and Table 5-1: Operation Event Timeline, and with greater 

detail in Section 5-5. 

19

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

20

September 13, 2004 

Figure 5-3: Timeline (Simplified)– Saturday Operation

Figure 5-2: NOAA N Spacecraft in Horizontal Position on TOC 
NOAA N-PRIME Mishap Investigation-Final Report 

6:00-6:45 am Between 6:07am and 6:13 am, the RTE, Technician Supervisor, I&T Manager, 

and two technicians arrived. The PA arrived at 6:42 am. NASA QAR was 

notified at home at about the 6:30 am time frame, but his first entry into the High 

Bay did not occur until 8:25 am. LMSSC safety was not notified of this operation 

at all.

6:07-6:45 am The TOC was moved from the common staging area (Ante Room) into the 

NOAA High Bay and prepared to support the NOAA N-PRIME activity. These 

activities were completed prior to 6:45am. The cart was prepared by the lead 

technician (PQC), and signed off in the procedure by the PQC, RTE, and PA, 

signifying that the TOC was “assured” of proper configuration per procedure (TIMH 3278200). (At this point, the PQC, RTE, and PA should have recognized 

that the TOC was not properly configured and then should have proceeded to 

properly configured the TOC by installing the 24 attachment bolts.) 

6:45-7:06 am The spacecraft was lifted and secured to the TOC. 

7:06-7:16 am The spacecraft was rotated 180 degrees about the vertical axis while still in the 

vertical configuration. 

7:16-7:28 am The spacecraft was rotated from the vertical to the horizontal position. As the 

spacecraft reached about 13 degrees of rotation, it slipped off the TOC and fell to 

the floor. 

Table 5-1: Operation Event Timeline 

The newly modified spacecraft vertical lifting sling, to be used for only the second time, would 

be installed to the crane, then attached to the spacecraft, and the spacecraft detached from its 

handling cart and lifted high enough to permit moving the TOC on its air bearing jacks under the 

suspended spacecraft. The lifting sling modification reduced the required number of technicians 

to attach and remove the sling from the spacecraft from seven (7) to four (4), reducing the 

number of technicians working around the spacecraft, but has no other significance to this 

operation because their sole task was to hold tag lines to keep the lifting sling legs away from the 

spacecraft during the sling attach/detach operation. 

Once positioned, the spacecraft would be lowered onto the TOC, 44 of 88 attachment bolts 

installed and torqued, and the lifting fixture detached from the spacecraft. (Note: The installation 

of every other bolt attaching the TIROS flight adapter to the TIROS TOC Adapter was common 

practice, and in fact is common throughout the aerospace industry because the handling stresses 

and loads are minor compared to the flight loads the interfaces are designed to accommodate. 

(Its significance is relative to a discussion of ‘empty bolt holes’, which is included later in the 

investigation analysis.) 

The spacecraft would then be rotated about its vertical axis to position the side-mounted 

instrument package, including MHS, such that it would be facing upward after the spacecraft was 

rotated to the horizontal position. This rotation was standard practice but is not included in the 

procedures or planning paperwork. 

21

The spacecraft fell to the floor as it reached 13 degrees of tilt while being rotated from vertical to 

horizontal. It is clear that the spacecraft fell because the TOC adapter ring was not secured to the

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

TOC with the required 24 bolts. (Proximate Cause) 

5.2 Planning and Procedures 

The operation was planned during the period from Wednesday, 3 September 2003, through 

Friday, 5 September 2003. (Figure 5-4: Timeline for Wednesday -Friday Preparations) 

On Wednesday, September 3, the NOAA N-PRIME MHS Shimming activity schedule was 

moved ahead from the initial plan to begin on September 10 to the updated plan to begin on 

September 6, the date of the mishap. Some of the preparation work on the flight spacecraft, such 

as tensioning the V-band, actually began on Friday, September 5. This short-notice advance in 

the schedule date had several consequences. The TOC required repair to remove a red tag, 

leading to a repair that was acceptable but required a subtle change in how the spacecraft was 

mated to the TOC. In addition, technicians were requested to perform operations on the flight 

spacecraft without released paperwork, leading to dissention among some of the NOAA I&T 

Team, and resulting in at least one key team member not working on Saturday. The accelerated 

pace may have led to less communication between the two programs sharing the TOC. Each of 

these factors is discussed in more detail in later sections. 

The following paragraphs describe the authorizing paperwork, the change in schedule, and the 

redlined procedures for Saturday’s operation. 

Figure 5-4: Timeline for Wednesday – Friday Preparations 

22

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

5.2.1 Change in Schedule

A problem with the originally scheduled activity for Saturday, September 6 on the NOAA N 

spacecraft resulted in moving the MHS Shimming activity, and hence the S/C hoisting and 

turnover activity, forward from Wednesday, September 10, to Saturday, September 6. The 

original NOAA N-PRIME activity on Saturday involved the Electrical Team, not the Mechanical 

Team, resulting in a late call for technician support for the activity. Discussions with LM and 

GSFC Project/Program personnel also indicated there was a desire to keep the spacecraft team

busy and productive during the NOAA-N downtime. 

The team for Saturday was hastily assembled, with the first call for mechanical technician 

support for the NOAA N-PRIME activity made on Thursday, September 4, indicating a very late 

decision to move the activity date forward. This accelerated schedule created an environment 

where the technicians were requested to perform activities on the flight spacecraft without 

approved paperwork, causing dissention among certain team members. This led to at least one 

key crewmember not participating in the Saturday operation and resulted in a briefing to NOAA 

I&T personnel on Friday on the need to work with approved paper and procedures. The project 

schedules published on September 3rd and 4th were never updated to indicate the new schedule. 

(Figures 5-5: TIROS N-PRIME Daily Schedule 9/03/2003 and 5-6: TIROS N-PRIME Daily 

Schedule 9/04/2003,)

The Program/Project had an opportunity to save schedule and costs by performing the pending 

Electromagnetic Compatibility (EMC) testing on the two spacecraft, NOAA N and N-PRIME, 

back-to-back. The test setup is somewhat complex, and significant savings could be realized if 

the back-to-back testing could be accomplished. This contributed to the decision to work 

Saturday, even though the originally planned activity could not be performed. 

The need to repair the red-tagged TOC was also accelerated, and led to a subtle change in the 

way the S/C and TOC were positioned for the actual mate. The TOC was maneuvered under the 

hanging S/C, instead of the procedure-directed crane movements to maneuver the spacecraft into 

position as it was being lowered. This change is not noted in the procedure, but it is not deemed 

significant. 

5.2.2 Program Directive and LOO 

23

Tasks on the TIROS NOAA program are authorized by the Program Office at GSFC via a 

Program Directive (PD). The document authorizing LMSSC to perform the MHS Shimming 

activity, Program Directive (PD) PD 03-58F-D2805, was initiated on Wednesday, September 3, 

with final approval occurring on Friday, September 5. LMSSC generates a Daily Schedule, 

which details activities on each of the TIROS NOAA spacecraft. These schedules are updated 

periodically, but not necessarily daily. The activity to perform MHS Shimming first appears on 

the NOAA N-PRIME Daily Schedule dated Wednesday, September 3, 2003, with the date of the 

activity planned for Wednesday, September 10 through Tuesday, September 23, identifying the 

task as “MHS shim/penalty test”. (Figure 5-5: TIROS N-PRIME Daily Schedule 9/3/03) This is 

consistent with the initiation of the PD on Wednesday, and provided a reasonable time to prepare 

for the activity the following Wednesday. Thursday’s Daily Schedule, 9/4/2003, shows the same

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

24

September 13, 2004 

planning date of September 10 for the activity. (Figure 5-6: TIROS N-PRIME Daily Schedule 

9/4/03.) The first two tasks of that sequence, “Close Panel 1” and “Hoist S/C to Turnover 

Cart/Tilt to Horizontal”, were originally scheduled for Wednesday and Thursday, September 10 

and 11. 

The LM paperwork authorization system flows from the PD to a Log of Operations (LOO). 

(Figure 5-7: Paperwork Relationships (Ref: ITOP-509 SV, LEO-MET Integration and Test 

Operations Rev A, dated 7/26/00). 

The LOO details the sequence of activities on a spacecraft flow, and describes the sequence that 

various operations are intended to occur. The new activity to perform the MHS Shimming was 

hand entered into the LOO (LOO 006) on September 5, but does not indicate the planned date for 

the activity. This is consistent with the preparation and approval dates of September 3 through 5. 

From the LOO, the next level of documentation is either an Operations Order (Op Order) or a 

Technical Instruction (TI). An Op Order is normally used to specify a detailed procedure 

(sequence of steps) for an operation that will not be repeatedly performed. A TI is used for 

detailed sequences of steps that will be performed repeatedly. The NOAA N-PRIME use of the 

LOO and TI’s is in accord with LM requirements, with the exception that critical detailed steps 

were redlined into the TI without the prescribed review and approval. 

5.2.3 Redline Use to Create New Procedure

Proprietary Information 
NOAA N-PRIME Mishap Investigation-Final Report 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

5.2.4 Procedure Review and Approval 

Proprietary Information 

5.2.5 Inconsistent Approach to Hazard Identification 

Proprietary Information 

5.3 Crew Makeup 

The crew makeup for the Saturday operation was comprised of experienced personnel, 

but as a direct result of the late decision to move the NOAA N-PRIME MHS shimming 

activity ahead in the schedule, the crew makeup was not optimum. Technicians had been 

requested to perform preparation work on the flight spacecraft without approved 

paperwork, such as tensioning the V-Band securing the spacecraft to its flight payload 

adapter and closing spacecraft panels #1 and #4. (Note: it is normal procedure to leave 

the V-Band at a low tension to minimize the long-term stress in the band.) Concerns 

about such requests were addressed on Friday by supervision calling an all-hands meeting 

to reiterate the company policy regarding work on flight spacecraft. This discussion 

occurred very late in the preparations for the rescheduled Saturday work and contributed 

to the difficulty in assembling a crew for the lift and turnover activity. Specifically, this 

concern led to one very experienced individual declining to participate in the Saturday

activity. The Technician Supervisor filled in as a Technician Equivalent, permitted by

LM Command Media when a full complement of technicians cannot be assembled. Also, 

some team members were notified as late as quitting time the day before the operation 

that they were to perform the next day. This created an atypical mix of authority among 

the various roles created dynamics that were not conducive to open discussion and shared 

responsibility. (Finding L-5, L-6, and L-11) 

5.3.1 Crew Experience 

In spite of the dynamics leading up to the final crew selection, the experience of each 

member was extensive. (Table 5-2: Crew Experience) The large number of lifts and 

26

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

TOC operations that each crew member had experienced was probably the basis for 

almost all crew members commenting to the effect that, “This was just another routine 

operation.” This feeling is felt to have led to complacency being prevalent among the 

crew, and led to a lack of attentiveness and attention to detail. 

Name Previous

Lifts 

Previous 

TOC Ops 

Previous TOC 

Configurations 

I&T Manager 30 5 0 

RTE >25 8 8

Technician Supervisor / 

Technician Equivalent 

75 4 0

Technician – Lead (PQC) 40 10 10 

Technician (Shared W/DMSP) 75 25 25 

PA (QA) 45 >20 10 

----- ------ -----

TOTAL OPERATIONS 290 72 53 

Ref: LMSSC Response to MIB 10-08-2003 AI-23 (with additional “TOC 

Configurations” info added by the MIB) 

Table 5-2: Crew Experience

Of special note is the fact that the RTE and the Technician Supervisor were among the 

least experienced in the use of the Turnover Cart, but still had experienced a significant 

number of operations. The RTE was involved in eight (8) previous TOC configuration 

operations, but had always relied on the PQC for those operations to perform the 

configurations. That person was not working that Saturday. This experience mix may 

explain the miscommunication between the RTE and the Technician Supervisor in a 

conversation regarding the existence of empty holes with threads. The reference (by the 

Technician Supervisor) regarding empty bolt holes may have been toward the missing 24 

bolts, but seems to have been interpreted (by the RTE) as referring to the 44 of 88 bolts in 

the payload adapter intentionally not installed. Even as the Technician Supervisor 

commented that there were empty bolt holes, the rest of the team and the RTE in 

particular dismissed the comment and did not pursue the issue further. (Finding L-2) In 

this instance, the Technician Supervisor lacked the knowledge to recognize the problem. 

The rest of the team, due to complacency and channelized attention, failed to pursue the 

apparent warning. (MISSED OPPORTUNITY #1) 

An additional factor was that the RTE was planning on only 1⁄2 day support, as he had 

family related constraints for the afternoon. He had agreed to conduct the lifting and 

rotation activity, with another RTE taking over for the MHS shimming activity for the 

afternoon. Likewise the DMSP technician was scheduled to support another program on 

that day. This may have contributed to the fast pace of the operation, and may have 

influenced several decisions. (Finding L-4) 

The I&T Manager decision to be present to observe the lifting activity is particularly 

27

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

puzzling since he had significant lifting experience. The reason for his attendance was to 

observe the use of the newly configured lifting sling (requiring fewer technicians to 

attach and detach). The lack of procedure modifications to account for the change in 

crew size is indicative of the lack of importance placed on procedures. This is also 

indicative of both the lack of supervision discipline and the tolerance for inadequate 

procedure discipline and QA witness. 

The experience of the Government QA Representative (QAR), substituting for the 

DCMA person specified in the procedure, was similar to the LMSSC technicians. He had 

witnessed the turnover operation at LMSSC 20-25 times, but had never witnessed the 

configuration of the Turnover Cart (installation of the adapter and its 24 bolts). 

The TOC configuration activity does not require witness since it is GSE work, so it is not 

clear whether the TOC verification that was performed via paperwork versus actual 

inspection would have triggered a concern or not. However, the GSE configuration work 

does require LMSSC PA (QA) to buy off that the cart is properly configured. For this 

reason, it is not specifically felt that his non-presence early in the preparation was 

significant.

5.3.2 Crew Responsibilities 

The following crew responsibilities are extracted from several sources, including the I&T 

Operations Practice and the Technical Instruction (procedure) being exercised. Some

specific responsibilities of particular interest are:

1) Product Assurance Inspector: “Inspection shall verify that the test procedure 

has evidence of work performed and/or completed data entry. Verifications may 

be done either during or after the operation, as long as the operations and/or data 

readings can be verified.” (Ref: TI-MH 3278200 para 3.9.2) 

2) Quality Engineer: “All test equipment, test setups, and collected data shall be 

subject to monitoring, review and validation by an authorized Quality Engineer 

(QE) per 326412. The signature or stamp of the cognizant QE shall indicate 

validation.” (Ref TI-MH 3278200, Para 3.9.) 

3) Torque witness: “Shall be the cognizant mechanical integration technician or 

mechanical engineer.” 

The following extractions expand on the above summaries: 

The Responsible Test Engineer (RTE) is directly responsible for assuring adherence to 

the applicable safety requirements and safe conduct of the procedure. The RTE is also 

responsible for conducting a pre-operational safety briefing for all personnel involved in 

the test activity. The pre-operational briefing will include, but not be limited to, the 

following: 

28

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

• Safety items in operation/test, including safety-critical and hazardous operations 

as well as hazardous or toxic chemicals that will be used. 

• Sequence of events and tasks. 

• Job assignments for the tasks. 

• Required safety equipment, if any, will be explained. 

(Ref TI-MH 3278200, Para 3.3.1) 

The RTE shall: 

• Direct the operations of this procedure. 

• Sign approval of performance data, as indicated in the procedure (e.g., RTE.) 

• Generate TPCN’s as required to incorporate test redlines. 

• Annotate any out-of-sequence operations with an explanatory note on the 

applicable test procedure paragraphs. 

Technicians: 

• Technicians performing this procedure shall be assigned based on individual 

qualifications and the approval of the RTE. 

• PQC stamps shall be applied to the test procedure or data sheets only as follows: 

• At the designated points in this procedure as each task is completed (e.g., 

PQC____). 

• On all supplemental data sheets attached to this procedure. 

• Adjacent to the last operation completed at the end of each work shift. 

(Ref: TI-MH 3278200 para 3.9.2) 

Torque witness: 

• Shall be the cognizant mechanical integration technician or mechanical engineer. 

Product Assurance Inspector: 

• Check the calibration stickers on the test equipment to verify that calibration 

will remain current through the life of the test. 

• Witness those operations steps that have a PA_____ entry, and shall verify 

acceptance of measurement and data recording of all performance data. 

• Mandatory inspection verification paragraphs shall be indicated in the test

procedure (e.g., PA______). Inspection shall verify that the test procedure has 

evidence of work performed and/or completed data entry. Verifications may 

be done either during or after the operation, as long as the operations and/or 

data readings can be verified. (Ref: TI-MH 3278200 para 3.9.2) 

Quality Engineer (QE): 

• All test equipment, test setups, and collected data shall be subject to 

monitoring, review and validation by an authorized Quality Engineer (QE) per 

326412. The signature or stamp of the cognizant QE shall indicate validation. 

(Ref TI-MH 3278200, Para 3.9.) 

29

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

5.3.3 Staffing and Notification of Safety, PA/QA), and DCMA/QAR (Gov’t Rep) 

Several required members of the crew were not involved in the basic planning leading up 

to the Saturday morning activity, particularly Safety, Product Assurance (Quality 

Assurance), and the Government Representative (DMCA or QAR), as required by the 

procedure. Attempts were made to contact Quality Assurance and the QAR on Saturday 

morning as the operation was in the preparation stages. The PA (QA) inspector was the 

only one who actually participated. 

The MIB observed an inadequate emphasis on safety and Quality Assurance within the 

TIROS program. Few resources are allocated to this function and few requirements for 

safety oversight exist. Further, little LM and GSFC programmatic supervision was 

provided for the safety representatives. A shrinking of the Quality Assurance activity 

was also observed as the program work diminished, with three inspectors now being 

shared between the TIROS and DMSP projects. (Findings L-10, G-3, and G-8) 

5.3.3.1 Safety 

The procedure for installation of the spacecraft on the Turnover Cart (TOC) specified “1 

Safety Engineer (or designee)” as part of the Required Personnel list. (Figure 5-16: TIMH-3278200 Staffing Requirement). Safety was not notified of the operation, and a note 

was entered in the margin of the procedure to that effect (Figure 5-17: Procedure Record 

Documented That Safety Not Notified). In fact, the primary Safety Engineer was on 

travel supporting a launch planned by the DMSP program. An alternate Safety Engineer 

was specified during the primary Engineer’s absence, but no attempt was made to contact 

either the primary or the alternate. It was clear from this, and from previous operations 

indicating that Safety was notified but did not attend, that Safety was not considered an 

essential element of spacecraft lifting operations. This lack of involvement was not 

questioned by supervision. (Findings L-3, L-9, L-10, and L-13) 

30

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Figure 5-16: TI-MH-3278200 Staffing Requirement 

In practice, the role of Systems Safety appears to be to review procedures to ensure the 

appropriate safety precautions are included, and to produce and coordinate the launch 

area safety data package, commonly known as the Missile System Prelaunch Safety 

Package (MSPSP). It was not common practice for Systems Safety to attend and oversee 

critical operations unless there is something unique about it. (Finding L-10 and G-8) 

In addition, the required number of technicians specified (7) was reduced but not noted. 

The reduced staff was considered acceptable because the lifting sling had been modified 

to eliminate the need for four (4) of the seven (7) technicians. The sole responsibility of 

the eliminated technicians was stated to be holding the lifting sling legs away from the 

spacecraft using four (4) tag lines, one at each corner. While the eliminated technicians 

specified roles are not significant in the lifting operation with the newly modified sling, 

the absence of four (4) additional pairs of eyes and ears cannot be completely overlooked. 

Figure 5-17: Procedure Record Documented That Safety Not Notified

Proprietary Information 

5.3.3.2 Product Assurance (PA or QA) 

31

The Quality Assurance Inspector, variously referred to as Product Assurance (PA) or 

Quality Assurance (QA), was not present during the initial preparations and was called

shortly after the preparations began on Saturday morning. He was on site and present in 

the building when called, but was on his way to support his regularly scheduled activity 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

on another program in another building. He consented to support the NOAA N-PRIME 

activity, and changed his plans accordingly. He entered the high bay 6:42am. 

By the time he arrived, the preparations of the TOC and the spacecraft were basically

complete, and the operation was ready to proceed. He had to make the decision of 

whether to delay the operation to verify the procedure steps he was supposed to witness, 

which where already completed by the technicians, or to stamp-off the procedures based 

on his trust and knowledge of the technician crew and the RTE. He stamped off the 

procedure without actually inspecting or witnessing the TOC preparation activities or 

configuration. This practice apparently was tolerated by management and supervision, as 

DCMA had written numerous Corrective Action Reports (CARs) addressing stamping 

violations. (Refer to Findings G-2 and G-9, and Section 9.2) 

5.3.3.3 DCMA or QAR (Gov’t Rep) 

The GSFC Quality Assurance Representative (QAR) was notified at home on Saturday 

morning after preparations were underway, but prior to the actual lift. An informal 

(email) agreement between the GSFC Systems Assurance Manager (SAM) and the 

DCMA had relieved the DCMA Representative from having to support weekend 

activities, and therefore the RTE called the QAR. The QAR gave his verbal OK to 

proceed with the lifting operation and indicated that he would be in as soon as he could. 

The assumptions behind this OK to proceed are unclear, but it is concluded by the MIB 

that the QAR understood the critical activity requiring his presence was the actual MHS 

instrument shimming, rather than the movement and lifting/rotation of the spacecraft. It 

is questionable whether he fully understood the function, scope, and practices of the 

DCMA representative in agreeing to substitute for the DCMA representative on 

weekends. The MIB concluded that he was unaware of the Mandatory Inspection Point 

(MIP) that required Government Representative presence for all movements of the 

spacecraft. He arrived on the scene at 8:25am, after the spacecraft had already fallen. 

(Finding G-2) 

Since the operation was consistently characterized as routine and low risk, even though it 

involved moving the spacecraft, there is strong evidence of complacency involved in the 

permission to proceed. (Findings L-4 and L-9) 

An additional factor that probably had an influence on the QAR’s oversight and decision 

making was the fact that he was the retired LM Product Assurance Manager. In that role, 

he was very well acquainted with the personnel and processes, and may not have 

exercised a fully objective oversight of the LM activities. (Finding G-6) 

DCMA had written numerous Corrective Action Reports (CARs), but had failed to 

follow up for corrective action implementation. Included in the list of CARs were 

several instances of stamping, or buy-off, violations. This contributed to the lax process 

environment. A more complete discussion and analysis of the DCMA CARs is included 

in Section 9. (Finding G-4) 

32

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

33

September 13, 2004 

5.4 Turnover Cart 

The following sections recount the sequence of events leading to the cart configuration 

on the day of the mishap.

5.4.1 Cart Use by NOAA N 

The Turnover Cart (TOC) was in use by the NOAA N spacecraft from August 6 through

August 26, after which the TOC was relocated from the NOAA High Bay to the Ante 

Room or common area between the TIROS NOAA and the DMSP High Bays. The same

basic operation had been performed, except that the MHS Shimming was done during the 

MHS installation, rather than requiring the MHS removal, then reinstallation with 

shimming. The timeline and Hi-Bay area layout is shown in Figures 5-18: TIROS and 

DMSP Timeline, and 5-19: LMSSC Hi-Bay Area. It was after this use by NOAA N that 

the TOC reconfiguration by DMSP was initiated and interrupted, leading to the 24 bolts

being missing.

5.4.2 Cart Reconfiguration by DMSP and Red Tag Procedures 

The DMSP and TIROS programs have routinely shared Mechanical Ground Support

Equipment (MGSE) throughout their histories because of the similarity of the spacecraft. 

However, each spacecraft and unique test configurations required the use of unique 

adapters between the spacecraft flight payload adapter and the TOC itself. The two 

programs are currently housed in the same building and share a common Ante Room

(storage/staging area) between their respective high bays. It was common practice to

utilize each other’s MGSE when beneficial to the program. Communication between the 

I&T Managers of each program was informal and substandard. Communication issues

surrounding the use of shared ground equipment had apparently existed between the 

DMSP and TIROS programs for some time, and had been the cause of conflict in the

past. No common schedule existed with which to avoid equipment conflicts and

overlaps. (Finding L-5)
NOAA N-PRIME Mishap Investigation-Final R

eport 

Figure 5-18: TIROS and DMSP Timeline 

Proprietary Information 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final R

eport 

September 13, 2004 

Figure 5-19: LMSSC Hi-Bay Area 
NOAA N-PRIME Mishap Investigation-Final Report 

After completion of NOAA N activities using the TOC, the cart was returned to the common 

Ante Room. The DMSP Program decided to use this TOC for its activities because their own 

TOC was red-tagged with a problem. Activities to reconfigure the TOC for their own 

configuration began on August 27. The reconfiguration was interrupted part way through the 

process of the TIROS adapter ring removal, in order to install the DMSP adapter, when it was 

discovered the TIROS TOC was red tagged and it was determined that it would be easier for 

DMSP to clear the red tag on its own TOC. This change in plan left the TIROS adapter ring 

sitting on the TIROS TOC with its 24 attachment bolts removed. (These are the 24 bolts in 

question.) No red tag nor any other indication was added to the TIROS TOC to indicate the 

incomplete configuration; the TIROS TOC remained red-tagged due to a floor jack problem. 

(Figure 5-20: TOC showing Adapter Plate with 24 Missing Bolts; Figure 5-21: DMSP TOC 

Configuration with TIROS Adapter Plate; and Figure 5-22: TOC Diagram side view) None of 

this was communicated to the TIROS folks, nor was it required by the LM Command Media 

because the over-riding philosophy was that each user was required to verify or ensure the GSE 

configuration was appropriate for its own specific use each time it was used. 

Finally, no real communication or documentation process existed for handling the red-tagging, 

repair and maintenance of ground support equipment. In this particular case, the repair of the 

cart did not return it to its full capacity—a restriction that was not communicated beyond the 

DMSP I&T manager (requester of repair) and the Technician Supervisor (repairer). The red tag 

process is so informal that the Red Tags that were utilized informally were discarded after their 

particular concerns were resolved, leaving no record of the Red Tag process. (Finding L-5) 

36

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Figure 5-20: TOC Showing Adapter Plate with 24 Missing Bolts 

37

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Figure 5-21: DMSP TOC Configuration (with TIROS Adapter Plate in Position) 

Figure 5-22: Adapter, Adapter Plate, and Southworth TOC Interface Place 

Proprietary Information 

Several versions addressing where the missing bolts went after they were removed from the TOC 

were heard during interviews. Two examples heard were that: 1) the bolts went to the DMSP 

cart; and 2) that the bolts went into the common area storage cabinet. Since the LM system

considers the GSE as uncontrolled until its configuration is verified by the using project for each 

use, the actual version of the bolt story is not important in establishing the cause of this mishap. 

5.4.3 Cart Rework by NOAA N-PRIME 

TIROS NOAA N-PRIME personnel worked off the red tag on the TIROS TOC on Thursday and 

Friday, September 4 and 5. The repair consisted of replacing the damaged jack. Because a jack 

of the same load rating was not available, a jack with a lower rating was utilized. Engineering 

analysis concluded the replacement jack was of sufficient capacity for static use of the cart, but 

analysis showed it was insufficient to permit movement of the TOC with the spacecraft attached. 

This limitation resulted in a slight change to the spacecraft lift and mating procedure. Instead of 

using the crane to maneuver the spacecraft over the cart and lower into position as specified, the 

spacecraft was raised and the TOC positioned under the suspended spacecraft. No redlines or

other notations were made in the procedure to indicate this change. (Ref TI-MH 3278200 Section 

5.6.4) This change is not deemed significant, but is an indicator of the lack of procedure 

discipline. 

5.5 Day of the Mishap 

The Saturday operation proceeded as planned, with the exception of the availability and late 

notification of PA and the NASA Quality Assurance Inspector (QAR--Customer 

Representative), and the lack of notification of Systems Safety. A detailed view of the Saturday 

timeline is presented in Figure 5-23: Saturday Detailed Timeline. The vertical axis is separated 

by specific procedure in order to group items from specific authorizing documents: the Program

Directive (PD), the Log of Operations (LOO), and the two primary Technical Instruction 

procedures (TI-MHS3278200 and TO-MH-3278200). Several parallel preparation activities 

were completed on Friday, making the spacecraft ready to begin the operation early on Saturday 

morning. The apparent jumping up and down indicates the difficulty in following the correct 

sequence of steps. 

The precise timeline for Saturday, the day of the incident, is somewhat speculative, but was 

reconstructed using both witness statements and by the supporting high bay door entry log. The 

38

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

pace of the activity seems quite fast for such critical and delicate lifting operations, but 

interviewees indicated that the operation went very smoothly and was not rushed. 

5.5.1 Preparation 6:00 – 6:45am: 

The work shift began at 6:00am on Saturday, September 6. The starting time was normal for the 

technicians, and is consistent with normal weekday start times. As was normal work practice, 

the technicians preferred to start early to leave time for personal activities later in the day. 

Personnel began entering the High Bay at 6:07am, as confirmed by the entry badge reader, and 

continued to enter until approximately 6:13am (Figure 5-24: Door Entry Log). During that time, 

the Responsible Test Engineer (RTE), Technician Supervisor (Technician Equivalent), 

Integration and Test Manager (I&T Mgr), and one Technician logged in to the High Bay. The 

arrival times of other technicians are not known because ‘tailgating,’ or following another person 

through the opened door, is an accepted practice. 

The Lockheed Martin (LM) Quality Assurance Inspector (PA) was notified by phone at 

approximately 6:30am, thereby confirming that his first entry into the High Bay was at 6:42am,

as recorded by the badge reader. He was on-site to cover another program, but agreed to support 

the NOAA N-PRIME activity. It is also known that NASA QAR was notified at home at about 

the same 6:30am time frame, and that his first entry into the High Bay did not occur until 

8:25am. LM safety was not notified of this operation at all, including on Saturday as required by 

the procedure (Note: Further discussion of this is noted in Section 5.2, Planning and Procedures, 

and 5.3, Crew Makeup. 

5.5.2 TOC Preparation and Assurance 6:07 – 6:45am: 

The activities required to “Assure Turnover Cart Configuration” are specified in the procedure 

TI-MH 3278200, step 5.6.1. The first step in that section refers to the “Spacecraft Turnover 

Assembly Section of this procedure”, which is Section 5.5. (Figure 5-25: Excerpt from Section 

5.6 Referring to Section 5.5) During this activity, the TOC was moved from the common 

staging area (Ante Room) into the NOAA High Bay and prepared to support the NOAA NPRIME activity. These activities were completed prior to 6:45am. The cart was prepared by the 

lead technician (PQC), and signed off in the procedure by the PQC, RTE, and PA. The 

following discussion reflects the steps in the procedure required to accomplish this preparation. 

Note that Step 5.6.1 refers to another section of the procedure, but does not provide the 

paragraph number of the section. 

The preparation of the TOC was intended to be performed in accordance with TI-MH- 3278200, 

Section 5.5. The note at the beginning of this section states: 

“NOTE: During this section, the following disciplines will be responsible for the 

verifications identified during buy-off in Table 5.5-1: 

39

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

40

September 13, 2004 

PQC – All steps followed and performed 

RTE – All steps performed and any torque witnessing 

PA – All steps successfully completed” 

The rest of this section is Export Controlled
NOAA N-PRIME Mishap Investigation-Final R

eport 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final R

eport 

Red lettering represents 

missed 

opportunities to install 24 bolts. 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final R

eport 

Figure 5-23: Saturday Detailed Timeline 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final R

eport 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

September 13, 2004 

Figure 5-24: Door Entry Log 
NOAA N-PRIME Mishap Investigation-Final Report 

Figure 5-25: Excerpt from Section 5.6 Referring to Section 5.5) 

Export Controlled 

Figure 5-26: Excerpt from Section 5.5 Assuring TOC Configuration 

Export Controlled 

5.5.3 Hoist and Secure S/C 6:45 – 7:06 am: 

The most probable start time of the actual hoisting activity was 6:45am. This time is established 

because PA arrived in time to stamp off the procedure asserting the acceptable configuration of 

the TOC (6:42am), PA was there for the beginning of the lift, and the I&T Manager reentered the 

High Bay at 6:45am, in time for the lift to begin (per his request to be present for the lift). This 

established the team as constituted for the lift activity. A time of 7:06 is ascribed as the end of

the hoisting activity based on backing up from the next two operations: Rotate Vertically, and 

Rotate to Horizontal. 

It would be during this time that the RTE would have conducted the required Safety Briefing. 

The Error Prevention Program adopted by LMSSC utilized the AESOP acronym as a memory 

tool addressing the major elements that help to focus the participants. Each crew member carries 

a card reminding them of the elements of AESOP, which are: 

A Assignment: Clear? Complete? Risks?

E Equipment: What? Availability? Working?

S Situation: Overall Assessment (Go, Ragged Edge, Stop) 

O Obstacles: Potential Problems? Look ahead! 

P Personnel: Who? Experience? Risks?

 Review I’M SAFE: 

Illness * Fatigue * Eating 

 

The AESOP process is intended to be implemented by stepping through each letter of the 

acronym to ensure all aspects of the ensuing activity are understood by all members of the 

activity. 

There are conflicting reports on the fidelity and extent of the briefing. Some reports indicate a 

briefing was held; others do not remember a briefing. This briefing is intended to cover 

individual roles and responsibilities, safety equipment, sequence of activities, etc. (See Section 

5.3.2) The conclusion is that if a briefing was held, it probably did not include the entire crew, 

and was most probably just a short discussion among a select few individuals. Also, it is most 

46 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

probable that the AESOP process was not followed since so many members of the team do not 

remember participating in a briefing. The lack of a briefing sufficient enough for most of the 

team members to remember it is deemed inadequate supervision. (Finding L-7) 

Personal interviews and answers to MIB questions during presentations to the Board indicated a 

strong reluctance to speak up and hold up an operation unless an individual was absolutely sure 

something was wrong. Merely having an uncomfortable feeling about the pending operation was 

not an acceptable reason for speaking up. This lead to the ineffectiveness of the Error Prevention 

Program. 

During the operation, the Technician Supervisor commented that there were empty bolt holes, a 

conversation that was overheard by several of the technicians. The team and the RTE in 

particular dismissed the comment and did not pursue the issue further. 

Evidence that the operation proceeded at an accelerated pace was expressed by the majority of 

the operations team, although the precise reason for the acceleration was varied. It is the MIB’s 

belief that the RTE was anticipating his departure and may have hurried through the procedures. 

The fact that all of the preparations were completed, the entire lifting/mating operation was 

completed, and the TOC vertical and horizontal rotation activities were begun in the space of one 

hour was recognized as highly unusual, yet raised no flags of caution. 

5.5.4 Rotate Vertically 7:06 – 7:16 am: 

A time of 7:06 is ascribed as the start of the vertical rotation to orient the spacecraft instrument 

package to an orientation that will put it on the upward side of the spacecraft when the spacecraft 

is rotated to the horizontal position. The TOC is known to take 5-10 minutes to rotate 180 

degrees. While the exact magnitude of the rotation is not known, 10 minutes is a reasonable 

period to allocate to the activity, including preparation. The spacecraft did not slip or otherwise 

‘wobble’ because of the weight of the spacecraft and the large contact surface area creating 

significant friction between the unbolted plates. This activity can be likened to slowly rotating a 

carousel or “Lazy Susan” with an item or items sitting on it with no means of securing other than 

simple friction. (Note: This activity is not described nor specified in the procedure, but was 

determined to be standard practice by the I&T Team. Normally, it was done to put the 

instrument package in the minimum risk position for horizontal rotation, but in this case, it was 

necessary for the MHS Shimming Operation.) 

The team was using Appendix C of the TI-MH 3278200 procedure for operation of the TOC. 

That section provides a general description of how to operate the TOC, rather than a step-by-step 

procedure. The section requires: 1 RTE, 4 Technicians, 1 Quality Inspector, and 1 Customer 

Representative. The large number of technicians is required because the TOC in normally 

maneuvered with a technician located at each corner. For this NOAA N-PRIME activity, the 

TOC had been maneuvered under the suspended spacecraft, and would not be maneuvered with 

the spacecraft attached because of the substandard floor jack repair. This is a second area in 

which the number of personnel prescribed in the procedure was not adhered to, but is not seen as 

a contributor to the incident. 

47 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

5.5.5 Rotate to Horizontal 7:16 – 7:28 am: 

A time of 7:16 is ascribed as the start of horizontal rotation. This time was determined by 

backing up from the actual time of the incident, along with calculating the time it took the TOC 

to reach 13 degrees of rotation. The cart is described as taking approximately 45 to 60 minutes 

to rotate from vertical to horizontal. Based on the 13 degree angle at the time of the incident, the 

rotation time ascribed is 13/90ths of 45 minutes, or approximately 7-8 minutes for actual time of 

rotation, plus a short time for the TOC operator to power the cart and begin the rotation. This 

establishes the probable time of rotation. It was not determined whether the angular rotation rate 

is linear during the entire 90 degrees, but this estimated duration is only relevant for helping to 

distribute the activities between attaching the lift fixture and the incident appropriately 

throughout the period to aid in gaining a sense of the pace of the activity. No other 

documentation exists to aid in timing determinations. 

In the 21 minutes between 6:45 and 7:06, the following steps and sections of the procedure were 

performed (Ref TI-MH 3278200, Section 5.6) (See also Table 5-1: Operations Event Timeline, 

and Figure 5-23: Saturday Detailed Timeline). (Note: The steps below are abbreviated 

descriptions because the individual steps do not have titles.) 

5.6.2 Attach S/C vertical lift fixture per 5.1 

5.6.3 Position personnel for Lift 

5.6.4 Lift S/C over TOC 

5.6.5 Offload crane to 1000 lbs 

5.6.6 Install and torque 44 places* 

5.6.7 Offload all crane load 

5.6.8 Remove lift fixture 

5.6.9 Torque check 44 fasteners* 

5.6.10 Continue with authorizing document 

 *Activities in close proximity to missing bolt holes 

In addition to the steps specified in the procedure above, the crew is known to have performed 

two additional steps, one of which is significant. They wiped the mating surfaces between the 

spacecraft Payload Adapter and the TOC Adapter Ring with isopropyl alcohol (a standard but 

undocumented procedure). This is significant because this action wiped directly over the 

missing-bolt holes. The heads of these missing bolts normally protrude above the adapter plate 

surface. The fact that the technicians wiped this surface without encountering the normal 

interference of the bolt heads is significant. A skill-based error was made by the crew: the 

technicians were narrowly focused on their individual tasks and did not notice or consider the 

state of the hardware or the operation outside of those tasks. This type of error is common in 

highly structured, repetitive tasks and each technician, when interviewed, commented on the 

large number of times these procedures had been successfully attempted in the past. In addition, 

each technician commented that the RTE had full authority and responsibility for the operation 

and that their role was to follow the RTE’s instructions; a culture proved to promote the type of 

channelized attention observed. While it is absolutely clear that it is the RTEs function to direct 

the tasks for each crew member, each crew member is also qualified for specific functions. 

Qualified technicians are expected to exercise their expertise in the performance of their tasks. 

(Finding L-2) (MISSED OPPORTUNITY #3)

48 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

The second added step was the rotation of the S/C about its vertical axis to put the instrument 

package in the preferred orientation with the instruments on the ‘up’ side of the spacecraft when 

in the horizontal position. This missing step is significant only as it points to the fidelity of the 

procedures. The outcome would have been the same with or without the spacecraft rotation 

except for the possible extent of the resultant damage. 

No details are available to further document the actual times of each of the above steps in the 

procedure except witness statements that the lift occurred ‘about 7am’. This is consistent with 

the activities that occurred, and the actual time of the incident.

The Board feels very strongly that in order for a small crew of two engineers and two technicians 

to accomplish the sequence of activities necessary to hoist the spacecraft and prepare for rotation 

to horizontal, including torquing and re-torquing the 44 fasteners holding the spacecraft to the 

adapter ring, the pace of the activity was very fast. In fact, members of the crew recognized that, 

“This was the smoothest this operation has ever gone.” (MISSED OPPORTUINTY #4)

The time of 7:28am is established as the time of the incident because the Technician Supervisor 

stated that ‘he immediately left the high bay to summon the I&T Manager.’ He actually reentered the High Bay at 7:29am.

49 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

5.6 Summary of Missed Opportunities 

Missed Opportunities have been identified above during the course of the analysis. Table 5-3: 

Missed Opportunities, summarizes those opportunities, grouping them to provide a sense of the 

pace and tenor of the operation. 

No. MISSED OPPORTUNITY SECTION

No. 

1 Conversation Between RTE and Technician 

Supervisor: 

At least one of the crew members was uncertain 

and nervous about seeing ‘holes with threads’, but 

did not have sufficient experience with the TIROS 

TOC-to-spacecraft configuration to recognize the 

reality of the situation.

5.3 

2 TOC Verification:

The verification of the TOC was not performed as

required by procedure. PA was not present to 

witness as required. 

5.5.2 

3 Mating Surface Wipe-down: 

Experienced Technicians performing the mating 

surface wipe down with isopropyl alcohol failed to 

realize that they were wiping over holes where bolt 

heads normally protruded and would make the 

wipe down more difficult. 

5.5.5 

4 Ease of Spacecraft Mate: 

Most of the crew members were aware of how 

smoothly the spacecraft mating operations went, 

but none recognized that it was due to the lack of 

impediment normally caused by the presence of 

the 24 adapter mating bolts. 

5.5.5 

Table 5-3: Missed Opportunities 

5.8 Summary 

Several elements contributed to the NOAA N-PRIME incident, the most significant of which 

were the lack of proper TOC verification, including the lack of proper PA witness, the change in 

schedule and its effect on the crew makeup, the failure of the crew to recognize missing bolts 

while performing the interface surface wipe down, the failure to notify in a timely fashion or at 

50 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

all the Safety, PA, and Government representatives, and the improper use of procedure redlines 

leading to a difficult-to-follow sequence of events. The interplay of the several elements allowed 

a situation to exist where the extensively experienced crew was not focusing on the activity at 

hand. There were missed opportunities that could have averted this mishap. The missed 

opportunities are summarized in Section 5.6. 

Table 5-4, Findings Summary (LMSSC), and Table 5-5, Findings Summary (government) 

provide a cross-reference between the Findings discussed in Section 7 and the descriptions in 

Section 5. 

Table 5-4: Findings Summary (LMSSC) 

NO. PARA. FINDING

L-1 

L-2 

L-3 

5.5.2 

5.3.1 

5.3.2 

5.3.3.1 

5.5.2 

5.3.3.1 

UNSAFE ACTS

1) RTE committed a “decision error” by not following procedures. 

2) Technicians committed “skill based error” by not noticing the missing 

bolts while wiping down interface plate and bolting down the spacecraft.

3) A) PQC and PA inspector committed “routine violations” by signing-off

on operations without witnessing and verifying TOC configuration. 

B) The safety representative was not present as called for in the 

procedure. 

L-4 

L-5 

5.3.1 

5.3.3.3 

5.3 

5.4.2 

PRECONDITIONS FOR UNSAFE ACTS

4) Adverse mental states in the forms of “channelized attention”, 

“complacency”, and “get-home-itis” resulted in an accelerated pace of 

operations and procedure execution. 

5) “Substandard crew resources” included poor hand-off between the DMSP 

and TIROS Projects regarding the TOC, late identification of personnel to 

work Saturday, and poor red-tagging process for GSE.

51 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 5-4: Findings Summary (LMSSC) 

L-6 

L-7 

L-8 

L-9 

5.2 

5.3 

5.5.3 

5.5.2 

5.3.3.1 

5.3.3.3 

UNSAFE SUPERVISION FACTORS

6) “Planned an inappropriate operation”- The team was formed late in a 

harried fashion with an atypical mix of personnel. 

7) “Inadequate supervision” was manifested in the lack of clear definition 

and enforcement of roles and responsibilities among the team individuals, 

consequently individuals failed to fulfill their expected roles and 

responsibilities. 

8) “Failure to correct known problems” was a supervisory failure to correct 

similar known problems. PA supervisors routinely allowed PA inspector 

sign-off after the fact. I&T supervisors routinely allowed poor test 

documentation. 

9) “Supervisory Violation” was committed by repeatedly waiving required 

presence of quality assurance and safety and bypassing Government 

Mandatory Inspection Points. 

NO. PARA. FINDING

L-10 

L-11 

L-12 

L-13 

5.3.3.1 

5.2 

5.3 

5.2 

5.3.3.1 

ORGANIZATIONAL INFLUENCES

10) In “resource management”, MIB observed inadequate emphasis on safety, 

and inadequate quality assurance support to provide effective coverage. 

11) The “Organization climate” in the I&T domain with an operational 

program has engendered an unhealthy environment that led to complacent 

and overconfident attitudes toward routine operations. 

12) Lack of effective “Organizational Processes” in the form of guideline and

safeguards to regulate the I&T environment. 

13) Ineffective System Safety Program. 

Table 5-4: Findings Summary (LMSSC) 

52 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 5-5: Findings Summary (government) 

NO. PARA. FINDING

G-1 

G-2 

G-3 

G-4 

5.2.4 

5.3.3.3 

5.3.3.1 

5.3.3.3 

Sect. 8 

UNSAFE OVERSIGHT FACTORS

1) The government quality assurance and safety provided “inadequate 

oversight”. Oversight function became “issue driven”. Procedures 

rarely reviewed; non-conformances not trended; rarely make impromptu 

inspections.

2) In substituting for the DCMA, the QAR failed to enforce a Government 

Inspection Point by failing to enforce his presence at the operation. 

3) Government has very limited safety oversight. 

4) Government “failed to correct known problems” such as PA signoff 

after the fact and poor test documentation. 

G-5 

G-6 

G-7 

G-8 

G-9 

5.2 

5.3.3.3 

5.3.3.3 

Sect. 

8.2 

Table 8-4 

5.3.3.1 

5.3.3.2 

Sect. 8.2 

Table 8-4 

ORGANIZATIONAL INFLUENCES

5) Deficient “resource management” include: rapid trade-offs between the 

schedules, staffing and milestones for the two remaining satellites 

exacerbated the already fast operational tempo of the LMSSC I&T 

team, lack of resources in the safety area. 

6) Unhealthy “organizational climate” factors include: using retired 

LMSSC employees as government representatives, lax and casual 

oversight toward an I&T environment with routine operations. 

7) Government lacks “organizational processes” to effectively monitor, 

verify, and audit the performance and effectiveness of the I&T 

processes and activities. 

8) The government safety program placed an overemphasis on launch site 

safety and inadequate in-plant safety.

9) Deficient DCMA CAR assessment and reporting processes. 

Table 5-5: Findings Summary (government) 

53 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 6 

METHOD OF INVESTIGATION 

Three days after the mishap, on September 9, 2003, Dr. Ghassem Asrar, NASA Associate 

Administrator for Earth Science established the NOAA N-PRIME Mishap Investigation Board 

(MIB) in the public interest to gather information, conduct necessary analyses, and determine the 

facts of the mishap. (Appendix B) The board was chartered to determine the causes(s) of the 

mishap and to recommend preventive measures and actions to preclude recurrence of similar 

mishaps. In addition, the board would also investigate or determine the existence of potential 

systematic problems/practices with system reliability and quality assurance activities at the 

facility. 

To identify the causes at work in the NOAA N-PRIME Mishap, the MIB collected evidence 

from the following sources: 

• Witness interviews. All witnesses present at the mishap were interviewed along with key 

supervisory personnel and those connected with the events in question. Prior to being 

interviewed, each witness informed that this is a safety investigation to determine the 

cause(s) of the mishap and not legal liability, and that NASA will make every effort to keep 

testimony confidential. Alternatively read or informed of the following: 

"The purpose of this safety investigation is to determine the root cause(s) of the mishap that 

occurred on September 6, 2003, and to develop recommendations toward the prevention of 

similar mishaps in the future. It is not our purpose to place blame or to determine legal 

liability. Your testimony is entirely voluntary, but we hope that you will assist the board to 

the maximum extent of your knowledge in this matter. Your testimony will be documented 

and retained as part of the mishap investigation report background files but will not be

released as part of the investigation board report. NASA will make every effort to keep your 

testimony confidential and privileged to the greatest extent permitted by law. However, the 

ultimate decision as to whether your testimony may be released may reside with a court or 

administrative body outside of NASA." 

Specific witness statements are not included in this text, as they are considered privileged 

and confidential. 

• Test Procedures and Handling Instructions: The MIB reviewed all procedures that were used 

on NOAA N-PRIME at the time of the mishap (i.e., TI-MH-3278200 and TI-MHS-3278200), 

as well as related procedures called out by those documents both before and after the step at 

which the mishap occurred. These reviews included the Program Directives, Log of 

Operations, and similar documents. The review was not limited to NOAA N-PRIME, but 

also included a review of pertinent NOAA N procedures (also undergoing testing in the same

integration and test area within the LMSSC-Sunnyvale facility) to identify whether similar 

actions had taken place in its processing. 

• Non-Conformance Reports: The MIB reviewed all reports that could indicate a possible 

systemic problem. 

54 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

• Corrective Action Requests: The MIB reviewed DCMA Corrective Action Requests (CARs) 

from April 2001 to September 2003 to identify systematic problems with the LMSSC quality 

system at Sunnyvale facility. 

• POES Management Plans: The MIB reviewed the Quality Assurance Plan, Safety Plan, and 

Configuration Management Plan. 

• Other sources examined. Other sources reviewed included project schedules, contract 

requirements, LMC Command Media (i.e., corporate-level policies), experience of the Board 

members in working these types of programs, etc. 

Two approaches were taken to determine the causes of the mishap. The first was an extensive 

analysis of the sequence of events prior to and on the day of the mishap; the planned operational 

scenario vs. the actual execution; and the planning activities, including scheduling, crew 

assembly and test documentation preparation. Missed opportunities that could have averted this 

mishap were also examined. This analysis is presented in Section 5. 

The MIB also utilized the Human Factors Analysis and Classification System (HFACS) (2000) 

to examine the potential causes of the mishap. HFACS provides a comprehensive framework for 

identifying and analyzing human error. Further, it provides a method to categorize the findings 

or deficiencies and to formulate intervention strategies. HFACS is built upon James Reason’s 

(1990) concept of latent and active failures or the so-called ‘Swiss Cheese” model. As shown in 

Figure 6-1, this model describes four levels of failure, each more removed from the actual 

accident than the last and yet each still influential due to its effect on the preceding level. 

• The first level, or that most proximate to the accident, contains the “Unsafe Acts” of 

the operators. These acts are the active failures that lead directly to the occurrence. 

• The second level of the model addresses the conditions and practices of the operators 

allowing the unsafe acts to occur, or “Preconditions for Unsafe Acts.” 

• Such preconditions can often be traced back to the supervisory level of the 

organization, which leads to the third level called “Unsafe Supervision.” 

• The final level of the Reason model deals with “Organizational Influences,” which 

captures the organization’s potentially adverse influence on the performance of the 

supervisors, operators, etc. 

These last three levels describe latent failures, or those problems that may exist undetected or 

unchanged for some time before being manifested in an accident scenario. Within the four levels

defined by Reason, HFACS creates greater resolution and granularity to the accident cause 

categories. While some categories are specific to aviation and pilot errors, most can be

generalized to a broader class of accidents. 

Using the evidence cited above, the MIB evaluated each sublevel of the HFACS approach and 

drew findings based upon the preponderance of evidence as to the presence of that cause. Levels 

3 and 4 of the model were evaluated for both organizations involved in the mishap, LMSSC and 

the Government. A more detailed description of the HFACS model is also provided below. 

55 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Latent failures

Organizational Influences

Unsafe supervision

Latent failures

Figure 6-1: Reason’s Swiss Cheese Model** Reason, 1990

Preconditions

Latent failures

Unsafe acts

Active failures

Failed

or

absent

defenses MISHAP

Latent failures

Organizational Influences

Unsafe supervision

Latent failures

Figure 6-1: Reason’s Swiss Cheese Model** Reason, 1990

Preconditions

Latent failures

Unsafe acts

Active failures

Failed

or

absent

defenses MISHAP

56 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

DESCRIPTION OF HFACS 

For completeness of presentation, this section summarizes the HFACS methodology. A more 

detailed description of HFACS can be found in “The Human Factors Analysis and Classification 

System-HFACS” by Scott A. Shappell, February 2000, US Department of Transportation, 

DOT/FAA/AM-00/7. 

Human Factors Analysis and Classification System (HFACS) Methodology

The Human Factors Analysis and Classification System (HFACS) provide a comprehensive 

framework for identifying and analyzing human error. Its use was first tested in aviation 

accidents, in which an estimated 70-80% is attributable to human error. Over 300 Naval aviation 

accidents and numerous others from other military and civil aviation have been assessed using 

the current instantiation of this system. 

The HFACS framework is built upon James Reason’s (1990) concept of latent and active failures 

or the so-called ‘Swiss Cheese” model. What convinced the MIB to use this model to analyze 

the mishap was its ability to address both active and latent failures within the causal sequence of 

events. As shown in Figure 6-1, this model describes four levels of failure, each more removed 

from the actual accident than the last and yet each still influential due to its effect on the 

preceding level. The first level, or that most proximate to the accident, contains the “Unsafe 

Acts” of the operators. These acts are the active failures that lead directly to the occurrence. The 

next three levels describe latent failures, or those problems that may exist undetected or 

unchanged for some time before being manifested in an accident scenario. Accordingly, the 

second level of the model addresses the conditions and practices of the operators allowing the 

unsafe acts to occur, or “Preconditions for Unsafe Acts.” Such preconditions can often be traced 

back to the supervisory level of the organization, which leads to the third level called “Unsafe 

Supervision.” And finally, the organization as a whole may adversely influence the performance 

of the supervisors, operators, etc. and hence, the final level of the Reason model deals with 

“Organizational Influences.” 

Within these levels defined by Reason, HFACS creates greater resolution and granularity to the 

accident cause categories. While some categories are specific to aviation and pilot errors, most 

can be generalized to a broader class of accidents. These categories are described below. 

Unsafe Acts 

HFACS divides the Unsafe Acts level into two categories as shown in Figure 6-2: errors, which 

represent the actions or activities of the individuals who directly perpetrate the accident; and 

violations, which describe disobeying rules and/or standards of conduct or operation by these 

same individuals. 

The error category is further refined as shown in Figure 6-2, by distinguishing among decision, 

skill-based and perceptual errors. A decision error refers to planned and executed behavior that 

proves to be inadequate or inappropriate to the situation at hand, due to poor choosing or 

insufficient information. A skill-based error, on the other hand, describes planned behavior that 

57 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

is poorly executed. Finally, perceptual errors refer to misjudgments concerning sensory input, 

because of degraded or unusual information. Within each of these error categories, HFACS 

provides examples of specific accident causes. These are shown in Figure 6- 3. 

The violation category can be divided into two subcategories: routine, which are those violations 

that are habitual, normal, and often tolerated by authority; and exceptional, which are violations 

that are atypical or aberrations. Specific examples of violations are also shown in Figure 6-3. 

Figure 6-2: HFACS Unsafe Acts* 

* Shappell and Wiegmann 2000

UNSAFE 

ACTS 

Errors Violations 

Decision Skill- Based Perceptual Routine Exceptional 

ERRORS

Skill-based errors 

• Breakdown in visual scan

• Failed to prioritize attention 

• Inadvertent use of flight controls 

• Omitted step in procedure 

• Omitted checklist item

• Poor technique 

• Over-controlled the aircraft 

Decision Errors 

• Improper procedure 

• Misdiagnosed emergency

• Wrong response to emergency 

• Exceeded ability 

• Inappropriate maneuver 

• Poor decision

Perceptual Errors (due to) 

• Misjudged distance /altitude/airspeed

• Spatial disorientation

• Visual illusion

VIOLATIONS

• Failed to adhere to brief 

• Failed to use the radar altimeter 

• Flew an unauthorized approach 

• Flew an overaggressive maneuver

• Failed to properly prepare for the flight

• Briefed unauthorized flight

• Non current/qualified for the mission 

• Intentionally exceeded the limits of the aircraft 

• Continued low-altitude flight in VMC 

• Unauthorized low-altitude canyon running 

Figure 6-3: Examples of Unsafe Acts*

* Shappell and Wiegmann, 2000 

58 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Preconditions for Unsafe Acts

In the “Preconditions” level, HFACS identifies the two main contributors shown in Figure 6-4. 

The first, named Substandard Conditions of Operators, addresses the mental, physiological, and 

physical states that may have adversely affected the performance of the operators. Specific 

examples in each category are shown in Figure 6-5. The second contributor is the Substandard 

Practice of Operators, which can be further divided into the practices of crew resource 

management and the practices relative to personal readiness. Crew resource management 

describes the practices of good communication and team coordination (i.e., preparedness of the 

team), whereas personal readiness refers to mental and physical preparedness of individual team

members. Figure 6-5 shows examples for these two subcategories. 

PRECONDITIONS 

for UNSAFE ACTS

Substandard Condition

of Operators

Substandard Practices

of Operators

Adverse

Mental State 

Adverse

Physiological

State

Physical/Mental 

Limitations 

Crew Resource

Management

Personal

Readiness

PRECONDITIONS 

for UNSAFE ACTS

Substandard Condition

of Operators

Substandard Practices

of Operators

Adverse

Mental State 

Adverse

Physiological

State

Physical/Mental 

Limitations 

Crew Resource

Management

Personal

Readiness

 Figure 6-4: Categories of Preconditions for Unsafe Acts* 

*Shappell and Wiegmann, 2000

SUBSTANDARD CONDITIONS OF OPERATORS

Adverse Mental States 

• Channelized attention

• Complacency 

• Distraction 

• Mental fatigue 

• Get-home-itis 

• Haste 

• Loss of situational awareness 

• Misplaced motivation 

• Task saturation 

Adverse Physiological state 

• Impaired physiological state 

• Medical illness 

• Physiological incapacitation 

• Physical fatigue

Physical/Mental Limitation

• Insufficient reaction time 

• Visual limitation 

• Incompatible intelligence/aptitude

• Incompatible physical capability 

SUBSTANDARD PRACTICE OF OPERATORS

 

Crew Resource Management 

• Failed to back-up 

• Failed to communicate/coordinate

• Failed to conduct adequate brief 

• Failed to use all available resources 

• Failure of leadership

• Misinterpretation of traffic calls

Personal Readiness 

• Excessive physical training 

• Self-medicating 

• Violation of crew rest requirement 

• Violation of bottle-to-throttle requirement 

Figure 6-5: Examples of Preconditions of Unsafe Acts* 

* Shappell and Wiegmann, 2000

59 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Unsafe Supervision

For Unsafe Supervision, HFACS describes four sublevels in Figure 6-6: inadequate supervision, 

planned inappropriate operations, failure to correct a known problem, and supervisory violations. 

Inadequate supervision addresses the question of whether the supervisory chain of command 

made it possible for the individual team members to succeed in terms of oversight, training, 

guidance, etc. Planned inappropriate operations refer to the decisions that, by reacting to 

pressures of schedule, operational tempo, cost cutting, etc., places the operation at greater risk of 

failure. The third subcategory identifies the instances when a supervisor fails to address 

deficiencies that are known to him or her, for example in individual performance, equipment 

maintenance, or other areas related to safety and reliability. Supervisory violations identify 

instances when rules or regulations are purposely violated. Examples in each of these areas are 

shown in Figure 6-7. 

UNSAFE

SUPERVISION

Inadequate 

Supervision

Planned Inappropriate

Operation

Failed to

Correct Problems

Supervisory

Violations

UNSAFE

SUPERVISION

Inadequate 

Supervision

Planned Inappropriate

Operation

Failed to

Correct Problems

Supervisory

Violations

Figure 6-6: Categories of Unsafe Supervision* 

*Shappell and Wiegmann 2000

Inadequate Supervision 

•Failed to provide guidance 

•Failed to provide operation doctrine

•Failed to provide oversight 

•Failed to provide training 

•Failed to track qualifications

•Failed to track performance 

Planned Inappropriate Operations 

•Failed to provide correct data

•Failed to provide adequate brief time

•Improper manning

•Mission not in accordance with rules/regulations 

•Provided inadequate opportunity for crew rest

Failed to Correct a Known Problem 

•Failed to correct document in error 

•Failed to identify an at-risk aviator 

•Failed to initiate corrective action

•Failed to report unsafe tendencies 

Supervisory Violations 

•Authorized unnecessary hazard 

•Failed to enforce rules and regulations

•Authorized unqualified crew for flight

Figure 6-7: Examples of Unsafe Supervision* * Shappell and Wiegmann, 2000 

Organizational Influences

HFACS suggests the latent failures that commonly occur in the upper-levels of management and 

that affect supervisory practices and other controls are issues in resource management, 

organizational climate, and organizational process as shown in Figure 6-8. Resource 

management issues are those that relate to policy decisions for the allocation of assets, a classic 

60 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

example being the tradeoff between safety preservation and cost/profit. Climate refers to the 

work environments within which personnel perform their functions, and can include 

communication channels, official policies (e.g., relative to hiring and firing), and unofficial 

policies (e.g., unspoken rules). The final area is that of organizational process, which describes 

the ways and methods that tasks are accomplished and work is controlled within the 

organization. These include standard operating procedures, incentives, operational tempo, and 

scheduling, among others. Further examples in each of the organizational influences are shown 

in Figure 6-9. 

ORGANIZATIONAL

INFLUENCES

Resource

Management

Organizational 

Climate

Organizational 

Process

ORGANIZATIONAL

INFLUENCES

Resource

Management

Organizational 

Climate

Organizational 

Process

Figure 6-8: Organizational Factors Influencing Accidents* 

*Shappell and Wiegmann 2000

Resource/Acquisition Management 

Human Resources 

•Selection 

•Staffing/manning 

•Training

Monetary/budget resources 

•Excessive cost cutting 

•Lack of funding 

Equipment/facility resources 

•Poor Design

•Purchasing of unsuitable equipment

Organizational Climate 

Structure 

•Chain-of-command 

•Delegation of authority 

•Communication 

•Formal accountability for actions 

Policies 

•Hiring and firing 

•Promotion 

•Drugs and alcohol

Culture 

•Norms and rules 

•Values and beliefs 

•Organizational justice 

Organizational Process 

Operations

•Operational tempo 

•Time pressure

•Production quotas 

•Incentives 

•Measurement/appraisal 

•Schedules 

•Deficient planning

Procedures

•Standards

•Clearly defined objectives 

•Documentation 

•Instructions 

Oversight

•Risk management

•Safety programs

Figure 6-9: Examples of Organizational Influences* 

* Shappell and Wiegmann, 2000

61 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 7

FINDINGS

The proximate cause of The NOAA N-PRIME mishap was the failure of the LMSSC operations 

team to follow procedures to properly configure the TOC, such that the 24 bolts that were 

needed to secure the TOC adapter plate to the TOC were not installed. 

The MIB identified root causes along the four levels of active or latent failures as ascribed by the 

HFACS framework. The supporting findings for these root causes are presented in this section. 

Figure 7-1 illustrates the Level 1 active failures or unsafe acts leading to the mishap and Figures 

7-2, 7-3, 7-4, & 7-5 trace each of the active failures to the three levels of latent failures as 

ascribed by the HFACS framework. The findings of this analysis are numbered sequentially and 

are described in more detail in the text that follows. 

SATELLITE FELL

OFF TURNOVER

CART

Failed to install 24 bolts attaching

the spacecraft adapter plate to 

The Turnover Cart base plate

1) Decision Error

RTE “assure the configuration” 

of the TOC based on paper 

Work rather than following 

procedure to verify torque

values

2) Skill-Based Error

Technicians failed to notice 

the missing bolts while wiping

down the adapter plates and

bolting down the spacecraft to

The spacecraft adapter plate 

3a) Routine Violation

PQC and PA signed-off the 

procedure step of “assure the

configuration” of the TOC

without witnessing and verifying 

3b) Routine Violation

Safety Representative not

present as called for in

the procedure

PROXIMATE CAUSE

SATELLITE FELL

OFF TURNOVER

CART

Failed to install 24 bolts attaching

the spacecraft adapter plate to 

The Turnover Cart base plate

1) Decision Error

RTE “assure the configuration” 

of the TOC based on paper 

Work rather than following 

procedure to verify torque

values

2) Skill-Based Error

Technicians failed to notice 

the missing bolts while wiping

down the adapter plates and

bolting down the spacecraft to

The spacecraft adapter plate 

3a) Routine Violation

PQC and PA signed-off the 

procedure step of “assure the

configuration” of the TOC

without witnessing and verifying 

3b) Routine Violation

Safety Representative not

present as called for in

the procedure

SATELLITE FELL

OFF TURNOVER

CART

Failed to install 24 bolts attaching

the spacecraft adapter plate to 

The Turnover Cart base plate

1) Decision Error

RTE “assure the configuration” 

of the TOC based on paper 

Work rather than following 

procedure to verify torque

values

2) Skill-Based Error

Technicians failed to notice 

the missing bolts while wiping

down the adapter plates and

bolting down the spacecraft to

The spacecraft adapter plate 

3a) Routine Violation

PQC and PA signed-off the 

procedure step of “assure the

configuration” of the TOC

without witnessing and verifying 

3b) Routine Violation

Safety Representative not

present as called for in

the procedure

PROXIMATE CAUSE

Figure 7-1: Level 1 Active Failures or Unsafe Acts Leading to the NOAA N-Prime Mishap 

62 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

63 

September 13, 2004 

Figure 7-2: Latent Failures Contributory to the Decision Error of the RTE 

1) Decision Error

RTE “assure the configuration”

of the TOC based on paper work

rather than following procedure to

verify torque values

4) Adverse Mental State

Complacency: Routine operation;

Pre-conceived notion that the TOC 

was already properly configured

External constraints: Signed up

for 1⁄2 day of work

5) Substandard Crew Resources

Poor communication between DMSP

and TIROS project in sharing of GSE

Poor documentation and red-tagging 

for GSE configuration control

Extensively redlined procedures

Ambiguous use of “assure” in the 

procedure

6) Planned Inappropriate Operation

Team formed late in harried fashion

with atypical mix of personnel and poor

test documentation

8) Failed to Correct Known Problems

Failed to recognize and correct GSE

configuration control problems

Failed to recognize and correct poor

procedure development/redline process

Failed to correct similar problems 

documented in NCRs and CARs

11) Organizational Climate

Dynamic I&T environment thrives on 

optimal schedule utilization

Operational program/routine operation

fosters complacent attitudes and lax

discipline

12) Organizational Processes

Lack of processes to I&T operational 

tempo, team formation and notification

12) Organizational Processes

Lack of process for GSE configuration 

control

Lack of process for procedure

development, redlining and approval

12) Organizational Processes

Lack of organizational oversight to

monitor, trend, verify, and audit the 

performance and effectiveness of 

I&T procedures and activities

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences

1) Decision Error

RTE “assure the configuration”

of the TOC based on paper work

rather than following procedure to

verify torque values

4) Adverse Mental State

Complacency: Routine operation;

Pre-conceived notion that the TOC 

was already properly configured

External constraints: Signed up

for 1⁄2 day of work

5) Substandard Crew Resources

Poor communication between DMSP

and TIROS project in sharing of GSE

Poor documentation and red-tagging 

for GSE configuration control

Extensively redlined procedures

Ambiguous use of “assure” in the 

procedure

6) Planned Inappropriate Operation

Team formed late in harried fashion

with atypical mix of personnel and poor

test documentation

8) Failed to Correct Known Problems

Failed to recognize and correct GSE

configuration control problems

Failed to recognize and correct poor

procedure development/redline process

Failed to correct similar problems 

documented in NCRs and CARs

11) Organizational Climate

Dynamic I&T environment thrives on 

optimal schedule utilization

Operational program/routine operation

fosters complacent attitudes and lax

discipline

12) Organizational Processes

Lack of processes to I&T operational 

tempo, team formation and notification

12) Organizational Processes

Lack of process for GSE configuration 

control

Lack of process for procedure

development, redlining and approval

12) Organizational Processes

Lack of organizational oversight to

monitor, trend, verify, and audit the 

performance and effectiveness of 

I&T procedures and activities

1) Decision Error

RTE “assure the configuration”

of the TOC based on paper work

rather than following procedure to

verify torque values

4) Adverse Mental State

Complacency: Routine operation;

Pre-conceived notion that the TOC 

was already properly configured

External constraints: Signed up

for 1⁄2 day of work

5) Substandard Crew Resources

Poor communication between DMSP

and TIROS project in sharing of GSE

Poor documentation and red-tagging 

for GSE configuration control

Extensively redlined procedures

Ambiguous use of “assure” in the 

procedure

6) Planned Inappropriate Operation

Team formed late in harried fashion

with atypical mix of personnel and poor

test documentation

8) Failed to Correct Known Problems

Failed to recognize and correct GSE

configuration control problems

Failed to recognize and correct poor

procedure development/redline process

Failed to correct similar problems 

documented in NCRs and CARs

11) Organizational Climate

Dynamic I&T environment thrives on 

optimal schedule utilization

Operational program/routine operation

fosters complacent attitudes and lax

discipline

12) Organizational Processes

Lack of processes to I&T operational 

tempo, team formation and notification

12) Organizational Processes

Lack of process for GSE configuration 

control

Lack of process for procedure

development, redlining and approval

12) Organizational Processes

Lack of organizational oversight to

monitor, trend, verify, and audit the 

performance and effectiveness of 

I&T procedures and activities

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences
NOAA N-PRIME Mishap Investigation-Final Report 

64 

September 13, 2004 

Unsafe Acts 

Precondition for Unsafe Acts 

Unsafe Supervision 

Organizational Influences 

2) Skilled-Based Error

Technicians failed to notice 

the missing bolts while wiping 

down the TOC adapter plate and 

bolting down the spacecraft to

The TOC adapter plate 

4) Adverse Mental State 5) Substandard Crew Resources

Channelized attention: Focused No Pre-Operational safety briefing only on task at hand Failure of RTE leadership Complacency: Routine operation 

One technician signed up only 

for 1⁄2 day of work for TIROS 

6) Planned Inappropriate Operation

Team formed late in harried fashion. 

A team member normally have a 

leadership role among technicians was 

not present 

11) Organizational Climate

Operational program/routine operations 

fosters complacent attitudes and lax 

discipline 

12) Organizational Processes

Lack of processes to regulate team formation

and notification

Figure 7-3: Latent Failures Contributory to the Skill-Based Error of the Technicians
NOAA N-PRIME Mishap Investigation-Final Report 

September 13, 2004 

3a) Routine Violation

PQC and PA signed-off the 

procedure step of “assure

the configuration” of the TOC 

without witnessing and

verifying

4) Adverse Mental State

Complacency: Routine operation/ have

confidence in RTE, signed off “assure 

the configuration” of the TOC without 

personal verification

5) Substandard Crew Resources

Lack notification of personnel

Late arrival of PA inspector

No Pre-operational safety briefing

Failure of RTE leadership

7) Inadequate Supervision

Lack of supervisory enforcement 

to ensure team I&T members 

understand and uphold

their roles and responsibilities

9) Supervisory Violation

RTE and I&T manager present failed

to enforce rules of QA presence, 

allowed sign-off procedure after the

fact and without personal verification 

11) Organizational Climate

Operational program fosters complacent

attitudes and lax discipline

12) Organizational Processes

Lack of process and training to assure

procedure execution discipline

10) Resource Management

Inadequate quality assurance support. 

11) Organizational Climate

Operational program fosters complacent

attitude and lax discipline

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences 3a) Routine Violation

PQC and PA signed-off the 

procedure step of “assure

the configuration” of the TOC 

without witnessing and

verifying

4) Adverse Mental State

Complacency: Routine operation/ have

confidence in RTE, signed off “assure 

the configuration” of the TOC without 

personal verification

5) Substandard Crew Resources

Lack notification of personnel

Late arrival of PA inspector

No Pre-operational safety briefing

Failure of RTE leadership

7) Inadequate Supervision

Lack of supervisory enforcement 

to ensure team I&T members 

understand and uphold

their roles and responsibilities

9) Supervisory Violation

RTE and I&T manager present failed

to enforce rules of QA presence, 

allowed sign-off procedure after the

fact and without personal verification 

11) Organizational Climate

Operational program fosters complacent

attitudes and lax discipline

12) Organizational Processes

Lack of process and training to assure

procedure execution discipline

10) Resource Management

Inadequate quality assurance support. 

11) Organizational Climate

Operational program fosters complacent

attitude and lax discipline

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences

65 Figure 7-4: Latent Failures Contributory to the Routine Violations of the PQC and PA
NOAA N-PRIME Mishap Investigation-Final Report 

66 

September 13, 2004 

3b) Routine Violation

Safety representative was not present

as called for in procedure

5) Substandard Crew Resources

Safety representative was not notified of

the operations

9) Supervisory Violation

RTE failed to notify safety

representative when 

planning for the operation. 

Routinely waives safety presence

10) Resource Management

Inadequate safety support. Safety

support limited by program 

resources available 

13) Ineffective Safety Program

Safety program primarily targeted

for launch operations. System 

safety program lack definition and

recognition. 

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

3b) Routine Violation Organizational Influences

Safety representative was not present

as called for in procedure

5) Substandard Crew Resources

Safety representative was not notified of

the operations

9) Supervisory Violation

RTE failed to notify safety

representative when 

planning for the operation. 

Routinely waives safety presence

10) Resource Management

Inadequate safety support. Safety

support limited by program 

resources available 

13) Ineffective Safety Program

Safety program primarily targeted

for launch operations. System 

safety program lack definition and

recognition. 

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences

Unsafe Acts

Precondition for Unsafe Acts

Unsafe Supervision

Organizational Influences

Figure 7-5: Latent Failures Contributory to the Absence of Safety Representative
NOAA N-PRIME Mishap Investigation-Final Report 

The following presents the findings for LMSSC, according to the HFACS framework. 

Level 1: UNSAFE ACTS 

Findings: 

The following unsafe acts are determined to be causal to the NOAA N-PRIME mishap. 

L-1) Decision Error: The RTE decided to “assure” the cart configuration through an 

examination of paperwork from a prior operation rather than through physical and visual 

verification. This decision is contrary to the standard operating procedure for this 

operation and to the RTE’s on-the-job training. The RTE had little actual experience 

with preparing the TOC for operations and assumed that using paperwork to assure the 

TOC status was sufficient. 

The RTE made a second decision error in dismissing a comment by the Technician 

Supervisor concerning empty bolt holes during the operation rather than investigating 

further. The rest of the team, following the RTE’s lead, likewise failed to pursue the 

apparent warning. 

L-2) Skill-based Error: The technicians who performed the operations of wiping down the 

TOC adapter plate and bolting down the spacecraft to the adapter plate committed a skillbased error in failing to identify the missing bolts - The technicians, with the exception 

Technician Supervisor noted above, failed to notice the missing bolts, even though they 

were working within inches of where the bolts were supposed to be. This happened 

several times, including when they wiped down the adapter plate with alcohol, lowered 

the spacecraft to the adapter plate, and bolted it down. This is a skill-based error because 

the technicians appeared to work in an “automatized” manner. They were narrowly 

focused on their individual tasks and did not notice or consider the state of the hardware 

or the operation outside of those tasks. Each member of the crew, when interviewed,

commented on the large number of times these procedures had been successfully 

completed in the past. In addition, each technician commented that the RTE had full 

authority and responsibility for the operation and that their role was to follow the RTE’s 

instructions, a culture known to promote the type of channelized attention observed. 

L-3a) Routine Violation: The PQC and the PA signed-off on “assure the configuration” of the 

TOC procedure step without personally validating the TOC configuration or, in the case 

of the PA, even being present at the time this step of the procedures was completed 

during the operation. For both the PQC and the PA, such a validation of the TOC 

configuration minimally should involve visual verification of the attachment bolts. This 

investigation uncovered that such violations (e.g., procedure sign-off without personal 

cognizance) have occurred prior to and during the NOAA N-PRIME mishap. 

L-3b) Routine Violation: The safety representative was not present as called for in the 

procedure. Again, this investigation determines that such a violation is routine. 

Operations often proceed without safety representative presence, contrary to specific 

67 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

procedural requirements. It is unclear whether or not the presence of a safety 

representative would have averted the mishap. Nonetheless, this is a safety related 

mishap; a safety representative is expected play a substantial role in prevention of such 

accidents. 

Statement of Cause: 

These elements described above led the MIB to conclude that decision and skill-based errors and 

routine violations by the NOAA N-PRIME I&T team were manifested as a failure to adhere to 

procedures. 

Level 2: PRECONDITIONS FOR UNSAFE ACTS 

Findings: 

At the second level of the HFACS, the MIB found that the preconditions for unsafe acts that 

were present at the time of the mishap were twofold: substandard condition of the operators in 

the form adverse mental states and substandard practices of the operators in the form of

substandard crew resources management. 

L-4) Adverse Mental States: In the former, adverse mental states, it is apparent that 

complacency was impairment to many on the team directly performing the operation and 

to those providing supervision or oversight to this team. Evidence of complacency was 

seen in both the LMSSC and Government project teams, as the operation was 

consistently characterized as routine and low risk, even though it involved moving the 

spacecraft. Also affecting performance the day of the mishap were adverse mental states 

of channelized attention of the technicians as described above and the feeling of external 

constraints on the RTE, who only signed on for a half day’s duty due to family related 

constraints. Likewise the DMSP technician was scheduled for a half day’s work as he 

was to support another program in the afternoon. Evidence that the operation proceeded 

at an accelerated pace was expressed by the majority of the operations team, although the 

precise reason for the acceleration was varied. It is the MIB’s belief that the RTE was 

anticipating his departure and may have hurried through the procedures. 

Another significant adverse mental state that may have affected the outcome of the 

operation was fatigue of the operators. Although no-one claimed to be fatigued, the 

operation began during a circadian trough (e.g., 6 am), which is known to degrade 

performance. 

L-5) Substandard Crew Resource Management: The MIB found the substandard practice of 

operators referred to as substandard crew resource management was also a precondition 

for the unsafe acts. The MIB found evidence of poor coordination and communication 

among individuals on the crew: 

a. Sharing of GSE between DMSP and TIROS: It was common practice for the 

technicians to informally notify each other of the state of the equipment. In the 

circumstances surrounding this mishap, the DMSP technician who de-configured 

68 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

the TOC did not communicate his actions to the TIROS technicians or RTE, but 

he was not required by the procedure to do so. The informal nature of the 

handling of the GSE does not provide clean, robust hand-offs between operation 

teams. 

b. Red Tagging of TOC: No real formal communication or documentation process 

exists for handling the red tagging, repair and maintenance of ground support 

equipment. Typical aerospace practice for controlling critical GSE requires a 

logbook to track the TOC configuration, repairs and red tagging for any restricted 

use. In this particular case, the repair of the cart did not return it to its full 

capacity—a restriction not communicated beyond the I&T manager (requester of 

repair) and the Technician Supervisor (repairer). 

c. Formation of ad-hoc operations team: Team members were notified as late as 

quitting time on Friday that they would be needed for Saturday operations. Some

personnel, namely the safety representative and the product assurance inspector, 

were not notified at all. Furthermore, the original plan was to switch RTE’s 

midway through Saturday operations, as well as having only one of the 

Technicians for half the day (had to get back to support a DMSP operation). The 

lead technician that would normally have been a member of the team and that 

usually “assures” the TOC for this RTE was absent due to his dispute with the 

RTE that occurred due to the RTE’s request to work on the satellite without the 

proper paper orders 

d. Pre-Operational Safety Briefing: It is standard LMSSC policy, captured in 

paragraph 3.3.1 of the procedure, to require the RTE to conduct a pre-operational 

safety briefing for all personnel involved in a test activity. This ensures everyone 

working on the activity understands what needs to be done and their role in 

accomplishing the work. From the interviews, it was clear this pre-operational 

safety briefing was not conducted with the crew as a group, nor was there 

evidence it was conducted individually. It appears the RTE simply directed the 

crewmembers in what to do and when they should do it. This was true of the PA 

Inspector as well, as he did not arrive until after the TOC had been configured by 

the RTE. 

e. Failure of Leadership: As the operations lead, the RTE failed to exercise proper 

leadership in preparing for the operation and enforcing procedure execution 

discipline during the operation, resulting in: late and poor preparation of test 

documentation, late notification and formation of the crew, lack of notification of 

safety, late arrival of PA inspector, late notification of government QA 

representative and failure to properly verify the TOC configuration. 

f. Technological Environment: The MIB found that the technological environment 

in which the operation was conducted was causal in the mishap. The operating 

procedures called for the RTE to “configure/assure” the TOC, but was silent on 

how this procedural step must be accomplished. The procedure did not specify 

that “assure” must include visual inspection, for instance. In addition, the 

operations team was utilizing a heavily redlined procedure that required 

considerable “jumping” from step to step, and had not been previously practiced. 

The poorly written procedure and novel redlines were preconditions to the 

decision errors made by the RTE. 

69 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Statement of Cause: 

It is the Board’s finding that the adverse mental states of complacency, external constraints and 

channelized attention and the practice of poor communication and coordination among the 

extended operations teams set up preconditions that allowed the NOAA N-PRIME mishap to 

occur. 

Level 3: UNSAFE SUPERVISION FACTORS—LMSSC 

Findings: 

The factors identified in the third level, unsafe supervision that allowed the preconditions and 

unsafe acts of the previous levels, fall into all four subcategories. 

L-6) Planned Inappropriate Operations: The MIB believes that planning for the lift/turnover 

operation was hurried and resulted in a hastily formed operations team. The team as 

constituted for the Saturday morning operation included a Technician Supervisor who 

was acting as part of the technician crew and the I&T manager who was acting as an 

observer. It did not include the most experienced lead technician, due to his dispute with 

the RTE the day before concerning inappropriate requests to work on the satellite without 

written orders. Although all team members were experienced and competent, this 

atypical mix of authority among the various roles created dynamics that were not 

conducive to open discussion and shared responsibility. Moreover, a team member that 

typically would have a leadership role among the technicians was not present during the 

operation, as he had had a conflict with the RTE over the hurried nature of the operations 

the day before. The MIB believes this inappropriate nature of the operation, even though 

planned, had a clear contribution to the conditions surrounding the mishap. This hastily 

planned operation was casual to the substandard crew resource management practices 

described above. 

L-7) Inadequate Supervision: Here, the supervisory inadequacies are related to ensuring that 

operation team members understand and uphold their assigned roles and responsibilities. 

The teaming of an RTE, a lead technician, a PA inspector, a government quality 

assurance representative and other technicians is typical for I&T operations in the 

aerospace industry. Although the roles and responsibilities of the individuals of this 

team, i.e., RTE as the operations lead, lead technician as chief operator, and PA inspector 

as the independent quality assurance agent are generally understood, the dynamics of this 

team are often governed by the personalities and experiences of the individuals, and the 

conditions of the operation. For instance, the I&T manager, present at the operation, 

should have enforced adherence to the procedures. The PA manager should have known 

who was “watching the shop” when he was on leave, and his designee should have 

ensured the presence of the safety engineer and quality inspector. The MIB concluded 

that the lack of clear definition, proper training and reinforcement of the roles and 

responsibilities of these individuals and their lack of enforcement and support by the 

supervisory chain are contributory factors to this mishap. Whether it was personality or 

lack of training, the individuals of the team failed to fulfill their expected roles and 

70 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

responsibilities. Lax or inadequate supervision was a contributing factor to the adverse 

mental states described above. 

L-8) Failure to Correct Known Problems: Given the prevalence of some of the problems 

identified above and through interviews, the MIB concluded that the supervisors failed to 

recognize and correct these adverse trends. In particular, the PA supervisors allowed 

routine conduct of operations with PA inspector sign-off after the fact, routine waiving of 

safety presence and routine late notification of government inspectors. The I&T 

supervisors allowed routine poor test documentation and routine misuse of procedure 

redlines. In addition, similar problems and deficiencies have been documented under the 

contractor’s Non-Conformance Report and the DCMA Corrective Action Requests, and 

the supervisors failed to recognize and correct them effectively. This failure to correct 

known problem is a direct contributor to the substandard crew resource management 

practices described above. 

L-9) Supervisory Violations: As discussed above, rules requiring the presence of quality 

assurance and safety were broken and the procedures for assuring the configuration of the 

cart were not followed, and not for the first time according to interview statements. The 

supervisors present, the RTE and the I&T manager, did not act to prevent this or rectify 

it. 

Statement of Cause: 

The MIB believes that inappropriate planning, in adequate supervision, failure to correct known 

problems, and supervisory violations by LMSSC resulted in improper staffing, poor process 

discipline and disregard for rules during the operation. These factors at the supervision level 

not only did not prevent the conditions or acts that led to the mishap but actively promoted them 

Level 4: ORGANIZATIONAL INFLUENCES- LMSSC 

Findings: 

Organizational influences in the Lockheed Martin Corporation that were at work in the NOAA 

N-PRIME accident include issues in resource management, climate, and processes. 

L-10) Resource Management: The MIB observed an inadequate emphasis on safety within the 

TIROS program. Few resources are allocated to this function and few requirements for 

safety oversight exist. Further, little programmatic supervision was provided for the 

safety representatives. A shrinking of the quality assurance activity was also observed as 

the program work diminished, with three inspectors now being shared between the 

TIROS and DMSP projects. In these two key oversight elements, LMSSC failed to 

provide the appropriate emphasis and corresponding resources. 

71 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

L-11) Organizational Climate: The organizational climate in the I&T domain is dynamic and 

requires flexibility and agility. Last minute changes in schedule, late notification of 

personnel to support activity changes and weekend work are typical. However, within 

POES/TIROS program, an operational program with 14 spacecraft launched, the I&T 

operations associated with the two remaining spacecraft have become routine. This 

perception was shared by almost everyone the MIB interviewed. Though difficult to 

establish clear linkages, the MIB nonetheless believes that the dynamics of the I&T 

function engendered an environment that led to complacent and overconfident attitudes 

toward routine operations and ultimately, reduced process discipline and vigilance on the 

part of the crew. 

L-12) Organizational processes: The final organizational influence the MIB felt adversely 

affected the outcome of the operation was the lack of established and effective process 

guidelines and safeguards for project and program operations. Key processes that were 

found to be inadequate include those that regulate operational tempo, operations 

planning, procedure development, use of redlines, and GSE configurations. For instance, 

the operation during which the mishap occurred was conducted using extensively 

redlined procedures. The procedures were essentially new at the time of the operation - 

that is, they had never been used in that particular instantiation in any prior operation. 

The re-written procedure had been approved through the appropriate channels even 

though such an extensive use of redlines was unprecedented. Such approval had been 

given without hazard or safety analyses having been performed. Similarly, trends in prior 

incidences of non-conformances or mishaps that were the result of wrongly implemented 

or poorly written procedures were not periodically examined by the organization; lacking 

is the organizational oversight to monitor, verify and audit the performance and 

effectiveness of the procedures and activities. These deficient organizational processes 

led to the inappropriately planned operation and failure to correct known problems. 

L-13) Ineffective Safety Program: Finally, throughout this investigation, the MIB finds the 

system safety program to be very ineffective. The current safety program is primarily 

targeted for launch site safety. The system safety program lacks definition and 

organizational recognition. 

Statement of Cause: 

The MIB finds that the limited resources and emphasis dedicated to quality assurance and 

system safety; an unhealthy organizational climate that bred complacency and overconfidence; 

and the lack of effective operations and oversight processes were the root causes of the mishap. 

The MIB finds the LMSSC system safety program, in particular, to be ineffective.

Following HFACS framework, the following identifies the findings for the government. Given 

the government provides an oversight function, only the unsafe oversight and organizational 

influences of the HFACS levels are analysis. 

72 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Level 3: UNSAFE OVERSIGHT FACTORS-Government

Findings: 

The role and responsibilities of the Government in providing oversight and guidance to 

contractor project execution and operations were examined within the HFACS framework. For 

the NOAA N-PRIME mishap, the factors identified include Inadequate Oversight and Failure to 

Correct Known Problems. 

G-1) Inadequate Oversight: In the area of oversight, the MIB found that the government 

quality assurance and safety oversight at GSFC had become issue driven due to the 

maturity of the project. Once issues were brought to their attention, the QA/safety 

personnel worked their resolution but there was very little proactive oversight, audit, 

inspection, etc. of the LMSSC operations. Procedures were rarely reviewed, nonconformances were not trended, and the GSFC personnel rarely made impromptu 

inspections of the LMSSC operations. 

G-2) Limited Quality Assurance Oversight: The MIB also believes the change from processing 

two satellites simultaneously to working on one satellite at a time led to a reduction in 

DCMA and GSFC QAR support. Although this did not change the number of 

Government Mandatory Inspection Points (GMIP), the MIB felt that the quality of

oversight resources may have been impacted. The most significant piece of evidence 

indicative of a failed oversight process and barrier to safety problems was the absence of 

the QAR during the operation that resulted in the mishap. Although his presence may not 

have prevented the mishap, he inappropriately waived a Mandatory Inspection Point. In 

practice, the QAR was assumed to have the authority to act as the DCMA representative 

by the POES Project Office, DCMA, and LMSSC. On the day of the mishap, when 

LMSSC called the QAR, he told them to proceed and he was on his way in. By 

effectively waiving the requirement to witness the spacecraft lift, he failed to enforce a 

DCMA requirement. 

G-3) Limited System Safety Oversight: The MIB found that the government provides very 

limited system safety oversight. Again, safety oversight is predominantly targeted for 

launch site operations. 

G-4) Failure to Correct Known Problems: Supporting the Failure to Correct Problems factor, 

the MIB was told that the some of the problems associated with procedure discipline and 

safety and program assurance oversight were known to the in-house Government QAR 

but were not communicated to the NASA project. Given his prior association with the 

LMSSC project personnel, the Government QAR tended to work the problems directly 

with LMSSC rather than pass on the information. In some cases, known problems were 

not addressed due to familiarity with the LMSSC personnel and reliance on personal 

relationships. Given the prevalence of some of the contractor deficiencies identified in 

this investigation, however, it is the MIB’s assessment that these deficiencies constitute 

problems which the government in-plant representative, DCMA, and the GSFC 

QA/Safety function should have recognized and demanded correction. In particular, 

73 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

these deficiencies have been identified and recorded through the contractor NCRs and the 

DCMA CARs. 

Statement of Cause: 

Both the in-plant government representation and the GSFC QA/safety function failed to provide 

adequate oversight to identify and correct deficiencies in LMSSC operational processes, and 

thus failed to address or prevent the conditions that allowed the mishap to occur. 

Level 4: ORGANIZATIONAL INFLUENCES—GOVERNMENT

Findings: 

Resource management, organizational climate, and processes within the government are also 

implicated as factors for the mishap. 

G-5) Resource Management: the GSFC project, in working to deal with a declining workload 

and resources, allowed and even encouraged trade-offs between the schedules, staffing 

and milestones for the two remaining satellites in the POES (TIROS) project. These 

constant and rapid trade-offs exacerbated the already fast operational tempo of the 

LMSSC I&T team. The MIB believes these frequent trade-offs may have inadvertently 

provided incentives to LMSSC to take on additional risk. Resource management also 

appears to be an issue in other areas: the project has very few resources in the safety area, 

and the assurance manager recently took on an additional project, thus reducing his time

on POES (TIROS). Although in this latter case, an additional quality engineer was 

added, the level of experience of this resource was below that of the assurance manager. 

Insufficient resources also led DCMA to cut back support for weekends. It is the MIB’s 

assessment that the current on-site DCMA and quality assurance staffing levels are 

insufficient to provide effective coverage. 

G-6) Organizational climate was found to be an issue, primarily in the government on-site 

structure. There is no Project in-plant civil servant government presence. The Project inplant government representatives (one in quality assurance, two in I&T) were past 

employees of LMSSC and were hired as outside contractors by the GSFC Project Office. 

With a wealth of relevant experience and familiarity with the inner workings of the 

LMSSC organization, these employees are well qualified for their positions technically. 

However, their past associations with the company might precipitate undue complacency 

due to familiarity. In particular, the QAR was formerly the Product Assurance Manager 

for this facility. His experience and familiarity with the LMSSC staff clearly might 

facilitate problem resolution as they are encountered. This runs the risk however of 

relaxing the proper checks and balances of the independent quality assurance function, 

such as overlooking problems by not properly reporting deficiencies; placing 

inappropriate confidence in former colleagues; or skipping inspection requirements. As a 

particular example in this mishap, the QAR waived his presence at the spacecraft lift. 

The MIB also has misgivings about organizational climate engendered by the complacent 

attitudes toward routine operations. In this instance, the contractor’s emphasis toward

74 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

schedule utilization and casual disregard for process and personal discipline are reflected 

by the government’s lax and casual oversight 

G-7) Organizational Processes: The MIB found no effective guidelines regarding the 

government’s oversight role in I&T operations planning, procedure development, and 

procedure execution discipline. The coordination between contractor and DCMA/QAR in 

preparing for this operation was late and confusing. It is not clear the role that the 

government should play in the review of procedures and test documentation and the 

enforcement of contractor procedure execution discipline. The MIB found no effective 

process in place to follow up on closure of DCMA generated Corrective Action Requests 

(CARs), SAC generated audit deficiencies and action items from an external review 

(TIROS Anomaly Review/Appendix L). Although the POES Project and the contractor 

track and trend closure of contractor generated Non-Conformance Report (NCRs) for 

timeliness, there is no process in place to analyze and trend NCRs for cause and to 

identify systemic problems. Likewise lacking is the government organizational oversight 

to monitor, verify and audit the performance and effectiveness of the I&T processes and 

activities. 

G-8) The Government System Safety Program placed an overemphasis on launch site safety 

and inadequate in-plant system safety. The LMSSC Safety Plan had not been updated for 

many years, with little or no change since the move to LMSSC at Sunnyvale. In addition, 

no Operational Hazard Analysis or similar analysis of potential in-plant hazards was ever 

performed. Such an analysis would identify the procedures and operations that are 

considered hazardous and appropriate mitigation measures implemented. 

G-9) The MIB found the DCMA CAR Assessment and Reporting Process and other DCMA 

audit processes to be deficient in identifying troubling trends in the LMSSC facility. 

Review of CARs indicated repeated requirement violations and bypassing of Mandatory 

Inspection Points by the contractor. The DCMA Technical Assessment Group (TAG) 

facility audits, the DCMA annual safety audits, and the DCMA facility summary reports 

of CARs prior to the mishap, however, all indicated a healthy facility environment, with 

no noteworthy problems reported. 

Statement of Cause: 

The MIB finds inadequate resource management, an unhealthy organizational climate and the 

lack of effective oversight processes led to inadequate QA and system safety oversight, casual 

oversight discipline, and inadequate follow-up to monitor contractor performance deficiencies 

within the I&T environment. 

Tables 7-1 and 7-2 summarize the findings for LMSSC and the Government. 

75 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 7-1: Findings Summary (LMSSC) 

UNSAFE ACTS 

L-1) Decision error: The RTE committed a “decision error” by not following procedures. 

L-2) Skill based error: Technicians committed “skill based error” by not noticing the missing bolts 

 while wiping down TOC adapter plate and bolting down the spacecraft. 

L-3a) Routine violation: PQC and PA inspector committed “routine violations” by signing-off on 

 operations without witnessing and verifying TOC configuration. 

L-3b) Routine violation: Safety representative, as required by the procedure, was not present. 

PRECONDITIONS FOR UNSAFE ACTS

L-4) Adverse mental states in the forms of “channelized attention”, “complacency”, and “external 

 constraints” resulted in an accelerated pace of operations and procedure execution. 

L-5) Substandard crew resources included poor hand-off between the DMSP and TIROS Projects 

 regarding the TOC, poor red-tagging process for GSE, late identification of personnel to work 

 Saturday, lack of pre-operational safety briefing, and lack of leadership on the part of the RTE. 

UNSAFE SUPERVISION

L-6) Planned inappropriate operation: The team was formed late in a harried fashion with an atypical 

 mix of personnel and poor test documentation. 

L-7) Inadequate supervision was manifested in the lack of clear definition and enforcement of roles 

 and responsibilities among the team individuals, consequently individuals failed to fulfill their 

 expected roles and responsibilities. 

L-8) Failure to correct known problems was a supervisory failure to correct similar known problems. 

 PA supervisors routinely allowed PA inspector sign-off after the fact. I&T supervisors routinely 

 allowed poor test documentation. 

L-9) Supervisory violation was committed by repeatedly waiving required presence of quality 

 assurance and safety and bypassing Government Mandatory Inspection Points. 

ORGANIZATIONAL INFLUENCES 

L-10) In Resource management, MIB observed inadequate emphasis on safety, and inadequate quality 

 assurance support to provide effective coverage. 

L-11) The Organization climate in the I&T domain with an operational program has engendered an 

 environment that led to complacent and overconfident attitudes toward routine operations.

L-12) Lack of effective “Organizational processes” in the form of guideline and safeguards to regulate 

 the I&T environment. Lack of organizational oversight to monitor, verify and audit the 

 performance and effectiveness of the procedures and activities. 

L-13) Ineffective System safety program. 

76 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 7-2: Findings Summary (Government) 

UNSAFE OVERSIGHT

G-1) Inadequate Oversight: The government quality assurance and safety provided inadequate 

 oversight. Oversight function became issue driven. Procedures rarely reviewed; non- 

 conformances not trended; rarely make impromptu inspections. 

G-2) Inadequate QA Oversight: In substituting for the DCMA, the QAR failed to enforce a 

 Government Inspection Point by failing to enforce his presence at the operation.. 

G-3 Limited System Safety Oversight: Government has very limited safety oversight. 

G-4) Government “Failed to correct known problems”, such as PA signoff after the fact and poor 

 test documentation. 

ORGANIZATIONAL INFLUENCES

G-5) Deficient “Resource management” include: rapid trade-offs between the schedules, staffing 

 and milestones for the two remaining satellites exacerbated the already fast operational 

 tempo of the LMSSC I&T team; lack of resources in the safety area; insufficient DCMA 

 and quality assurance staffing to provide effective coverage. 

G-6) “Organizational climate” factors include: On-site organizational structure of using retired 

 LMSSC employees as government representatives; lax and casual oversight toward an I&T 

 environment with routine operations. 

G-7) Lacks Government oversight guidelines regarding I&T operations planning, procedure 

 development, and procedure execution discipline. Lacks Organizational processes to 

 effectively monitor, verify, and audit the performance and effectiveness of the I&T 

 processes and activities. 

G-8) The Government safety program placed an overemphasis on launch site safety and 

 inadequate in-plant safety. 

G-9) Deficient DCMA CAR assessment and reporting processes. 

77 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 8 

RECOMMENDATIONS 

Although the unsafe act of failure to follow procedures by the individuals of the NOAA N-Prime 

I&T team was the immediate cause of the mishap, it is the assessment of the MIB that 

deficiencies at the process, supervisory and organizational levels were contributory factors. 

These deficiencies within the program are wide in scope and are rooted in failure to follow sound 

engineering processes and lack of effective supervisory and organizational practices. This 

section identifies the MIB recommendations to remedy these deficiencies. These 

recommendations include basic process improvements for procedure development, review and 

approval; establishment of configuration management for ground support equipment; sensitizing 

individual, supervisory and organizational awareness to process discipline; and enforcement of 

rules and regulation through training programs and improvements in organizational structures 

and practices, for the contractor LMSSC and for the government. Sections 8.1 and 8.2 present 

the MIB recommendations for LMSSC and the government, respectively. 

8.1 Recommendations-LMSSC 

Natural fallout of the HFACS analysis exercise is the linkage between the findings or 

deficiencies among the four levels of human failures. With active or latent failures (i.e., lack of 

effective safeguards), the HFACS provides a comprehensive framework to categorize these 

deficiencies and to formulate intervention strategies. The following present the MIB 

recommendations for the LMSSC organization and correlate the associated findings under the 

HFACS framework. 

1) Provide a formal training program for certifying all test conductors and for training all I&T 

personnel in their roles and responsibilities. Provide periodic refresher training to reinforce 

these roles and responsibilities 

This recommendation addresses the deficiencies identified in findings L-1, 2, 3a, & 4. 

As the immediate cause of the mishap, individual process discipline must be restored at the 

organizational level. In particular, organizational practices must be established to reinforce 

the role and responsibility of contractor and government inspectors as independent 

verification agents, serving as a safety net for the individuals executing the procedure. 

Procedure execution discipline must be enforced and operational signoff without personal 

cognizance must be banned. This recommendation directly addresses the organizational 

deficiency that the lack of clear definition of the roles and responsibilities for the individuals 

in the I&T structure was a significant causal factor leading to the decision errors and routine 

violations of the PQC and the PA. Periodic refresher training would serve to reinforce these 

roles and responsibilities and to dispel any complacency from settling in over time. 

To serve in the leadership role, the test conductor or RTE must be certified to have the 

appropriate qualifying experience as well as the supervisory discipline and training. As test 

78 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

conductor, the RTE must be a role model of process discipline and safety consciousness for 

the rest of the team. 

Central to this training is the emphasis that safety is everyone’s responsibility, where there is 

no such thing as a routine operation from the safety perspective. Skill based errors (e.g., not 

noticing the missing bolts) as committed by the technicians were preventable. While focus 

must be exercised to perform the task at hand, circumspection must also be exercised to 

discern extraordinary signs of hazard or other untoward events. 

2) Provide supervisory training to promote an active supervisory role in identifying, monitoring 

and correcting poor processes discipline and other deficiencies. 

This recommendation addresses the deficiencies identified in findings L-6, 7, 8, and 9 

 

Given the prevalence of the deficiencies identified through this investigation, it is the MIB’s 

assessment that the supervisors were aware of these deficiencies and failed to correct them. 

The PA and I&T supervisory chain should take a more active role in monitoring and 

correcting the deficiencies identified by the Non-Conformance Reports as well as the 

Corrective Action Requests. Inadequate supervision permitted the planning of an 

inappropriate operation in a harried fashion with an atypical crew mix and poor 

documentation. In addition, supervisory violation was committed in failure to properly 

notifying the safety representative and quality assurance support as required. Supervisors 

must set an example and enforce standards of process discipline. This training should 

address all supervisory deficiencies identified. 

3) Establish effective process guidelines and safeguards to regulate the I&T environment. 

Process guidelines and safeguards should be developed for operations planning, procedure 

development, redlining, procedure execution discipline, GSE configuration management. 

This recommendation addresses the deficiencies identified in findings L-5, 6, 11 & 12. 

The basis of poor crew resources management (finding L-5), planned an inappropriate 

operation (finding L-6), and poor process discipline due to adverse organizational climates 

(L-11) are the results of deficient processes and their enforcement. This recommendation 

addresses the organizational deficiency that the lack of effective operational processes for the 

I&T environment led to inappropriately planned operations and specific poor I&T practices 

and disciplines. Effective guidelines and safeguards should serve to discourage tendencies 

toward compromising process discipline under the influence of the dynamics of an I&T 

environment and the complacency of an operational system. 

a) Operations Planning- Establish clear guidelines for the assembly and notification of an 

I&T team in support of an operation. The team must be of proper make-up, experience 

and training. 

b) Procedure Development- A major MIB finding is the lack of sound procedure 

development, redlining and execution practices. The MIB recommends that all 

procedures be reviewed for clarity and completeness, to include: 

79 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

• Define intended actions for ‘assure’, ‘verify’ and other interpretable terms. 

• Conduct a systematic review and identification of hazards and safety critical 

operations. Perform an operational FMEA as part of this process. 

• Uniquely identify hazardous and critical operations, for the overall procedure and 

specific steps. 

• Provide emergency instructions when operations are capable of causing personnel 

injury or equipment damage if not expeditiously shutdown, safed, or secured should a 

malfunction occur. 

• Include a mandatory pre-operational briefing with a checklist and step in the 

procedure. 

• Provide verification that all constraints for an operation are closed with a separate 

step in the procedure. 

• For repeatable procedures, provide a sign off/buy off for each step. 

c) Redlining- The MIB recommends a review of the redlining process and training of 

personnel on purpose and usage, to include: 

• Clarify when redlines are appropriate. 

• New procedures should not be developed as a redline. 

• Sign off on redlines should require review by other than the operators conducting or 

witnessing the operation. 

d) Procedure Execution Discipline- The MIB recommends improvements in procedure 

execution discipline, to include: 

• Ban operation signoff without personal cognizance 

• Ensure procedure instructions are clearly communicated and understood by all 

personnel directly involved with the operation 

• PA, Safety and Government sign off should not be allowed to be waived without 

approval of Project Office 

e) GSE Configuration Management- Establish an effective GSE configuration management 

program to provide: a) centralized GSE configuration management, maintenance and 

repair record keeping; b) verification of maintenance and repair for GSE; and c) 

centralized control of GSE shared by different programs.. 

4) Review and staff PA and safety support according to requirements. It is the MIB’s 

assessment that the current PA and safety staffing level is insufficient to provide effective 

coverage. 

 

This recommendation addresses the organizational deficiency of resource management 

identified in finding L-10. 

5) Establish an effective safety program with a well-defined system safety policy and 

mandatory requirements. Safety awareness must be promoted to all levels of the 

organization through a training program or a training module within other applicable training 

programs. Safety needs to be an integral part of the organization and operation at Sunnyvale. 

This recommendation addresses the deficiency identified in finding L-3b and L-13. 

80 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

This recommendation addresses the deficiency that at all four levels of the HFACS analysis, 

some aspects of the lack of safety awareness, support, supervision, guidelines, and 

requirements were identified as contributing factors to this mishap. The current safety 

program is ineffective as it is loosely structured, lacks definition and the level of support is 

resource dependent. 

6) Establish an effective monitoring, trending, verification and audit program to manage the 

performance and deficiencies of the I&T activities. 

This recommendation addresses the deficiencies identified in finding L-8&12. 

This recommendation addresses the lack of effective organizational oversight to manage the 

performance and risks associated with the I&T environment. In particular, problems and 

deficiencies identified by the contractor and the government were not examined for effective 

closure. Though problems that had significant consequences were addressed promptly (as in 

this mishap, the LMSSC AIT was convened promptly), there is not an effective program to 

track, examine, categorize and trend the nature and closure of the contractor identified nonconformance reports (NCRs) or the DCMA generated Corrective Action Requests (CARs). 

These NCRs, CARs as well as other lesser accidents and near misses should be monitored, 

trended and evaluated for clues of foreshadowing systemic problems. This program should 

also monitor and track the closure of actions, deficiencies, and recommendations resulted 

from outside audits and reviews, such as this mishap investigation, the TIROS Anomaly 

Review, and the Supplier Assurance Contract (SAC) assessments. 

Complementing the monitoring and trending program, an effective program must be 

established to verify and audit compliances to LMSSC Command Media or other established 

processes or guidelines. To counteract complacency and lapses into poor practices, periodic 

audits of processes and activities must be conducted to prevent deviation from command 

media and to demonstrate compliance to new requirements. 

7) Closed Circuit Video Monitoring- It is the MIB’s assessment that closed circuit video

monitoring within the I&T area is an effective aid for promoting process discipline, 

improving safety awareness, facilitating effective oversight, and investigating mishaps. 

Closed circuit video monitoring within the I&T area is recommended. 

This recommendation addresses deficiencies identified in findings L-7 and L-12. 

8) Establish a training program to disseminate lessons learned from this and other mishaps. 

This program would constitute an important supplement to safety awareness. This also 

constitutes and the organizational oversight 

This recommendation addresses the deficiencies identified in findings L-1, 2, 3, 4, 12. 

Table 8-1 summarizes the MIB recommendations for LMSSC and the associate findings that the 

recommendation is intended to remedy. 

81 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 8-1: Summary of Recommendation for LMSSC and Corresponding Findings

Recommendations Findings

1) Provide a formal training program for certifying 

all test conductor and for training all I&T 

personnel of their roles and responsibilities. 

Provide periodic refresher training to reinforce 

these roles and responsibilities 

L-1) RTE decision error 

L-2) Technician skill-based error 

L-3a) PA and PQC routine violations 

L-4) Adverse mental states 

2) Provide supervisory training to promote an active 

supervisory role in identifying, monitoring and 

correcting poor process discipline and other 

deficiencies.

L-6) Planned inappropriate operation 

L-7) Inadequate supervision 

L-8) Failure to correct known problem 

L-9) Supervisory violation 

3) Establish effective process guidelines for 

regulating the I&T environment, including 

operations planning, procedure development, 

redlining, procedure execution discipline, and 

configuration management 

L-5) Substandard crew resources 

management

L-6) Planned inappropriate operation 

L-11) Organizational climate 

L-12) Lack of organizational process to 

regulate the I&T environment 

4) Review and staff PA and safety personnel 

support according to requirements. 

L-10) Resource management: limited 

quality assurance and safety support 

5) Establish an effective safety program with a welldefined system safety policy and mandatory 

requirements. Safety awareness must be 

promoted to all levels of the organization through 

a training program or a training module within 

other applicable training programs. 

L-3b) Routine violation: absence of 

safety 

11) Organizational climate 

13) Ineffective safety program

6) Establish an effective monitoring, trending, 

verification and audit program to manage the 

performance and deficiencies of the I&T 

activities. 

L-8) Failure to correct known problems 

L-12) Lack of organizational processes to 

monitor, verify and audit the performance 

and effectiveness of the procedures and 

activities 

7) Closed Circuit Video Monitoring as an aid to 

supervision and promote performance monitoring

L-7) Inadequate supervision 

L-12) Lack of organizational processes to 

monitor, verify and audit the performance 

and effectiveness of the procedures and 

activities 

8) Establish a training program to disseminate 

lessons learned from this and other mishaps 

L-1) RTE decision 

L-2) Technician skill-based errors 

L-3) PQC and PA routine violations 

L-4) Adverse mental states 

L-12) Organizational oversight 

82 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

8.2 Recommendations-Government 

The lack of effective government supervision or oversight cannot be overlooked in this mishap. 

In particular, as with the contractor quality inspectors, the roles and responsibilities of 

government inspectors as independent verification agents must be reinforced. The MIB 

recommends the following to improve the government oversight effectiveness: 

1) Provide a dedicated, full time government in-plant representative as indication of 

commitment and support. Given the common work-areas and equipment, the MIB 

recommends this individual be shared with the DMSP program. It would be left to the 

discretion of the project office whether this individual would be a civil servant assigned from

the project office, or a civil servant drawn from the DCMA staff. Since this individual will 

oversee all in-plant contract staff, the MIB recommends this position be evaluated for a 

higher grade level than is currently authorized. 

This recommendation addresses the deficiency identified in finding G-6. 

This recommendation addresses the government in-plant organizational structure weakness 

of having only former LMSSC contract employee on staff. A civil servant in-plant 

representative, serving as the lead for the in-plant staff, would strengthen the authority of the 

office and would provide more effective supervision of the in-plant staff. 

2) Establish and document clear roles and responsibilities for DCMA and the contractor in-plant 

representatives. Review and revise DCMA Letter of Delegation and the government in-plant 

representative Letter of Assignment. 

This recommendation address the deficiencies identified in finding G-2 and G-6. 

This recommendation addresses the government in-plant structure’s apparent lack of a clear 

definition of the roles and responsibilities of the in-plant representative as an independent 

government quality assurance agent, and in particular as a DCMA substitute. In the instance 

of this mishap, the QAR failed to be fulfill his obligation as the DCMA representative and 

failed to be present at the spacecraft lift operation. 

3) NASA should provide sufficient resource for DCMA to effectively implement the plan to 

fulfill the Letter of Delegation. The NASA Project should provide sufficient on-site quality 

assurance support to effectively implement the plan to fulfill the Letter of Assignment. It is 

the MIB’s assessment that the current on-site DCMA and quality assurance staffing levels 

are insufficient to provide effective coverage. 

This recommendation addresses the deficiency identified in finding G-5. 

This recommendation addresses the organizational deficiency of resource management, and 

consequently providing inadequate oversight. 

83 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

4) Establish effective oversight guidelines and safeguards for the I&T environment. Effective 

oversight guidelines should be developed for operations planning, procedure development, 

and procedure execution discipline. 

This recommendation addresses the deficiencies identified in findings G-1, 4, 6 and 7. 

Effective government oversight guidelines should be established to eliminate the current 

practice of issue driven mode of oversight (finding G-1), to correct known contractor 

problems (finding G-4), and to counterbalance the dynamics of the I&T culture and the 

complacency of an operational system (finding G-6). 

a) Operations Planning- Establish government oversight guidelines relative to the 

government’s role in operations planning: 

• Clarify and establish guidelines for government notification of operations planning 

activities and review of test documentation. 

• Establish clear communications protocol between the contractor and government of 

planned activities, schedule and support requirements. There was clear indication of 

schedule confusion relative to this mishap. 

b) Procedure Development- Establish government oversight guidelines relative to review of 

procedures: 

• Clarify and establish guidelines for government review of procedures. 

• Institute government review and sign off of critical procedures. 

c) Procedure Execution Discipline- Enforce procedure execution discipline: 

• Ban operation signoff without personal cognizance. 

• Government inspections should not be waived without the consent of the government 

project System Assurance Manager (SAM) 

5) Establish an effective safety oversight program. Ensure that the System Safety Program Plan 

is updated and hazard analysis is performed. Increase safety oversight and provide safety 

training. 

This recommendation addresses the efficiencies identified in findings G-3, 5 and 8. 

The government has the same deficiency as the contractor in the lack of an effective system

safety oversight program. The government must set the pace in safety consciousness. It is 

not acceptable that safety support is tailored to resources available. 

6) Coordinate with the contractor to implement an effective oversight program to monitor, 

trend, verify and audit the contractor performance and deficiencies. 

This recommendation addresses the deficiencies identified in findings G-4 and 7. 

 

The government should coordinate with the contractor to implement an effective program to 

track, examine, categorize and trend the nature and closure of the contractor identified non84 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

conformance reports (NCRs) or the DCMA generated Corrective Action Requests (CARs). 

These NCRs, CARs as well as other lesser accidents and near misses should be monitored, 

trended and evaluated for clues of foreshadowing systemic problems. This program should 

also monitor and track the closure of actions, deficiencies and recommendations from outside 

audits and reviews, such as this mishap investigation, the TIROS Anomaly Review, and the 

Supplier Assurance Contract (SAC) assessments. 

The government should audit contractor process and procedures to assure contract 

compliance and compliance with local requirements. 

7) DCMA to evaluate the effectiveness of their assessment processes that seem to have missed 

identifying unhealthy trends and potential systemic problems. Formulate corrective 

measures. 

This recommendation addresses the deficiency identify in finding G-8. 

Review of CARs indicated repeated requirement violations and bypassing of Mandatory 

Inspection Points by the contractor. The DCMA Technical Assessment Group (TAG) 

facility audits, the DCMA annual safety audits, and the DCMA facility summary reports of 

CARs prior to the mishap, however, all indicated a healthy facility environment, with no 

noteworthy problems reported. 

8) Periodic independent reviews should be conducted to review Project status and performance. 

This recommendation addresses the deficiencies identified in finding G-7. 

 

This is just another aspect of providing oversight to an operational program. For an 

operational project such as POES, without the vigorous review schedule and process of 

developmental programs, such periodic reviews are essential to provide an independent 

assessment of performance and to combat lapses of complacency. 

Table 8-2 summarizes the MIB recommendations for the government and the associate findings 

that the recommendation is intended to remedy. 

85 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

Table 8-2: Summary of Recommendation for the Government and Associate Findings

Recommendations Findings

1) Provide a dedicated, full time government inplant representative as indication of commitment

and support. 

G-6) Organizational structure of using 

retired LMSSC personnel as government 

representative 

2) Establish and document clear roles and 

responsibilities for the contractor in-plant 

representatives. Revise Letter of Delegation for 

DCMA and Letter of Assignment for QAR. 

G-2) Inadequate QA oversight: Absence 

of QAR at the operation. 

G-6) Organization structure: Lack of clear 

definition of roles and responsibility of 

the QAR. 

3) Provide sufficient resource for DCMA to fulfill 

the Letter of Delegation and for the on-site 

government support to fulfill the Letter of 

Assignment

G-5) Resource management: Insufficient 

DCMA and on-site QA to provide 

effective coverage. 

4) Establish effective oversight guideline for I&T 

operations planning, procedure development, and 

procedure execution discipline. 

G-1) Inadequate oversight 

G-4) Failure to correct known contractor 

problems. 

G-6) Adverse organizational climate 

G-7) Lack of government oversight 

guidelines regarding I&T operations 

planning, procedure review, and 

procedure execution discipline. 

5) Establish an effective safety oversight program. 

Ensure Safety Program Plan is updated and 

hazard analysis is performed. Increase safety 

oversight and provide safety training. 

G-3) Limited safety oversight 

G-5) Lack of resources in safety 

G-8) Ineffective safety program

6) Coordinate with the contractor to implement an 

effective oversight program to monitor, trend, 

verify and audit the contractor performance and 

deficiencies.

G-4) Failure to correct known problems 

G-7) Lack of organizational process to 

monitor, verify and audit the performance 

and effectiveness of the procedures and 

activities. 

7) DCMA to evaluate the effectiveness of their 

assessment processes and formulate corrective 

measures. 

G-9) Deficient DCMA CAR assessment 

and reporting process 

8) Periodic independent reviews should be 

conducted to review Project status and 

performance 

G-7) Lack of organizational process to 

monitor, verify and audit the performance 

and effectiveness of the procedures and 

activities. 

86 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 9 

OBSERVATIONS 

Proprietary Information 

87 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 10 

ACRONYMS LIST

AESOP Assignment, Equipment, Situation, Obstacle, and Personnel 

AIT Accident Investigation Team

ARC Ames Research Center 

C/A Corrective Action 

CAR Corrective Action Request 

CM Configuration Management 

DCMA Defense Contractor Management Agency 

DMSP Defense Meteorological Satellite Program 

ESD Electrostatic Discharge 

FMEA Failure Mode Effects Analysis 

GMIP Government Mandatory Inspection Points 

GOES Geostationary Operational Environmental Satellite 

GOVT Government

GSE Ground Support Equipment 

GSFC Goddard Space Flight Center 

HFACS Human Factors Analysis and Classification System 

I&T Integration and Test 

LEO Low Earth Orbit 

L/V Launch Vehicle 

LMC Lockheed Martin Corporation 

LMSSC Lockheed Martin Space Systems Company 

LOO Log Of Operations 

MH Mechanical Handling 

MHS Microwave Humidity Sounder 

MIP Mandatory Inspection Point 

MIB Mishap Investigation Board 

MOU Memorandum of Understanding 

MSPSP Missile System Pre-launch Safety Package 

NCR Non-Conformance Report 

NESDIS National Environmental Satellite, data and Information Service 

NOAA National Oceanic and Atmospheric Administration

NPD NASA Program Directive 

NPG NASA Program Guideline 

PA Product Assurance 

POES Polar Operational Environmental Satellite 

PQC Product Quality Control 

QA Quality Assurance 

QAR Quality Assurance Representative 

QE Quality Engineer 

RTE Responsible Test Engineer 

SAC Supplier Assurance Contract

88 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

S/C Spacecraft 

SEU Single Event Upset 

SV Sunnyvale 

TAG Technical Assessment Group 

TI Test Instruction 

TI-MH Test Instruction for mechanical handling procedure 

TI-NSET Test Instruction for setting up spacecraft for electrical testing 

TI-PNL Test Instruction for opening up spacecraft panels 1& 4 

TIROS Television Infrared Observational Satellites 

TOC Turnover Cart 

TP Test Procedure 

TPCN Technical Procedure Change Notice 

USAF SMC United States Air Force, Space and Missile Systems Center 

89 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

SECTION 11 

REFERENCES LIST

Mishap Investigation Board Documents

1) Background Briefing to the Mishap Investigation Board for NOAA N-PRIME; by POES 

Program Manager, Karen Halterman, Sept. 12, 2003 

2) NOAA N-PRIME MIB Preliminary Summary of Interviews; by T. Panontin & Co. Sept. 16, 

2003 

3) NOAA N-PRIME Mishap Interviews 

4) NOAA Guidelines to NASA; Wilfred Mazur, Sept. 23, 2003 

5) Briefing Package “NOAA N’ Investigation Interim Report, 11/04/03” 

6) Briefing Package “NOAA N’ Investigation Interim Report, 12/09/03” 

7) MIB Activities Summary 

8) MIB Action Items 

9) MIB Action Item Responses 

LMSSC Documents

10) Configuration Management Plan for TIROS; 2308226, Rev J, June 2001 

11) Performance Assurance Implementation Plan for NOAA-K,L,M; 2629700, Rev B, May 1997 

12) Quality Assurance Program Plan for NOAA-K,L,M; 3267412, Rev B, May, 1989 

13) Quality Assurance Program Plan for NOAA-K,L,M; 3267412, Rev E, Sept. 2001 

14) Reliability Program Plan for NOAA-K,L,M; 3267411, May, 1989 

15) System Safety Implementation Plan for NOAA-KLM Satellites; 3267408, Rev B, October 

1994 

16) Handling Procedure, K,L,M Spacecraft and Components; MH 3278200, Rev A, March 1996 

17) Test Instruction, Mechanical Handling for NOAA-N,N’; TI-MH-3278200, Rev B, Mar. 2003 

17-N) TI-MH-3278200; as run procedure for N 

17-N’) TI-MH-3278200; as run procedure for N-PRIME 

18) NOAA-N’ as run Test Documentation 

19) TIROS N-PRIME Accident Investigation Final Report; October 7, 2003 

20) TIROS NOAA-N’ Accident Investigation Team (AIT) Out-brief; October 7, 2003 

21) Briefing Package “TIROS Program Response to NASA NOAA N-PRIME Investigation 

Team Requests”; by A. Lauer, Sept. 16, 2003 

22) Photos of Mishap 

NASA/NOAA/DCMA Documents

23) Performance Assurance Requirements for the NOAA-KLMNN’ Satellites; GSFC S-480-

26.1, Nov. 1994 

24) GSFC Specification for Reliability and Quality Assurance for the TIROS Program, NOAAHIJ; GSFC S-480-4 B, July 1981 

90 

September 13, 2004 
NOAA N-PRIME Mishap Investigation-Final Report 

25) NASA Procedures and Guidelines for Mishap Reporting, Investigation, and Record Keeping; 

NPG 8621.1, June 2000 

26) Management of Government Quality Assurance Functions for NASA Contracts; NHB 

5300.4(2B-2), April 1993 

27) Letter of Contract Delegation; to DCMA from GSFC; June 2000 

28) Government Inspection Requirements; to LMSSC from DCMA; August 2001 

29) Email Exchanges between GSFC and DCMA regarding weekend support; Jan. 2003 

30) Letter of Assignment for GSFC Quality Assurance Representative; by GSFC, Jan. 2000 

31) POES GFE Assurance Review Report; 480/54021-00/WD, Feb. 2000 

32) Supplier Assurance Assessment Report on LMSSC; Supplier Assurance Contract, May 2002 

33) POES Project Briefing Package to MIB (Topics on Organization, Resident Office, LMSSC, 

Event History, Award Fees, Sherman-Scolese Action Plan, QA Support, Risk Tracking Log, 

MIB Action Item Responses); Sept. 2003 

34) LMSSC Performance Evaluation Board Letters; Sept. 2002-June 2003 

35) POES Integration & Test Schedules 

36) POES Program Schedules; August 31, 2003 

37) POES Program Plan; Sept. 2000 

38) POES Program Contract Modifications; Sept. 2002- Sept. 2003 

DMSP Documents

39) DMSP Deficiency Reports; July-September, 2003 

40) DMSP Program Management Review and Award Fees; 2001-2003 

Other Documents

41) The Human Factors Analysis and Classification System-HFACS; Shappell & Wiegman, 

DOT/FAA/AM-00/7, Feb. 2000 

42) TIROS Anomaly Review, Final Presentation; Sherman & Scolese, June, 2001 

43) Preventing Human Error Study Guide; Prevention Institute, 2000 

91 

September 13, 2004