**EO-1 Anomaly Resolution Report for WARP Anomaly on 6/21/01**

**1. Introduction**

The Wide-band Advanced Recorder Processor (WARP) on-board the EO-1
spacecraft went into an anomalous state at the end of the day on June
21, 2001 (GMT 2001:172:23:32). The WARP was reporting Error Detection
and Correction(EDAC) uncorrectable errors which continued until a reset
of the WARP was implemented on the afternoon of June 29, 2001 (GMT
2001:180:20:00). The errors went undetected by the Flight Operations
Team because limit checking on the EDAC uncorrectable counter had been
eliminated. The presence of uncorrectable errors on the WARP caused the
science data taken during the anomaly to be mostly unusable.

**2. Timeline of Events**

172:23:32 GMT - WARP began to report EDAC uncorrectable errors.

173 - 178 GMT - WARP dumps received at Ground Stations show no sign of
problems,...the Reed-Solomon encapsulation accurately delivered
corrupted science data it was given on-board to the ground without
telecommunications errors. FOT Console Operators are not shown limit
violations or other ground system reporting showing a problem with the
data.

178:13:00 GMT - Data Processing personnel report sequence breaks in
science data from Ground Station tapes.

178:17:00 GMT - WARP engineers are assembled to diagnose the problem.

179:13:00 GMT - WARP engineers build decision tree to guide diagnostics
actions.

179:17:00 GMT - WARP team requests FOT perform a memory test diag which
includes a memory format as part of the procedure. The problem clears.

179:19:00 GMT - WARP team performs fill data test to prove WARP is back
on-line.

180:03:00 GMT - FOT successfully dumps a DCE at GSFC Ground Station.

**3. Resolution Team Members**

Convened Team on June 28, 2001 and also met on June 29, 2001, July
17,2001, and August 13, 2001.

Team Lead: Dan Mandl, EO-1 Mission Manager

Participant Org Expertise

======================================

Terry Smith GSFC WARP

Evan Webb GSFC WARP

Mike Cunningham Litton WARP

Chris Dailey Litton WARP

Steven Slegel GSFC WARP

Thomas Moore Honeywell FOT/Technical Lead

Seth Shulman CSC FOT/Navigation

Rob Bote Honeywell FOT/Mission Planning

Larry Alexander Compaq EO-1 Data Processing Lead

**4. Analysis of Cause**

The anomalous state went undetected until data tapes from the Ground
Stations were received at the Mission Operations Center (MOC) and
analyzed by Level Zero Data Processing System (DPS) personnel. Tapes
from June 21-June 22 began L0 processing on June 27 and the presence of
numerous sequence breaks within all files were noted on June 28. Early
on June 29, WARP engineers came up with the following preliminary
diagnosis:

The errors occur only on Memory Card #2 (outermost card)

The errors occur on all 6 of the 4-Mbit arrays.

The errors are recurrent

There are over 200,000 errors per playback set (about 20% of the data)

The errors appear to occur in 80 byte blocks

The engineers prepared an operations instruction to read the WARP Memory
Mask register. This test showed that there were no signs of corruption
in the register value. Next they ran a WARP Memory Built-In-Test on
Memory Card #2 in the range mode. During the course of this test, the
WARP memory is reformatted. After this, they ran the DCE Self-Test
(RS-422 Card Data Injection) which generates card test data. It then
became evident that the WARP problem had disappeared,...the WARP
returned to a nominal state with no uncorrectable errors. The WARP team
filled the entire memory (48 Gbits) and monitored the EDAC errors on
playback to prove the return to nominal state.

The WARP team attributed the cause of the problem to a stuck bit within
a state machine inside one of the memory boards. The team reasoned that
the problem did not appear to be hardware related, so if the problem
occurred again, it would probably not occur in the same way or the same
location. In fact, there are no mechanisms within the WARP to isolate
the state of state machines at failure, so it would be almost impossible
to plan a diagnostic set of dumps to be taken in the event of another
failure.

The problem has not reoccurred since then.

**5. Resultant Actions**

An internal review was conducted of all the WARP telemetry parameters
and limit settings to assure that the correct parameters are in fact
being monitored with the right limit values. The limit setting of 1 was
instituted for the value of the uncorrectable EDAC error counter (i.e.,
if the value=1, then the limit is violated and reported to the console
operators).

It was originally thought that we should perform a routine reformat of
the WARP memory as a precaution (either once per day or after every
sequence of predetermined DCEs). However, this would preclude
troubleshooting if the error reoccurred, so the deletion of files after
downlink from the ATS load was retained as the nominal operational mode
that had been used since early in the mission.

Since then, the WARP has been reformatted three times: once during the
ACE safehold anomaly (9/14/01); once when an operator error resulted in
entry into low power mode by the WARP internal protection mechanisms;
and once for the weekend shutdown surrounding the Leonids meteor shower.
The operator error has been documented in a Root Cause and Corrective
Action report and resolved by additional training.

**6. Specific CCB Actions and Schedule**

The limit for WRMEDACECU=1 for checking uncorrectable EDAC errors was
implemented on July 12, 2001 under DBCR \# 458.