**PART 12. FAILURES & ANOMALIES**

**1. INTRODUCTION**

During the course of the EO-1 mission lifetime, there were only six
failures and anomalies that occurred. These six occurrences are
described below.

**2. ALI FOCAL PLANE CONTAMINATION**

Contamination of the focal plane by an unknown substance was recognized
in October 1998 during the characterization and calibration of the ALI
at Lincoln Laboratory. This contaminant may be completely eliminated by
raising the temperature of the focal plane above 260 K. Prior to launch,
several bake outs occurred in an attempt to eliminate the source of the
contaminant. In January 1999, the entire instrument was baked out at
303K while under vacuum for one week and then later for an additional
two days. The focal plane was also baked out for three hours at 273 K in
October 1999 and for one day at 273K in July 2000 during spacecraft
thermal vacuum testing at GSFC.

In the event that on-orbit bake outs would become necessary, an
additional heater was added to the focal plane radiator in February
1999. This heater, along with others on the instrument, can raise the
temperature of the focal plane to 270 K on orbit.

The location of ALI contamination has been identified as the top
surfaces of the spectral filters overlaying the focal plane detectors. A
Charge Coupled Device (CCD) camera was placed at the focus of a
collimator and images of the filter surfaces were obtained. The
resulting pictures clearly show that a residue had formed on the surface
of the filters during contamination build-up. Additionally, all evidence
of the residue has been eliminated as a result of the bake out. This
conclusion is supported by the levels of post-bake out data returning to
baseline levels once the focal plane had been cooled to 220 K.

Focal plane contamination appears in three forms: pixel-to-pixel
variation, mean level shifts, and bowing. Further details of these three
forms of contamination can be found in the [ALI Validation Report
(Section 3.1.1.5)](../ALI%20Contamination%20Anomaly.doc).

Much information pertaining to the contamination of the ALI focal plane
was obtained during ground testing between October 1998 and November
2001. The contaminant appeared to condense on the surfaces of the
spectral filters lying above the detectors when the focal plane was
operated at 220K. However, once the focal plane is warmed above 260 K
the contaminant "boils off" and detector responses return to baseline
levels. This implies that mirror surfaces, when maintained above 273 K
at all times, will not collect contaminants during ground testing or
during orbital operations.

Although the source of the contamination is not known, the leading
suspect is the black paint (Z306) coating the inside of the telescope to
reduce stray light. Bake out of the telescope surround structure was
limited to 70 hours and it is possible that residual outgassing of the
paint may be sticking onto the filter surfaces when the focal plane is
cold.

As a result of ALI focal plane contamination, the ALI operations
procedure incorporated an imaging procedure whereby, every 15 days and
for a 15 hour period, the cover was opened and a bake out was performed.
That procedure resulted in the elimination of focal plane residue such
that no loss of imaging quality was experienced.

**3. HYPERION CYROCOOLER SENSOR**

On 10 January 2001, there was a failure of the Cryocooler's compressor
motor 'positive' direction stroke measurement position sensor. This
failure caused the cooler electronics control algorithm to malfunction
which ultimately led to a higher than normal temperature signature from
the cooler. The Hyperion Cryocooler was operated over several following
weeks with an interim work-around to bypass the failed positive stroke
sensor whereby the duty cycle was limited to approximately 50% (1 day
on, 1 day off) and the maximum drive limit was reduced from 89% to 75%
to provide conservative operating margins. Unfortunately, run in this
fashion, the cooler was often unable to maintain proper temperature
control throughout the late cycle DCE sequences. In addition, this was
very operations intensive and required constant monitoring of cryocooler
housekeeping data. Some new RTS\'s were written and tested by TRW which
would provide a more efficient solution. They were loaded and
operational as of 1/31/01 and provided a permanent fix via a patch to
the cooler software. In the new configuration, the cooler attained a
proper temperature in roughly the same time (\~4 hrs) as prior to the
anomaly event and easily maintained that temperature during DCE's.
Shortly after this, another cooler software patch was uploaded to fine
tune the RTS's to allow for near full continuous on-time cooler
operation.

The long term consequence of the position sensor failure and corrective
action taken by the instrument contractor was to modify the Hyperion
operations procedure to perform a Cyrocooler duty cycle turn off every
15 days for a 15 hour period. This procedure allowed the cooler to
"de-ice" and therefore regain its normal cooling capability.

**4. WARP -- SOFTWARE**

The Wide-band Advanced Recorder Processor (WARP) went into an anomalous
state at the end of the day on June 21, 2001. The WARP was reporting
Error Detection and Correction (EDAC) uncorrectable errors which
continued until a reset of the WARP was implemented on the afternoon of
June 29, 2001. The errors went undetected by the Flight Operations Team
because limit checking on the EDAC uncorrectable counter had been
eliminated. The presence of uncorrectable errors on the WARP caused the
science data taken during the anomaly to be mostly unusable.

Early on June 29, WARP engineers came up with the following preliminary
diagnosis:

-   The errors occur only on Memory Card #2 (outermost card)

-   The errors occur on all 6 of the 4-Mbit arrays.

-   The errors are recurrent

-   There are over 200,000 errors per playback set (about 20% of the
    data)

-   The errors appear to occur in 80 byte blocks

The engineers prepared an operations instruction to read the WARP Memory
Mask register. This test showed that there were no signs of corruption
in the register value. Next they ran a WARP Memory Built-In-Test on
Memory Card #2 in the range mode. During the course of this test, the
WARP memory is reformatted. After this, they ran the DCE Self-Test
(RS-422 Card Data Injection) which generates card test data. It then
became evident that the WARP problem had disappeared and the WARP had
returned to a nominal state with no uncorrectable errors. The WARP team
filled the entire memory (48 Gbits) and monitored the EDAC errors on
playback to prove the return to nominal state.

The WARP team attributed the cause of the problem to a stuck bit within
a state machine inside one of the memory boards. The team reasoned that
the problem did not appear to be hardware related, so if the problem
occurred again, it would probably not occur in the same way or the same
location. In fact, there are no mechanisms within the WARP to isolate
the state of state machines at failure, so it would be almost impossible
to plan a diagnostic set of dumps to be taken in the event of another
failure.

Fortunately, the problem did not reoccur.

An internal review was conducted of all the WARP telemetry parameters
and limit settings to assure that the correct parameters were in fact
being monitored with the right limit values. The limit setting of 1 was
instituted for the value of the uncorrectable EDAC error counter (i.e.,
if the value=1, then the limit is violated and reported to the console
operators).

It was originally thought that a routine reformat of the WARP memory
should be performed as a precaution (either once per day or after every
sequence of predetermined DCEs). However, this would preclude
troubleshooting if the error reoccurred, so the deletion of files after
downlink from the ATS load was retained as the nominal operational mode
that had been used since early in the mission.

After the anomaly occurred, the WARP was reformatted three times: once
during the ACE safehold anomaly (9/14/01); once when an operator error
resulted in entry into low power mode by the WARP internal protection
mechanisms; and once for the weekend shutdown surrounding the Leonids
meteor shower. The operator error was documented in a Root Cause and
Corrective Action report and resolved by additional training.

A full accounting of this anomaly is contained in the [WARP Anomaly
Resolution
Report](../EO-1%20WARP%20Anomaly%20of%206-21-01%20Resolution%20Summary%20as%20of%2011-29-01.doc).

**5. ATTITUDE CONTROL SYSTEM -- HARDWARE**

The EO-1 spacecraft went into an anomalous state on 9-14-01. The
Attitude Control Electronics (ACE) went into a hung (latch-up) state.
Later, following some actions by the ACS team, EO-1 went into safehold.
As a result, 7 Data Collection Events (DCE's) were lost involving all
three instruments and also an additional 3 DCE's were affected in the
Hyperion portion of those DCE's that were lost.

The team concluded that the ACE lost power despite the fact that the
status display for the ACE indicated that the power to the ACE was
nominal. After extensive deliberation about possible causes, it was
decided that there are two electronic parts which could have caused the
anomaly. One of the parts is a DC/DC converter from Interpoint and the
other part is a solid state power controller (SSPC) from DDC. A
teleconference was held with both vendors, led by Amri Henrnandez of the
Power Systems Branch supported by many of the salient members of the
Anomaly Resolution team members. A brief summary of the findings after
the meeting is as follows.

A.  Interpoint: The EO-1 ACE Low Voltage Power Controller (LVPC) uses a
    MHF+2805S to provide +5v power to the digital electronics in the
    ACE. It also provides drive power to the low current +28v switched
    services in the LVPC. The internal control logic for this converter
    is mostly analog and it does not include an integrated controller.
    After walking through the design schematic with Interpoint
    engineers, it was agreed that a latched event is not likely to
    happen due to the implementation of the control circuitry. A
    credible failure of the part is most likely to be permanent. After
    the ACE power was rebooted all signals returned to nominal with no
    signs of degradation.

B.  DDC: The EO-1 Power Subsystem Electronics (PSE) uses a
    RP-21015DO-602, Solid State Power Controller (SSPC), to distribute
    power to the ACE.

As a result of the findings, it was decided that the most likely cause
of the anomaly was a Single Event Upset on the SSPC that resulted in the
ACE going into a latch-up state. It was also noted that there are many
of these SSPC's on EO-1 and that adequate contingencies should be
established to assure that any future occurrences are covered.

A summary of actions taken, as a result of the meetings to address the
EO-1 Attitude Control Electronics (ACE) anomaly, was that three new
Telemetry and Statistic Monitors (TSM's) were added and one TSM was
modified. There were approximately five contingency flow diagrams that
were modified also.

A full accounting of this anomaly is contained in the [Anomaly
Resolution Report for ACE Anomaly dated
11/18/01](../EO-1%20ACE%20Anomaly%20of%209-14-01%20Resolution%20Summary%20as%20of%2011-18-01.doc).

**6. ALI SOLAR CALIBRATION APERTURE SELECTOR**

The Aperture Selector is an opaque slide plate located on the Aperture
Cover used to vary sunlight on the solar calibration diffuser plate
located within the instrument. During the first 19½ months on-orbit, the
ALI performed 40 solar calibrations flawlessly. On July 5, 2002, during
the 41^st^ solar calibration, the aperture selector plate failed to
fully close. Due to the latency in data reception and processing, one
more solar calibration was performed with the plate appearing in a
slightly different but stationary position. The initial failure occurred
with Slot #5 97.43% open. At this point, the solar calibration script
was changed to eliminate the activation command to the aperture selector
motor. Over a period of about six weeks, three more solar calibrations
were taken with the new script. Plate motion was not evident during any
calibrations after the failure, but a combination of induced forces
evidently caused the plate to move in small, uncontrolled increments so
as to increase exposure of slots over time until no further movement
occurred. At this point Slot #6 was 49.27% open. Six solar calibrations
performed after this point in time showed the system response to be
stable thereby giving further verification that the selector plate had
eventually reached a fixed, stationary position. Since August 15, 2002
solar calibrations have remained stable within ±1%.

A subsequent failure mode analysis by MIT Lincoln Labs concluded that
the probable failure mode was that epoxy staking of the aperture plate
to the ball nut was overstressed during ground test thermal soak at
-30°C and failed during subsequent ground, launch, and orbital thermal
and vibration environments. This failure in turn allowed the ball-nut to
become unthreaded from the brass spacer and move away from the plate
until it jammed due to particle entrapment thereby allowing the plate to
freely slide along the ball screw until reaching the jammed ball-nut.

After the aperture plate uncontrolled motion stabilized, operational
solar calibrations resumed albeit at a constant illumination level
rather than over the entire dynamic range as before. Also as a result,
the ability to perform a multiple step linearity check was lost.
Notwithstanding the failure, radiometric response stability could
continue to be monitored if the plate position remained fixed. In
addition, it was concluded that there is no impact on nominal DCE dark
current and imaging, no impact on special dark collections, and no
impact on lunar calibration dark current and imaging.

MIT Lincoln Labs gave a presentation on the ALI solar calibration
aperture selector failure on November 4, 2002. Their three sets of
presentation charts can be viewed on the following links.

-   [Advanced Land Imager Solar Calibration Mechanism
    Anomaly](../ALI_Solar_Calibration_Mechanism_Anomaly.ppt)

-   [ALI Solar Calibration Aperture Selector Failure
    Overview](../ALI_Aperture_Selector_Failure_Overview.ppt)

-   [ALI Aperture Selector Failure
    Analysis](../ALI_Aperture_Selector_Failure_Overview.ppt)

Subsequent to this failure, the operations procedures were changed such
that ALI solar calibrations were no longer performed. All other
operations procedures remained unchanged. The reason for this change was
to avoid the possibility that the moveable aperture plate would stick in
an unfavorable position so as to interfere with normal operations.

**7. WARP -- Hardware**

A WARP anomaly occurred on August 25, 2004 (Day 238) that caused the
loss of 8 DCEs and 1 R/T pass (no engineering data loss).

The anomaly occurred during an Autonomous Sciencecraft Experiment (ASE)
long duration test, and since ASE was hosted on the WARP (and
controlling EO-1), the first indication of the problem was that no
telemetry was received during the Svalbard Ground Station (SGS) pass on
the above date at 09:38z. Below is an overview of that day\'s events:

At 238-09:38z: SGS pass -- No telemetry during which a SatWatch alert
was triggered.

At 238-11:00z: \"Blind\" TRDS pass -- S/C health was OK, but the WARP
was not generating any telemetry packets. Power Output Module #2 (OM-2)
showed that the WARP should be powered on.

At 238-12:05z: \"Blind\" TDRS pass -- Sent WARP \"No-op\" and Memory
Dump commands with no response.

At 238-12:55z: SGS Pass (scheduled pass, but we had to perform a
\"Blind\" acquisition since WARP and ASE were not responding) -- Dumped
VR/DS-1. Attempted to send /WARPRESET with no response. Sent /WARPPWR
OFF with expected response on Power Page LOS w/S-Band transponder on.

At 238-14:32z: SGS Pass -- Dumped VR/DS-0. Uplinked Automatic Time
Sequence (ATS) load with S-Band passes only. Switched ATS buffers.

Telemetry playback analysis showed that the last packet generated by the
WARP RSN (packet 03D) was at 238-07:01:38z and that Output Module #2
(OM-2) current dropped by about 1 amp compared to normal. No WARP
activities were scheduled to occur at or near that time. At this point
the prime suspect for the anomaly was the Solid State Power Controller
(SSPC) and not ASE. EO-1 had a similar SSPC problem in September 2001
that caused a power cutoff to the Attitude Control Electronics (ACE).

At 238-17:47z: SGS Pass -- The Engineering Team determined that the best
course was to cut power to the WARP at the SSPC and attempt to reset.
The reset was performed at this time and the WARP returned to standard
operations.

At 238-18:43z: \"Blind\" TDRS pass -- Final WARP configurations (Memory
Scrub and EDAC log in overwrite mode). Cleared 1773 WARP and 1773
WARP/RSN Bus errors. Switched ATS buffers.

At 238-21:19:33z: Uplinked ATS load to resume normal imaging operations
at the next available scene.

This anomaly was fully explained as traceable to a transient Single
Event Upset failure of the WARP SSPC and was similar to the Attitude
Control Electronic (ACE) latchup that occurred in September 2001. The
WARP was powered down and restarted. This power reset action cleared the
problem. This type of problem occurred so infrequently that no standard
recovery operating procedure was needed.