EO-1 Anomaly Resolution Report for ACE Anomaly of 9-14-01

**1. Introduction:**

The EO-1 spacecraft went into an anomalous state on GMT 257 (9-14-01).
The Attitude Control Electronics (ACE) went into a hung state. Later,
following some actions by the ACS team, EO-1 went into safehold. As a
result, 7 Data Collection Events (DCE's) were lost involving all three
instruments and also an additional 3 DCE's were affected in that
Hyperion portion of those DCE's were lost. The time line of the events
that day was as follows:

**2. Timeline of Events:**

257-16:04 GMT Lost contact with Attitude Control Electronics (ACE)

257-16:52 GMT EO-1 enters science collection mode

257-17:31 GMT TSM 50 trips because Battery State of Charge (BSOC) \< 85%
and sends event message that battery voltage is 26.42, bus voltage is
26.35 and BSOC is 79.9951%

257-18:15 GMT TSM 50 trips because BSOC\<70% and causes load shed

257-18:23 to 18:32 GMT Pass with Alaska, Glen Bock on console notices
limit violations, S/C events and loss of ACE communications. Escalates
problem and schedules additional TDRSS time

257-19:01 GMT Begin TDRSS pass. ACS experts arrive approximately at this
time

257-19:37 GMT Power Subsystem Electronics(PSE) indicate that ACE is on
and S/C pulling 1.4 amps which is less that it should be, thus it is
deduced that ACE not drawing sufficient power

257-19:39 GMT Begin a series of attempts to perform warm and cold reset
of ACE

257-20:02 GMT Powered ACE off via ground command. PSE FDC powers ACE on.
Telemetry received from ACE, Ace entered safehold and drove arrays to
index. S/C transitioned to power positive, sun pointing.

257-20:45 GMT TSM 50 (Low BSOC) returned to normal, BSOC 80.8% battery
voltage 30.04

257 and 258 GMT Friday evening was spent recovering from safehold

258 GMT Saturday morning was spent turning instruments back on at which
time operations were nominal

**3. Tiger Team Members:**

Convened tiger team on 9/17 and also met on 9/20 and 9/26 and 10/9.
Tiger team members were as follows:

Lead: Dan Mandl GSFC

Participant Org Expertise

=========================================

Amri Hernandez GSFC Power

David Ward GSFC Mechanical

Stu Frye MitreTek Systems

Bruce Zink Swales Systems

EJ Bickley Swales Systems

Paul Sanneman Swales ACS

Kathy Blackman Hammers ACS

Teresa Hunt Swales ACS

Dave Speer Litton ACS

Kurt Smithgall Swales Electrical

Chris Xenophontos Litton PSE SW

Louise Basha GFSC Flight SW

Emory Stagmer Litton Flight SW

Joan Frank Honeywell FOT/Administrative Lead

Tom Moore Honeywell FOT/Technical Lead

Seth Shulman CSC FOT/Navigation

Rob Bote Honeywell FOT/Mission Planning

Momi Ono TRW Hyperion

Carol Segal TRW Hyperion

Jeff Mendenhall Lincoln Lab ALI

**4. Analysis of Cause:**

The team concluded that the ACE lost power despite the fact that the
status display for the ACE indicated that the power to the ACE was
nominal. After extensive deliberation about possible causes, it was
decided that there are two electronic parts which could have caused the
anomaly. One of the parts is a DC/DC converter from Interpoint and the
other part is a solid state power controller from DDC. A teleconference
was held with both vendors, led by Amri Henrnandez of the Power Systems
Branch supported by many of the salient members of the Anomaly
Resolution team members listed above. A brief summary of the findings
after the meeting were as follows.

A.  Interpoint: The EO-1 ACE Low Voltage Power Controller (LVPC) uses a
    MHF+2805S to provide +5v power to the digital electronics in the
    ACE. It also provides drive power to the low current +28v switched
    services in the LVPC. The internal control logic for this converter
    is mostly analog and it does not include an integrated controller.
    After walking through the design schematic with Interpoint
    engineers, it was agreed that a latched event is not likely to
    happen due to the implementation of the control circuitry. A
    credible failure of the part is most likely to be permanent. After
    the ACE power was rebooted all signals returned to nominal with no
    signs of degradation.

B.  DDC: The EO-1 Power Subsystem Electronics (PSE) uses a
    RP-21015DO-602, Solid State Power Controller(SSPC), to distribute
    power to the ACE. Figure 1 shows the functional equivalent of this
    switch. The circuit diagram is composed of three sections which
    perform control functions on a MOSFET. Section I

![](media/image1.wmf){width="5.906944444444444in"
height="4.455555555555556in"}

performs a current sense function. Section II performs a circuit breaker
function and section III performs an instantaneous current trip
function. The following is an explanation of how each section works:

> I . Main current sense circuit section. Current through the 'Sense
> Resistor' is converter to a 25mV/15A scale. This voltage level is then
> converted to a current signal that feeds sections II and III.
>
> II . Circuit breaker function for the SSPC. Current through the
> component follows an I²T curve similar to circuit breakers
> (exponential curve). In other words, the closer the load current is to
> the high limit the sooner the switch will trip off. Likewise, the
> further the load current is from the limit the longer it takes to
> trip. During normal operation of a circuit breaker function, a trip
> through this path will be reported to the user via status line SB in
> Fig 1.
>
> III\. Instantaneous current trip circuit section. To protect the
> internal hardware in the switch as well as the load, there is a clamp
> to the current level of the I²T exponential curve. This instantaneous
> clamp or trip is directly applied to the MOSFET gate (G) to source (S)
> and if tripped would not report the status of the MOSFET via status
> bit SB.

![](media/image2.wmf){width="5.665277777777778in"
height="3.865972222222222in"}

One of the key components of the SSPC is a circuit breaker function
which follows an I²T curve to determine the trip level and time. Figure
2 is a simplified trip curve for this SSPC. This means that if the load
current is at the 100% limit of the part then it will take \>10 seconds
to trip. If however, the current 300%, for example, it will take less
than 10 seconds but more than 0.01 seconds. If the current is at 800% of
its limit then the Instantaneous trip can turn off the SSPC in about
25usec. The SSPC has two status bits, one which shows the state of the
switch (enabled/disabled, see figure 1) and the other which shows if the
current through the part is above 5% (not shown in circuit diagrams).
During the anomaly the status lines showed that the switch was enabled
and that the current was below 5% of its capacity. The command line was
also enabled as expected by the spacecraft Fault Detection and
Correction (FDC) routine. During steady state operation of the SSPC, any
latched command and any latched I²T performance of the switch is
reported to the user through the status line SB in figure 1. It is
important to realize that the switch state is [not]{.underline} sensed
at the output of the switch but at the input of the "latched I²T trip"
driver as shown in figure 1.

 

However, during a short circuit, for example, the instantaneous trip
portion of the SSPC will turn off the MOSFET in about 25u seconds using
the path outlined as section III in Figure 1. This circuit almost
immediately compares the sense current from section I to an internal
threshold that represents the \>800% Imax limit. If the threshold is
exceeded then a latched off state is directly driven across the MOSFET's
gate-to-source. This action bypasses the status line, therefore the user
will not know the true state of the switch if only relying on status
line SB.

 

Given the two separated control paths, sections II and III of figure 1,
it is easily seen how a fast energy event can hit any of the marked
locations, A, B and C, latch the Instantaneous current trip and not
report the state change through status line SB. The exponential
characteristic of the I²T filter is slow enough not to detect current
spikes via path II.

In summary, the anomaly could happen in one of three ways:

 

A)    The event happened at the current sensor location (A). The current
is sensed with a resolution of 25mv/15A. The small voltage level
variations was converted into current in section I, the instant latched
trip section, II, features is a high bandwidth low capacitance
application that ends in a custom Set/Reset latch. Once this latch drove
the MOSFET off, the switch was disabled but the status line did not
confirm the change in state.

 

B)     The event happened before the current threshold comparator (B).
The signal propagated through the circuit in section III but not through
the circuit in section II (due to the exponential characteristic of the
circuit filter). Again, This scenario turned the MOSFET off with no
report on the status line (due to section III path already discussed).

 

C)    The event happened at the instantaneous trip circuit Set/Reset
latch (C). A direct state change at this location latched the switch off
but the status line would not reflect the change.

Note: For additional detailed description of the SSPC, please refer to
the attached DDC data sheet.

As a result of the findings it was decided that the most likely cause of
the anomaly was a Single Event Upset on the SSPC. It was also noted that
there are many of these SSPC's on EO-1 and that we should take a look to
assure that contingencies would cover any future occurrences.

**5. Resultant Actions:**

The table below was constructed by EJ Bickley. Resultant change requests
submitted to CCB for evaluation were added.

+------+---+---------------+------+-------------+---------------------+
| **Ou | * | **Symptom of  | *    | **Expected  | ##                  |
| tput | * | SSPC failure  | *FSW | Spacecraft  |  Recommended Change |
| M    | P | (no BUS       | Rec  | Action**    |                     |
| odul | S | power)**      | over |             |                     |
| e/Se | E |               | y?** |             |                     |
| rvic | F |               |      |             |                     |
| e/Na | D |               |      |             |                     |
| me** | C |               |      |             |                     |
|      | ? |               |      |             |                     |
|      | * |               |      |             |                     |
|      | * |               |      |             |                     |
+------+---+---------------+------+-------------+---------------------+
| 1/1  | N | ALI 1773 BUS  | N    | None        | Develop a new TSM   |
| /ALI |   | errors. Event |      |             | that will enable    |
|      |   | messages for  |      |             | the ALI outgas      |
|      |   | aborted RTS   |      |             | heater (RTS 61).    |
|      |   | sequences     |      |             | TSM will key on ALI |
|      |   | related to    |      |             | 1773 BUS errors for |
|      |   | ALI.          |      |             | \> 20 minutes.      |
|      |   |               |      |             |                     |
|      |   |               |      |             | **Reconfirm need    |
|      |   |               |      |             | with Mendenhall and |
|      |   |               |      |             | possible submission |
|      |   |               |      |             | of Change           |
|      |   |               |      |             | Request(CR)**       |
+------+---+---------------+------+-------------+---------------------+
| 1/2/ | N | LEISA AC 1773 | N    | None        | None. Normal        |
| L    |   | BUS errors.   |      |             | operation via RTS   |
| EISA |   | Event         |      |             | clears failure in   |
| AC   |   | messages for  |      |             | PSE.                |
|      |   | aborted RTS   |      |             |                     |
|      |   | sequences     |      |             |                     |
|      |   | related to    |      |             |                     |
|      |   | LEISA AC.     |      |             |                     |
+------+---+---------------+------+-------------+---------------------+
| 1/   | Y | No downlink   | N    | None        | Refine loss of      |
| 3/S- |   | available.    |      |             | communication       |
| band |   |               |      |             | contingency to      |
| XMTR |   |               |      |             | account for SSPC    |
|      |   |               |      |             | power outage. **T.  |
|      |   |               |      |             | Moore to mod Loss   |
|      |   |               |      |             | of Com Contingency  |
|      |   |               |      |             | Flow diagram and    |
|      |   |               |      |             | submit CR**         |
+------+---+---------------+------+-------------+---------------------+
| 1/4  | Y | ACE RSD       | N    | BSOC drops  | New TSM under       |
| /ACE |   | packet no     |      | to \<70%,   | development to      |
| LVCP |   | longer        |      | RTS 2       | activate RTS 2      |
|      |   | received,     |      | executes.   | based on loss of    |
|      |   | 1773 BUS      |      | TSM #215    | ACE telemetry       |
|      |   | errors,       |      | will        | packets to the M5.  |
|      |   | significant   |      | eventually  | **Seth Shulman      |
|      |   | drop in BSOC  |      | power cycle | submit CR to add    |
|      |   |               |      | the ACE     | new TSM and anther  |
|      |   |               |      | LVPC        | CR to mod TSM 215** |
|      |   |               |      | service     |                     |
+------+---+---------------+------+-------------+---------------------+
| 1/5/ | N | X-band RSN    | N    | None        | Power OFF SSPC,     |
| X-   |   | 1773 errors   |      |             | then perform normal |
| band |   | and no X-band |      |             | turn ON using       |
| RSN  |   | downlink      |      |             | ground based STOL   |
|      |   | av            |      |             | procedures. **T.    |
|      |   | ailable.Event |      |             | Moore to submit CR  |
|      |   | messages for  |      |             | to modify X-band    |
|      |   | aborted RTS   |      |             | contingency flow    |
|      |   | sequences     |      |             | diagram.**          |
|      |   | related to    |      |             |                     |
|      |   | X-band RSN.   |      |             |                     |
+------+---+---------------+------+-------------+---------------------+
| 1/6  | N | Pitch control | Y    | ACS FDC     | None.               |
| /PPT |   | problem       |      | Test #23    |                     |
|      |   |               |      | will        |                     |
|      |   |               |      | configure   |                     |
|      |   |               |      | back to     |                     |
|      |   |               |      | Pitch RWA   |                     |
|      |   |               |      | control     |                     |
+------+---+---------------+------+-------------+---------------------+
| 2/1/ | N | WARP RSN and  | N    | None. TSM   | Power OFF WARP      |
| WARP |   | WARP M5 1773  |      | 74 should   | SSPC, then perform  |
|      |   | errors and no |      | not trip as | normal turn ON      |
|      |   | X-band        |      | no WARP     | using ground based  |
|      |   | downlink      |      | data is     | STOL procedures.    |
|      |   | available.    |      | processed.  | **T. Moore to       |
|      |   |               |      | TSM 74 will | submit CR to modify |
|      |   |               |      | execute RTS | WARP contingency    |
|      |   |               |      | 21 and      | flow diagram if     |
|      |   |               |      | power OFF   | needed.**           |
|      |   |               |      | the WARP.   |                     |
+------+---+---------------+------+-------------+---------------------+
| 2/2  | Y | Loss of BUS   | N    | Instruments | Operations executes |
| /CDH |   | controller    |      | remain ON,  | loss of             |
| LVPC |   | places ACE    |      | PSE FDC     | communication       |
|      |   | into SAFEHOLD |      | trip to     | contingency. Only   |
|      |   | control of    |      | load shed   | action which will   |
|      |   | spacecraft.   |      | instruments | recover C&DH LVPC   |
|      |   | PSE           |      | will not    | is the special      |
|      |   | configuration |      | occur if    | command power cycle |
|      |   | remains       |      | ACE is      | to the PSE.         |
|      |   | static.       |      | maintaining | Splinter meeting    |
|      |   | Ground sees   |      | proper      | 10/10 to decide how |
|      |   | no RF         |      | SAFEHOLD    | contingency should  |
|      |   | downlink with |      | attitude.   | work**. T. Moore to |
|      |   | S/C during    |      | PSE takes   | submit CR with      |
|      |   | next          |      | NO action   | modification to     |
|      |   | scheduled     |      | to power    | C&DH contingency    |
|      |   | pass.         |      | OFF         | diagram.**          |
|      |   |               |      | i           |                     |
|      |   |               |      | nstruments. |                     |
|      |   |               |      | Hyperion    |                     |
|      |   |               |      | performs    |                     |
|      |   |               |      | safing      |                     |
|      |   |               |      | activity    |                     |
|      |   |               |      | due to loss |                     |
|      |   |               |      | of BC. ALI  |                     |
|      |   |               |      | takes no    |                     |
|      |   |               |      | autonomous  |                     |
|      |   |               |      | action base |                     |
|      |   |               |      | on loss of  |                     |
|      |   |               |      | BC . LEISA  |                     |
|      |   |               |      | AC remains  |                     |
|      |   |               |      | ON if       |                     |
|      |   |               |      | powered ON  |                     |
|      |   |               |      | and will    |                     |
|      |   |               |      | overheat    |                     |
|      |   |               |      | without a   |                     |
|      |   |               |      | BC to       |                     |
|      |   |               |      | command it  |                     |
|      |   |               |      | OFF.        |                     |
+------+---+---------------+------+-------------+---------------------+
| 2/3  | N | HYPERION 1773 | N    | None        | .Turn SSPC off. Run |
| /HYP |   | BUS errors.   |      |             | normal turn-on      |
| ELEC |   | Event         |      |             | procedures. **T.    |
|      |   | messages for  |      |             | Moore to submit CR  |
|      |   | aborted RTS   |      |             | to modify Hyperion  |
|      |   | sequences     |      |             | contingency         |
|      |   | related to    |      |             | diagram.**          |
|      |   | HYPERION.     |      |             |                     |
+------+---+---------------+------+-------------+---------------------+
| 2/4  | Y | HYPERION      | N    | None        | New TSM to monitor  |
| /HYP |   | fails to      |      |             | Hyperion temps ();  |
| HTR  |   | mai           |      |             | if these indicate   |
|      |   | ntain nominal |      |             | all Hyperion is     |
|      |   | operating     |      |             | getting very cold,  |
|      |   | temperature.  |      |             | then send command   |
|      |   | Red low limit |      |             | to power cycle      |
|      |   | violations    |      |             | Hyperion Heater     |
|      |   | after about 1 |      |             | SSPC. **T. Moore    |
|      |   | orbit.        |      |             | and M. Ono to       |
|      |   |               |      |             | submit CR to make   |
|      |   |               |      |             | flight SW change.** |
+------+---+---------------+------+-------------+---------------------+
| 2/5  | Y | Unable to     | N    | RTS 2       | New TSM to monitor  |
| /RWA |   | control S/C   |      | starts when | RWA temp or         |
|      |   | attitude. ACE |      | TSM 66      | regulated voltage   |
|      |   | transitions   |      | trips (ACE  | telemetry; if these |
|      |   | to SAFEHOLD   |      | in          | indicate all 3 RWAs |
|      |   | (based on FDC |      | SAFEHOLD)   | are OFF, send       |
|      |   | failure?),    |      |             | command to power    |
|      |   | but power     |      |             | cycle PSE SSPC.     |
|      |   | still         |      |             | **Seth Shulman to   |
|      |   | degrades,     |      |             | submit CR.**        |
|      |   | although SADE |      |             |                     |
|      |   | can track sun |      |             |                     |
|      |   | (with higher  |      |             |                     |
|      |   | margin of     |      |             |                     |
|      |   | error).       |      |             |                     |
+------+---+---------------+------+-------------+---------------------+
| 2/6/ | Y | Motion        | N    | ACS FDC     | Depending on SAD    |
| SADE |   | failure of SA |      | test #87    | position, SHM       |
|      |   | reducing      |      | would first | control would       |
|      |   | power into    |      | perform     | probably still be   |
|      |   | battery. ACE  |      | Notify      | power positive,     |
|      |   | transitions   |      | Ground      | giving FOT chance   |
|      |   | to SAFEHOLD   |      | (2800 sec)  | to diagnose and     |
|      |   | in 58 minutes |      | then put    | power cycyle SAD    |
|      |   | (based on FDC |      | ACE in      | service at PSE.     |
|      |   | failure) and  |      | Safehold    | **Seth Shulman to   |
|      |   | points array  |      | (3500       | submit CR to modify |
|      |   | using wheels. |      | sec);RTS 2  | appropriate         |
|      |   |               |      | starts when | contingency flow    |
|      |   |               |      | TSM 66      | diagram**           |
|      |   |               |      | trips (ACE  |                     |
|      |   |               |      | in          |                     |
|      |   |               |      | SAFEHOLD)   |                     |
+------+---+---------------+------+-------------+---------------------+

A summary of the needed actions as a result of the meetings to address
the EO-1 Attitude Control Electronics (ACE) anomaly was that we needed 3
new Telemetry and Statistic Monitors (TSM's) and one TSM that has to be
modified. There were approximately 5 contingency flow diagrams that
needed to be modified also.

**6. Specific CCB Actions and Schedule:**

The following are the Operational Database Change Requests (ODBCR) that
specifically resulted from the above.

ODBCR \# Title Disposition

=======================================================

#440, Mod to TSM 215 - Approved to remove the battery voltage test from
TSM

215\. Requires a table load and a code patch for the telemetry points
that

monitor multiple items.

#441, New TSM to Check for ACE Packets - Approved to safe the
instruments

and load shed in the event of another failure of this type.

#442, New RWA TSM/RTS Sequence - Approved to power cycle the RWA in the

event of low regulated motor drive electronics voltages. This fix will

require a code patch plus new TSM/RTS pair in addition to an update to
the

RTS_COLD_RESTART procedure.

#443, S-Band SSPC Contingency - Approved and assigned to Dan Mandl to

refine the procedures in the event of the loss of S-band comm to toggle
the

SSPC switch. In the mean time, use E.J. Bickley\'s spreadsheet as a
guide

to FOT responses to future events.

#444, X-Bank SSPC Contingency - Approved and assigned to FOT.

#445, WARP SSPC Contingency - Approved and assigned to FOT.

#446, Hyperion SSPC Contingency - Approved and assigned to FOT. Don\'t

forget to include turn-off procedures in addition to turn-on procedures.

#447, New Hyperion TSM - Approved and assigned to FOT. Need to make sure

it only runs once without ground intervention, so put a disable command
as

the last entry or a 60-100 minute wait in the new TSM/RTS pair. FOT to

discuss with Momi Ono (TRW) before implementing.

#448, SADE SSPC Contingency - Approved and assigned to FOT. Power
cycling

is not enough in this case, need to explicitly get the SA moving again

somehow via this procedure.

#449, ALI SSPC Contingency - Approved and assigned to FOT. There is an

open question about whether to automatically turn ALI heaters on if
there

is a power anomaly detected. FOT to discuss with Jeff Mendenhall at LL.

ODBCR's 439, 440, 441, 442, 447 to be completed by November 30, 2001 if
EO-1 has an extended mission.

ODBCR's 443, 444, 445, 446, 448 and 449 to be completed by October 30,
2001