# Summary

An Anomaly Review Team (ART) was formed to investigate the apparent
spontaneous switch of the Power Supply Electronics (PSE) Mode Controller
(MC) onboard the POLAR spacecraft in September 2003. The anomaly
occurred while in a real-time contact following a long duration eclipse.
Two parameters were observed to change state from primary to secondary
values. The primary focus of this ART was on the possible reasons for
the MC switch. This report documents the history and status of the MC,
the investigation undertaken by the ART, and the findings of that
investigation. The conclusion is that an unknown failure within the
power system has caused a permanent switch to the secondary MC. This
switch over will not impact the operation of the power system or the
overall spacecraft operations for the remainder of the mission.

# Background

The Electrical Power Subsystem (EPS) consists of power generating (solar
array), control (PSE), and storage functions (batteries). The sun's
solar energy is directly transferred from the solar arrays to the load
bus. The main load bus provides regulated +28V direct current power to
the instruments and spacecraft subsystems.

The PSE is the central controlling element in the EPS. It controls and
maintains the +28V regulated bus by either shunting excess solar power
or by boosting the power load bus voltage with stored energy from the
batteries. Failure detection circuitry provides automatic switchover
from primary to secondary control functions.

The MC maintains the regulated bus by controlling the charge, boost, and
shunt functions within the EPS. Both primary and secondary MC circuits
are active simultaneously to provide on-line redundancy of the Shunt
Control Amplifiers (SCAs) driving the shunts. The failure detection
circuitry protects against either improper bus voltage or mode overlap
(e.g. shunt drives active while in boost mode). If a malfunction is
detected within the primary MC, the secondary MC takes over active
control and begins regulating to +28V. A simplified functional block
diagram for the MC is shown in Figure 1.

Over and under-voltage protection is provided by automatically switching
the MC and it's associated shunt control amplifier. Over-voltage in
excess of +29.5V and under-voltage of less than 27.0V causes a
switchover. There was no indication in spacecraft telemetry that either
of these situations occurred.

Mode overlap protection switches the MC to backup whenever the batteries
are commanded by the MC to provide power to the bus (boost mode) while
the shunt drive is active (shunt mode). Switching from secondary to
prime can only be accomplished by ground command. It is not documented
whether this mode overlap protection also switches the associated SCA to
the backup, as is done when the PSE detects under- and over-voltage
conditions. This is the mechanism that is suspected of causing the
fail-over.

![](media/image1.jpeg){width="6.990277777777778in"
height="4.227777777777778in"}

### Figure 1: Mode Controller Block Diagram

The anomaly occurred at 06:24:44 on September 7, 2003 during a real-time
contact. During this contact two parameters in the power subsystem were
observed to change from primary to secondary values. At that time the
anomaly was considered to be caused by a single event upset because all
EPS parameters were nominal both before and after the event.

The first parameter (OMCSTAT) changed its value from PRIMARY (raw value
of 1) to SECNDARY (0). This parameter represents the status of the
PSE-MC. The internally redundant Mode Controller is responsible for
transitioning the PSE seamlessly between three power modes: Charge,
Boost, and Shunt.

The second parameter (OSHDRACS) changed its value from AMP_A (1) to
AMP_B (0). This parameter represents the connection status of Shunt
Drive A, the first of two Shunt Drives (Shunt Drive B is the other). The
Shunt Drive can be connected to either Shunt Control Amplifier A (AMP_A)
or Shunt Control Amplifier B (AMP_B).

# Current Status

Currently the secondary MC is still active and using SCA B. All attempts
to command the MC and SCA to primary have stopped.

# ART Charter

1)  Analyze all relevant telemetry and other data to assess, if
    possible, the status of the mode controller, and the potential for
    recovery of its functionality.

2)  Review all relevant telemetry to determine, if possible, the reason
    for the switchover.

3)  Determine whether any additional commanding of the primary MC should
    be done.

4)  Issue a recommendation regarding the configuration of the EPS
    control circuitry.

5)  Produce a report detailing the findings of the ART.

# ART Membership

Scott Glubke / GSFC 444 / SSMO Spacecraft Engineering Lead (ART
Chairman)

Steve Odendahl / GSFC 444 / POLAR Mission Director\
Patrick Crouse / GSFC 444 / SSMO Deputy Project Manager

Anisa Ahmad / GSFC 563 / Power Systems Engineer\
Steve Hearn / LMSO / GGS-POLAR Spacecraft Lead Engineer

Barbara Giles / GSFC 692 / POLAR Project Scientist

Mike Machado / HTSI / POLAR FOT Lead

# ART Investigation

## Telemetry review

The Flight operations Team (FOT) first noticed the anomaly when the
ground system event monitor noted that two parameters in the EPS had
changed state during a real time contact. At the time of the anomaly the
EPS was in charge mode following a 130-minute eclipse. The solar arrays
were in full sun and providing able charge current (12.8 amps). The main
bus was regulated at +28.1V. All other EPS parameters appeared normal.
The FOT verified that the rest of the spacecraft subsystems and
instruments were nominal. Due to the severity of the eclipse season,
which lasted until September 19, 2003, no action was taken before then
to reconfigure the PSE because changing the PSE requires taking the
instrument high voltage off-line and thus interrupting science.

## Failure Analysis

Subsequent ART meetings discussed the anomaly and various related
parameters both before and after the anomaly. There were no obvious
indications as to why the switchover occurred.

The GSFC Power System Engineer, Anisa Ahmad, reviewed the anomaly with
other engineers within her branch to see if anyone had similar
experiences on other programs. The Polar Orbiting Environmental
Satellite (POES) had a similar anomaly during ground testing in February
1999. The POES spacecraft has a similar power system design and was
built by the same company.

The following is a brief summary of the POES failure investigation:

*The POES spacecraft has the following Mode Controller Fault Detection &
Correction (MC-FDC) Circuitry:*

1.  *If VBUS \< 27.0V for 2+/-0.5 sec, then (a) MC is changed from
    primary to backup, and (b) PSA-A is disconnected so both Partial
    Shunt Drive -1 and Partial Shunt Drive-2 are driven by PSA-B.*

2.  *If VBUS \> 29.5V for 130 msec OR If VBUS \> 30.5V for 100 msec,
    then*

    a.  *MC is changed from primary to backup, and*

    b.  *PSA-A is disconnected so both Partial Shunt Drive-1 and Partial
        Shunt Drive-2 are driven by PSA-B.*

3.  *IF MC PRI is driving BVR AND PSA-B driving shunts for \>0.5 sec,
    then disconnect PSA-B so that Partial Shunt Drive-2 is also
    connected to PSA-A.*

4.  *If BVR is active AND BCRs are active, then*

    a.  *MC is changed from primary to backup and*

    b.  *PSA-A is disconnected so both Partial Shunt Drive-1 and Partial
        Shunt Drive-2 are driven by PSA-B.*

5.  *If BVR is active AND PSAs are active, then*

    a.  *MC is changed from primary to backup and*

    b.  *PSA-A is disconnected so both Partial Shunt Drive-1 and Partial
        Shunt Drive-2 are driven by PSA-B.*

*In summary, the two mode overlap conditions monitored by the MC-FDC
are:*

1.  *The first is the PSA-A DRIVE being active while either the primary
    Mode Controller (MC) Boost or Charge modes are active. After 1.5 to
    2.0 seconds, the FD commands the MC to backup, and switches the
    source of PSA DRIVE-1 from the PSA-A DRIVE output of the MC to the
    PSA-B DRIVE output.*

2.  *The second is the PSA-B DRIVE being active while either the Primary
    MC Boost or Charge modes are active. In this case, the FD switches
    the source of PSA DRIVE-2 from the PSA-B DRIVE output of the MC to
    the PSA-A DRIVE output after approximately 500 milliseconds. This
    detector is inhibited if the MC has been previously commanded to
    backup.*

The POES anomaly is narrowed down to the presence of negative voltage on
PSA Drive lines. The PSA drive voltages are sensed by MC-FDC to
determine if the PSAs are ON or OFF. A voltage greater than 1.9 volts is
interpreted as PSAs are ON. Otherwise, they are OFF. This signal is
processed by an operational amplifier IC LM139. The positive rail of
this circuitry is tied to about +20 volts and the negative rail is tied
to spacecraft single point ground (SPG). The LM139 IC has a specified
limit that any of its inverting or non-inverting terminals shall not see
a voltage \>300mV lower than the voltage at the negative rail. If it
exceeds, then the output of this op-amp goes HIGH and is interpreted
falsely as if PSAs are ON, even though the PSA drive signals is
\<-300mV. The MC-FDC issues commands (if PSAs are On while BVR is ON) to
reconfigure the power system thinking that in fact mode controller has
failed. However, in reality, the mode controller has not failed. The
MC-FDC circuitry should have been designed such that it would disregard
a negative voltage and not misinterpret it as a failure.

The anomaly investigation concluded that

1.  the electrical ground support equipment caused the POES anomaly,

2.  the worst case conditions on orbit should not cause the spacecraft
    to experience the MC switchover.

It was also noted that barring any other failures such a switchover
would not affect the spacecraft operations and that the MC could be
returned to the primary side by ground command.

Because of the science impact of cycling the high voltages, the
operation to switch the MC back to the primary side was delayed until a
planned maneuver in January 2004 when the instrument were planned to be
off. The SCA would be left on the secondary side.

## Additional Testing/Investigation

On January 12, 2004 the FOT attempted to switch the MC back to the
primary side at the beginning of a planned pass. The command to switch
Mode Controllers was sent from the proper STOL procedure at 23:20:53
GMT. The command was successfully received by the spacecraft, but failed
to change the state of the Mode Controller. The command was sent a
second time from the STOL procedure at 23:23:06 GMT. Again, the
spacecraft acknowledged receipt of the command, but no change of state
occurred. No further action was taken during that contact.

The FOT reviewed the command definition to determine if a reset,
reconfiguration, or constraint removal prior to sending the MC switch
command was required. No other commands or resets were identified.
Because the command was received, acknowledged, and processed by the
spacecraft, the MC should have switched back to the primary side. A
review of the telemetry did not show even a momentary switch back to the
primary MC. The MC-FDC reacts much faster than the telemetry updates so
even a momentary switch may not be seen in telemetry.

The power systems group reviewed the initial data again in addition to
further researching the MC design. A final suggestion was to command the
shunt drives back to the primary SCA. If this command failed to switch
back to primary, then it can be concluded that most probable
degradations/failures that would have occurred may be:

a.  A component failure in the primary mode controller circuitry.

b.  The op-amp(s) used in the fault detection (in specific mode overlap
    portion) circuitry would have degraded. The op-amp used changes its
    output states even when its non-inverting input becomes more
    negative compared the op-amp's power supply negative rail.

If either one of the above is the cause for the failure, then the FOT
cannot command either mode controller or the shunt drive-A to primary
sides.

On March 22, 2004 at 19:43:51 GMT, the FOT sent a command to the
spacecraft to reconnect Shunt Drive A to PSE Amp A. The command was
received and accepted by the spacecraft, but the configuration of the
Shunt Drive did not change. No other change to the Power subsystem was
observed. The command was sent a second time at 19:44:59 GMT without
affect. No further commanding was attempted.

# ART Discussion & Findings

The exact cause of the MC switchover cannot be determined due to the
limited information available in telemetry. All attempts to switch back
to the primary side have failed. The most probable location of the
failure is within the fault detection circuitry. This is based on all
the EPS telemetry being nominal and with FDC limits both before and
after the anomaly was first noticed.

# Final Disposition & Recommendation

The final recommendation of the ART is to complete the mission using the
secondary power system mode controller and shunt drive amplifier. The
spacecraft is currently healthy and operating properly on the secondary
electronics. No other actions are necessary.