# Presentation: 910339

## TES Instrument Operations Status/Plans

- Robert Murdock
- TES IOT
- October 27, 2008
- Pasadena, California

## Contents

- TES Key Engineering Activities (Past Year)
- TES Executed Science Activities (Past Year)
- TES Instrument Operations and Health Monitoring
- ICS Management
- Anomalies
- Future Operations Plans
- IST/EMOS-Related Problems & Concerns
- TES Operations Personnel Updates

## TES Key Engineering Activities (Past Year)

- Performed De-icing of Focal Plane Detectors
- #17: Nov 26 - 28, 2007 – Nominal Results
- #18: May 28 – 30, 2008 – Nominal Results
- Uploaded (U/L) new Global Survey (GS) Test Macro (Macro ID 7) – May 14, 2008
- Precursor to GS (Version 18) Macro
- No scans below 60 degrees south latitude
- No long scans
- Reduces the chance of tripping an over-velocity error on the instrument
- Limits the potential for persistent ICS motor over current events to less than 6
- ICS Performance (PERF) flag enabled
- Executed on May 15, 2008
- U/L Revised ICS Performance Macro (Macro ID 327)  – June 5, 2008
- To emulate the updated GS Macro (Version 18)
- No long scans
- Executed June 7, 2008
- U/L updated GS (Macro ID 10, Version 18) – June 5, 2008
- No scans below 60 degrees south latitude
- No long scans
- Executed June 6, 2008
- Recovered from an ICS over-velocity fault (Re-activation of the ICS) – June 12, 2008
- U/L updated Global Survey (Version 19) – July 29, 2008
- No scans below 60 degrees south latitude and above 70 degrees north latitude
- Executed July 30, 2008

## TES Executed Science Activities 9/07 to 9/08

- 192 Executed Global Surveys (198 opportunities, 97% completed)
- Two cancelled due to Focal Plane De-icing
- Three cancelled due to ICS Velocity Fault – June 9, 2008
- 1 cancelled due to Aura Drag Make-Up (DMU) – June 26, 2008
- 787 Executed Special Observations
- 96 Transects
- 21 Stares
- 670 Step and Stares

## Special Observation Campaigns 9/07 to 9/08

- Sept 10, 2007 -  Year-long Transect observations scheduled over Beijing and SW Beijing for detecting pollutants and atmospheric chemistry studies.
- Oct 13 & 15, 2007 Transect observations scheduled near Table Mountain sonde facility.
- Oct 27 & 31, 2007 A Step&Stare and a Transect observations scheduled near S. California and Table Mountain for monitoring California fires.
- Feb 7,  23, & Mar 10, 26, 2008 Transect observations in support of WAVE-2008 campaign.
- Mar 24 – Apr 30, 2008 Step&Stare and Stare observations covering N America in support of ARCTAS campaign.
- Apr 01 – May 15, 2008 Step&Stare observations covering SE Asia and tropics in support of OP3 campaign.
- May 23 - June, 2008 Stare and Transect observations over megacities, Tokyo, Riverside, Dhaka, Mumbai, and Lagos.
- June 13 – , 2008 Step&Stare and Stare observations covering N America in support of ARCTAS campaign and continuations of N. America Special Observations in the summer.
- June 13 – , 2008 Step&Stare observations covering SE Asia and tropics in support of OP3 campaign.
- July 16 & 23, 2008 Stare/Transect observations over Mauna Loa in support the H2O isotope measurement campaign.
- July 25 & 27, 2008 Transect observations covering California central valley (Fresno).

## TES Instrument Operations and Health Monitoring

**TES Instrument Operations:**
- Scheduled  stored sequence activities on the Master Time-line using the MMS System from the TES Instrument Support Terminal (IST) at JPL
- Stored activities are generally scheduled a month in advance except for Special Observations (SOs), which are scheduled weekly due to changing ground track files
- Global Surveys – executed every other day
- Special Observations – executed after receiving requests from the science team
- ICS and PCS Performance Sequences – executed every other day
- Filter Wheel Movement Maintenance Sequence – executed every other day
- Radiometric Assessment Calibration – executed every other day
- DRAM and EEPROM Maintenance Sequences – executed bi-weekly
- Performed Routine real-time activities at the IST via DMZ System:
- Focal Plane De-icing
- New & Updated Macro (Sequence) Loads
- Sequence Dumps
- New Macro Parameter Table (MPT) Loads
- Other Table Loads (e.g., Fault Threshold)
- Table Dumps
- New macro executions (e.g., Lube Macro)
- Participated in daily instrument status meeting held with Management to report critical subsystem parameters
- Reports on instrument data are generated from real-time telemetry and TES VCDU data when real-time data is not accessible
- Instrument Events or Error Messages are summarized
- Participated in weekly ICS Meeting Reviews with TES Management and ICS Working Group
- U/L, executed, and validated new load files and investigated anomalies on the TES Engineering Model (EM) before use in flight

## TES Instrument Operations and Health Monitoring (Cont)

- TES Health Monitoring:
- All instrument subsystems are currently NOMINAL:
- Power:
- PCAA Quiet Bus
- PCAB Quiet and Noisy Bus
- PCAC Quiet Bus
- VME Voltages and Currents
- ICS Motor Current (Nominal, but being closely watched and managed).
- Laser:
- Laser a Power and Temperatures
- Mechanical Coolers:
- Cold-head
- Base-plate and electronic temperatures
- Drive stroke and vibration are Nominal.
- Passive Coolers:
- 180K and 230K passive cooler radiator temperatures
- Temperatures:
- RCS
- Beam-splitter
- Optical Bench
- Filter Wheels
- Focal Plane Assemblies
- PCS motor and mirrors
- Fore-optics and Translator temperatures

## TES Instrument Operations and Health Monitoring (Cont)

- ICS Status Since Removal of all Long Scans
- Decreased average GS ICS motor currents
- Increased idle motor currents (current measurements between scans)
- Occasional ICS Motor Over-Current Events are being generated
- NO over-velocity events detected
- Decreased in ICS background heating (HCP) heater cycle counts (believe correlated to increase in ICS idle motor current)

## Slide 9

## Slide 10

## ICS Management

- Although Higher ICS current conditions and over-current events with greater persistence have been seen before, there is cause for concern.
- TES generated 3 new 2Tranlate command macros:
- Macros to move the translator near the center of the ICS, away from the set-up scan positions:
- Macro ID 4 moves the translator to the negative side of the ICS near the center
- Macro ID 5 moves the translator to the positive side of the ICS near the center
- These macros are contingency macros that can be used to move the translator away from a stall
- Move the translator beyond the set-up scan positions
- Lube Macro (Macro ID 3) moves the translator to a position just beyond the negative set-up scan position followed by movement just beyond the positive set-up scan position to allow bearing ball swapping & lubrication that is needed to reduce risk of generating a fault.
- This implementation is consistent with the ICS Working Group recommendation in May ’08 to perform occasional long scans to move the bearings enough so that they get ball crossings
- This is also consistent with TES Mission Operations stated plan in June ’08 to consider how to implement the occasional long scans, and to investigate how to lessen the risk of an over-current fault in the process of a long scan
- All three macros were U/L on 9/29/08, and were successfully executed in Real-time on 10/13/08
- There are no observed significant changes in the ICS current conditions after executing the lube macro; results are still inconclusive
- TES is proposing to continue to execute the lube macro

## Anomalies

- The Aura spacecraft FMU/SSR anomaly.
- The anomaly was detected at 1600 GMT on December 4th, initially impacting low rate science for MLS and HIRDLS, housekeeping data and GBAD data.
- TES lost approximately 17 minutes of science data on December 5th.
- Science collection by the instruments was continued
- The Aura FOT and NGST characterize the anomaly and proposed a resolution (reset or swapping of FMU).
- TES planned placing itself in Safe mode during attempts to reset or swapping of the FMU.
- The TES instrument has been minimally impacted by the Aura FMU/SSR anomaly, primarily in the sporadic dropout of Aura GBAD/attitude data, which is used in ground processing.  From 12/17 to the current date TES data has been complete

## TES Anomalies (Cont)

- TES went into Safe mode following an ICS Over-velocity fault trip on June 9 at 14:04 GMT, during the execution of the Radiometric Performance macro.
- TES Velocity Check fault is designed to limit the travel of the ICS translator when the velocity is not tightly controlled
- The TES Velocity Check is defined with a maximum persistence of 9 Real Time Interrupts (RTIs; 0.01 seconds on TES)
- When the over-velocity fault condition persists for 10 RTIs, the ICS-only fault response (Macro 373) is triggered
- TES has seen an increase in the number of over-velocity events since December '07.
- TES Flight Software generates 'Events' when the instrument exceeds the over-velocity fault threshold (25% above commanded velocity)
- These over-velocity events tend to occur in two TES macros;
- ICS Performance Macros and Radiometric Assessment
- TES anticipated an ICS Over Velocity fault condition or stall
- TES uploaded two replacement macros, GS (Version 18) and ICS PERF (Macro ID 327) macros
- The old ICS PERF (Macro ID 325), with long scan, executed on-board just be the fault

## TES Anomalies (Cont)

- Corrective Action:
- Once all subsystem, except ICS, were reported nominal:
- The ICS was reactivated
- The new ICS Perf Macro (Macro ID 327, Without Long Scans) was put into operations
- Nominal Operations was resumed on 6/13/2008
- TES current conditions were reported nominal

## Future Operation Plans

- Continue with Global Survey Schedule
- Continue to support Special Observation Campaigns as they become defined by the TES Science Team
- Continue management of the ICS to reduce the chances for faults, to mitigate faults, and to recover from a stall
- Propose use of ICS Lube Macro again
- Anticipate Focal Plane Detector De-icing #19 in December 2008
- Continue to schedule (post GS):
- ICS and PCS Performance Sequences
- Filter Wheel Movement Maintenance Sequence
- Radiometric Assessment Calibration
- DRAM and EEPROM Maintenance Sequences

## IST/EMOS-Related Problems & Concerns

- None at this time

## TES Mission Operations Personnel Updates

- Robert Murdock (IOT Lead)
- Jeffrey Mart (IOT)
- Robert Valencia (IOT)
- Douglas Shepard (Project Manager)
- Reinhard Beer (Principal Investigator)
- David Rider (Instrument Scientist)

## TES ICS 

- The linear motion of the cube corners changes the optical path difference in the
- interferometer optical path creating the interferograms that are the fundamental quantity
- measured by TES. The basic elements of linear translator portion of the system are illustrated.
- The encoder with the suspect bearing is shown in the upper left corner of the figure.

- Teflon toroid
- separators

- 815Z Oil
- Degradation
- Products