# Presentation: 061682

## MLS Instrument Operations Status Update

- Dominick Miller** **
- Jet Propulsion Laboratory
- California Institute of Technology

- MLS Weekly Report

## Overview

**Milestones**
**Operations overview**
**Instrument performance**
**Data processing status**
**Mechanism life tests**
**Trends for the past year**
**Lessons learned**
**Changes since last year **
**MLS concerns, issues**
**Plans**
**Summary**

- MLS Weekly Report

## MLS Milestones

**Instrument Related Milestones**
**Instrument “first light” ****	****	****	****	****- Jul.   24,  2004**
**Full-up science observations start****	****	****	****- Aug. 13,  2005**
**TSSM life test completion ****	****	****	****	****- Oct.  24,  2005**
**Suspected HBT degradation leads to primary HCL band turn off****	****- Feb. 16,   2006**
**AAA life test completion ****	****	****	****	****- Mar. 06,   2006**
**MLS performs its first MCL activity****	****	****	****- Mar. 17,   2006**
**MLS Fault Management Cascade ****	****	****	****- Mar.  29,  2006**
**Anomalous behavior leads to ClO band shutdown ****	****	****- Apr.  08,  2006**
**ClO band on to observe southern polar winter****	****	****- May  18,  2006**
**Software Milestones**
**Version 1.51 (first public release) data processing starts ****	****- Jul.  24 2005**
**Version 1.51 beginning of data accessibility on GSFC DAAC,  ****	****- Feb. 15 2005**
**Version 1.52 of Level 2 (adjust for no Band 13) ****	****	****- Mar.  2006**
**Version 2 Release 0 of PGEs ****	****	****	****- Jun.  2006**
**Version 2 Release 1 of PGEs ****	****	****	****- Aug.  2006**
**Version 2 Release 2 of PGEs ****	****	****	****- Oct.   2006**

- MLS Weekly Report

## Instrument Operations Overview

**MLS has a minimal amount of routine commanding**
**Monthly AAA reconditioning (three ground commands)**
**Bi-annual EEPROM checksum verification**
**Special Activity Commanding as needed **
**Reconfiguration for special calibration opportunities**
**Moon crossing activities**
**Baseline update measurements**
**Linearity measurements**
**Cloud measurements during pitch maneuvers**
**Update instrument operational heaters**
**Change signal paths through switch network**
**Update signal chain gain settings**

- MLS Weekly Report

## Instrument Science Performance Overview 

**Performance parameters are routinely trended**
**Small reductions in signal gains have been noted **
**Gain adjustments implemented in April 2006 for most bands to boost the signals to approximately launch levels**
**Adequate signal/gain adjustment remaining for a 5 year mission**
**Based on current trends continuing**
**Bands 10/29 and 13 discussed separately**
**Next gain adjustment expected in 1-2 years**

**Instrument Science Performance** **Overview**

- MLS Weekly Report

## EOS MLS Instrument Performance Daily Operations Calendar 

- Percentage Science Data per Day (Not adjusted for planned operations)

- MLS Weekly Report

## EOS MLS Instrument PerformancePercentage Data Usable by Month 

- Average Percentage Science Data per Month (Not adjusted for planned operations)

- MLS Weekly Report

## Level 2 Data Processing Status

- MLS Weekly Report

## MLS Mechanism Life Tests

**THz and GHz Mirror Mechanisms**
**Life Test completed and report issued in Oct. 2005**
**Acceptably low amount of wear after one life test**
**GHz Antenna Actuator Assembly (AAA) **
**Life Test completed – Mar. 2006**
**In Feb. 2004, ball screw lubricant depletion led to a temporary suspension of life test**
**In Apr. 2004, life test resumed after:**
**Replenishing ball screw with a different lubricant formulation – this was done to both the flight unit and life test unit**
**Modifying life test fixture to offset the gravity loading**
**Lowering the test temperature to reflect flight unit thermal design**
**Adding a shroud to reduce O2 levels at the mechanism**
**Visual inspection of the ball screw showed good amount of lubricant and no indication of further wear after Apr’04 **
**Chemical analysis of the ball screw lubricant throughout the test indicated very small increase in metal content and oxidation stabilized over the second half of life test**
**Results indicate that the GHz AAA had an acceptably low amount of wear after one life test **

- MLS Weekly Report

## Instrument Performance Trends 

**Over 500 engineering telemetry values**
**Over 540 filterbank channels **
**Over 500 digital auto-correlator channels (trended daily as 16 high level parameters)**
**Most trends are nominal**
**Three non-nominal trends have been noted**
**The 640 GHz receiver LO IF power has temporarily dropped on three occasions but has recovered each time**
**The 640 GHz IF s****ignal feeding Bands 10 & 29 (ClO & HOCl) shows increased thermal sensitivity and anomalous behavior**
**The 640 GHz I****F Band 13 (HCl) electronics show a rapidly decreasing output signal level**

**Instrument Performance** **Trends**

- MLS Weekly Report

**Notes:**
NOTE:  Our best understanding of the gain changes at present is that they are due to normal aging of low-drop voltage regulators in the Filterbank Assemblies

## Instrument Overview Signal Flow Block Diagram

- MLS Weekly Report

## MLS Trend Examples – 1 

**Typical Signal Trends**

- MLS Weekly Report

## MLS Trend Examples – 2

**Spectrometer Module Temperature**

- MLS Weekly Report

## MLS Trend Examples - 3 

**GHz Module Temperature**

- MLS Weekly Report

## MLS Trend Examples - 4 

**THz Module Temperature**

- MLS Weekly Report

## MLS Trend Examples - 5 

**Bus V and I, Assembly Supply Voltages**

- MLS Weekly Report

## MLS Trend Examples - 6 

**Control Voltages**

- MLS Weekly Report

## MLS Trend Examples – 7 

**Additional Signal Chain Trends**

- MLS Weekly Report

## MLS Trend Examples – 8 

**Mechanism Jitter Trends APE and GME**

- MLS Weekly Report

## MLS Trend Examples – 9 

**Mechanism Jitter Trends**

- MLS Weekly Report

## MLS Trend Examples – 10

**R4 Receiver LO IF Power Monitor**

- First Drop

- Third Drop

- Second Drop

- MLS Weekly Report

## MLS Trend Examples – 11 Band 13 (HCl)

** ****At the end of 2005, the MLS Band 13 (HCl) signal was observed to degrade more rapidly than previous trends**
**After monitoring Band 13 for two additional months and seeing no recovery (similar to Band 10), the decision was made to turn off Band 13 on Feb. 16,  2006.  **
**Several additional tests revealed that Band 13 life was being conserved while Band 13 was turned off.  **
**Since Band 14 overlapped the Band 13 HCl measurement (with slightly degraded measurement performance), it was decided to use Band 14 to retrieve HCl data and to use Band 13 as a monthly “spot check” on Band 14. **
**Band 13 monthly measurements were suspended in May when Bands 10/29 started exhibiting anomalous behavior.  This was done to avoid detrimental thermal cycling of Bands 10 & 29**

- MLS Weekly Report

## MLS Trend Examples – 12 Band 13 (HCl)

- MLS Weekly Report

## MLS Trend Examples – 13Bands 10 & 29 (ClO & HOCl)

**Following degradation in their output power levels in early April, MLS Bands 10 and 29 (both part of the same signal path) were turned off to determine whether their operational lifetime can be extended by duty-cycling. **
**The results of duty-cycling were inconclusive for extending lifetime but did show a significant negative effect due to thermal cycling.  On 17 April 2006: MLS 640 GHz Bands 10 (ClO) and 29 (HOCl) were turned off.**
**These bands were turned back on in mid-late May to observe the onset of Antarctic chlorine activation. **** **
**At the current rate of decay, Bands 10 & 29 should last for the duration of the mission**
**If the signal feeding Bands 10 & 29 degrades below a useable level:**
**The ClO measurements will be retrieved from  band 5 of the MLS 190 GHz radiometer with ~2x larger noise than the 640 GHz band 10 measurements**
**HOCl is not measured by other MLS bands.**_** **_

- MLS Weekly Report

## MLS Trend Examples – 14Band 1, Nominal Trend

- MLS Weekly Report

## MLS Trend Examples – 15Band 10 (ClO) Trend 

- MLS Weekly Report

## MLS Trend Examples – 16 Band 29 (HOCl) Trend 

- MLS Weekly Report

## MLS Trend Examples – 17Band 10 (ClO) Trend Focus 

**MLS Trend Examples – 17****Band 10 (ClO) Trend Focus**

- MLS Weekly Report

## Lessons learned

**MLS Fault Management (FM)**
**Implemented late in the FSW cycle**
**Two types of FM, Mechanism and RIU_Off**
**FM needs to be disabled before and enabled after configuration changes**
**FM was tested during I&T and thought to be adequate**
**Not tested extensively during I&T**
**Live monitoring in I&T reduces the necessity to use FM **
**Frequent configuration changes during I&T would have significantly increased test time**
**What was learned **
**Enabling Mechanism FM and RIU_Off FM at the same time can lead to multiple additional RIU’s being turned off if two or more RIU’s fault at a given time (RIU Off Cascade). **
**The cause of this problem has been identified as a bug in the MLS FSW**
**RIU_Off cascades to date have all been the result of multiple mechanisms faulting at the same time.**
**Changes**
**A FSW fix has been developed but would require a large number of contacts to upload the new FSW build to the instrument**
**An operational work around is in place **
**Updated MLS FM products and procedures now enable Mechanism FM separately from RIU_Off FM **
**These updates also provide the operator with notification of mechanism faults and the opportunity to intervene and avert an RIU_Off cascade**

- MLS Weekly Report

## Updates since last year

**MLS Operations Personnel Updates**
**Richard Lay has moved on to another project at JPL**
**Dominick Miller has taken over as the  MLS operations team lead for Richard Lay **
**Mario Loo has taken on additional responsibility as an operations team member while maintaining the MLS FSW and testbed operations. **
**Gerson Melgar has rejoined the MLS team 1⁄2 time to help out with operations **
**Documentation Updates**
**MLS Operations User’s Guide updates are in process to reflect operational changes since launch**
**Operational constraints**
**Recovery steps **
**IST system changes**
**Contact List Updates – included on next slide **
**Product and Procedural Updates**
**New calibration scan tables and procedures **
**New procedures for controlling Bands 10, 29 and 13**
**Fault management products and procedures**
**Updates will be made to MLS_DSP_Safe.prc**
**Pre-existing red limit response procedures are still valid**
**Telemetry pages **
** **
**9/80 Work Week**
**If mandated, the instrument would be monitored consistent with the 9/80 schedule versus 5/40**
**At this time 9/80 is optional and MLS Ops team choices would still have 5/40 scheduled monitoring**
**MLS has also 24/7 software monitoring of PDS files as they are delivered to JPL**
**Includes automated text messaging to Operations team and Instrument Scientist **

- MLS Weekly Report

## Contact Information

|  | Office | Cell Phone | EMAIL Address  |
| --- | --- | --- | --- |
| Dominick Miller (Operations Lead) | (818) 393-3564 | (818) 653-7852 | dominick.miller@jpl.nasa.gov |
| Mario Loo (Instrument FSW & Operations Team) | (818) 354-5607 | (818) 687-8107 | mario.s.loo@jpl.nasa.gov |
| Gerson Melgar (Operations Team) | (818) 354-6350 | (818) 653-3078 | gerson.melgar@jpl.nasa.gov |
| Robert Jarnot (Instrument Scientist) | (818) 354-5204 | (818) 653-9266 | robert.f.jarnot@jpl.nasa.gov |
| Herb Pickett  (THz Principal Investigator) | (818) 354-6861 | (818) 640-6068 | herbert.m.pickett@jpl.nasa.gov |
| Vince Perun (Ground Software) | (818) 354-2271 | N/A | vincent.s.perun@jpl.nasa.gov |
| Dave Cuddy (Project Manager) | (818) 354-2099 | (818) 648-9762 | david.cuddy@jpl.nasa.gov |
| Nathaniel Livesey (Principal Investigator) | (818) 354-4214 | (818) 219-6394 | nathaniel.j.livesey@jpl.nasa.gov |

- MLS Weekly Report

## MLS Concerns & Issues - 1

**Instrument Support Terminal Status**** **
**IST Online connectivity is still having problems with the new DMZ **
**Remote Desktop disconnects and Workspace data resets are still occurring ~1-2 times per week **
**Remote Desktop Workspace sessions are not updating as they should**
**Workspace windows have no data or clock updates from the Remote Desktop disconnect to Remote Desktop reconnect**
**Denver Raytheon has been working these issues**
**New items: **
**GMT Clock “freezes” have been noted on two occasions while stripchart data continues to update with the appearance of stale data**
**MLS IOT can no longer see spacecraft related telemetry pages, directories restricted**
**While not a requirement, the following capabilities were available but are not any longer:**
**Dual head monitor capability**
**AOS/LOS clock display**
**It is possible to perform nominal operations with the current configuration but, operations during anomalous behavior would most definitely be complicated and more prone to errors without these two capabilities**
**New Online DMZ parameters need to be defined/redefined and documented:**
**Product upload server, Toronto is gone**
**CM directories, EOC/DMZ/both?  **

- MLS Weekly Report

## MLS Concerns & Issues - 2

**Auto-Operation Impacts **
**One lost packet per dump**
**If lost packet is science data - minor impact**
**If lost packet is engineering/calibration data**
**24.7 seconds of MLS science data would have no calibration values**
**One “granule” of science data would be unusable do to data discontinuities and the science processing algorithms**
**First order calculations show that ~ 2% of packets lost would be engineering/calibration data packets, ~ 2 packets/7days**
**May be able to interpolate missing engineering data but would require significant data processing software modifications**
**The MLS team has a strong apprehension against “tossing” good science data for the sake of saving money**
**MLS would like to like to see further investigations into fixing this problem at the spacecraft/ground system level as opposed the instrument level**

- MLS Weekly Report

## Instrument Plans

**Support Aura inclination maneuvers, AVE campaigns and other spacecraft activities**
**Continue with routine MLS activities**
**Perform MLS special calibration activities**
**Moon Scans**
**Baseline updates**
**Calibration Synthesizer test**
**Other tests as needed**
**Bands 10 & 29**
**Band 10 was turned on for the southern polar winter and is doing well at this time**
**For the duration of the southern polar winter, the instrument will be operated in a manner that minimizes Band 10 thermal variations**
**The general feel for Band 10 is to leave it on after the polar winter, but this decision will be revisited  **
**Band 13**
**Past measurements show that Band 13 life is being conserved while Band 13 is off**
**The one day per month duty cycle for Band 13 has temporarily been suspended due to the thermal variations it causes on Bands 10 and 29**
**Resuming the one day per month Band 13 measurements will be revisited after the southern polar winter**

- MLS Weekly Report

## Summary

**Status Summary**** **
**While Bands 10 and 13 have presented some challenges, the overall MLS instrument performance continues to be very good**
**All science products are still being retrieved with only a very slight degradation in the quality of the HCL measurement**
**MLS calibration tests confirm the health of the instrument**
**MLS data processing systems are operating smoothly **
**All Level 1 and Level 2 files routinely archived & available from GSFC DAAC**

- MLS Weekly Report