# Presentation: 659629

## Exploring the Sun-Earth System 

- Exploring the Sun-Earth System

- An alliance of solar, heliospheric and geospace missions for the coordinated study of Sun-Earth system dynamics.

## Re-engineering of flight operations and data systems for Polar, Wind and Geotail

- Outline:
- Review of ISTP/CDHF
- HQ directive for FY02-FY05
- Implications of that directive
- Re-engineering the Polar, Wind and Geotail operations
- Issues and concerns

## Review of the ISTP Data System(an independent entity of ISTP to serve the worldwide SEC community)

- In addition to
- data processing and distribution services for GGS (Polar, Wind, Geotail) & IMP-8,
- data distribution services for SOHO and Cluster,
- and operations and science coordination,
- ISTP served as a one-stop data source by
- consolidating and distributing data for 15 additional spacecraft, observatories and T&M programs
- and by providing extensive data and media integrity and quality services.

**Notes:**
Dedication: October 30, 1991

## Directive from HQ for FY02-FY05

- The end of ISTP as a program.
- Continuation of Polar science activities as the apogee progresses through the equatorial regions.
- Reduction of NASA support in science participation in Geotail.
- Placement of Wind at L1 as a "hot spare" for monitoring the solar wind and limited support to science teams for special campaigns.
- Termination of the ISTP Theory and Ground Based Investigations program.
**Termination of the ISTP/CDHF as an independently funded facility.**

- The Senior Review 2001 evaluated separately the five elements of ISTP/GGS and recommended a substantial restructuring.

**Notes:**
Have copies of the two "do not transmit letters" available.

## Directive from HQ for FY02-FY05

- Accept higher risk levels than during the prime mission phase
- minimize operations and data processing costs in order to maximize  the use of funds for instrument calibration, data analysis and science interpretation.
- Accept a lower data capture rate from 99+%
- a 95% data capture that lessens demands on the operations and data processing staffs is acceptable.

- HQ recommended that the continuing missions,

- In addition, HQ reorganized the funding authority such that each spacecraft project scientist has budget responsibility for implementing the complete science, operations, data processing and data distribution program elements.

**Notes:**
Current capture rate is over 99%.

## Implications of HQ Directives

| FY01
Actual,        not inc. SOMO | FY02
cost estimate,
after initial reductions | FY02
budget allocation | FY03
budget allocation | FY04
budget allocation | FY05
budget allocation |
| --- | --- | --- | --- | --- | --- |
| $5.28M | $3.94M+ | $4.12 | $2.8M | $2.3 | $2.3 |

- Polar, Wind and Geotail MO

- SOHO and Cluster have effectively pulled out of the CDHF.
- The ISTP flight operations and data system costs considerably exceeded funding to be provided for FY02 and beyond for PWG.
- HQ provided funding for re-engineering the PWG flight operations and data systems to the Polar project.
- SEC projects were encouraged to investigate new approaches to "find the best deal".

**Notes:**
For Polar, the budget provided represents a substantial reduction over the funds expended in FY01 and will require a substantially different approach to mission operations and data processing if a viable science program is to be maintained. The budgets for Wind and Geotail are near or are at “phase down” levels and require an overall reduction to a “minimum maintenance” mode of operation.

## Immediate Courses of Action

- The ISTP project office was disbanded and reduction of past ISTP activities occurred during October and November of 2001.
- ISTP ground system services immediately reduced were:
- QuickLook data processing for special requests only,
- key parameter CD distribution reduced from ~12,300 to 156 per year,
- no key parameter re-processing,
- no key parameter software updates,
- no ground based or collaborative mission data processing or ingestion,
- no dedicated program assistance center,
- no system software updates excepting security patches,
- no test or development environment, and
- no off-hours data processing or problem response
- services of the ISTP SPOF and Command Management System consolidated with the MOC and the project scientist's office.

**Notes:**
Under the current operating environment, these reductions are not sufficient to bring the Polar, Wind and Geotail mission operations and data processing costs in under budget. 

## Immediate Courses of Action

- Contacted instrument teams regarding their requirements for various operations and data services.
- Produced prioritized requirements document for Polar, Wind and Geotail.
- Conducted five feasibility studies for alternative approaches
- 1) the present ground system management under CSOC
- 2) SPDF at GSFC under the direction of Bob McGuire
- 3) LASP at U Colorado under the direction of Bill Peterson
- 4) UC Berkeley under the direction of Bob Lin
- 5) the NSSTC under the direction of Dennis Gallagher

- The University of Maryland and Johns Hopkins University Applied Physics Lab declined to conduct a study.

**Notes:**
Under the current operating environment, these reductions are not sufficient to bring the Polar, Wind and Geotail mission operations and data processing costs in under budget. 

## Results of Feasibility Studies

- The following conclusions and actions resulted from the review of the studies:
-  More software and system re-engineering needs to be pursued than provided for by the CSOC/CDHF study and CSOC estimates for flight operations.
- Seven re-engineering projects were identified that can, potentially, reduce the number of FTEs by three-quarters.
-  The re-engineering work should be performed under local control.
-  The UC Berkeley capabilities for hosting mission operations remain of interest.

- It has been determined that a consolidation of all systems under the PWG Mission Operations Center (MOC), as suggested by the SPDF study, offers the most cost effective solution with the least disruption and the least risk to spacecraft health and safety.

## New Data System Overview

- The following Polar, Wind and Geotail services provided by ISTP are to be retained:
-  Near Real Time (NRT) data, open line access
-  Quicklook (spacecraft playback) data, online access
-  Level Zero data processing, online and CD distribution
-  Geotail Sirius data processing, online access
-  Key Parameter data processing, online and limited CD distribution
-  Ancillary data processing, online access

## Plan for Re-engineering

## Plan for Re-engineering

- A mission operations and data processing re-engineering team has been formed:
- Polar, Wind and Geotail project scientists, sys admin & programmer (code 690)
- Space Physics Data Facility computer scientists (code 630)
- Information Systems Center computer scientists (codes 586 and 587)

|  | Re-engineering task | Responsible person |
| --- | --- | --- |
| 1. | unattended spacecraft contacts for data playbacks | Steve Odendahl,  Polar Mission Director |
| 2. | cross-training of flight operations personnel | Steve Odendahl,  Polar Mission Director |
| 3. | re-hosting the CMS for security and obsolescence issues | Rick Burley, Code 630 |
| 4. | automation of KP processing | Jim Byrnes, Code 587 |
| 5. | simplifying online distribution of LZ and ancillary data | Bobby Candey, Code 630 |
| 6. | automation of CD production | Bobby Candey, Code 630 |
| 7. | streamlining LZ processing to include NRT and QL | TBD |

## Unattended Spacecraft Contacts for Data Playbacks

- Notes for instrument teams:
- Fewer attended contacts/fewer “double” attended contacts may mean less convenient or delayed command scheduling.
- There may be a request to the instrument teams to scrub their list of monitored parameters.

- 7-9 console operators,
- covering 24x7, typically 4 shifts of two operators,
- Wind: 1 contact/day,
- Polar: 4 contacts/day,
- All attended contacts

- 4 console operators,
- covering 16x5,
- Wind: 3 contacts/week,
- Polar: 3-4 contacts/day,
- TBD number of unattended contacts

- _Current_

- _After_

## Cross-training of FOT Personnel

- Notes for instrument teams:
- Cross training of FOT personnel should bring better service for day-to-day operations.

- 7 console operators

- 3 CMS operators

- 3 spacecraft engineers

- 2 operations managers

- 1 DSN scheduler

- 4 console operators

- 1 CMS operator

- 3 spacecraft engineers

- 2 operations managers

- 1 DSN scheduler

- _Current_

- _After_

## Re-hosting the CMS for Security and Obsolescence Issues

- Notes for instrument teams:
- Core CMS remains the same, no changes planned for command input from instrument teams.
- Re-hosting should be transparent; possible requests to participate in parallel testing/ops period.

- spof1.gsfc.nasa.gov
- outside firewall, receives/relays
-  commands & verifications

- _Current_

- _After_

- CMS
- inside firewall, verifies command sequences, etc.,
- on older VMS machine

- Decnet “push”,
- identified as security risk

- spof7.gsfc.nasa.gov
- outside firewall, receives/relays
-  commands & verifications

- CMS
- inside firewall, verifies command sequences, etc.,
- re-hosted on new PC

- TCP/IP

- FORMATS
- secure relay across firewall

- Bldg. 2

- Bldg. 3

- Bldg. 3

- Bldg. 3

## Automation of KP Processing

- _Current_

- _After_

- Notes for instrument teams:
- Software port process _should_ generally be transparent to instrument teams.
- Instrument teams which retain KP software programming expertise may be asked to provide consultation services to GSFC team.
- Verification of file format and content by instrument teams will be required after porting.
- Routine quality checking of KP file production will reside with instrument teams.
- Possible data loss.

- TAE GUI interfaces on VAX
- to shift of operators

- Custom software & Oracle
- on Dec Alpha control processing

- Frozen software library

- Extensive quality checking

- Automated file processing

- Consolidation to single machine with ready access to data

- Software library converted to
- collection of individual processes

- Limited quality checking

## Automation of KP Processing (more details)

- _Current_

- _After_

## Simplifying Online Data Distribution

- _Current_

- _After_

- Notes for instrument teams:
- Data by ftp pull only; data push and various user interfaces to be terminated.
- LZ data older than 2-3 months to be gzip compressed
- LZ long file names, there has been a suggestion for a different file naming convention.
- No index files or SFDUs
- All data will be public including NRT, QL, LZ, etc.

## Simplifying Online Data Distribution - more details

- _Current_

- _After_

## Automation of CD Production

- Notes for instrument teams:
- DVDs for Polar_all, Wind_all and Geotail_all distribution
- Current (downsized) CD distribution to instrument teams to be retained
- CD directory structure to be retained
- No index files or SFDUs
- Quality control of CD product to be performed by receiver; slow replacement

- _Current_

- _After_

## Automation of CD Production - more details

- _Current_

- _After_

## Streamlining LZ Processing to Include NRT and QL

- Notes for instrument teams:
- Software port process _should_ be transparent to instrument teams.
- File types and formats would remain identical.
- Verification of file format and content, by instrument teams, will be required after porting.
- Routine quality checking of LZ data files will reside with instrument teams.
- Reprocessing/replacement of LZ data may be limited to 1-2 months after receipt of files.
- There will be some data loss.

- under study for possible implementation

- Wind/Polar
- NRT on older microVaxs

- Wind/Geotail QL & LZ

- Bldg. 23

- Bldg. 3

- _Current_

- _After_

- NRT for Wind & Polar rehosted to Dec Alpha

- LZ for Wind & Polar rehosted to Dec Alpha

- Wind/Polar
- NRT on older microVaxs

- Wind/Polar
- NRT on older microVaxs

- Wind/Polar NRT
- (on older microVaxs)

- Unix using LabView & Oracle

- Polar QL & LZ

- Bldg. 3

- Geotail QL & LZ

## Approximate Schedule for                 Re-engineering Activities

- Mar

- Apr

- May

- Jun

- Jul

- Aug

- Sep

- Oct

- Nov

- Dec

- 2002

- unattended spacecraft contacts for data playbacks

- cross-training of flight operations personnel

- re-hosting the CMS

- automation of KP processing

- automation of CD production

- streamlining LZ processing to include NRT and QL

- simplifying online distribution

- increasing priority

## Summary

- Process of re-defining the Polar, Wind and Geotail flight operations and data systems requirements is complete.
- Feasibility studies were conducted to explore fresh approaches.
- Identified areas, responsible parties and funding for re-engineering tasks.
- Initiated re-engineering tasks.
- The re-engineered flight operations and data system should retain most, if not all, Polar, Wind and Geotail specific processing functions previously provided by ISTP.
- The re-engineered system is expected to properly support the PI teams and be affordable.

## Concerns on the Project Side

- Unintended impacts to instrument teams.
- If any re-engineering tasks are not successful, either technically or fiscally, can the impact be afforded?
- Can we find a contractual environment for operations that is affordable and legal?
- Can the re-engineered system collapse to an affordable Wind-only environment after Polar and Geotail end of missions.
- Where should the separation be between project data processing and NSSDC data distribution?
- What is the minimum data recovery percentage acceptable in light of fiscal constraints imposed on extended mission programs.

## Concerns on the Instrument Team Side