# Presentation: 195473

## Warts, Bumps, and Blemishes

***Experimenting with Sensor Webs Using EO-1******2 March 2004***

*Stuart Frye**Mitretek Systems*
*EO-1 Systems Engineer*

## Contents

***EO-1 Sensor Web Functionality***
***Mission Systems Configuration***
***System Modeling for Autonomy Migration***
***Interface Scripts and Glue Code***
***Mission Planning Activities***
***Ground System Accommodations/Upgrades***
***Flight Software Changes***
***Issues, Warts, Bumps, and Blemishes***
***Lessons Learned***

## EO-1 Sensor Web Functions

***EO-1 Sensor Web Functions***

**Trigger Detection/**
**Posting/Clearinghouse**

**New Triggers**

**Data Delivery Status**

**T**
**R**
**I**
**G**
**G**
**E**
**R**

**I**
**N**
**F**
**O**
**R**
**M**
**A**
**T**
**I**
**O**
**N**

**O**
**B**
**S**
**E****R****V****A****T****I****O****N**

**R**
**E**
**Q**
**U**
**E**
**S**
**T**
**S**

- Detect/LocateTriggering Events	Hot Pixels	Cloud Coverage	Edges/Boundaries	Etc.
- Post Trigger Info, Send to Subscribers
- Use Raw Sensor Data, Derived Products, or Model Outputs as Source (Direct Broad-Cast/Rapid Delivery...)

- Watch for Triggering Events
- Parse Science Goals For Data Requests
- Negotiate Follow-up Observations with Participating Platforms
- Track Status of Data Acquisition/Delivery

- Implement On-boardScheduling/Processing/Feature Identification/Diagnosis Software
- Coordinate Normal Operations with Experiment Activities
- Determine Tie-breakersfor Competing Requests
- Perform SpecialReacquisition Maneuvers/Sequences

**Observation Execution/**
**Processing/Assimilation**

**Sentinel Monitoring/**
**Scheduling/Status**

## EO-1 Mission Systems

***EO-1 Mission Systems***

**EO-1**

## Automated Sequence Generation

***Mission goals ***
***E.g. – image Kilauea (Lat/Lon)***
***To Command Sequence***

**2003:233:16:49:57 CMD ACSETWHLBIAS(INERTIAL,X=0.341589,Y=1.1749,Z=-0.118046); **
**2003:233:17:56:57 CMD ACGOTOMANEUVER(ORBITAL,TIME=900,XLIMDEG=0.02,YLIMDEG=0.062699,...); **
**2003:233:18:07:06 CMD I_SETFPEPOWER(POWER_MASK=5); **
**2003:233:18:07:06 CMD YHEASTBY; **
**2003:233:18:07:16 CMD YHEASETSWIR(GAINA=1,GAINB=1,GAINC=1,GAIND=1,...); **
**2003:233:18:07:26 CMD YHEASETVNIR(VNIRALV8,VNIRBLV8,VNIRCLV8,VNIRDLV8); **
**2003:233:18:11:06 CMD I_CONFIGFPE(CONFIG_COMMAND=16908); ...**
**2003:233:18:17:06 CMD BCMMODESCRS422; **
**2003:233:18:17:16 CMD WRMSREC(IDWS=65535,IDWV=65535,...); **
**2003:233:18:17:54 CMD I_SET_FPE_DG(DURATION=-1); **
**	****...**

## Uses Model of Activities

***Resources***
***States******	******	***
***	***
***Other Activities***
***These models are then combined to model the world as it changes due to activities***

- Uses 14 files; uses XXX memory
- requires target pointing

- Activity: Science Image Acquisition

- variable, dependent
- on activity duration

## Night collect

**Warp ****mode**

**Hyp ****State**

**Night collect**

**Day collect**

**Downlink**

**Hyperion ****Preparation**

**Target ****in view**

**X-band****Ground ****Station**

**WARP****file ****count**

**WARP****data****volume**

## CASPER Planning

***CASPER can implement nominal procedures through decompositions (similar to scripts)***
***In order to image: do x, then y, then z...***
***CASPER can also perform planning “from scratch” via search***
***If want ACS-mode state variable = standby, consider adding an activity that changes ACS-mode (then the requirements of these activities may be new conflicts,...)***
***Most commonly used search framework “iterative repair”***

## Activities, Constraints, Repairs

| Constraint | Property that must hold for plan to be valid | Must always use less power than available |
| --- | --- | --- |
| Conflict | Violation of a constraint | Current plan uses more power than available from 18:00-18:30 |
| Repair Method | Modification to plan that may remove conflict | Delete activity using power during conflict |
| Repair Choice | Which activity to delete | Delete largest user? |

**contributors**

**a)**

**b)**

**conflict**

**Power Usage**

**Activities**

## Constraint Resolution Tree

- Activity

- Open
- Temporal Constraint

- TimeLine

- Disconnect

- Connect

- Delete

- Apply Dependency

- Violated
- Dependency

- Move

- Add

- Unground
- Parameter

- Abstract

- Detail

- Lift

- Ground

- Violated
- Temporal Constraint

- Mismatched
- Decomposition

- Redetail

- Place

- Unplaced
- Activity

- Activity
- Type

- Decomposition

- Value

- Constraint

- Activity

- Activity

- Activity

- Activity

- Activity

- Start Time

- Duration

- Start Time Interval

- Constraint

**Repair Method:**

**Method Choices:**

***Constraint Resolution Tree***

**Conflict Type:**

## Repair Algorithm

- Start
- (if conflicts exist and user time-limit not exceeded)

- ...

- Select a conflict

- Select a repair method

- ...

- move

- ...

- ...

- Select an activity

- Select a start time

- Perform the
- action, collect
- the new conflicts,
- and repeat

## Interface Scripts and Glue Code

***Interface Scripts and Glue Code***

***SGM-generated observation requests are ingested to MOPSS***
***Target information and maneuver files exported from MOPSS are encapsulated into CASPER Goal Files***
***Tie-breaker selections received from SGM are encapsulated in commands and uplinked to spacecraft***
***CASPER replans for triggered target and executes new plan ******on-board***

***PERL Scripts handle traffic between SGM and EO-1 MOC via formatted Email running on Secure Shell***
***Scripts send target Lat/Lon to Matlab and return computed target In-view times to SGM***
***Matlab interfaces converted from GUI versions to command line callable routines***

## New Mission Planning Activities

***New Mission Planning Activities***

***Experiment Time Slots Need to be Integrated into Commercial Observation Schedule for Every Experiment***
***Placeholder Target and Comm Requests Are Inserted to Pre-Populate Schedule***
***SGM-generated Records ******Are Ingested and ******Placeholders ******Overwritten***
***Image Sequences ******Are Exported as******Input to Goal File******Generation  Scripts for CASPER-Scheduled Requests***

***Exported Sequences Are Removed from MOPSS ******Command Load***
***Overlap Between Exported CASPER Sequences and Command Load Sequences Are Verified/De-Conflicted ***
***Including Verifying Continuity of Scene Information from Exported Sequences in Load Reports for Downstream Antenna Operations, Science Data Processing, and Scene Tracking Systems***
***Need to pick which comm contact to use to load/jump to new ******on-board code to avoid other overlapping operations on processor***

***Don’t forget to push the blue button at 8:07GMT***

## Ground System Accommodations/Upgrades

***Ground System Accommodations/Upgrades***

***Created new procedures for sending sensor web triggers to spacecraft, loading new code on-board, jumping to new executables ***
***Modified command uplink acknowledgement scheme and timeout settings to handle large code uploads***
***Modified command database for new autonomy commands***
***Modified telemetry database for new autonomy telemetry***
***Modified Systems Test and Operations******Language (STOL) procedures to ******perform code load, checksum, ******uncompress, jump, goal/script ******activation, WARP reset***
***Modified max slew rate from .25 to .433 deg/sec (Re-image scenario)***
***Increased number of retransmit entries in FEDS command queue***
***Upgraded trending system to pickup new telemetry mnemonics***

***No, not THAT blue button!***

## FSW Overview (Block Diagram)

***FSW Overview (Block Diagram)***

## WARP Software Architecture

- VxWorks / Tornado (OS)

**MSSP I/F**
**Task**
**(MP)**

**PM I/F**
**Task**
**(PM)**

**CFBIU**
**I/F Task**
**(CF)**

**Memory**
**Scrub Task**
**(MS)**

**Health&**
**Safety**
**Task**
**(HS)**

**Recorder**
**Management**
**Task**
**(RM)**

**Memory**
**Dwell Task**
**(MD)**

**Checksum**
**Task**
**(CS)**

**Software**
**Manager ****Task**
**(SM)**

**1773 RT**
**Task**
**(RT)**

- Newly Developed
- Task for EO-1 WARP

- Re-Used Task from
- MIDEX/MAP

- Interrupt-Driven
- Device Driver

**Time Code**
**Task**
**(TC)**

***WARP Software Architecture***

- 1773 RT-
- Driver

- MSSP
- Driver

- PM
- Driver

- CFBIU
- Driver

## Integrated “Plug and Play”, using SCL as adapter

- WARP Software Bus (SB)

- VxWorks / Tornado (OS)

**Existing**
**WARP**
**Drivers**
**...**

**Mem Dwell**
** ****Cloud**
**Cover Task**

** ****Bridge**
** ****Task**

- SCL Software Bus

**SCL**
**Script**
**Tasks**

**CASPER**
** ****& Science****Task**

**SCL**
**Command**
**Tasks**

**SCL**
**Telemetry**
**Tasks**

- EO-1 TLM Channels VC0 & VC3
- 2 APIDs for SCL Control
- 2 APIDs for SCL real-time control

- SCL Commands(to C&DH M5 via WARP Remote Terminal)

**WARP ****Remote ****Terminal**

- New

- Existing

**Here is how we implemented it on EO-1, an existing on-orbit satellite, as an experiment**

**Onboard planning and scheduling tool**

**Livingstone****Task**

**Onboard diagnostic tool**

## Flight Software Lab

***Flight Software Lab***

***Developed capability to reload WARP Flight Software kernel and patch to boot from new image using hijacked existing command***
***Developed C&DH patches (next page)***
***Integrated Spacecraft Control Language (SCL)  and CASPER spacecraft autonomy software with WARP flight code***
***Developed utilities for encapsulating executables into S records for memory load STOL commands***
***Upgraded VirtualSat to simulate additional command, telemetry, and event message traffic***
***Implemented remote access for integration work via (Tight) Virtual Network Computing ***
***Implemented file transfers for code loads via Secure Shell***
***Developed ability to compress and decompress executable code loads to reduce uplink bandwidth requirements***
***Procured and integrated two additional ******test strings***
***2 C&DH Mongoose 5, 2 WARP Mongoose 5,******2 VirtualSat simulators, 1 Spare Mongoose 5***

***Now I see why they didn’t fly that board!***

## On-Board Changes to C&DH

***On-Board Changes to C&DH***

***Software Routing updates to allow ******Commanding from the WARP***
***Telemetry Filter Table modifications ******to accommodate CASPER/SCL ******Telemetry Downlink and On-board ******Recording***
***New Telemetry Statistics Monitor (TSM) to automatically enable sun maneuver avoidance TSM upon daylight entry every orbit***

***It didn’t work that way in the lab!***

## On-Board Changes to WARP

***On-Board Changes to WARP***

***Reloaded entire WARP code image and jumped to it via patch***
***Modified Memory Dwell task and S-band playback function in WARP Flight Code to read science data into RAM from near-line bulk storage***
***Created various SCL and CASPER-related tasks***
***Hijacked telemetry packets and commands for SCL and CASPER use***
***Loaded new CASPER, SCL and cloud assessment algorithm on-board***

***Added Event Messages for status reporting***
***Modified checksum configuration on WARP for upload verification***
***Increased WARP Watchdog timeout to prevent reset when booting to new larger code***
***Turned Off CPU hogging and changed Memory Dwell task check-in error to an event – had caused warm restart ***
***Implemented a decompression******utility on-board based on zlib library inflate function***

***Explain to me again why I can’t playback science data over S-band or run memory diagnostics with CASPER running***

## System Engineering Issues

***CASPER knows spacecraft state and resources***
***Doesn’t do navigation, orbit propagation,...***
***Doesn’t do momentum management/maneuver planning***
***Has to coordinate file naming conventions with Command Load observations***
***Changeover from Command Load to CASPER control***
***Better coordination required because more complex activity sequences are being undertaken***
***Operational sequences are not independent***

## Warts (1 of 3)

***FSW lab hardware not identical to flight hardware***
***WARP Flight Processor has 256Mbytes RAM, but breadboard in lab has 32M memory for integration work – limits use of full on-board memory***
***Off-line WARP bulk memory cards not procured for EO-1 lab (>$1M) limits testing for image data file manipulation code***
***Insufficient memory in Flight Software Lab Breadboard caused several month delay in integration effort***
***Sensors and Mechanisms simulated using VirtualSat***
***Cannot duplicate on-board dynamics in lab (e.g., CPU starvation)***
***Unexpected spacecraft reactions encountered during experiments***
***On-orbit debugging required***
***Had to use outgassing periods every 16 days to run experiments***
***Always a stretch to define scope, schedule support, deliver tested code and unzip/jump/verify procedures in time for uplink***

## Bumps (2 of 3)

***Code loads to testbeds in FSW Lab slow at first - sped up by implementation of ICEPROMS and/or Ethernet on Mongoose boards***
***Takes 3-4 days to uplink code loads to spacecraft ***
***15-20 ground station contacts***
***TDRS not reliable for large uplinks – can only use ground stations***
***6Mbyte code loads to spacecraft compress by about 6-1***
***Encountered problems verifying large uplinks***
***Not enough time to do full dump and compare***
***Using checksums was labor intensive and discrepancies hard to isolate***
***Made for some exciting tests....***
***WARP reboots during dumps causes dump flag to hang on C&DH***
***Had to stuff WARP dump bit to YES, then send abort to clear C&DH flag***
***Still ran experiments on non-verified code – Oh Well!***

## And Blemishes (3 of 3)

***On-board Cloud Detection takes 15 minutes to run on-board***
***Not sufficient for look-ahead/assess/retarget scenarios***
***Next load of FSW will allow selectable readout of hyperspectral bands and selectable readouts of particular rows of the image data file to speed up***
***Special care has to be taken to avoid invoking on-board memory operations during command load event windows***
***No code loads, script updates, dumps, jumps, or other activation/deactivation memory operations during WARP Record or Playback events***
***Crashed WARP once – memory starvation issue***
***Spacecraft was under CASPER control***
***Crash occurred during image sequence – Watchdog check-in***
***Left spacecraft maneuvered with instruments on and covers open***
***Had to recover manually during next communications contacts***

## Lessons Learned

***Build excess CPU and memory capacity into Flight Segment***
***Enables sensor web/autonomy improvements post-launch***
***Include at least 2 flight processors on-board in future designs***
***Can do development work without disturbing C&DH operations***
***If 2******nd****** processor is not executing new FSW properly, reboot to old code***
***Build ground FSW Lab with identical hardware to Flight Segment***
***Minimize time spent on development of support tools and utilities during early part of software effort***
***Concentrate on primary functionality until better tools would save time***
***Learn through failure if it’s safe to do so – if you wait until you’re 100% sure of success, you may never get anything done***
***Setup safeguards to auto-recover via command load after crashes are encountered during experiments***
***Need to setup process for delivery of science data from experiments - problematic in commercial data sales setting***