# Presentation: 169963

## Architecture Analysis of Evolving Complex Systems of SystemsTechnical PresentationSoftware Assurance Symposium2008

**Principal Investigator (PI): Dr. Mikael Lindvall, FC-MD**
**NASA POC: Sally Godfrey, GSFC**
**Team members: ****Chris Ackermann, Dr. Arnab Ray, Lyly Yonkwa, Dharma Ganesan (FC-MD)**** ****William C. Stratton, Deane E. Sibol (APL) **
- Fraunhofer Center for Experimental Software Engineering Maryland (FC-MD)
- Fraunhofer Institute for Experimental Software Engineering (IESE)
- Johns Hopkins University Applied Physics Laboratory Space Department Ground Applications Group (APL)

## Outline

- Motivation
- Background: (static) SAVE
- Dynamic SAVE Vision
- Dynamic SAVE examples
- Applicability Throughout the Life Cycle

## Problem/Approach

- Systems are often difficult to understand
- Systems of systems adds to the challenge
- Makes system verification difficult
- Interfaces often source of problems
- Approach
- Architecture analysis focusing on interfaces
- The new tool, Dynamic SAVE,
- extends the already existing *static* Software Architecture Visualization and Evaluation (SAVE) tool

## Background: The (static) SAVE Tool Software Architecture Visualization and Evaluation

- Does the actual implementation match the planned architecture?
- Define a *planned* architecture
- Create an *actual* architecture from source code
- Identify architectural violations through comparison
- Applied to APL’s Common Ground System, Research Infusion project
- Conclusions (Aerospace 2007)
- The SAVE approach is useful and practical
- One can quickly model, visualize, analyze, find static architecture violations
- Good for single software applications
**But for systems of systems, some questions remain unanswered... **

## Application-Specific Modules

- Encapsulation of client/server interface

- Encapsulation of socket communications

- A Planned Architecture

## The Actual Application Architecture

- Where’s socket implemented?

## The Actual Architecture vs. The Planned

## The Common Ground System

##  Dyn-SAVE Vision

-  Dyn-SAVE Vision

- Telemetry**Server**

- Telemetry**Client**

- Specify Planned
- Behavior

- Form Actual
- Behavior

- Specify Level of Abstraction
- For analysis

- Capture Dynamic
- Information

- Compare Planned and Actual
- Behavior

**Who does socket communicate with?**
**Is communication according to specification?**
**Check ****Sequences****, Parameters, Values, Timing**

## DynSAVE in practice

- The Common Ground System

- These systems all have ICDs
- (Interface Control Documents)

## Focus on:Interface Control Documents

- NASA systems often developed by different teams
- Interface Control Documents (ICD) is key, but
- ICDs often interpreted differently because
- ICDs implicit, lack important details etc.
- Cause subtle critical deviations from specified behavior
- Deviations difficult to detect
- Emerging behavior difficult to predict
- Can result in severe problems, e.g. lost data, performance
- Need to
- Detect deviations before deployment
- (Specify expected and actual behavior before creating ICD!)

## Some Research Questions

- Sequence diagrams
- Can we use sequence diagrams to model, abstract, and detect such deviations?
- Can sequence diagrams express what we need?
- Iterative modeling
- Can we start with abstract models, add details as necessary?

## Approach

- Collect concrete examples from APL
- Model planned behavior
- Use specification from ICD
- Capture actual traces
- Use Archive_Server and Eng_Dump
- Generate Client scenarios, observe how Server responds
- Identify common patterns

##  Planned sequence diagram

- The “simplest” diagram that describes the planned communication behavior described in the ICD

## Example 1: Illegal filter

- An illegal extra filter is sent after BeginPlayback and Data messages  have been sent.
- The illegal filter is difficult to detect because it is in packet 869.

**Notes:**
This is ignored by the server (in fact, it might not even get read off the socket).
Server gets stuck, block on that client
Client might expect filter to be in effect,

## Detailed sequence diagramexperimental notation

- 	1. Start time must be less than stop time
- 	2. Data type of each of the received data messages must match specification

- Detailed sequence diagramexperimental notation

## Example 2: Illegal Type specification

- STF ordered – STP received.

**=STF)**

**Notes:**

STF used to be a supported type, but no longer is
Old legacy clients might still request it, if they have not updated to the new ICD
Unexpectedly, you’ll see that the server returns STPs
The server simply ignored the invalid filter and used its default which is TYPE STP
The correct server behavior should have been to just close the socket on an invalid filter  
This can be a significant problem because the client might assume that same type of data that was requested will be returned and process the incoming data according to that type
In fact, several internal clients, including Eng_Dump, are indeed coded in this fashion and do not verify that the type returned and about to be processed is of the type requested 
Obviously, processing data of the incorrect type will cause incorrect data to be generated or even client application crashes

## Adding Timing Constraints

## Checking for Timing Problems

## CFDP – A Mission Data System Protocol

- CFDP provides reliable downloads of recorded on-board data
- The implementation is distributed across flight and ground systems
- The protocol runs on top of unreliable CCSDS command and telemetry layer
- At APL, CFDP is mostly automated, but...
- Operators turn off CFDP uplink during critical command load sequences
- Operators freeze and thaw timers - pending transactions don’t time out between contacts
- Improper CFDP operation can lead to unnecessary retransmissions, wasting precious downlink bandwidth

## DynSAVE monitoring of CFDP

- Monitors macro-level behaviors without affecting flight or ground software
- Detects improper CFDP operation, for example:
- timers were not frozen and uplink was disabled on the ground for an extended period, causing multiple retransmissions when the uplink was finally enabled again
- Detects issues in CFDP implementation, e.g.:
- sender continues to send file data after the transaction has been cancelled
- These types of behaviors can go undetected (file transfers still work) but are important to detect (they can result in data loss!)

- DynSAVE

- X

- X

## Planned CFDP Sequence

- Sample Rules:
- Check that received FD are not NAKed
- Check for duplicate FDs
- Check that we have all FDs upon FIN

## Slide 23

| FileData: 430704-431700 |
| --- |
| FileData: 432698-433694 |
| FileData: 433695-434691 |
| FileData: 434692-435688 |
| FileData: 435689-436685 |
| FileData: 436686-437682 |
| FileData: 437683-438679 |
| FileData: 438680-439676 |
| FileData: 439677-440673 |
| FileData: 440674-441670 |
| FileData: 441671-442667 |
| FileData: 442668-443664 |
| FileData: 443665-444661 |
| FileData: 444662-445658 |
| FileData: 445659-446655 |
| FileData: 446656-447652 |
| FileData: 447653-448649 |
| FileData: 448650-449646 |
| FileData: 449647-450643 |
| FileData: 450644-451640 |
| FileData: 451641-452637 |
| FileData: 452638-453634 |
| FileData: 454632-455628 |
| FileData: 455629-456625 |
| FileData: 456626-457622 |
| FileData: 457623-458619 |
| FileData: 458620-459616 |
| FileData: 459617-460613 |
| FileData: 461611-462607 |
| FileData: 463605-464601 |
| FileData: 464602-465598 |
| FileData: 465599-466595 |
| FileData: 466596-467592 |
| FileData: 467593-468589 |
| FileData: 468590-469586 |
| FileData: 470584-471580 |
| FileData: 471581-472577 |
| FileData: 472578-473574 |
| FileData: 473575-474571 |
| FileData: 475569-476565 |
| FileData: 476566-477562 |
| FileData: 477563-478559 |
| FileData: 478560-479556 |
| FileData: 479557-480553 |
| FileData: 480554-481550 |
| FileData: 481551-482547 |
| FileData: 482548-483544 |
| FileData: 483545-484541 |
| FileData: 484542-485538 |
| FileData: 485539-486535 |
| FileData: 486536-487532 |
| FileData: 487533-488529 |
| FileData: 488530-489526 |
| FileData: 489527-490523 |
| FileData: 491521-492517 |
| FileData: 492518-493514 |
| FileData: 493515-494511 |
| FileData: 494512-495508 |
| FileData: 495509-496505 |
| FileData: 498500-499496 |
| FileData: 499497-499999 |
| EOF: Condition Code=No Error |
| ACK(EOF): Condition Code=No Error |
| NAK: 19940-20937;27916-28913;36889-37886;56829-59820;72781-73778;76769-77766;82751-85742;101694-102691;111664-112661;115652-116649;121634-122631;130607-131604;139580-140577;146559-147556;153538-154535;155532-156529;170487-171484;197406-198403;203388-204385;220337-498500 |

## Actual CFPD Sequence

- Needed FDs: 502
- Sent FDs: 840
- Potential Waste: ~70%? – Further analysis needed.

## Zoom in on CFDP sequence

**Rule 2 Violation: **
**duplicate FDs!**

## Life Cycle Support

**System**
**Architecture**
**Use DynSAVE to**
**Specify and Test **
**Communication**
**Add to ICD**

**Sub-System**
**Development**
**Use DynSAVE to**
**Develop and Test**
**based on ICD**

**System**
**Integration and Test**
**Use DynSAVE to**
**test based**
**on ICD**

- Initial use of Dyn SAVE

## Create System Architecture

- No Server, No Client Exist
- Use DynSAVE to
- Specify Planned communication
- Sequences
- Parameters, Values
- Timing constrains
- Create Tests
- Correct, Incorrect behavior
- Specific incorrectness
- Automatically generate defects
- Ensure that communication protocol can handle all tests
- Add Diagram, Specification, Tests to ICD
- “Generate” information for ICD

- Client

- Server

- Communication

## Status

- Dyn-SAVE works for telemetry protocol
- Currently adding functionality to evaluate CFDP protocols
- Applying Dyn-SAVE to APL’s systems
- We’d like to apply to other systems

## Summary

- Analyze, Visualize, and Evaluate
- structure and behavior using
- static and dynamic information
- individual systems as well as systems of systems
- Next steps:
- Refine software tool support
- Use approach to review, improve ICD
- E.g. add planned sequence diagrams, rules to ICD
- Apply to other systems to get feedback