# Presentation: 571763

## Product-Line Verification of Safety-Critical Software 

**Robyn Lutz, JPL & ISU**

**This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office**

***Technical Presentation***
***NASA OSMA Software Assurance Symposium ***
***September 8-12, 2008***

**NASA, GRAIL**

**Elliott, JPL: MSAP**

- SAS08_Product_Line_Verif_Safety_Lutz

**National Aeronautics and Space Administration**

**Notes:**
Contingency Software in Autonomous Systems
Objective:  1) enhance diagnostic techniques to identify failures 2) provide software contingencies to mitigate failures 3) perform tool-based verification of contingency software and 4) apply results to ARP (and MSL) to pave the way to more resilient, adaptive unmanned systems.
The contingency plan for the autonomous rotorcraft before SW contingencies is to terminate flight - good because the rotorcraft doesn’t fly away and crash somewhere (like highway 101) when it runs out of fuel.  Challenging because flight termination means “full down collective and throttle idle setting” which can result in hard landing.

Our SW contingencies support mission completion or improved flight termination (landing in some cases).

Summary:  Mitigate failures via software contingencies resulting in safer, more reliable autonomous vehicles in space and in FAA national airspace.
Uniqueness:  1) Adding intelligent diagnostic capabilities without introducing non-determinism.  2) Responding to anomalous situations currently beyond the scope of the nominal fault protection. 3) Contingency planning using the SAFE (Software Adjusts Failed Equipment) approach.  

## Product-Line Verification of Safety-Critical Software Problem

**Product-Line Verification of Safety-Critical Software** Problem

- Industrial experience shows that it is the verifiable conformance of each system to the product-line specifications that makes or breaks the product-line practice
- Verification that the software for each project satisfies its intended product-line constraints is thus essential
- How should we verify that delivered software conforms to the product-line and how do we document that conformance?
- How should we verify that safety-critical software built using product-line assets is safe?

**National Aeronautics and Space Administration**

## Identify candidate suite of product-line verification techniques, focusing on safety 
Tailor as appropriate to NASA needs 
Apply, evaluate, and report results in collaboration with project 
Produce examples and data for training and transfer of techniques to projects

**Product-Line Verification of Safety-Critical Software**
- Approach

- Identify candidate suite of product-line verification techniques, focusing on safety
- Tailor as appropriate to NASA needs
- Apply, evaluate, and report results in collaboration with project
- Produce examples and data for training and transfer of techniques to projects

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical SoftwareDefinition of product line

** ****A *****software product line***** is defined to be “a set of software-intensive systems sharing a common, managed set of features that satisfy the particular needs of a specific market segment or mission and that are developed from a common set of core assets in a prescribed way” [Clements and Northrop, 2002]. **

**Product-Line Verification of Safety-Critical Software**Definition of product line

** **

**National Aeronautics and Space Administration**

*MSAP Project*: Worked this year with Multi-Mission System Architecture Platform project at JPL
- to identify (1) “negative lessons learned” from industrial use of product lines (risks & challenges) and (2) relevant mitigation strategies
- findings requested by MSAP managers to use in product-line business case
- MSAP work slowed when key personnel moved off project
*GRAIL Project*: Now working with Gravity Recovery and Interior Lab (Discovery Lunar mission, 2011; twin spacecraft; inheritance from GRACE’s earth-study techniques and from Lockheed Martin’s product-line spacecraft, especially MRO)
- to perform trend-tracking for GRAIL of software changes and anomalies across product line (beginning with MRO):  on-going in FY09
- to evaluate use of DDP to customize product-line risks/mitigations to GRAIL: on-going in FY09; collaborative with M. Feather

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software**Overview

** **

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical SoftwarePlanned Application on GRAIL

- Industrial product-line verification techniques help predict quality/defects in a new system in a PL (such as GRAIL) based on similarities/differences with earlier systems in that PL (MRO, Phoenix, etc.)
- Application to GRAIL:  what can PL approach to GRAIL tell us about *what to expect *in terms of *quality/defects/concerns* for software verification?

- SAS08_Product_Line_Verif_Safety_Lutz

**National Aeronautics and Space Administration**

** **

**Product-Line Verification of Safety-Critical Software**Planned Application on GRAIL

## Report : “Survey of Product-Line Verification and Validation Techniques” (2007)

- Best practices extracted from **reports of industrial experience **
- Verification challenges & verification enablers identified for NASA product lines based on experts’ recommendations
- List of resources assembled (conferences, industrial and defense industry experiences,  annotated bibliography)
*Report: “Tool-Support Survey for Product-Line Verification and Validation Techniques” (2007)*
**Commercial and academic tools **for product-line development
- Configuration management and change management tools for product lines
- Tool-supported testing in product lines (both domain testing and application testing)
- Advice from product-line experts regarding tool support

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software **
- Quality of data allows generalization of recommendations

**National Aeronautics and Space Administration**

** **

## Report : “An Evaluation Report for Three Product-Line Tools ”, Simon Chong (6/08)
Widespread agreement that better tool support for product lines is needed for project applications
Report evaluated the functionality, usability and performance of 3 candidate product-line tools (2 commercial, 1 academic) identified in Task 2
FORM (from Pohang University of Science & Technology), Pure::Variants (from PureSystems, GmbH), and Gears (from BigLever)
Downloaded and experimented with each tool 
Emphasis on practical use:  e.g.,  setup, support, user interface, support for commercial tools, documentation
Recommendation for application plan based on measured criteria:  Gears

** **

**Product-Line Verification of Safety-Critical Software**
- Accomplishments this year

*Report : “An Evaluation Report for Three Product-Line Tools ”, Simon Chong (6/08)*
- Widespread agreement that better tool support for product lines is needed for project applications
- Report evaluated the functionality, usability and performance of 3 candidate product-line tools (2 commercial, 1 academic) identified in Task 2
- FORM (from Pohang University of Science & Technology), Pure::Variants (from PureSystems, GmbH), and Gears (from BigLever)
- Downloaded and experimented with each tool
- Emphasis on practical use:  e.g.,  setup, support, user interface, support for commercial tools, documentation
- Recommendation for application plan based on measured criteria:  Gears

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical SoftwareAccomplishments this year

*Paper : “Enabling Verifiable Conformance for Product Lines” , R. Lutz*
- To be presented at 12th Int’l Software Product Line Conference, Sept. 8-12, 2008
- Recommendations for process characteristics that will make it *easier to show *that the system being built conforms to the product-line specifications
- Draws on information from four product lines or product-line-like sets of systems (3 JPL, 1 NASA)
- a set of interferometer (spaceborne telescope) software systems with reused architecture, documentation, and code
- a proposed NASA mission consisting of a cluster of microsatellites
- a multi-mission hardware and software architecture
- a ground data system  providing services for multiple spacecraft missions

- SAS08_Product_Line_Verif_Safety_Lutz

** **

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical Software  Accomplishments this year

- Many will be **safety** or mission critical
- Many will be **long-lived**
- Many will be **highly autonomous**
- Recommends process characteristics to handle challenges in the appropriate development phases
- To establish verifiable conformance of **safety requirements **in the new product need rigorous traceability to the product-line safety requirements
- To maintain conformance for **long-lived missions  **need to accurately capture the requirements delta information
- To verify conformance of **highly autonomous product lines **need  good documentation of the product-line behavior, preferably as a machine-readable model , and use small configuration  units

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software **** **Accomplishments this year

**National Aeronautics and Space Administration**

** **

## Product-Line Verification of Safety-Critical Software Synergistic activities

- Served on program committees for Software Product Line Testing workshop, 2007, and Software Product Line Conference, 2008

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software **Synergistic activities

**National Aeronautics and Space Administration**

** **

## Product-Line Verification of Safety-Critical Software Findings: Enabling Verification of Inherited Software

- Key to being able to perform verification on product lines
***Controlling requirements changes in baseline assets***
- Having this information in the application requirements specification allows scoping of features and code that will have to be developed beyond the product-line assets
***Sufficient documentation*** ***is essential****.*
- Verification that the new system conforms to the product-line requirements, architecture, and test suite requires good documentation of the new system.
***Access to source code***
- For thorough testing of the product-line components, need source code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl]

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software **Findings: Enabling Verification of Inherited Software

**National Aeronautics and Space Administration**

** **

## Rigorous traceability 
Key to being able to perform verification on product lines [Bergey]. 
Capturing the requirements delta
Having this information in the application requirements specification allows scoping of features and code that will have to be developed beyond the product-line assets [Pohl] . 
Sufficient documentation is essential.  
Verification that the new system conforms to the product-line requirements, architecture, and test suite requires good documentation of the new system. 
Access to source code 
For thorough testing of the product-line components, need source code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl].

***Rigorous traceability***
- Key to being able to perform verification on product lines [Bergey].
***Capturing the requirements delta***
- Having this information in the application requirements specification allows scoping of features and code that will have to be developed beyond the product-line assets [Pohl] .
***Sufficient documentation*** ***is essential****.*
- Verification that the new system conforms to the product-line requirements, architecture, and test suite requires good documentation of the new system.
***Access to source code***
- For thorough testing of the product-line components, need source code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl].

**Product-Line Verification of Safety-Critical Software**
- Findings: Enabling PL Verification

**National Aeronautics and Space Administration**

## Start small
Successful product-line engineering was first applied on a legacy system “in which there is frequent change occurring at relatively high cost.  Such a domain is often an isolatable section of the system where the changes can be encapsulated and where a single group of software developers is responsible for making the changes” [Weiss and Lai].
Use small configuration units
Rockwell Collin described the tradeoff in their recent experience building a large product line: “Having more items leads to a finer grained decomposition of functionality, which, in turn, leads to higher levels of reuse and systems that are easier to test and get through airworthiness qualifications. However, having more items also means there are more configuration items to keep track of and maintain, more complexity, and a steeper integration curve” [Clements and Bergey]. 

***Start small***
- Successful product-line engineering was first applied on a legacy system “in which there is frequent change occurring at relatively high cost.  Such a domain is often an isolatable section of the system where the changes can be encapsulated and where a single group of software developers is responsible for making the changes” [Weiss and Lai].
***Use small configuration units***
- Rockwell Collin described the tradeoff in their recent experience building a large product line: “Having more items leads to a finer grained decomposition of functionality, which, in turn, leads to higher levels of reuse and systems that are easier to test and get through airworthiness qualifications. However, having more items also means there are more configuration items to keep track of and maintain, more complexity, and a steeper integration curve” [Clements and Bergey].

**Product-Line Verification of Safety-Critical Software**Findings: Enabling PL Verification (cont.)

**National Aeronautics and Space Administration**

## Build family-specific tools 
“Invest in family-specific tools to make the production of family members rapid using the production process”  [Weiss & Lai]. They recommend domain-specific modeling languages and code compilation or composition from the models.  
Demand tool  interoperability 
Clements and Northrop emphasize that it is the integrated tool suite rather than the standalone tools that provide the needed development environment. “The interoperability of a chosen set of tools is critical for the automated production of products in a product line from core assets.” 

***Build family-specific tools***
- “Invest in family-specific tools to make the production of family members rapid using the production process”  [Weiss & Lai]. They recommend domain-specific modeling languages and code compilation or composition from the models.
***Demand tool  interoperability ***
- Clements and Northrop emphasize that it is the integrated tool suite rather than the standalone tools that provide the needed development environment. “The interoperability of a chosen set of tools is critical for the automated production of products in a product line from core assets.”

**Product-Line Verification of Safety-Critical Software **
- Findings: Tool Support for PL Verification

**National Aeronautics and Space Administration**

## Plan verification tasks to reuse test suites  
Testing in a product line begins early as the domain-engineering product-line assets (the reusable components) are built. Then, each time those assets are assembled into a new product-line application, more testing occurs. “During application engineering, assets from this test infrastructure should be reused to speed up testing and thus overall product development” [Knauber and Hettrick]. 
Make tools for configuration management (including variability management and change management) a high priority   
The payoff for using a product line is safe and efficient reuse.  To enable and control this reuse, product-line developers agree that tool-supported configuration management is required.   The challenge is that, “Traditional CM tools are meant to handle variation over time, but they are not good at handling simultaneously supported variations.” [Clements and Northrop] 

***Plan verification tasks to reuse test suites****  *
- Testing in a product line begins early as the domain-engineering product-line assets (the reusable components) are built. Then, each time those assets are assembled into a new product-line application, more testing occurs. “During application engineering, assets from this test infrastructure should be reused to speed up testing and thus overall product development” [Knauber and Hettrick].
***Make tools for configuration management (including variability management and change management*****) *****a high priority***
- The payoff for using a product line is safe and efficient reuse.  To enable and control this reuse, product-line developers agree that tool-supported configuration management is required.   The challenge is that, “Traditional CM tools are meant to handle variation over time, but they are not good at handling simultaneously supported variations.” [Clements and Northrop]

**Product-Line Verification of Safety-Critical Software**Findings: Tool Support for PL Verification

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical SoftwareTechnical challenges of approach

- SAS08_Product_Line_Verif_Safety_Lutz

| Categories | Challenges | Techniques  |
| --- | --- | --- |
| Organizational | Ownership of assets unclear
Structure doesn’t match project needs | Divide into subdomains
Consider separate integration organization
Rigorous two-way traceability  |
| Cultural | Reluctance to invest for other projects
Feedback from user to owner of assets fails | Create conscious PL culture
Learn from experience and problem reports |
| Requirements | Products needs new features
Time can erode product line | Establish strong change board to judge tradeoffs 
Capture requirements delta early for new projects 
Invest in tool support for change management |
| Contractor/supplier | Low incentive for  conformance to product-line specifications
Less control than in-house | Motivate/reward conformance to product-line specifications
Include source code in supplier agreements  |

**Challenges and techniques for verifiable conformance of a product line**

** **

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical Software  Challenges for work to be more widely applicable

- External ownership of product line development can make access to requirements documents, data, code, more difficult
- Reluctance to invest for other projects
- Projects that are willing to use other’s products but are reluctant to make investments for other projects can impede product-line development and verification.
- Difficulty controlling the delta
- CCB usually  first line of defense in deciding whether  new features, alternative design mechanisms, or different coding standards will move systems away from the product-line standard set of verified components.
- Balancing commonality and common sense
- Folding changes in one product back into the product line can lead to degradation of the product line and the erosion of commonality.  On the other hand, propagating each change to the product line can make re-verification costs unmanageable.

- SAS08_Product_Line_Verif_Safety_Lutz

**Product-Line Verification of Safety-Critical Software **** **Challenges for work to be more widely applicable

**National Aeronautics and Space Administration**

** **

## TRL 3
“Analytical and experimental critical function and/or characteristic proof of concept”
MSAP provided limited proof-of-concept of usefulness of product-line engineering verification techniques for inherited software
GRAIL application goal for next year: validation in relevant environment (TRL 5)

- TRL 3
- “Analytical and experimental critical function and/or characteristic proof of concept”
- MSAP provided limited proof-of-concept of usefulness of product-line engineering verification techniques for inherited software
- GRAIL application goal for next year: validation in relevant environment (TRL 5)

**Product-Line Verification of Safety-Critical Software**** **
- Technology Readiness Level

**National Aeronautics and Space Administration**

## NASA is reusing more product-line, inherited, & reused software assets 
We need better ways to verify compliance of delivered software against the constraints levied on all systems in that product line 
We need better ways to verify that problems with similar, earlier systems won’t recur in our new system 
Work focuses on identifying product-line verification techniques that can help solve problems now for current missions 

- NASA is reusing more *product-line, inherited, & reused *software assets
- We need better ways to *verify compliance *of delivered software against the constraints levied on all systems in that product line
- We need better ways to *verify* that problems with similar, earlier systems *won’t recur *in our new system
- Work focuses on identifying product-line verification techniques that *can help solve problems now *for current missions

**Product-Line Verification of Safety-Critical Software**
- Relevance to NASA

**National Aeronautics and Space Administration**

## Product-Line Verification of Safety-Critical Software Planned capability of research

*Customize industrial, product-line techniques to verify correctness of a new system’s software inheritance, and apply to NASA systems: *
- GRAIL (FY09 focus)
- JUNO (same heritage)
- Future NASA spacecraft
-     with same LMA product-line
-     heritage
- Exploration systems reusing same baseline and inherited software

- SAS08_Product_Line_Verif_Safety_Lutz

** **

**National Aeronautics and Space Administration**