# Presentation: 197982

## SAS 2007 Executive
Galaxy Global Corporation
Presented by:
Valerie Jones &
Jennifer Murray
NASA POC: Jeffrey Northey
SAS_07_EvalRaTools_RAPhase_Jones2

***Galaxy Global Corporation***
- Presented by:
- Valerie Jones &
- Jennifer Murray
- NASA POC: Jeffrey Northey
*SAS_07_EvalRaTools_RAPhase_Jones2*

**Evaluation of Current Requirements Analysis Tools Capabilities for IV&V in the Requirements Analysis Phase**

## Problem Statement

- A Requirement Analysis tool that performs comprehensive automated analysis does not exist
- Existing tools available for use do not provide guidance or automated assistance in identifying potentially problematic requirements
- Requirements analysis is a long and tedious process.
- An automated tool could decrease the time required during the requirements analysis phase as well as improve the process.

## Tool Evaluation Study

- Tools Evaluated:
- NASA E-Smart/ARM
- Lexior
- QuARS
- Requirements Assistant
- SAT
- TEKChecker
- Tools evaluated on a predefined set of technical and quality attributes.
- Model problem defined to do comparison between manual analysis and analysis with automated assistance

## Tool Study Results

- Two evaluation studies were performed
- Results indicated automated analysis tools correctly identified technical issues consistent with manual analysis.
- Requirements Assistant performed best technically in both studies
- Analysis results were consistent with manual analysis and additional key issues identified
- Key issues identified including:  missing requirements and inconsistencies in requirements

## RA Technical Results

- Comparing to original 21 TIMs submitted for project.
- 12/21 were found correctly.
- 6/21 were flagged but for other reasons.
- 5/21 were not flagged at all.
- In addition to matching TIM’s generated, tool also found missed defects that were not reported originally.
- RA was the only tool to identify missing and inconsistent requirements

## Technical  Results (Project 2)

- Requirements Assistant (RA)
- Correctly Identified 2 RIDs.
- Could possible collect additional 2 RIDs if entire document was analyzed.
- One was found via traceability. (Outside of tools capabilities)
- Additional 13 issues identified and presented to project manager

## RA Differences	

- RA was the only tool evaluated that takes domain information specific to the system into consideration
- Allows for creation of a domain corpus to define known information about the system
- Input files allow tool to identify missing or inconsistent requirements
- RA analyzes starting at the sentence level, paragraph level then entire document as a whole.
- Utilizes knowledgebase of rules developed of 25 years of experience on various systems

## File Definitions

- Milfile - This file contains the aspects that the reviewer wants to address.
- Milsub - an entry, that should be considered when the word “system” is found in a requirement.
- Actor - an actor in the system (not necessarily humans).
- Actuator - a list of hardware that the actor(s) work on.
- Quant – Units of measure
- Function - An array of functions that the system has, or should have (in the reviewer’s opinion).
- Gloss - This contains the glossary of terms in the requirements document. (if provided). The file only contains the glossary terms, not their definition.
- List_Sensor – A list of sensors known in the system.

## Tool Pre-Processing	

- Input document must be in standard text format
- Input files are currently entered manually
- Input files are not required but increase the tools ability to correctly identify errors

## Tool Output

- Examples – Poorly written requirements1. Appropriate system standards shall be used where necessary.Remarks of The Requirements AssistantTM:a. appropriate – What is appropriate?b. standards – Specifically which standards are required?c. where necessary – Under what conditions will these standards be applied?2. It is required that the red light be illuminated once temperatures exceed about 25 degrees. Remarks of The Requirements AssistantTM:a. It is required – Is this a requirement?b. Illuminated – Is this for a certain duration?c. Temperatures – temperatures of..?d. About – accuracy?e. Degrees – Celsius, Fahrenheit?f. Is there also a requirement how/when the red light will be switched off?

## Examples cont.

- Off-line tooling allowing the end-user to manipulate ... in order to... compensate for ... stability.Remarks of The Requirements AssistantTM:a. manipulate – exactly what is manipulated?b. In order to – this is not a requirement but a goal.c. Compensate for – how much?
- Remarks of The Requirements AssistantTM:No requirements on reliability, safety and EMC were found in the set of requirements.

## Various Output Previews

- The Requirements AssistantTM provides three different previews of its analysis results: View by quality attributes: 	a. Consistency	b. Completeness	c. Testability	d. Accuracy	e. Readability

## Various Output Options cont.

- View by sentence:
- Poor words (in **bold**): The FSW will send** appropriate **error messages, and set the fault condition to “failed”, should the **alive **and Idle discrete lines** fail **to signal **ready **” and “Idle” within a parameterized amount of time [General Remark: 4 Upper limit? General Remark: and, and, and, complex sentence ]
- The FSW will send commands, and receive health & status and science telemetry information from** most XXX** instrumentation through this interface.General Remark: Quantity-like in paragraph most in sentenceNon Requirement **["will"]**: null
- View by topic
- DELIVERY AND RETURN "The** XXX **shall deliver at least 2850 kg (6,283 lbm) (gross) of **cargo.**
- DELIVERY AND RETURN "The** XXX **shall return at least 2,858 kg (6,283 lbm) of ** cargo.**

## Various Output Options cont.

- View by concept:
- 	Cargo Processing Concept:
- DELIVERY AND RETURN "The** XXX **shall deliver at least 2850 kg (6,283 lbm) (gross) of **cargo.**
- DELIVERY AND RETURN "The** XXX **shall return at least 2,858 kg (6,283 lbm) of ** cargo.**

## Benefits of RA

- Leads analyst to identify key issues outside capabilities of other tools evaluated
- Could allow the analyst to prioritize analysis based on identification of potentially problematic requirements first.
- Better results for both novice and experienced analysts
- Novice analyst identified key issues with a total analysis time of 3 days compared to a team of analysts working for 3 weeks a (50% decrease in analysis time).
- Developers would have positive ROI if using this tool for tracking requirements across stages of development
- Can assist in Requirements Validation with UML modeling.
- Knowledge base behind the tool allows for very versatile use for various aspects of software development to stay consistent

## Recommendations

- Proposed facility adopt Requirements Assistant (RA) for further use.
- RA requires additional upgrades before it can be used facility wide.
- Upgrades
- Improve user interface for analyst efficiency
- Increase speed
- UML support
- Domain term library

## Proof of Concept

- Current tasking is to evaluate RA capabilities for requirements validation and verification to UML modeling of systems.
- Working with NASA IV&V Facility project performing requirements validation and verification of 4 System Reference Models utilizing RA capabilities.
- Initial results from first model analyzed were provided to the project
- Project implemented additional updates based on findings
- Through the POC we will work to develop an appropriate user interface for the tool and the IV&V analyst.

## Proof of Concept Model

## What’s Next?

- Continue the Proof of Concept Study for Requirements Validation
- Recommended updates to Requirements Assistant for user interface development.
- Evaluating Eclipse platform
- Develop additional requirements.
- Proposal submitted for Research Infusion.