# Presentation: 909684

## NASA Software Assurance Training

***April 8, 2005***

***Susan J. Sekira***
***GSFC Software Assurance Lead***

**Notes:**
This course was developed in response to changes within the Agency and software’s increasing visibility and criticality in NASA missions and science objectives.  It’s intended to raise awareness levels among software assurance practitioners as part of NASA Software Process Improvement Initiative.

It’s designed to be given at each Center by their respective Software Assurance Point of Contact.  This course is the first of 3 Software Assurance courses being developed for NASA.  The targeted  audience is Project Managers, Software Managers, Software Engineers, Safety Engineers, and Systems Engineers.

## Course Agenda

**Course Objectives**
**Overview of Software Assurance**
**Acquirer Software Assurance**
**Provider Software Assurance**
**Software Measurement**
**NASA Software Assurance Classification Assessment**
**Supplementary Information**
**Summary**

**Notes:**
The following Course Agenda includes classroom instruction, as well as a team exercise developed to apply the concepts learned today. 

In addition to a high level overview of Software Assurance, Software Assurance requirements are discussed from both an Acquirer and Provider perspective.  Specifically, what are the software assurance requirements for the organization responsible for accepting software products vs. what are the requirements for the organization that actually designs, develops, implements, tests, operates, and maintains software products.

A brief overview of software measurement/data collection is also provided, along with sample metrics and hints for successful measurement programs.

NASA’s 4-step Software Assurance Classification Assessment was recently developed to identify and evaluate the characteristics of software in determining the software’s classification and level of software assurance to be applied.  We’ll discuss the various classes, evaluation criteria, and their applicability to your projects and software systems.

## Course Objectives

## Software Assurance Course Objectives

**To establish a common framework and knowledge base among the NASA software community**
- Establish roles and responsibilities
- Enable consistency in implementation and performance
- Eliminate misconceptions about software assurance
**To highlight the benefits of software assurance throughout the development life cycle**
**To encourage and strengthen implementation of software assurance practices**

***General Audience:  Project Managers, Software Managers, Software Engineers, Safety Engineers, and Systems Engineers***

**Notes:**
Self explanatory

It is particularly important that the NASA software community share a common understanding of what’s required to implement a successful Software Assurance Program that is cost effective, while providing safe and quality products.   Everything from resources, key deliverables, metrics, lessons learned, etc.

## Overview of

**Software Assurance**

## NASA Software Assurance Challenges

**Performance Based Contracting**
- Lack of software assurance requirements in proposals
- Limited insight into interim products
**Increased code size and complexity**
**Resources **
- Constrained budgets
- Compressed schedules
- Limited availability of trained personnel
**Increased visibility**
- Loss of mission
- Loss of credibility
**Achieving “high quality” software   **
- 	(and knowing what that means!)

***MARS***

**Notes:**
Before we review Software Assurance, let’s take a look at some of the challenges we’re facing in the software arena.
NASA has undergone change and numerous challenges in the last decade, including the concept and implementation of Performance Based Contracting and its’ impact to our level of insight into software development.
Software code has increased in both size and complexity as NASA continues to infuse software in autonomous operations and other critical system applications.
Limitations on budget and schedule are certainly not new to NASA and we continue to confront limited resources when it comes to acquiring and retaining trained software personnel.
Increased visibility within the public eye, as well as on Capitol Hill, has also intensified scrutiny and criticism beyond our comfort level.  Our highly  publicized failures and internal/external investigations have precipitated both a loss of credibility and a lack of confidence.  
Finally, achieving “high quality” software is subjective and widely misunderstood across the Agency.  We as “One NASA” need to know what it means to be “high quality” and how to achieve this.

## Agency Response and Actions...

**Renewed emphasis on process improvement initiatives (e.g., CMM, CMMI)**
**Improved and accepted tools and techniques **
**Early Software Assurance involvement during Software Acquisition activities**
**Redefinition of NASA Software Assurance and updates to standard**
**Approved NASA Software Engineering Requirements**

***NASA Software Assurance Standard was updated in July 2004!***

**Notes:**
Let’s look at how NASA has responded to recent software challenges:  

As an Agency, NASA has recognized that successful software products have mature and effective development and maintenance processes to support them.  Process improvement initiatives are now commonplace within NASA, along with tools,  techniques, and methodologies (e.g., CMM, CMMI, and Six Sigma) to assist practitioners.

Experience has taught us that early involvement in the development life cycle increases our chances of success.  In other words, applying software assurance resources and defining requirements early in the process leads to a common framework of requirements, deliverables, and expectations.

NASA policies, standards, and requirements for software have been revisited and updated to reflect changes in our Agency and the software environment.  NASA approved NASA NPR 7150.2, NASA Software Engineering Requirements and in July 2004, NASA approved extensive updates to the Software Assurance Standard.  An overview of this Standard, its requirements and benefits, are provided next.

## What is Software Assurance?

**SQ**

**V&V**

**Software**
**Safety**

**Software**
**Reliability**

**IV&V**

**Software Assurance is an umbrella risk identification and mitigation strategy for safety and mission assurance of all NASA’s software**

- SQ = Software Quality
- V&V = Verification and Validation
- IV&V = Independent Verification and Validation

**Notes:**
So what is Software Assurance?  Per the NASA Software Assurance Standard, it’s defined as an umbrella risk identification and mitigation strategy for safety and mission assurance of all NASA’s software.

The Software Assurance process is the planned and systematic set of activities that ensure conformance of software life cycle processes and products to requirements, standards, and procedures.  It applies to software developed or acquired for NASA, either internally or externally, including the incorporation of open source, COTS, GOTS, or MOTS into NASA systems.  

For NASA, this includes the disciplines of Software Quality (with the functions of Software Quality Engineering, Software Quality Assurance, and Software Quality Control), Software Safety, Software Reliability, Software Verification and Validation, and IV&V.  Each discipline brings its own perspective to the process; the collective effect of all these efforts provides assurance of mission safety, reliability, and quality.

## What’s in it for Me?

**Strives to improve the quality of the product while employing risk mitigation techniques**
**Focuses on opportunities for early error detection, problem prevention, and risk identification and mitigation**
**Provides project management insight into the software development processes and products throughout the life cycle**
**Reviews and assesses interim products – can’t build quality in at the end!**
**Improves the quality of future products/services**

*The level of Software Assurance needed is commensurate with the software classification as well as software size, complexity, criticality, and risk*

***Software Assurance:***

**Notes:**
If you’re a Project Manager, you’re probably asking, “So what does this mean to me?  Why so early in the development life cycle?”

Software Assurance assists in risk mitigation by minimizing defects and preventing problems throughout the development life cycle (with the ultimate goal to improve the quality of the product).  To do this, Software Assurance  utilizes project-based measurement data to monitor each product and process for possible improvements.  

Software Assurance provides project management continuous insight into these software development processes and products.  This is accomplished by  reviewing and assessing interim products and associated processes, as opposed to waiting until final inspection.  As you know, you can’t build in quality at that end!

Software Assurance can improve not only the quality of the current product, but future products/services.  It’s a journey and an opportunity for continuous improvement.
Tailoring the implementation of software assurance requirements is acceptable based on the program/project classification and various software characteristics.  More on this later in the course...

## Value of Software Assurance

**Earlier detection is cheaper to fix with lesser impact to schedules**
**Increases chances of delivering full functionality within cost and schedule**
**Safety issues are identified and addressed early as part of overall system safety **
**Software assessments can be used to determine milestone completion**
**Metrics provide indicators of the quality and maturity of the software/system**

**Relative Cost of Finding, Fixing Errors**
**Adapted from Software V&V, Steven Rakitin**

- 0

- 10

- 20

- 30

- 40

- 50

- 60

- ...

- 100

- Requirements

- Design

- Code

- Test

- Maintenance

**Phase Error Found & Fixed**

** ****Cost to Fix**

- $1

- $5

- $20

- $50

- $100+

**Notes:**
Let’s take a closer look at some of the potential benefits. 
It’s a well known fact that early error detection is cheaper to fix than later in the development life cycle.  And it can also have a lesser impact to overall development schedules.  That’s not to imply that there’s no disruption in schedule, but you may have more schedule slack (or time to recover) earlier in the development life cycle, rather than during acceptance testing.

With full life cycle coverage, you also increase the probability of delivering full functionality within cost and schedule.  Continuous support in all of the SA disciplines will provide a consistent level of insight into processes and products.  There should be no unknowns or hidden issues.

Given that software safety is part of system safety, safety issues are also identified early and tracked (with time to mitigate any risks or safety hazards).

By conducting software assessments, audits, inspections, etc., you have more information from which to predict and assess milestone completion.  The more information, the better!

And metrics provide another avenue for assessing quality and software maturity.

## Who are the Practitioners?

**Software Assurance practitioners INCLUDE a wide range of personnel, employed throughout the software development life cycle**

- Software Engineers

- Safety Engineers

- Systems Engineers

- IV&V Personnel

- Software Managers

***Software Assurance personnel aren’t just those Software Quality folks!!***

- Software Quality Personnel

**Notes:**
So who are the Software Assurance practitioners?

One common misconception is that Software Assurance personnel are just those Software Quality folks!  

In fact, Software Assurance personnel include most everyone in this class today.  They are the Software Managers, the Safety, Software, and Systems Engineers, as well as the Software Quality Engineers and IV&V personnel from the NASA West Virginia Facility.  They include Software Reliability and V&V personnel and perhaps other folks on your development team.  A Center’s organizational structure and governing documents for each program/project further define who’s who is software assurance.

We all have important roles and responsibilities and it’s imperative that we understand the players and their contributions to Software Assurance.  Only through a common framework and knowledge base can we ensure consistency in implementation and performance and realize the benefits of Software Assurance.   

## The Disciplines of

**Software Assurance**

## Software Quality

**Assures that quality is built into the software                        through the functions of**
- Software quality assurance
- Software quality engineering
- Software quality control
**Ensures conformance of software life cycle processes and products to requirements, standards, and procedures**
**Performs process and product activities throughout the life cycle to provide objective insight into the maturity and quality of the software processes and products**
**Promotes continuous process improvement**

**Notes:**
Let’s begin with a discussion of Software Quality (perhaps the best known of the 5 software assurance disciplines).  

Software Quality is the planned and systematic set of activities to ensure quality is built into the software.  This is accomplished through the functions of Software quality assurance, Software quality engineering, and Software quality control.  

The software quality discipline consists of product assurance and process assurance activities with the emphasis on conformance to requirements, standards, and procedures.  Activities are conducted throughout the life cycle to provide objective insight into the maturity and quality of the software processes and procedures.  

Strengths and weaknesses identified during assessments can then be used to promote continuous process improvement on current and future software systems.

## Sample Process and Product Activities

**All required plans are developed in accordance with specified requirements, standards, or procedures**
**All software requirements are documented and traceable from system requirements to design, code, and test **
**Software development records are maintained and up-to-date**
**Configuration baselines are managed, maintained, and accurate**
**Software quality metrics are in place and used to manage the software development effort**

**All plans (e.g., Configuration Management Plan, Software Management Plan) and procedures are implemented according to specified standards and procedures.**
**Engineering peer reviews and management reviews are conducted and action items are tracked to closure**
**Tests are planned, documented, and conducted using approved test procedures and tools**
**Project risks are documented and addressed in accordance with the risk management plan**

**Process**

**Product**

*SQ identifies strengths, weaknesses, and areas for improvement!*

**Notes:**
The following examples are provided to enhance your understanding of the various types of process and product activities that may be conducted.

PRESENTER:  Choose several bullets to highlight.  This chart is not intended to be exhaustive.  No need to spend a lot of time on this chart.

## Software Safety

**Software Safety is a systematic approach to identifying, analyzing, tracking, mitigating and controlling software hazards and hazardous functions (data and commands) to ensure safer software operation within a system.**** **

***Software Safety entails...***

- Ensuring that software safety requirements are clearly identified, documented, traced and controlled throughout the software lifecycle

- Analysis of the consistency, completeness, correctness and testability of software safety requirements

- Testing of software safety critical components on actual hardware to ensure that the safety requirements were sufficiently implemented and that applicable controls are in place to verify all safety conditions

- Continuous analysis of proposed changes on system safety** **

***Software Safety is a function of System Safety!***

**Notes:**
Software Safety has grown in importance as more and more NASA systems have critical functions either controlled by software or adversely impacted by a software failure.

The discipline of Software Safety functions as part of the overall System Safety program and coordination is essential  between system safety personnel and software safety personnel.  At some Centers, these personnel may be one in the same.  

It’s also important to note that Software Safety assessments begin early in the development life cycle and continue as proposed changes are made.

PRESENTER:  Be prepared to discuss/provide examples of what is considered software safety-critical.  For example, flight software that detects and reports a specific hazardous state (e.g., instrument overheating) and attempts to take corrective action...

## What is Safety-Critical Software?

**Reside in a safety-critical system (as determined by a hazard analysis) AND at least one of the following:**
- Cause or contribute to a hazard
- Provide control or mitigation for hazards
- Control safety-critical functions
- Process safety-critical commands or data
- Detect and report, or take corrective action, if the system reaches a specific hazardous state
- Mitigate damage if a hazard occurs
- Reside on the same system (processor) as safety-critical software

***Does the software meet any of the following criteria:***

**Notes:**
Well, how do I know if my software is safety critical??

The determination of safety criticality is based on NASA’s Software Safety Litmus Test which was derived from the definition of Hazard Severity in NPG 8715.3, the NASA Safety Manual.

To determine if the software is safety critical, and to what extent, review the following criteria...

## What is Safety-Critical Software? – cont.

**Process data or analyze trends that lead directly to safety decisions (e.g., determining when to turn power off to a wind tunnel to prevent system destruction)**
**Provide full or partial verification and validation of safety-critical systems, including hardware and software subsystems**

- If the software is determined to be safety-critical, then the project must adhere to the NASA-STD-8719.13, NASA Software Safety Standard

***Or does the software meet any of the following criteria:***

**Notes:**
If the software meets any of the 3 criteria, the software is considered safety-critical.  Refer to the NASA Software Safety Standard and Guidebook for more information.

## Safety-Critical Software Examples

***At GSFC, the following software is considered safety-critical:***
**Example 1:  **Software on the Hubble Robotic Vehicle (HRV) that controls the robotic arm for servicing the spacecraft
**Example 2: **Guidance and navigation software on HRV’s De-Orbiting Module (DM)** **used to dock the DM and HST

*NOTE:  Engineering telemetry data used to provide safety information, but not required for safety or hazard control, is not safety-critical!!*

**Notes:**
PRESENTER:  Include specific examples of safety-critical software at your Center.

## Software Reliability

**Software Reliability is concerned with incorporating and measuring reliability in the products produced throughout the life cycle**

**Systems Engineering** defines software requirements as they contribute to system robustness
**Software Engineering** develops software containing required redundancy and fault tolerance *AND* measures and analyzes software’s ability to withstand errors
**Software Quality** assures quality metrics/measures are documented, monitored, analyzed and tracked (e.g., error density) *AND* verifies that reliability requirements have been successfully demonstrated

***Specifically...***

**Notes:**
Software Reliability is a discipline exercised by Systems Engineering, Software Engineering, and Software Quality with the emphasis on incorporating and measuring reliability in the products produced by each process of the life cycle.  

To assure reliability...

For measures, including those for software reliability modeling, reference the IEEE Standard Dictionary of Measures to Produce Reliable Software (IEEE Std. 982.1)

## Verification and Validation (V&V)

**Software Verification and Validation **
- Ensures that software being developed or maintained satisfies functional and performance requirements
- Ensures that each phase of the development process yields the right products
**Every participant in the software life cycle process plays a role in some aspect of V&V!**
***V&V activities include, but are not limited to:***
- Analysis of system and software requirements
- Engineering peer reviews (e.g., code walkthroughs)
- Test planning and test execution
- Audits/assessments (e.g., baseline management)

**Notes:**
Software Verification and Validation is another discipline that engages participants across a variety of functional areas and/or groups.  

First and foremost, verification ensures that “you built it right”, while validation ensures that “you built the right thing”.  

When regard  to software, this means ensuring that the software satisfies functional, performance, and other requirements and that each phase of the development process yields the right product.

Software V&V entails a variety of activities, including engineering peer reviews, formal reviews, analyses of requirements and design, test planning, test execution, and formal audits and assessments.

PRESENTER:  See the V&V Activity Matrix in the Backup Charts!

## Independent V&V – IV&V

**IV&V is Verification and Validation performed by an organization that is technically, managerially, and financially independent of the development organization**
**IV&V focuses on mission critical software, provides additional reviews and analyses, and provides in-depth evaluations of life cycle products that have the highest level of risk**

- Validation of design to meet system needs/requirements
- Traceability of safety critical requirements
- Code analysis of mission-critical software components
- Design analysis of selected critical algorithms

***Examples:***

**Notes:**
This brings us to the discipline of Independent Verification and Validation (better known as IV&V).  

Within NASA, IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization.  For NASA, IV&V is performed and managed by the NASA IV&V Facility.  

IV&V focuses on safety and mission critical software by conducting additional reviews and in-depth evaluations of the life cycle products that carry the highest level of risk.  

All software projects that are identified as safety-critical or Class A software will be candidates for IV&V support and funding via Headquarters.  Software projects that are neither Class A nor safety-critical may request Headquarters funding, but funding is reserved for special circumstances.  Software intense projects not selected as candidates for IV&V may negotiate with the IV&V Facility for direct support, but must fund the support themselves.

Examples of representative IV&V tasks or activities are as follows...

## So what do I need to do?

***As a Manager, ask the following:***
**How much software is predicted to be on the project?  How much will be acquired vs. developed in-house?**
**What is the software’s classification (i.e., Class A – G or Exploratory)?**
**What is its criticality?  What functions will software control?  What hazard controls and mitigations?**
**What metrics are being collected?  Can they help me measure the maturity and quality of the software product?**
**Have I carefully planned for software assurance?  Are SQ, IV&V, and Software Safety personnel on-board? **
**Are my contracts/MOU’s written to assure the safety and quality of the software?**

**Notes:**
PRESENTER:  This is the point at which Project Managers/Leads need to sit up and understand that they have a role to play.

Having addressed each of the 5 disciplines of Software Assurance, what’s next?  What do you do with this information? 

As a Project Manager or Software Manager, you should ask yourself some basic questions to help you characterize the development effort.

For example....run through questions.

NOTES:  
Remind the audience that Software Assurance applies to both in-house and acquired software.  Make sure they know which organization at your Center provides SA/SQA resources and support.

## Getting Started...

***The NASA Software Assurance Standard [NASA-STD-8739.8] specifies requirements for software assurance for use by NASA projects, programs, and facilities***
***This standard...***
- Provides a software life cycle perspective for the minimum required software assurance procedures that contribute to quality software
- Provides specific requirements for each discipline of software assurance
- Provides the “Acquirer” and “Provider” requirements for software assurance to obtain the most cost effective, best quality, and safest products

**Notes:**
This brings us to the requirements themselves and the NASA Software Assurance Standard. This standard is approved for use by NASA Headquarters and all NASA Centers and is intended to provide a common framework for consistent practices across NASA programs.   

PRESENTER:  Run through the bullets...

## Acquirer vs. Provider

***Software Assurance begins with the Acquirer!***

_**Acquirer**_ **(Usually NASA or an Organization within the Agency)**
- Specifies the requirements and accepts the resulting software products

_**Provider**_**(May be a contractor, a university, a separate organization within NASA, or within the same organization as the Acquirer)**
- Designs, develops, implements, tests, operates, and maintains software products

**Notes:**
It’s important to distinguish between the Acquirer and the Provider.

In the NASA environment, the Acquirer is usually NASA or an organization within the Agency that specifies the requirements and accepts the software products.  There’s typically some sort of contractual agreement with defined requirements and deliverables.  

The Provider is the organization that designs, develops, implements, tests, operates, and maintains software products.  The Provider may be a contractor, a university, a separate organization within NASA or within the same organization as the Acquirer.  The latter example is sometimes the most confusing to understand, particularly when the development team is small.  

PRESENTER:  You may need to cite example development or maintenance projects at your Center; specifically, who acquires or provides software.

## Acquirer

**Software Assurance**

**Notes:**
Next we’re going to explore the role of the Acquirer, Key Activities, and associated Software Assurance Deliverables.

## Acquirer Software Assurance

***Getting started...***
**Identify a person responsible for software assurance (e.g., a software assurance manager)**
**Secure budget/resources for software assurance EARLY on (just like any other functional area)**
**Participate in the Initiation phase of the program/project to ensure appropriate oversight/insight into requirements, including needed deliverables**

*Notify your Safety and Mission Assurance Organization *
*to arrange for Software Assurance Expertise*

**Notes:**
As the Acquirer, you need to get started early!  This entails...

Identifying someone responsible for software assurance who is independent from the development effort.

Securing the budget and necessary resources to ensure success, including resources for staffing Software Engineers, Safety Engineers, Software Quality Engineers, IV&V personnel, etc.

With Software Assurance personnel onboard, you have the resources and expertise to define the SA requirements, identify the necessary deliverables, and provide you daily insight into the Provider’s development approach, activities, and progress.

When developing a Software Assurance Program, work with your Center S&MA Organization to arrange for Software Assurance Expertise.

## Acquirer Software Assurance

***Key Activities:***
**Performs Software Assurance Classification Assessment to identify and evaluate the characteristics of software in determining the software’s classification **
**Applies software classification to tailor software assurance requirements**
**Develops SA input to the Request for Proposal (RFP)**
**Reviews proposals for compliance to SA requirements**
**Establishes and maintains a software assurance plan for the acquirer **
**Ensures consistency between acquirer and provider SA plans**
**Ensures SA training for both the acquirer and provider**

**Notes:**
At a minimum, Software Assurance expertise will perform the following activities...

PRESENTER:  Remind the audience that these activities are a subset of what is prescribed in the NASA Software Assurance Standard.  In addition, key SA activities for Operations through Retirement were not highlighted in an effort to prevent information overload.

## Acquirer Software Assurance

***Key activities (cont.)***
**Provides surveillance to assure both the acquirer and provider SA organizations perform to their plans and procedures **
**Assures that problem reports, action items, and test anomalies are documented, addressed, analyzed, and tracked to closure**
**Assures that software products are reviewed and any metrics collected, analyzed, and trended**
**Reports at all formal software reviews (e.g., CDR)**
**Fosters effective communication with project management, other project assurance personnel, and the provider**

## Acquirer’s Software Assurance Deliverables

**SA Input to RFP**
**Software Assurance Plan**
- Addresses all SA disciplines
- Outlines activities and deliverables
**Software Assurance Status Reports**
- Results from software assurance activities
- Issues/risks/recommendations
**Software Assurance Records**
- Reports
- Analyses
- Metrics
**SA Input at Formal Reviews**

**Notes:**
From a project deliverable standpoint, you should receive the following...

Keep in mind that software assurance requirements are essential whether they’re incorporated in an RFP, MOA, or some other in-house agreement.  The key is to document, communicate, and agree upon the software assurance requirements early in the life cycle.

## Acquirer’s Role by Phase...

***INITIALIZATION,******PRE-AWARD ***
- Secure SA expert/resources
- Conduct software classification assessment
- Specify SA reqts for inclusion in RFP  - including provider deliverables for SA
- Plan acquirer’s SA activities – prepare Preliminary SA Plan

***POST-RFP; PRE-AWARD***
- Evaluate proposal for SA requirements
- Conduct Pre-award Surveys
- Participate in contract negotiations

***POST-AWARD; PRE-IMPLEMENTATION***
- Ensure provider SA plan meets requirements
- Verify consistency between acquirer and               provider SA plans
- Ensure trained SA personnel

***CONTRACT***
*** ******IMPLEMENTATION***
- Conduct surveillance
- Assure CM
- Assure provider SA
- Ensure acquirer performs tasks

***ACCEPTANCE***
- Conduct Audits prior to delivery
- Assure Facility readiness
- Assure acceptance documentation
- Capture Lessons Learned

***OPERATION***
- Review SA plan
- Ensure SA processes in place
- Conduct periodic audits of OPS

***MAINTENANCE***
- Ensure SA processes in place
- Assure Metrics are transferred to Maintenance Organization and maintained

***RETIREMENT***
-  Assure preparation, approval, and execution of retirement plan
- Ensure plans for archival or disposal of SA records

**ACQUIRER’S**
**ROLE**

**Notes:**
This last chart in the Acquirer Section summarizes the key Acquirer activities by life cycle phase.  Those activities in blue represent critical activities to be performed prior to software implementation.  All too often Software Assurance is brought onboard after implementation is underway.

PRESENTER:  The intent is to provide a snapshot or summary of the Acquirer’s roles and responsibilities.  

## Provider

**Software Assurance**

**Notes:**
Next we’re going to explore the role of the Provider, Key Activities, and Software Deliverables.

## Provider Software Assurance

***Key Activities:***
**Identifies one or more persons to direct and manage a software assurance program (e.g., a software assurance manager) – independent from the project**
**Develops a software assurance program that includes SQ, Software Safety, Software Reliability, V&V, and IV&V (when required)**
**Establishes and maintains a working relationship with project management and the acquirer**
**Establishes and maintains a software assurance plan that conforms to IEEE STD 730-2002**
**Conducts periodic reviews, audits, and assessments of the development processes and products**

**Notes:**
The first activity from a Provider perspective is to identify someone to direct and manage the Software Assurance Program.  This person must have a reporting channel to provider management that is independent of the provider’s project management and software development function.  

NOTE that for smaller projects, the software assurance plan may be incorporated in another project planning document.  Larger projects typically have a separate plan or multiple software assurance plans.  

## Provider Software Assurance

***Key activities (cont.)***
**Prepares and maintains software assurance records from software assurance activities**
**Participates in formal and informal reviews**
**Prepares software assurance status reports, highlighting**
- Assurance accomplishments from reviews, tests, etc.
- Significant  problems, their status, solutions, and remedial and preventive actions
- Trends in software quality metric data
- Plans for upcoming software assurance activities
- Recommendations and lessons learned
**Flows down software assurance requirements to any subcontractors and assures compliance**

**Notes:**
PRESENTER:  Remind the audience that these activities are a subset of what is prescribed in the NASA Software Assurance Standard.

## Provider Deliverables

**Sample Software Deliverables**
- Software Management Plan
- Configuration Management Plan
- Software Requirements Specification
- Requirements Traceability Matrix
- Test plans and procedures
- User’s Guides
- ** Etc.
**Software Assurance Deliverables**
- Software Assurance Plan
- Software Assurance Status Reports
- Software Assurance Records

- **  See NPR 7150.2 for more on software products and required content

**Notes:**
This slide provides a subset of expected provider deliverables from both a software development perspective and software assurance perspective.  This is by no means an exhaustive list.  

## Provider’s Role by Phase...

***CONTRACT***
*** ******IMPLEMENTATION***
- Establish communication with acquirer, project team
- Maintain SA plan
- Conduct /participate in reviews, inspections, audits and prepare SA  reports
- Collect, utilize metrics to identify trends, assess quality
- Deliver SA reports, maintain records

***INITIALIZATION,******PRE-AWARD ***
- Draft SA approach
- Respond to SA requirements in RFP

***POST-RFP; PRE-AWARD***
- Address SA questions from acquirer
- Participate in contract negotiations for SA

***POST-AWARD; PRE-IMPLEMENTATION***
- Develop SA plan
- Review Acquirer SA plan
- Ensure trained SA personnel

***ACCEPTANCE***
- Conduct audits prior to delivery
- Assure acceptance documentation
- Prepare Lessons learned

***OPERATION***
- Develop New/updated SA plan
- Ensure SA processes in place
- Conduct periodic audits of OPS

***MAINTENANCE***
- Ensure SA processes in place
- Conduct periodic audits or assessments

***RETIREMENT***
-  Send final SA records to project

**PROVIDER’S**
**ROLE**

**Notes:**

This last chart in the Provider Section summarizes the key Provider activities by life cycle phase.  

PRESENTER:  The intent is to provide a snapshot or summary of the Provider’s roles and responsibilities.  

## Communication is Essential!

**Verify Provider SA Plan is complete and compatible with Acquirer SA Plan**
**Review Provider plans and procedures **
**Review Provider SA reports and records from surveillance activities**
**Evaluate software products and deliverables throughout life cycle**
**Prepare SA Status reports for Project**
**Establish and maintain working relationship with Project, software assurance personnel, and the Provider**

**Develop Provider SA Plan per Acquirer requirements**
**Review Acquirer plans and procedures**
**Flow down requirements to any subcontractors**
**Prepare and maintain SA records (accessible by Acquirer)**
**Prepare SA Status reports for Project and Acquirer**
**Establish and maintain working relationship with Project, software assurance personnel, and the Acquirer**
**Share issues/concerns/risks**

**Acquirer SA**

**Provider SA**

- Software Assurance Plan

- Plans and Procedures

- Software Assurance Status Reports

- Software Assurance Records

- Software Products and
- Deliverables

- Meetings, Telecons, and
- and Project Reviews

**Notes:**
This chart highlights the relationship between Acquirer and Provider Software Assurance from a deliverable and communication perspective.  

Interface activities are highlighted in the blue rectangles, while deliverables flow between.

## Software Measurement

- “...when you can measure what you are speaking about, and express it in numbers, you know something about it; when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind...”

- Lord Kelvin, 1889

**Notes:**
Project managers use numbers to help in system-level decision-making.  In hardware, it could be MTTF, or percent of parts not delivered on time...

Software assurance information also needs to be quantified to assist effective decision-making.  In this section, we address basic principles of software measurement, and provide some simple metrics to provide you with an understanding of how measurement data is useful for assessing the quality of the software throughout the life cycle.

## Why Measure?

***Measurements***
- provide meaningful information to enable informed control decisions
- provide data from which to model and predict future software trends

**Are we there yet?**
- 	How much time and effort required to detect remaining errors?
- 	To what extent is product error free?
**What will it take to get there?**
**Why did/didn’t we get there?**
- 	Post mortem analysis – lessons learned

**Notes:**
What can measurements tell us?

## Sample Metrics

_**Requirements**_
- Ambiguity = weak phrases
- Completeness = TBD + TBA + TBR
- Volatility = excessive requirement changes
_**Design & Implementation**_
- Structure/Architecture = complexity & size
_**Verification**_** (e.g., formal reviews, peer reviews, testing)**
- Defect density (defects per 1000 lines of code)
- Problem report tracking = open, closed, severity

***Start simple with only a few metrics...***

**Notes:**
When developing metrics, always start simple...

With a good software tool (such as ARM) to analyze requirements, you can get a good idea of where you are relative to turning over the requirements to design, or, going back to the customer to get more clarification, etc.

You can use complexity measurements to determine if the software may be difficult to test or too complex. 

And of course, many metrics during the verification phase are relatively easy to collect with a defect data tracking tool.  Goals, questions, and actions for defect data are many.

## Requirements Metrics Completeness & Volatility Analysis

***CRR ***
***Looks Good!***
***(Stable)***

***CRR ***
***Excessive Changes!***
*** ******NOT Stable***

**Modifications to Requirements**

- 0

- 50

- 100

- 150

- 200

- 250

- 300

- 350

- 400

- 450

- 1Q94

- 2Q94

- 3Q94

- 4Q94

- 1Q95

- 2Q95

- 3Q95

**Calendar Quarter**

**Quantity**

- New

- Modified

- Deleted

**Total Number of Requirements****       **

- 0

- 100

- 200

- 300

- 400

- 500

- 600

- 700

- 800

- 900

- 1000

- 1Q94

- 2Q94

- 3Q94

- 4Q94

- 1Q95

- 2Q95

- 3Q95

**Calendar Quarter****    **

**Quantity   **

***Combination of BOTH views indicates risk area - requirements are NOT YET stable***

**Requirements Metrics** **Completeness & Volatility Analysis**

**Notes:**
These two charts illustrate how more than one type of information may be needed so that you do not misinterpret the data. The left chart would seem to imply that the requirements are stable, that is, few new requirements are being added. However, the chart on the right shows that many requirements are still undergoing modification.  

Your issue /goal:  Determine if development is ready to proceed to design

Metrics:  	How many new requirements are added over time?
	How many existing requirements are modified over time?

Analysis:  Utilize more than one metric. 

## Defect Trending

**Problem --> Opening Errors **
**Continues sharp climb**

**Expected**

**Notes:**

This chart provides insight into defect open and closure rates and the provider’s ability to work off anomalies.  The slope/climb of the defects submitted can serve as an indicator of software stability and the degree to which test is uncovering problems.  The number of defects is expected to increase with time, but should subside or level off as the system stabalizes and matures. Test activities should not be concluded if defect rates continue at an increasingly steady rate.

Other trends to examine include: 
 Severity / criticality of the defects
 Location – same type of error? 
-  Detection method

## Hints for Successful Measurement

**Define your measures of success early in your project and track your progress toward them**
**Collect only what you need, and use what you collect**
**Use more than one metric to interpret the entire picture**
**Use defect data trends to help you decide when to release a product**
**Measure complexity to help you optimize design decisions and create a more maintainable product**
**Ensure reasonable data collection, interpretation of data**
**Categorize defects to identify product and process weaknesses**

**Notes:**
We discussed complexity relative to testing and maintenance, and trending to indicate where things are going awry.  You can see that often more than one perspective is needed to put the full picture into context.  Most important, know what data you need to generate the types of measurements with which you can use, and collect only those.  Keep it simple!

## NASA Software Assurance

**Classification Assessment**

**Notes:**
This brings us to NASA’s Software Assurance Classification Assessment and a high-level overview of the process.  Given the overview nature of this class, it is recommended that you reference the NASA Software Assurance Standard for more details on each step of the assessment process.

## SA Classification Assessment Process

**The Software Assurance Classification Assessment was developed by NASA to identify and evaluate the characteristics of software to determine the software’s classification and level of software assurance to be applied**
**This assessment is conducted for all NASA software**
**The process entails 4 basic steps:**
- Perform the Software Safety Litmus Test to determine if the software has safety implications for the system, property, humans, or the environment
- Determine the software engineering class of software (Class A-H)
- Determine the software criticality score based on project/mission/software characteristics
- Classify the software and software assurance effort based on the results from Steps 1-3

**Notes:**
So where did this process originate and when does the assessment need to be performed?

The SA Classification Assessment Process is a new process that was developed and incorporated in the recent update to the NASA SA Standard in July 2004.  This assessment is conducted once the software program, project, or facility is conceived and initially approved. Depending on the complexity and criticality of the software, the assessment may be conducted for multiple software tasks within a system or facility.  

The assessment entails a basic 4-step process...

##  SA Classification Assessment (cont.)

**After each step of the process, record the results in a Software Assurance Classification Report and maintain as a quality record**
**Provide report to the SMA Director and Systems Management Office (SMO)**
***Report contents include:***
- Project name and date
- Names of software assurance manager and
- 	project manager
- Result of software safety litmus test
- Software Class (A-H)
- Software Criticality score
- Software Assurance Effort/Priority

**Notes:**
Results from each step are recorded along the way in a Software Assurance Classification Report.

PRESENTER:  Pass around a completed sample assessment report for one of your Center’s software programs or projects and walk them through the report contents (emphasizing the location of the results from each of the 4 steps)

## Software Assurance Classes

**Class A****	****Human Rated Software Systems**
**Class B****	****Non-Human Space Rated Software Systems**
**Class C****	****Mission Support Software**
**Class D****	****Analysis and Distribution Software**
**Class E****	****Development Support Software**
**Class F****	****	****General Purpose Computing Software (Multi-****	****	****	****Center or Multi-Program/Project)**
**Class G****	****General Purpose Computing Software (Singe****	****	****	****Center or Project)**
**Class H****	****General Purpose Desktop Software**

**Notes:**
Examples for each software class are as follows:

Class A – Applies to all space flight software subsystems to support human activity in space (e.g., navigation and control, crew escape, mission operations)
Class B – Flight and ground software that must perform reliably in order to accomplish primary mission objectives for robotic spacecraft (e.g., power systems, command and control ground systems)
Class C – Flight or ground software that is necessary for science return from a single (non-critical) instrument, or is used to analyze or process mission data, or other software for which a defect could adversely impact attainment of some secondary mission objectives (e.g., mission data processing and analysis, pre-launch integration and test software)
Class D – Non-space flight software (e.g., science data collection, storage, and distribution, analysis tools)
Class E – Software developed to explore a design concept or support development functions (e.g., earth science modeling, low TRL research software)
Class F – Examples include software supporting IFMP or Travel Manager
Class G – Examples include Center custom applications like a corrective action tracking system
Class H – Examples include Microsoft Word, Excel, and Adobe Acrobat

## Tailoring Guidelines for SA Requirements

**The level of Software Assurance needed is commensurate with the software classification, criticality, unique software characteristics (e.g., software size, complexity), and perceived risk**
**Based on the NASA SA Standard and NPR 7150.2, tailoring guidelines are being developed to assist assurance practitioners in defining the software assurance requirements required to meet specific project/mission/software objectives**
**The tailoring guidelines will be provided as part of the NASA Software Assurance Guidebook (currently under revision)**

## Supplementary

**Information**

**Notes:**

The remaining slides provide supplemental information on Center Contacts, Recommended Web Sites, Additional Training Opportunities, and Reference materials.

## GSFC Contacts

**Software Working Group (SWG) Representatives**
**Sally Godfrey  301 286-5706**
**Susan Sekira  301 286-6160**
**Center Software Assurance Lead (Code 303)**
**Susan Sekira  301 286-6160**
**Software Assurance Technology Center (Code 304)**
**Al Gallo, Manager  301 286-3756**
**Systems Safety and Reliability Office (Code 302)**
**Karen Fisher, Chief  301 286-7123**
**Flight Software Branch (Code 582)**
**Elaine Shell, Head  301 286-2628**

**Notes:**
PRESENTER:  Add your Center’s information.  At a minimum, provide contact information for your SWG member (s) and SA Lead.

## Recommended Web Sites

**NASA Software Assurance**
_**http://software.nasa.gov/**_
**NASA Software Working Group****  **
**	**_**[http://nasa-software-pbma-kms.intranets.com/](http://nasa-software-pbma-kms.intranets.com/)**_
**NASA Process Based Mission Assurance (PBMA)**
**	**_**[http://](http://pbma.hq.nasa.gov/pbmamaster.html)**__**[pbma.hq.nasa.gov/pbmamaster.html](http://pbma.hq.nasa.gov/pbmamaster.html)**_
**NASA Technical Standards**
**	**_**[http://](http://standards.nasa.gov/)**__**[standards.nasa.gov](http://standards.nasa.gov/)**_**/**
**Carnegie Mellon- Software Engineering Institute**
_**[http://](http://www.sei.cmu.edu/)**__**[www.sei.cmu.edu](http://www.sei.cmu.edu/)**__**[/](http://www.sei.cmu.edu/)**_
**Software Engineering Laboratory**
**	**_**[http://](http://sel.gsfc.nasa.gov/)**__**[sel.gsfc.nasa.gov](http://sel.gsfc.nasa.gov/)**_**/**

**Notes:**
PRESENTER:  Point out the various web sites and the assets they can expect to find.

## More Web Sites

**Software Assurance Technology Center**
**	**_**[http://](http://satc.gsfc.nasa.gov/)**__**[satc.gsfc.nasa.gov](http://satc.gsfc.nasa.gov/)**_**/**
**NASA Independent Verification and Validation**** **
- 	      _[http://www.ivv.nasa.gov](http://www.ivv.nasa.gov/)_
**GSFC Engineering Process Group**
**	**_**[http://software.gsfc.nasa.gov](http://software.gsfc.nasa.gov/)**_
**GSFC Software Assurance**
**	**_**[http://sw-assurance.gsfc.nasa.gov](http://sw-assurance.gsfc.nasa.gov/)**_

**Notes:**
PRESENTER:  Add your Center’s specific web sites.

## GSFC SA Groups and Initiatives

**Engineering Process Group (EPG)**
- Center-wide team that establishes and continuously improves system and software processes and products
**Software Assurance Technology Center (SATC)**
- Center of Excellence that provides project support, outreach, and research in software metrics and software assurance tools and techniques
**Quick-Look Project Team**
- Team of software experts organized to take a quick look at potential (or impending) software problem areas for a specific project

**Notes:**
PRESENTER:  This is an example for GSFC.  Add your Center’s SA Groups and Initiatives and discuss their charters and/or services/support.

## GSFC Procedures and Products

**GPG 8700.5, In-House Development and Maintenance of Software Products**
**580-PG-8730.3.1, ISC Product Development Handbook**
**GPG 8700.6, Engineering Peer Reviews**
**300-PG-7120.2.2, Mission Assurance Guidelines Tailoring**
**303-PG-7120.2.1, Developing and Implementing Software Quality Programs**
***Also available work instructions, checklists and templates for:***
**	****	****Software Quality (SQ) process and product assessments**
**	****	****SQ Checklists for evaluating processes and products**
**	****	****FSW Product Plans and Test Plan Templates**
**	****	****FSW Coding Standards for Ada and C**
**	****	****FSW CCB Process and Policy**
**	****	****FSW Development Life Cycle**
**	****	****ARM tool, code analysis and metrics, reliability tools (via SATC)**

**Notes:**
PRESENTER: This is an example for GSFC.  Add your Center’s specific SA procedures, checklists, etc.

## Additional Training Opportunities

**SOLAR Training****  **_**[https://](http://solar.msfc.gov/)**__**[solar.msfc.nasa](http://solar.msfc.gov/)**__**[. ](http://solar.msfc.gov/)**__**[gov](http://solar.msfc.gov/)**__**[/](http://solar.msfc.gov/)**_
**Software Assurance**
**Software System Safety**** **
**NASA Engineering Training Program (NET)  **_**http://net.larc.nasa.gov/**_
**Introduction to CMMI**
**Intermediate CMMI**
**Defining World-Class Processes**

**Notes:**
PRESENTER: Add any Center-specific training.

## Recommended Reference Documents

**Notes:**
The following recommended reference documents are available via the Agency Software Assurance Web site or the NASA Technical Standards Web site.  

The NASA policies, directives, standards, and guidebooks provide a common framework for consistent practices across NASA programs.  Software Assurance practitioners are encouraged to reference these documents often throughout the entire software life cycle!

## Additional References

- Conte, Dunsmore, Shen,*Software Metrics Engineering Metrics and Models*, Benjamin/Cummings, 1986
- Leveson, Nancy G., *SAFEWARE System Safety and Computers, *Addison-Wesley Publishing Company, Inc., 1995
- Lyu, Michael R., *Handbook of Reliability Engineering*, IEEE Computer Society Press, McGraw Hill, 1995
- Shulmeyer, Gordon G., and James I. McManus,*The Handbook of Software Quality Assurance,*Prentice Hall, 1999
- Kan, Stephen H., *Metrics and models in Software Quality Engineering,*Addison Wesley, 2002

**Notes:**
PRESENTER:  Add and/or delete references, as appropriate.

## Summary

**Notes:**
This brings us to the course summary and a reminder of the course objectives.

Hopefully you walk away today with a general awareness and understanding of the 5 disciplines of Software Assurance and the key activities for both the Acquirer and Provider.  We also discussed the value of software measurements and NASA’s new Software Assurance Classification Assessment Process and YOU’RE encouraged to share your knowledge and understanding with others in your organization or project.

## Benefits of Software Assurance

**Provides insight into the software development processes and products throughout the life cycle**
**Focuses on opportunities for**
- Early error detection
- Problem prevention
- Risk identification and mitigation
**Increases chances of delivering full functionality within cost and schedule**
**Enables improvement of future software products and services!!**

**Notes:**
We also emphasized the benefits of Software Assurance and what it can mean to your program or project throughout the software life cycle.  

PRESENTER:  Be sure to have the NASA Software Assurance Trifolds available for distribution.  They, too, provide a high-level overview, benefits, and keys to successfully implementing software assurance programs. 

## Example SA Activities

**Software Assurance Classification Process**
**Proposal and contract evaluations**
**Requirements traceability**
**Design and code analyses**
**Engineering peer reviews**
**Test planning and test execution**
**Software measurement and trends**
**Audits and assessments**

**Team work and communication!!**

**Notes:**
Throughout the course, we mentioned numerous software assurance activities for each discipline and those personnel responsible for each.  

They included, at a minimum... 

## Keys to a Successful SA Program

**Identify a person responsible for directing and managing the Software Assurance Program**
**Define software assurance requirements early in the life cycle**
**Develop a Software Assurance Plan**
**Monitor processes throughout the system and software development life cycle**
**Evaluate software and deliverables to assure that quality and safety are being built into the products**
**Ensure compliance with established standards and procedures**
**Establish metrics to help measure quality**

**Notes:**
The keys to successfully implementing software assurance is quite simple and straightforward.  First...

## More Keys...

**Assure that problems and risks are documented, reported, addressed, and tracked to closure**
**Prepare and maintain software assurance records and status reports**
**Capture Lessons Learned to improve the quality of future products/services!**

## Backup Slides

**Notes:**
PRESENTER:  Available for use, if needed.

## V&V Activity Matrix

**Notes:**
The various V&V activities are represented horizontally across the top, while the participants are highlighted vertically.  The action verbs in green across the bottom define the roles and responsibilities of the various participants.

This matrix is provided to illustrate the breadth of the V&V discipline and the associated personnel.

PRESENTER:  Choose examples to elaborate on (if appropriate).

## Fundamental Differences

- Provides **Center**-level services
- Focuses on ALL Project software
- Emphasizes compliance to  standards and procedures
- Reviews, monitors and audits all Project processes and products for completeness and accuracy
- Matrixed to the Project as part of the Project Team and provides daily insight/oversight
- Reports to Project and Center Director through S&MA

- Provides **Agency**-level services
- Focuses on MISSION CRITICAL Project software
- Emphasizes completeness and correctness of the product
- Reviews, analyzes, and provides in-depth evaluations   of life cycle products which have the highest risk
- Independent from the Project and provides analyses and evaluations per IV&V priorities
- Reports to Project, GPMC’s, and NASA Headquarters

**SQ**

**IV&V**

**Notes:**
Project Managers and Software Development Leads/Managers frequently inquire into the fundamental differences between the disciplines of SQ and IV&V.  This chart illustrates some of the key differences between the two in terms of scope, activities, staffing, and reporting.  

## Frequently Asked Questions

**How do we levy software assurance requirements on an existing software development activity?**
**How do these requirements apply to a small development effort?**
**Are these requirements applicable to in-house work?**
**Who determines the software classification for my software?**
**Do I need to document the classification?**

**Notes:**
Leftover from Version 1.0 of the training module...