# Presentation: 936702

## IV&V TransitionConcept Review

- December 12, 2003

## Introduction

- Transition Team Members
- Steve Raque, IV&V Facility, Lead
- Neal Barthelme, GSFC
- Paul Boldon, HQ Code Q
- Ken Costello, IV&V Facility
- Rick Grammier, JPL
- John Kelly, HQ Code AE
- Significant contributions also provided by Burt Sigal (JPL) and Sue Sekira (GSFC)
- Agenda
- Concept Review Purpose
- Background – IV&V Assessment
- Transition Plan
- Concept for NASA IV&V

## Concept Review Purpose

- The concept for Agency IV&V is a “system” composed of several “subsystems” (processes) that will be owned by different organizations
- An Agency approach to IV&V requires the evaluation and management of software risk across the Enterprises, OSMA, and Engineering
- Think of the Concept Review as the “System PDR” of the IV&V Transition
- The Concept Review is:
- An important step in establishing IV&V as an Agency Program by FY05
- Enables the establishment of a PCA between GSFC and Code Q
- A gate to effecting the transition
- Draft Transition Plan and Draft PCA reviewed by GSFC and Code Q
- After Concept Review, the Transition Plan and PCA will be formally transmitted to Code Q for approval
- The Transition Plan names the AA/OSMA as the overall approving official for the IV&V Concept with the responsibility to call upon other stakeholders and process owners to ensure the concept is accepted Agency-wide. Mr. O’Connor has delegated the review to the GSFC PMC

## Background

- In February 2003, the NASA Associate Deputy Administrator for Technology Programs requested Code Q to lead a team to assess the Agency’s IV&V capability
- The assessment team was made up of personnel from HQ, GSFC, JPL, KSC, and DCMA
- The assessment was conducted during the months of February and March 2003
- The goals of the assessment were the following:
- to better understand IV&V's potential
- to help shape IV&V’s continued role at NASA
- to determine how best to structure the reporting and support of the IV&V Facility to assure its technical, managerial and financial independence
- The team reviewed IV&V’s processes and products.  The assessment included interviews with both IV&V customers and with IV&V management.
- The final assessment team report was published on April 23, 2003
- The conclusions and recommendations of the assessment team were briefed to the NASA Associate Deputy Administrator for Technology Programs, the NASA Chief Engineer, the Associate Administrator for Safety and Mission Assurance, and the Director of the IV&V Facility on April 4, 2003.  At the conclusion of this briefing, the Director of the IV&V Facility was directed to evaluate the assessment team’s conclusions and recommendations and prepare a transition plan for implementation of agreed to changes based on the assessment team’s report.
- On May 13, 2003, the NASA Executive Council was briefed on the results of the assessment and the recommended IV&V management and funding changes
- During this meeting, and in September of 2003, the Executive Council decided that IV&V would become a Code Q Program delegated to GSFC, an IV&V Board of Directors (IBD) would be formed, IV&V would be funded from Corporate G&A, and measures to improve the effectiveness of IV&V would be put in place

## Transition Plan

- NASA Executive Council decisions are used as the transition goals
- Transition IV&V to a Code Q Program Delegated to GSFC
- Create an IV&V Board of Directors
- Transition IV&V Funding to Agency G&A
- Improve the effectiveness of IV&V processes
- The plan objectives are to describe the tasks, responsibilities, participants, schedule, and dependencies that will ensure:
-  A smooth transition and the goals are met
- The recommendations and concerns identified by the Assessment Team are addressed
- There is coordination with changes necessary to address IG-03-011
- AE and Q establish a process to provide IV&V with complete list of NPG 7120.5 and NTS 8719.13 projects on a recurring basis
- AE verify that programs and projects comply with IV&V policy
- The Inspector General (IG) has communicated to the Chief Engineer that IG-03-011 will be closed.  “The IG has determined that the actions taken by NASA (including this transition plan) are sufficient to disposition all of the audit's recommendations.” (Dec 5, 2003)

## Transition Task/Schedule Overview

## Key Tasks

- Develop Process for Compiling and Validating List of IV&V Candidate Projects (AE owns process)
- Develop Process for Scoping and Costing IV&V (IV&V owns process)
- Develop IV&V Implementation Process (IV&V owns process)
- Define Review, Process Control, and Reporting of IV&V (Code Q and GSFC own processes)
- Determine Relationships among IV&V, GSFC, HQ, Enterprises, Centers, and Projects
- The primary goal of the processes developed in tasks 3 and 4 is to improve the effectiveness of IV&V

## Concept for Agency IV&V

## Overall IV&V Process

## Ranked List of Projects Based on Criticality of Software

- Process owned by Code AE
- Use the knowledge base of Headquarters and Centers to compile a list of Projects with software under development or maintenance governed by NPG 7120.5 or 8719
- Objective is to cover a five year horizon
- Projects are ranked using an Agency-wide criticality criteria
- Projects are divided into four major classes – Human Rated (H), Robotic (R), Instruments (I), and Information or Data Analysis systems (D)
- Each project is given a score based on the characteristics of the project. Engineering notes can provide mitigations to the score or ranking.

## Annual Process for Compiling and Validating Ranked List of Projects

**A.1 – A.3 **
**Agency Projects**_** **_
**List (B/Q/AE)**

**C.1 – C.2**
**Integrated Ranked list**
** ****of Candidate Projects**
**(AE)**

**B.2 Enterprise &**
**Center Review**

**C3. Agency Review**

**IV&V Estimation **
**& Budget Approval **
**Process**

**SW**
**Assurance**

**SW Criticality **
**Criteria**
**Worksheet (new)**

**B.1 Centers**
** ****generates prioritized **
**project lists**

**A.4 AE Requests to Center**
** ****generate prioritized **
**project lists**

**SW**
**Engineering**

**C.4 Final Integrated**
**Ranked list of **
**Candidate Projects**
**(AE)**

## Ranking Process Frequently Asked Questions

- Q: Can the Centers submit a project in a different rank order than the position indicated by the worksheet “project criticality score”?
- A: Yes. The worksheet is an aid to promote consistency, not a substitute for engineering judgment.
- Q: How much of this process do you expect to see in updated NPDs or NPGs?
- A: The requirement to develop a ranked list of projects containing significant software and the responsible HQ office should be included in an NPD/NPG. Codes AE and Q will provide an appropriate level of direction on the use of the annual ranked list relevant documentation.

## Ranking Process Frequently Asked Questions

- Q: Why is a ranked list needed?
- A: The ranked list provides a manageable funding cut point which assures Agency resource are being wisely used to mitigate risks in its most critical projects.
- Q: Do you expect this list to have usefulness beyond the IV&V budget process?
- A: Yes.  I believe that the list will raise the awareness of critical software components and help Software Assurance and Engineering focus their efforts.

## Scoping and Costing IV&V

## Scoping and Costing IV&V

- Purpose is to provide the IV&V Board of Directors (IBD) with a life-cycle IV&V cost estimate for each candidate project
- The IV&V Facility determines the proper level of IV&V support to be applied to each of the projects and generates a cost estimate for each project on the Agency ranked list
- Different cost estimation techniques are used depending upon amount of project detail known (schedule, software size, etc.)
- Data is collected to improve the various cost estimation techniques

## Scoping and Costing IV&V

## IV&V Budget Review Process

## IV&V Budget Review and Approval Process

- IBD is chaired by AA/OSMA and has the Chief Engineer, the CIO, and the AAs for M, N, R, S, U, and Y as voting members. GSFC Center Director and IV&V Facility Director are ex officio members
- The IBD determines the appropriate level of funding for software IV&V and selects the Programs and Projects that require support from the IV&V Facility
- It is the intent of the IBD to apply the proper level of IV&V on Programs and Projects that have the most critical software development
- Lower priority Projects, which cannot be supported within the IBD established budget, will not receive IV&V from Corporate G&A
- IBD annually recommends 5 year IV&V budget to Code Q

## IV&V Services and Integration with the Project Life Cycle

## IV&V and the Project Life-Cycle

**IV&V and the Project Life-Cycle**

- Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk
- Dialog between the IV&V Facility and the Project must begin before SRR
- For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post-launch or major upgrades/maintenance (e.g. Shuttle, MER)

- System Requirements Review

- Preliminary Design
- Review

- Critical
- Design
- Review

- System Test

- S/W
- FQT

- Initial
- IVVP
- Signed

- Mission Readiness Review

- Concept
- Phase
- 1.3

- Requirements
- Phase
- 1.4

- Design
- Phase
- 1.5

- Implementation
- Phase
- 1.6

- Test
- Phase
- 1.7

- Operations Phase &
- Maintenance Phase
- 1.8, 1.9

- Baseline
- IVVP
- Signed

- - IV&V provides support and reports for Project milestones
-  Technical Analysis Reports document major phases
-  IVVP is updated to match changes in Project

- IV&V
- Provides
- CoFR

- IV&V Final Report

- IV&V Management & Planning and Phase Independent Support
- 1.1, 1.2

- System
- Retirement

- Launch

- Note: numbers correspond to IV&V WBS

## Management and Support Tasks

- IV&V Management
-  and Planning
- 1.1

- Phase Independent
- Support
- 1.2

- These activities help to ensure the proper application of IV&V on projects
- Executed throughout the Software Development Life Cycle
- Covers development of IV&V Plan and interfaces with projects
- The IV&V Plan (IVVP) documents the IV&V approach for the Project. Some sections of the IVVP require Project concurrence (the sections covering schedule, POCs/Interfaces, reports/deliverables to the Project, and items (e.g. software artifacts) needed by IV&V).
- Initial IVVP serves to set overall expectations, POCs, etc. It is not a full-blown plan (not enough software detail is usually known).
- Interfaces include project suppliers (such as contractors building software for NASA). Proper language must be included in development contracts to provide IV&V access to software artifacts.
- Other cross-cutting tasks such as Issue Tracking and Resolution, management review, project review support, etc.

- Project Software Development Life Cycle

## Primary Life Cycle Tasks

**Primary Life Cycle Tasks**

- These activities form the core of the effort on any project
- Includes tasks for all life cycles that examine primarily life cycle products
- Includes such things as requirements traceability analysis, design architecture analysis, code analysis, test analysis, etc.
- Results are documented thoroughly and presented to the project
- Issues and risks are tracked through closure (task under management and support).
- Technical reports are provided directly to the projects.
- Results/issues are communicated immediately. They are not held until formal technical reports.
- Best interface for resolving technical issues is between IV&V analysts and development personnel.

- Concept
- Phase
- 1.3

- Requirements
- Phase
- 1.4

- Design
- Phase
- 1.5

- Implementation
- Phase
- 1.6

- Test
- Phase
- 1.7

- System Requirements Review

- Preliminary Design
- Review

- Critical
- Design
- Review

- Integration
- & Test

- System Test

- Mission Readiness Review

- S/W FQT

## Close Out and Ongoing Analysis

- IV&V provides a final report documenting its findings and results
- Supports mission/launch/flight readiness reviews
- Effort not always complete at time of launch
- Extended missions require after launch support due to software being developed and uploaded during cruise phase
- Some missions have extended life times and perform ongoing software maintenance
- IVV support mirrors normal life cycle processes but on a smaller level and in an iterative manner
- Each planned change is a mini-IV&V project (Space Shuttle)

- Operations
- Phase
- 1.8

- Maintenance
- Phase
- 1.9

- Launch

- Begin Operations

- Operational Maintenance

- System Retirement

## IV&V WBS for NASA Missions

- The purposes of the IV&V Work Breakdown Structure are to:
- Provide a consistent approach to IV&V across the Agency (similar missions receive the same amount of risk mitigation)
- Provide a consistent and comprehensive basis for collection and reporting of metrics
- Help Projects anticipate and understand what IV&V will do for them and what will be needed from them
- The IV&V WBS was developed using industry standards and previous IV&V on NASA missions as reference
- IEEE Std. 1012-1998 IEEE Standard for Software Verification and Validation
- IEEE/EIA 12207.0-1996 Standard for Information Technology-Software life cycle processes
- Analysis of all missions supported by the IV&V Facility
- Baseline WBS Tasks for NASA Missions
- The baseline represents a set of tasks that are nominally performed on a NASA mission of the given type (mission types correspond to those used in ranking criteria)
- Used as the task set for pre-startup assessment budget estimates
- Tasks are tailored based on project characteristics (module/function level size, complexity, reuse, risk, etc.) as part of IV&V planning and criticality analysis tasks

## IV&V WBS

- The heading legend is as follows:H – Human Rated, R – Robotics, I – Instrument, D – Science and Data Analysis
- Items with an X are the baseline tasks for that type of mission
- Tasks not included in baseline for a particular mission (including 1.10 Optional Tasks) can be performed:
- As recommended by IV&V to mitigate particular project risks
- When paid for by Projects to provide additional risk mitigation
- As funded by the IBD or Code Q

| 1.1 | IV&V Management and Planning | H | R | I | D | 1.2 | Phase Independent Support | H | R | I | D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.1.1 | Independent Verification and Validation Planning | X | X | X | X | 1.2.1 | Issue and Risk Tracking | X | X | X | X |
| 1.1.2 | Baseline Change Assessment | X | X | X | X | 1.2.2 | IV&V Tool Support | X | X | X | X |
| 1.1.3 | Management Review of IV&V | X | X | X | X | 1.2.3 | Management and Technical Review Support | X | X | X | X |
| 1.1.4 | Final Report Generation | X | X | X | X | 1.2.4 | Criticality Analysis | X | X | X | X |
|  |  |  |  |  |  | 1.2.5 | Participate in Inspections & Walkthroughs | X | X | X | X |

## IV&V WBS (Continued)

| 1.3 | Concept Phase | H | R | I | D | 1.4 | Requirements Phase | H | R | I | D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.3.1 | Reusability Assessment | X | X | X | X | 1.4.1 | Traceability Analysis | X | X | X | X |
| 1.3.2 | Distributed Architecture Assessment | X | X | X |  | 1.4.2 | Software Requirements Evaluation | X | X | X | X |
| 1.3.3 | System Requirements Review | X | X |  |  | 1.4.3 | Interface Analysis | X | X | X | X |
| 1.3.4 | Concept Document Evaluation | X |  |  |  | 1.4.4 | System Test Plan Analysis | X | X | X | X |
| 1.3.5 | Software/User Requirements Allocation Analysis | X |  |  |  | 1.4.5 | Acceptance Test Plan Analysis | X | X |  |  |
| 1.3.6 | Traceability Analysis | X |  |  |  |  |  |  |  |  |  |

## IV&V WBS (Continued)

| 1.5 | Design Phase | H | R | I | D | 1.6 | Implementation Phase | H | R | I | D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.5.1 | Traceability Analysis | X | X | X | X | 1.6.1 | Traceability Analysis | X | X | X | X |
| 1.5.2 | Software Design Evaluation | X | X | X | X | 1.6.2 | Source Code and Documentation Evaluation | X | X | X | X |
| 1.5.3 | Interface Analysis | X | X | X | X | 1.6.3 | Interface Analysis | X | X | X | X |
| 1.5.4 | Software FQT Plan Analysis | X | X | X | X | 1.6.4 | System Test Case Analysis | X | X | X | X |
| 1.5.5 | Software Integration Test Plan Analysis | X | X | X | X | 1.6.5 | Software FQT Case Analysis | X | X | X | X |
| 1.5.6 | Database Analysis | X | X | X | X | 1.6.6 | Software Integration Test Case Analysis | X | X | X | X |
| 1.5.7 | Component Test Plan Analysis | X | X |  |  | 1.6.7 | Acceptance Test Case Analysis | X | X |  |  |
| 1.5.8 | Sizing and Timing Analysis | X |  |  |  | 1.6.8 | Software Integration Test Procedure Analysis | X | X |  |  |
| 1.5.9 | Data Flow Analysis | X |  |  |  | 1.6.9 | Software Integration Test Results Analysis | X | X |  |  |
|  |  |  |  |  |  | 1.6.10 | Component Test Case Analysis | X | X |  |  |
|  |  |  |  |  |  | 1.6.11 | Sizing and Timing Analysis | X |  |  |  |
|  |  |  |  |  |  | 1.6.12 | System Test Procedure Analysis | X* |  |  |  |
|  | *Performed for safety critical functions only |  |  |  |  | 1.6.13 | Software FQT Procedure Analysis | X* |  |  |  |

## IV&V WBS (Continued)

| 1.7 | Test Phase | H | R | I | D | 1.9 | Maintenance Phase | H | R | I | D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.7.1 | Traceability Analysis | X | X | X | X | 1.9.1 | IVVP Revision Analysis | X |  |  |  |
| 1.7.2 | Software FQT Results Analysis | X | X |  |  | 1.9.2 | Proposed Change Assessment | X |  |  |  |
| 1.7.3 | System Test Results Analysis | X | X |  |  | 1.9.3 | Task Iteration | X |  |  |  |
| 1.7.4 | Simulation Analysis | X |  |  |  | 1.9.4 | Anomaly Evaluation | X |  |  |  |
| 1.8 | Operations Phase |  |  |  |  | 1.9.5 | Migration Assessment | X |  |  |  |
| 1.8.1 | Evaluation of New Constraints | X |  |  |  | 1.9.6 | Retirement Assessment | X |  |  |  |
| 1.8.2 | Proposed Change Assessment | X |  |  |  |  |  |  |  |  |  |
| 1.8.3 | Operating Procedure Evaluation | X |  |  |  |  |  |  |  |  |  |

## IV&V WBS (Continued)

| 1.10 | Optional Tasks |  |  |
| --- | --- | --- | --- |
| 1.10.1 | Acceptance Test Design Analysis | 1.10.13 | Hazard Analysis |
| 1.10.2 | Acceptance Test Procedure Analysis | 1.10.14 | Independent Risk Assessment |
| 1.10.3 | Acceptance Test Results Analysis | 1.10.15 | Independent Testing |
| 1.10.4 | Algorithm Analysis | 1.10.16 | Operational Evaluation |
| 1.10.5 | Audit Support | 1.10.17 | Performance Monitoring |
| 1.10.6 | Component Test Design Analysis | 1.10.18 | Project Management Oversight Support |
| 1.10.7 | Component Test Procedure Analysis | 1.10.19 | Security Assessment |
| 1.10.8 | Component Test Results Analysis | 1.10.20 | Software FQT Design Analysis |
| 1.10.9 | Configuration Management Assessment | 1.10.21 | Software Integration Test Design Analysis |
| 1.10.10 | Cost Analysis | 1.10.22 | System Test Design Analysis |
| 1.10.11 | Disaster Recovery Plan Assessment | 1.10.23 | Training Documentation Evaluation |
| 1.10.12 | Feasibility Study Evaluation | 1.10.24 | User Documentation Evaluation |

## IV&V Facility Relationship to HQ

- IV&V reports annual performance and receives approved budget from IBD (Chaired by Code Q)
- PCA describes Program Authority
- AA/OSMA delegates Program to GSFC Center Director
- IV&V Facility Director is Program Manager
- Facility works with Code Q IV&V Liaison to coordinate IBD budget inputs and performance reporting

## IV&V/Center/Project Relationships

- IV&V-Project Relationship:
- IV&V still reports issues to Project first and treats Project as primary “customer” for technical findings and risks
- As a Code Q Program, IV&V will keep Center S&MA personnel informed of IV&V technical issues so that S&MA has a complete mission assurance picture
- IV&V-Center Relationship:
- Center Liaison facilitates the startup of IV&V on new projects
- Center Liaison and IV&V Facility Leads facilitate technical issue resolution
- Center Liaison promotes consistent approaches to IV&V on Projects, and promotes awareness of IV&V Center-wide
- S&MA, Projects, and IV&V provide technical status and issues to the GPMC
- IV&V reports to GSFC PMC as a Program Office

## IV&V Reporting Approach

- Note – Other channels of communication can be instituted on a
- Center-by Center basis as needed or required

## Center-level IV&V Issue Escalation Process

- Notes:	1. Center S&MA organizations provide alternate/additional path for issue escalation
- 	2. On a Center-by-Center basis, other individuals and organizations may be brought into this process

## Summary & Next Steps

- Agency Concept for IV&V fulfills all requirements from IV&V Assessment
- Concept approval clears the way for full implementation
- Next steps are for each entity that owns the various processes to document those processes to the level of detail necessary for implementation
- Submit final Transition Plan and PCA for Approval

## Backup Material

## Ranking Process Worksheet with examples

**Engineering Note:**
- This is a complex and new approach to launching an existing spacecraft with crew which presents above nominal risk. However, these changes do not effect the pre-launch processing (e.g., component score of 0).

## Ranking Process Worksheet with examples (continued)

**Engineering Note:**
- Redundancy management and crew interface are assessed to be at a nominal level of risk and not a significant challenge to the project (0 score for component).

**Engineering Note:**
- The is a modification to a fighter airplane.  New software is going to be processing aero-surface controls in order to achieve better performance. OSMA has assessed the aero-surface control, critical systems control and critical systems monitoring subsystems as being safety critical (Scores shown in ***Bold Italic***).

## Ranking Process Worksheet with examples (continued)

## Ranking Process Worksheet with examples (continued)