# Presentation: 479674

## Controlled Software Management

- HNF-PRO-309
- Rev. 5

**Notes:**
Slide A 1, blank

Why modify procedure?
Upgrade to meet NQA-1-2000 requirements (HNF-MP-599)

Improvements:
Addressed DNFSB Recommendation regarding safety software
Addressed DOE Order 414.1C (Supplemental SQA requirements)
Added requirements for Software SME/Owner training
Upgraded to meet NQA-1-2000 requirements
Provided clear approach for Grading SQA requirements
Addressed site software security requirements
Clarified roles and responsibilities
Added centralized configuration management process

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

## Manage software lifecycle activities; including:
Identify Requirements, 
Control Configuration, 
Acquire & Evaluate, 
Design & Implement, 
Verify & Validate, 
Operate & Maintain, and 
Retire software 

- Manage software lifecycle activities; including:
- Identify Requirements,
- Control Configuration,
- Acquire & Evaluate,
- Design & Implement,
- Verify & Validate,
- Operate & Maintain, and
- Retire software

**Notes:**
	DNFSB Recommendation 2002-1, Quality Assurance for Safety-Related Software, dated September 23, 2002, determined that “An integrated and effective SQA infrastructure still does not exist within DOE”.

2002-1 identified several actions:
Identify software important to safety, including software used to analyze the consequences of potential accidents.
Identify responsible organizations for software management, including SQA, technical support, configuration management, training, error reporting and  stewardship functions.
Establish requirements for a rigorous SQA process, including specific guidance: 
grading requirements according to safety significance and complexity; 
performing safety reviews, including failure analysis and fault tolerance; 
performing verification and validation testing; and
performing training to ensure proficiency of users. 

## Software Management Method

- Systematic
- Disciplined
- Quantifiable

**Notes:**
NQA-1-2000, Subpart 2.7, Section 400 requires that a Software Engineering Method be established which ensures that software lifecycle activities are planned and performed in a traceable and orderly manner.  

NQA-1-2000 defines software engineering as “the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software”.

Transition – The need for a systematic, disciplined and quantifiable approach to software design and maintenance is clearly demonstrated by the following examples.

Slide A 2, Software Engineering 

## Therac 25
Patriot Missile System
Ariane 5

- Therac 25
- Patriot Missile System
- Ariane 5

**Notes:**
Slide A 3, blank

These are non-Hanford examples.  If you have any examples here, please let me know.

## Therac 25

**Notes:**
Has anyone heard of the Therac 25 Medical Electron Accelerator?

Between June 1985 and January 1987, 4 patients died because the Therac-25 medical linear accelerator software failed to detect an improper shield configuration, exposing them to a 25,000 Mev X-Ray beam. 
Since much of the software was reused from Therac-6/ Therac-20 systems, which had operated safety for many years, the analysts assumed there were no design problems in the software. 
However, unlike the Therac-25, the Therac 6 & 20 had independent protective circuits for monitoring electron-beam scanning, plus mechanical interlocks to ensure safe operation.  
Fatal flaws in translating new functionality into safe, reliable software were encountered that we will use as a method to demonstrate the concepts presented in this course.

Slide A 4, Therac 25 

## Patriot Missile System

**Notes:**
On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks and killed 28 soldiers and injured 100.  The cause of the system failure was an inaccurate software timing calculation.  Although the timing error was a mere 0.34 (9.5 x E-08 sec error over 100 hours) seconds, a Scud travels at about 1,676 meters per second.  In .34 seconds, the SCUD travels 569 meters or over 1700 feet!  This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. 

Even worse than the design error, the timing error had been identified two weeks earlier!  However, the Patriot Project Office failed to respond to the error correctly.

The army immediately sent out a warning that “very long run times could affect targeting accuracy.”  However, the phrase "very long run times" was not specifically defined.  As a result, the Patriot operators did not know they were operating under dangerous conditions when the SCUD attack occurred.

The Patriot system was originally designed to operate in Europe against Soviet medium- to high-altitude aircraft and cruise missiles traveling at speeds up to about MACH 2 (1500 mph). To avoid detection it was designed to be mobile and only operate for a few hours at a time.  Under such conditions, the timing error would not have adversely affected performance, but under the conditions in the Gulf War, the missile battery had been operational for over 100 hours, resulting in the timing error being multiplied over a much greater period.

Slide A 5, Patriot Missile System 

## Ariane 5

**Notes:**
It took the European Space Agency 10 years and $7 billion to produce Ariane 5, a giant rocket capable of hurling multiple satellites into orbit with each launch.  All it took for mission failure was a small computer program trying to stuff a 64-bit number into a 16-bit space.
At 39 seconds after launch in June 1996, as the rocket reached an altitude of two and a half miles, the rocket made an abrupt course correction that was not needed, compensating for a wrong turn that had not taken place.  The flight control computer mistakenly thought the rocket needed a course change because of data coming from the inertial guidance system. 
The flight control error was caused by a failure of the guidance system that shut down due to an overflow error.  The shutdown occurred when the guidance system computer tried to convert the horizontal velocity of the rocket -- from a 64-bit format to a 16-bit format.  The number was too big for the assigned data register, and an overflow error resulted.  
Ordinarily, when a program converts data from one form to another, the conversions are protected by extra lines of code that watch for errors and recover gracefully.  But in this case, the programmers had decided that this particular velocity figure would never be large enough to cause trouble. After all, it never happened before. 
Unfortunately, Ariane 5 was a faster, more powerful rocket than Ariane 4.
Slide A 6, Ariane 5 
Transition:  Keeping in mind that software errors can be deadly serious, let’s turn back to the controls of HNF-PRO-309

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 7, blank

## Applies to all controlled software.
Controlled software includes nuclear, safety-related, high-risk and quality affecting software currently in use at the Hanford Site under the Project Hanford Management Contract (PHMC) 

- Applies to all controlled software.
- Controlled software includes nuclear, safety-related, high-risk and quality affecting software currently in use at the Hanford Site under the Project Hanford Management Contract (PHMC)

**Notes:**
Procedure applies to all controlled software as determined by the Controlled Software Determination Form.

The procedure applies to:
safety system software, 
safety analysis and design software, 
distributed control systems
quality affecting software used to design, characterize, or verify quality of analyses and work processes, including compliance with environmental permits and other regulatory requirements,
high-risk business applications that support site internet and server access, security, human resources, project management, payroll and accounting. 

## Exemptions

- Site licensed and metered software
- Firmware
- WIPP – Controlled by other procedures
- OCRWM – Controlled by other procedures
- Single-use spreadsheets

**Notes:**
Site licensed and metered software which are provided in accordance with Information Management (IRM) contract requirements and which reside under Chief Information Officer ownership. See site licensed software for a listing.
Licensed software that is purchased with, and integral to, the operation of measuring and test equipment that is not altered by the user organization, and where the functionality of the system is demonstrated through calibration over the operational range.
Software used for the Transuranic (TRU) Waste Management program that is controlled in accordance with WMP-400, Section 6.1.1.
Software used to support Spent Nuclear Fuel and Sludge Retrieval Projects that is controlled in accordance with the Office of Civilian Radioactive Waste Management (OCRWM) Quality Assurance Requirements Document (DOE/RW-0333P).
Spreadsheets that are wholly incorporated into technical reports, calc-notes or other documentation where the calculations, mathematical formulas, and input data can be exactly verified during the technical review of the report.  Such calculations are treated and considered to be manual calculations because the assumptions, formulas, inputs and outputs are verified as part of the calculation package technical review. 

Slide A 8, Exempt Spreadsheets 

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 9, blank

## Project/Facility/Function Manager 
Software SME 
Software Owner
Independent Technical Reviewer
Test Personnel
FH Chief Information Officer

- Project/Facility/Function Manager
- Software SME
- Software Owner
- Independent Technical Reviewer
- Test Personnel
- FH Chief Information Officer

**Notes:**
Project/Facility or Function Manager 
Appoint Software Subject Matter Expert (SME) to establish and maintain the software configuration management program at the company, project, function, or facility level. 
Software SME
Assign the responsibility for acquisition, or development of software to a Software Owner; responsibility for technical review to an Independent Technical Reviewer; and responsibility for testing to test personnel.
FH Chief Information Officer (CIO)
Review and approve appointment of Software SME.

Slide A 10, Roles & Responsibilities 

## Manage the Project 
Acquire
Evaluate Legacy/COTS
Accept for New Scope
Manage Requirements
Design

- Manage the Project
- Acquire
- Evaluate Legacy/COTS
- Accept for New Scope
- Manage Requirements
- Design

**Notes:**
Hand out procedures – refer to flowcharts in App. D - L

This is the outline that the presentation will follow

Section 5.1	Software Management Methodology – determine software category, assign resources, determine risk, and plan software acquisition and/or development.
Section 5.2	Software Acquisition – acquire software from a commercial supplier, government code center, or internet download (i.e. freeware/shareware).
Section 5.3	Legacy Evaluation - accept COTS and LEGACY software.
Section 5.4	Acceptance for New Usage/Scope - accept existing, approved software for new scope, usage or application.
Section 5.5	Software Requirements Management – define and manage software requirements to ensure that developed, modified or acquired software meets its intended use.
Section 5.6	Software Design – develop the software design consistent with the specified functional and performance requirements.

Continued on next slide

Slide A 11, Flowchart – Plan view 

## Implement
Test
Approve for Use
Install
Operate
Maintain

- Implement
- Test
- Approve for Use
- Install
- Operate
- Maintain

**Notes:**
Section 5.7	Software Implementation – implement the design in a computer program or code and establish appropriate user instructions.

Section 5.8	Software Testing – plan and perform testing to ensure the software fully meets the specified requirements.

Section 5.9	Approval for Use – review and approve a software product and document the formal release of the software for use.

Section 5.10	Software Installation – install software on operating computer platforms to ensure proper performance in the user environment.

Section 5.11	Software Operation– operate software in accordance with specified user limitations to ensure safe and reliable operation of the software.

Section 5.12	Software Maintenance – identify and report software problems, and take corrective action to modify software to resolve errors, or to add new enhancements to meet changing application requirements. 

## Software Project Management (5.1.2)

- Software Classification
- Risk Management
- Quality Planning

**Notes:**
Procedure clearly organized to support starting or entry/exit points in the procedure.
Step 5.1.2.1 Software Owner – Identifies need for a software application within the scope of this procedure.

Step 5.1.2.2 Software Owner - Determines the start or entry point. 
Operate – Approved for Use? Go to Section 5.11.
Install – Approved for Use but not installed? Go to Section 5.10.
Approve – 309 Rev. 4 software?   Go to Section 5.9.
Accept LEGACY – Legacy but Not approved for Use? Go to Section 5.3.
Accept for New Usage/Scope – Go to Section 5.4.
Modify – Need to modify software?  Go to Section 5.12.
Acquire or Develop – New Software?  Go to Step 5.1.3.

Slide A 12, Flowchart – Entry point

## Software Classification (Appendix A)

- Controlled Software Determination Checklist
- Level A – Safety Systems
- Level B – Safety-related analysis (SARP)
- Level C – “High-risk”
- Level D – “Medium-risk”
- Level E – “Low-risk”

**Notes:**
Software Classification Criteria:
Level A - Software used in high safety consequence applications where failure could have an adverse effect on safety class or safety significant systems, structures or components where failure could result in non-conservative safety analysis.
Level B – Software used in low safety consequence software applications where failure could cause a reduction in the degree of safety or defense-in-depth noncompliance with nuclear safety regulatory laws, environmental permits or regulations.
Level C – Software used to support general service applications or business practice where software failure or application error could result in the potential for high business risk.
Level D - Software used in general service applications, where software failure or application error could result in the potential for medium or low project risk. 
Level E – Software used to support general service applications that can be exactly verified by hand calculations.

Owner completes/SME approves/CIO approves
App A

Slide A 13, Appendix A

## Risk Management (5.1.3)

- Functional Requirements Document
- Controlled Software Determination Checklist
- Computer System Evaluation
- Acquisition/Development Strategy
- Alternatives Analysis

**Notes:**
Risk management involves the following activities
Defining the critical features, capabilities and interfaces.
Determining software category
Determining computer security requirements
Developing acquisition/development strategy
Performing Alternatives Analysis

Owner completes/SME approves
FRD
Owner completes/SME approves/CIO approves
System evaluation
Alternatives analysis

Slide A 14, Flowchart – Risk management 

## Quality Planning (5.1.4)

- A description of the software need
- Organizations responsible for software
- Configuration Management Methodology
- Supporting documentation to be prepared
- Required Software Reviews
- Programming standards, conventions, techniques or methodologies.

**Notes:**
A description of the overall nature and purpose of the software application.
Organizations responsible for performing the work and achieving software application quality, and their tasks and responsibilities.
Configuration management methodology.
Design basis for the software application.
Supporting documentation to be prepared and maintained.  Supporting documentation needs to be prepared in accordance with HNF-RD-8310, Document Control Program.
Required software application reviews.
Standards, conventions, techniques or methodologies that shall guide code development.

Continued on next slide

Slide A 15, Flowchart – Make or buy decision 

## Quality Planning – cont’d

- Error reporting/corrective action methods
- Preparation of Contingency and Disaster Recovery Plans (HNF-PRO-597)
- Data and Information Management Standards (HNF-PRO-656)
- Functional Security Requirements/ Application Development (HNF-PRO-311)

**Notes:**
Methods for error reporting and corrective action.
Identification of Disaster Recovery Plans as required by HNF-PRO-597, Preparing Contingency Plans and Disaster Recovery Plans.
Identification of data management methods as required by HNF-RD-12439, Data and Information Management Standards.
Software application controls that address the security systems for the protection of software and data as specified in HNF-PRO-311, Functional Security Requirements/Application Development. 

Owner completes/SME approves
SMP

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 16  - blank 

## Software Acquisition (5.2)

- Statement of Work (HNF-PRO-186)
- CIO Approval for Level A, B, or C software
- Purchase software (HNF-PRO-123)
- Virus scan files (HNF-PRO-592)
- License agreements

**Notes:**
Prepare Statement of Work for acquisition of software, software application, or software services, where required, in accordance with HNF-PRO-186, Statements of Work, and submit to the Software SME and Fluor Hanford Chief Information Officer for approval.   Ensure the Statement of Work flows down software quality assurance and configuration management requirements particularly in the area of services where software will be used.

Purchase software, software application, or software services in accordance with HNF-PRO-123, The Material Request/Purchase Requisition/Contract Requisition Process.  

Notify the FH CIO Office for a cyber security and new technology review and approval.
Upon approval, virus-scan any files or software downloaded from the Internet or received from any other outside source before opening it in accordance with HNF-PRO-592, Unclassified Computer Security Control Program Management Control Process.
Obtain disclaimers, copyrights or license agreements.

Continued on next slide

Slide A 17, Flowchart – Acquisition

## Acquisition – cont’d

- Establish primitive baseline
- Prepare Software Management Plan
- Manage software code media
- Archive completed code in Source Integrity/Visual Source Safe
- Manage documents (HNF-RD-8310)

**Notes:**
Slide A 18, Software Baseline

Establish primitive baseline.
Prepare Software Management Plan.
Manage the software code disks/CD-ROM in accordance with Software Management Plan (SMP) ensuring disk copies are stored in an area/facility other than where the application operates.
Archive the completed software code in either the Centralized Configuration Management System (i.e., Source Integrity EE or Visual Source Safe).
Prepare and manage the documents in accordance with HNF-RD-8310, Document Control Program.

Owner completes/SME approves
SMP
Owner completes/SME approves/CIO approves
SOW (A/B/C only)

## COTS/LEGACY Evaluation (5.3)

- Functional Requirements Document
- Software Evaluation Report
- intended use of software
- adequacy of software to support application
- adequacy of test plans/test reports
- adequacy of user documentation
- compensatory measures

**Notes:**
Slide A 19 Legacy/COTS Evaluation Phase

Evaluate the COTS or existing software baseline to determine the adequacy of the software design, test and user documentation to support operation and maintenance.

Slide A 20 Example of SER

Prepare a Software Evaluation Report (template - or equivalent) that:
Describes the intended use of the software, 
Describes the adequacy of the software to support user application requirements (reference Functional Requirements Document), 
Describes the adequacy of test plans and test cases required to validate the software for acceptability over the intended range of use, 
Describes the adequacy of user documentation, and
Describes compensatory measures to overcome inadequate or incomplete design, implementation and user documentation. 
Prepare documentation identified on the SEF, incorporating any required changes (updates/addenda).  If insufficient design information is available to prepare user documentation, develop compensatory measures. 

Owner completes/SME approves
FRD
Owner completes/ITR reviews/SME approves
SER

## Acceptance for New Scope (5.4)

- Functional Requirements Document
- Evaluate existing software baseline
- intended use of software
- adequacy of software to support application
- adequacy of SRS/SDD documentation
- adequacy of test plans/test reports
- adequacy of user documentation

**Notes:**
Slide A 21 – examples of new scope

Evaluate existing software baseline to determine if the current software design, test and user documentation supports operation and maintenance for the new scope or usage per risk and graded approach as outlined in Appendix C, Software Classification /Grading Criteria.  

Slide A 22 – process flow

If the software is not adequate to support user application requirements, modify software in accordance with Section 5.12.
If the SRS and/or SDD does not address the intended use, modify software documentation baseline in accordance with Section 5.12.
If the existing test plans, test cases and test reports are not adequate to validate the software for its intended use, perform acceptance testing in accordance with Section 5.8.

If the existing user documentation does not comply with the minimum requirements of Section 5.7.5, prepare User Documentation (template - or equivalent). 

Owner completes/SME approves
FRD

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 23, blank

## Requirements
Design
Implementation
Testing
Installation
Operation & Maintenance
Retirement

- Requirements
- Design
- Implementation
- Testing
- Installation
- Operation & Maintenance
- Retirement

**Notes:**
Introduction to “classic” lifecycle typical of a development effort

## Requirements Management (5.5)

- Software Requirements Specification
- Functionality
- Performance
- Design Constraints
- Attributes
- External Interfaces
- Software and Data Security Controls

**Notes:**
Slide A 24– Example of SRS

Prepare a Software Requirements Specification (SRS) (template) that defines the requirements the software must satisfy including the response of the software to anticipated classes of input data and the detail and information necessary to design the software.  
Include in the SRS requirements for:
Functionality - the functions the software is to perform. 
Performance - the time-related issues of software operation such as speed, recovery time, response time, etc.
Design constraints imposed on implementation phase activities - any elements that will restrict design options. 
Attributes - one-time-related issues of software operation such as portability, acceptance criteria, access control, maintainability, etc.
External Interfaces - interactions with people.
Software Controls that address the security systems for the protection of software and data as specified in HNF-PRO-311, Functional Security Requirements/Application Development.

Assign Independent Technical Reviewer to perform a SRS review and specify the method of documentation for the verification review.

Owner completes/ITR reviews/SME approves
SRS (A/B/C only)

## Identify critical characteristics for putting a payload into space.

- Identify critical characteristics for putting a payload into space.

**Notes:**
Slide A 25 – Critical Characteristics
Ariane 5:  The requirements document did not contain the Ariane 5 trajectory data as a functional requirement.

Slide A 26 – Critical Characteristics for Therac 25
Shield Configuration
Position 1 – X-ray is reflected off positioning mirror
Position 2 – X-ray is flattened and defined
Position 3 – electron beam scan

## Software Design (5.6) 

- Design Description
- Description of major components
- Technical Description
- Allowable Input/Output ranges
- Integration Points (HLAN)
- Data Model (Metadata)
- Hardware and Software Configuration

**Notes:**
Slide A 27, Flowchart – Design and implementation

Prepare a Design Description (template) for Level A, B, C and D software application that documents the overall structure (control and data flow) and the reduction of the overall structure into physical solutions (e.g., algorithms, equations, control logic, and data structures).  

Include in the Design Description the following information, as applicable:
A description of the major components of the software design as they relate to the software requirements, 
A technical description of the software application.
A description of the allowable or prescribed ranges for inputs and outputs,
List of Integration Points (e.g., HLAN interface),
Data Model (Metadata), and
Hardware and Software Configuration

Owner completes/ITR reviews/SME approves
SDD (A/B/C/D only)

## Technical Description

- Theoretical basis
- Mathematical model
- Control Flow
- Data Flow
- Control Logic
- Data Structures

**Notes:**
Slide A 28 Theoretical Basis – 
	FB-111 (fly by wire system “crashed” when  higher than anticipated G forces encountered) FB111 designed for high- speed (Mach 2.5), high altitude fighter bomber.  In Vietnam, missile technology had reached Mach 5  and 60,000 ft.  By the time the plane was deployed it could not outrun a Soviet missile.  As a result the US Air Force changed tactics,  Fly low to the ground and maneuver tight, high-G turns to avoid missile strikes.  Unfortunately, the software “crashed” when it encountered high-G turns in certain airframe configurations.  Because the FB111 was the first “fly by wire” platform, due to the high-altitude escape pod design, when the software “crashed” so did the aircraft.

Slide A 29, Mathematical model – numerical approximation of theory 
	(NDA software uses wrong numerical approximation)
Slide A 30 Control Flow – sequence and condition of function calls  
	(Therac25: Class 3 variable max. value 255 - Thus, on every 256th pass through “Set-Up Test” the variable overflows, and has a zero value. That means that on every 256th pass the code failed to test the shield configuration.)
Slide A 31 Data Flow – flow of information within and between subroutines 
	(NASA orbiter (Sept. 1999) mismatched units)
Slide A 32 Control Logic – 
	(Patriot  “non-detect” if radar return not in “range gate” )
Slide A 33 Data Structures – size and layout of data registers  
	Ariane 5 Guidance computer tried to convert a 64-bit format to a 16-bit format register. The programmers had decided that this particular velocity figure would never be large enough to cause trouble. Unluckily, Ariane 5 was a faster rocket than Ariane 4. 

## Independent Design Review

- Complete
- Verifiable
- Consistent
- Traceable to requirements

**Notes:**
Slide A 34, blank

Why did the Design Review for the Ariane 5 fail to detect the design error?  
The requirements document did not contain the Ariane 5 trajectory data as a functional requirement.

Polar Explorer – data units not specified
Patriot Missile System – time clock precision not specified
FB111 – Maneuvering High G Limits not specified

## Software Implementation (5.7) 

- Create source code
- Analyze and debug software
- Document code
- Code walkthrough (Level A/B/C only)
- Prepare user documentation

**Notes:**
Assign a programmer to create the source code based on the design.

Create the source code based on the design, and analyze and debug the code to identify and correct errors.

Document the code and user-generated parameters, scripts and metadata required to compile and generate the executable program.  This documentation may be provided as code listings or on electronic media.

Owner 
Assigns programmer

Programmer writes/ITR reviews
Source code

Create Source Code
Program developed on PDP-11, significant compromises made in code structure to simplify memory requirements.  No checks and balances to verify that input data was consistent.

Code Walkthrough
Program structure assumed primary sequence.  No one accounted for what would happen if logic was bypassed. If the machine produces a photon beam with the beam flattener not in position, a high output dose rate results.  

## User Documentation 

- Software name and version identifier
- Description of user interaction
- Description of required training
- Input/Output specifications
- Input/Output format

**Notes:**

Continued on next slide

## Description of system limitations
System configuration
Information for maintenance support
Description of user messages and how user can respond
Process for reporting problems

- Description of system limitations
- System configuration
- Information for maintenance support
- Description of user messages and how user can respond
- Process for reporting problems

**Notes:**
Slide A 35, flowchart

Adequate User Instructions is critical to safe operation.
The operator hit the one-key command "B" (for "beam on") to begin the treatment. After a moment, the machine shut down and the console displayed the message "Malfunction 54."  The operator did not have any other information available in the Therac-25 documentation to explain the meaning of Malfunction 54. The machine showed a substantial underdose on its dose monitor display because the ion chamber used to monitor the patient was saturated from excessive radiation. The operator was accustomed to the quirks of the machine, which would frequently stop or delay treatment.  In the past, the only consequences had been inconvenience. The operator immediately took the normal action when the machine merely paused, which was to hit the "P" key to proceed with the treatment. 

In actuality, but unknown to anyone at that time, the patient had received a massive overdose, concentrated in the center of the treatment area. After-the-fact simulations of the accident revealed possible doses of 16,500 to 25,000 rads in less than 1 second over an area of about 1 cm.

Owner writes/SME approves
User documentation

## Establish software baseline

- Manage software media
- Archive code in Source Integrity/Visual Source Safe
- Manage documentation as supporting documents
- Register baseline in Hanford Information System Inventory (HISI)

**Notes:**
Slide A 36, blank

Establish the software baseline by collecting existing software and supporting documentation (e.g., code disks/CD-ROM).   
Complete the following:
Manage the software code disks/CD-ROM in accordance with Software  Management Plan (SMP) ensuring disk copies are stored in an area/facility other than where the application operates.
Archive the completed software code in either the Centralized Configuration Management System (i.e., Source Integrity EE or Visual Source Safe) or CIO approved alternate archival system.
Prepare and manage the documents in accordance with HNF-RD-8310, Document Control Program.
Register/update the supporting documentation baseline in HISI.

Owner 
Establishes baseline

Identify Source Integrity/Visual Source Safe and HISI courses available

## Software Testing (5.8)

- Assign test personnel
- Prepare Test Plan
- Specify test methods
- Independent Technical Review
- Perform testing
- Prepare Acceptance Test Report

**Notes:**
Slide A 37, Flowchart - Test 

UNINTENDED EFFECT TESTING:  The $165M Mars Polar Lander mission failed due to inadequate testing. 
The three landing legs of the probe contain small microswitches which are triggered when the legs touch the surface. This signal commands the engines to cease firing. 
Post-accident tests have shown that when the legs are initially unfolded during the final descent, springs push them so hard that they "bounce" and trigger the microswitches by accident. As a result, the computer receives what it believes are indications of a successful touchdown, and it shuts off the engines. 
Since this false signal actually occurs high in the air, the engine shutdown automatically leads to a free fall and destructive high-speed impact. 
Ground testing prior to launch apparently never detected this because each of the tests was performed in isolation from other tests. One team verified that the legs unfolded properly. Another team verified that the microswitches functioned on landing.
APA 5th Edition. CSS561-Programming Concepts, Software Maintenance and Change Control: Derek Denaro & Samuel Manackaraj, September 23, 2003

Owner
Assigns test personnel

## Test Plan

- Test objectives and technical approach,
- Required ranges of input parameters,
- Requirements for testing logic branches,
- Requirements for hardware integration,
- Evaluation method(s) to be used,
- Sequence of required activities, and
- Acceptance criteria.

**Notes:**
Slide A 38, blank

Ariane 5:  It is possible to do ground testing by injecting simulated accelerometric signals in accordance with predicted flight parameters, while also using a turntable to simulate launcher angular movements. Had such a test been performed by the supplier or as part of the acceptance test, the failure mechanism would have been exposed.

Document test requirements by preparing a Test Plan (template - or equivalent). Include in the test plan the following, as applicable:

Test personnel write/ITR reviews/SME approves
Test Plan
Software identification (name and version)
Test requirements and technical approach,
Required ranges of input parameters,
Requirements for testing logic branches,
Requirements for hardware integration,
Requirements for unintended function test,
Evaluation method(s) to be used,
Sequence of required activities, and
Acceptance criteria for review of test results

## Test Methods 

- Hand calculations,
- Calculations using comparable proven problems,
- Comparison with confirmed published data,
- Comparisons with validated computer programs,
- Comparisons with experiments and tests, and
- Manual inspections of system performance.

**Notes:**
Review Test Plan for completeness, consistency, clarity and correctness.  Acceptable test methods may include:

        Comparison with hand calculations,
        Comparison with calculations using comparable proven problems,
        Comparison with information from confirmed published data,
        Comparisons with other validated computer programs,
        Comparisons with experiments and tests, and
        Manual inspections of system performance.

**Notes:**
Slide A 39 – case study
For the program specification – write a set of test cases that you feel would adequately test the program:

A program will read three integer values from the data input screen.  The three values represent the sides of the triangle.  The program will present a message stating whether the triangle is equilateral, isosceles or scalene.

An equilateral triangle has three equal sides.
An isosceles triangle has two equal sides.
A scalene triangle has three unequal sides.

Don’t reveal next slide until time’s been give to students to write test cases.
Slide A 40 – did you find all 14 test cases?

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 41, blank

## Acceptance Test Report 

- Demonstrate that test results address the specified test acceptance criteria
- Changes to methods, inputs, or  sequence
- Date of test
- Simulation models used, where applicable
- Hardware used and test configuration
- Test results and conclusions
- Requirements Traceability Matrix

**Notes:**
Slide A 42– test report example

Prepare an Acceptance Test Report (template - or equivalent) to document acceptable performance of software.  Include in the Test Report the following information:

Sufficient documentation to demonstrate acceptable performance.
Description of, or reference to, the applicable test plan, test cases, and/or test requirements and acceptance criteria.
Description of any changes in evaluation methods, inputs, or test sequence.
Date of test.
Simulation models used, where applicable.
Hardware used and its configuration during the test.
Test results and conclusions concerning acceptance and utilization of the software.
Requirements Traceability Matrix.
Identification and signature of test personnel. 

Test personnel prepare/ITR reviews/SME approves
ATR

## Register the software application in the Hanford Information System Inventory (HISI) by completing the required links to supporting documentation.
Software SME Approval for Level A/B/C/D Software
CIO Approval for Level A/B Software

- Register the software application in the Hanford Information System Inventory (HISI) by completing the required links to supporting documentation.
- Software SME Approval for Level A/B/C/D Software
- CIO Approval for Level A/B Software

**Notes:**

Slide A 43 – detailed process

Owner registers in HISI

SME approves
Level A/B/C/D

CIO approves
Level A/B

## NETWORK
Prepare PRRB Submission Package
Obtain PRRB approval
Perform installation
Prepare Software I&C Form
Archive completed software in Source Integrity/Visual Source Safe

- NETWORK
- Prepare PRRB Submission Package
- Obtain PRRB approval
- Perform installation
- Prepare Software I&C Form
- Archive completed software in Source Integrity/Visual Source Safe

- DESKTOP
- Perform installation
- Prepare Software I&C Form
- Archive completed software in Source Integrity/Visual Source Safe

**Notes:**
Slide A 44, blank

Determine HLAN interface –  Does this software directly interface with HLAN or control data transfers between HLAN servers? 

Standard software interfaces with the HLAN which are controlled through system software, such as network printing and file management, do not require Production Readiness Review Board (PRRB) review.

Owner prepares documents for PRRB

Test personnel install/SME approves 
ICF

## PRRB Submittal 

- PRRB Submission Form
- Implementation Plan

**Notes:**
NOTE:  The PRRB Submission Package includes the following:
PRRB Submission Form
Implementation Plan for applications with high consequence of failure or impact on HLAN users. 

## Software Operation (5.11)

- Manage Authorized User List
- Establish user access controls
- Document user qualification (Level A/B)
- Log usage for Level A/B Software
- Operate software
- Perform periodic operational testing (Level A)
- Software retirement

**Notes:**
Slide A 45– process flow
Users must be formally authorized and controlled
User qualification must be established
The use of A/B software must be formally documented

Periodic operational testing – 
Perform operational testing at specified intervals based on system reliability

Operational testing should also be considered when event driven:
  Changes in operating system
  Changes in hardware

Owner develops/SME approves
Authorized users list

Owner develops
Access controls
Provides training for Level A/B/C
Periodic testing

SME develops 
Retirement criteria

Owner requests/SME approves
Retirement

## Software Maintenance (5.12)

- User notification of required software change
- Software Owner resolves problem with Computer Technical Support
- Complete Problem Report to identify software problem and determine impact
- Report error in accordance with HNF-PRO-052
- Propose updates/changes by completing PR/CR
- SME approval of proposed software changes

**Notes:**
Slide A 46– process overview

ERROR REPORTING IS ESSENTIAL:  A 1986 GAO study found 99% of injuries caused by medical devices were not reported to the FDA. At that time, hospitals reported only about 51% of problems to the manufacturer. After the Therac-25 deaths, the FDA made a number of adjustments to its policies in an attempt to address the breakdowns in communication and product approval. In 1990, health- care facilities were required by law to report incidents to both the manufacturer and FDA. 

Continued on next slide

## Maintenance – cont’d

- Implement approved software change
- Define test cases required to verify minor change and to support regression testing
- Independent Technical Review
- Perform required testing
- Document testing on PR/CR or Test Report
- Review and approve PR/CR test documentation
- Update HISI when new or revised software products are approved for use

**Notes:**
Owner completes/SME approves
PR/CR

## Understand the consequence of inadequate software management control
Identify scope of procedure and approved exemptions
Specify Software Project Management activities
Determine process for software acceptance
Develop software test plan
Define Software Operation and Maintenance Requirements
Define Lifecycle phases and documentation requirements

- Understand the consequence of inadequate software management control
- Identify scope of procedure and approved exemptions
- Specify Software Project Management activities
- Determine process for software acceptance
- Develop software test plan
- Define Software Operation and Maintenance Requirements
- Define Lifecycle phases and documentation requirements

**Notes:**
Slide A 47, blank

## HNF-PRO-309 manages software lifecycle using a graded approach
Identify Requirements, 
Control Configuration, 
Acquire & Evaluate, 
Design & Implement, 
Verify & Validate, 
Operate & Maintain, and 
Retire software 

- HNF-PRO-309 manages software lifecycle using a graded approach
- Identify Requirements,
- Control Configuration,
- Acquire & Evaluate,
- Design & Implement,
- Verify & Validate,
- Operate & Maintain, and
- Retire software

**Notes:**
DNFSB Recommendation 2002-1, Quality Assurance for Safety-Related Software, dated September 23,2002, determined that “An integrated and effective SQA infrastructure still does not exist within DOE”.

2002-1 identified several actions:
Identify important to safety software including software used to analyze the consequences of potential accidents.
Identify responsible organizations for software management, including SQA, technical support, configuration management, training, error reporting and  stewardship functions.
Establish requirements for a rigorous SQA process, including specific guidance on the following: 
grading of requirements according to safety significance and complexity; 
performance of safety reviews, including failure analysis and fault tolerance; 
performance of verification and validation testing; and
Performance of training to ensure proficiency of users.