July 23, 2003

Mr. Ronald A. Milner, Chief Operating Officer

Office of Civilian Radioactive Waste Management

U.S. Department of Energy

1000 Independence Avenue, SW

Washington, DC 20585

Subject: U.S. NUCLEAR REGULATORY COMMISSION’S OBSERVATION AUDIT

REPORT NO. OAR–03–03,“OBSERVATION AUDIT OF BECHTEL SAIC

COMPANY LLC (BSC) ACTIVITIES FOR THE PERFORMANCE BASED AUDIT

OF SOFTWARE ACTIVITIES PERFORMED BY BSC IN LAS VEGAS, NEVADA,

AND AT TWO U. S. DEPARTMENT OF ENERGY LABORATORIES, AUDIT NO.

OQAP–BSC–03–07”

Dear Mr. Milner:

I am transmitting the U.S. Nuclear Regulatory Commission’s (NRC’s) Observation Audit

Report No. OAR–03–03. Staff from NRC’s Division of Waste Management observed the

U.S. Department of Energy (DOE), Office of Quality Assurance, software audit OQAP-BSC-03-

07 on June 2–13, 2003, at the Bechtel SAIC Company LLC (BSC), facility in Las Vegas,

Nevada, and at two DOE Laboratories. The objectives of this performance-based audit were to

assess: (1) software quality, including the implementation and effectiveness of the software lifecycle processes; (2) activities that are used to manage the acquisition, development,

qualification, and use of software supporting the Yucca Mountain Project license application;

and (3) BSC’s implementation of the Office of Civilian Radioactive Waste Management

(OCRWM) Quality Assurance Requirements and Description (QARD), DOE/RW-0333P,

Revision 13, regarding controlling software.

The audit team reviewed 28 areas regarding software life cycle elements plus the associated

technical and programmatic areas supporting them. The audit team rated each area reviewed

in three categories: 1) QARD requirements flow-down; 2) implementation of procedural

requirements; and 3) overall effectiveness of the area. The audit team weighted each of the

28 areas equally in determining the overall effectiveness of the software process.

The DOE audit team concluded that, overall, the software process was “Marginally Effective.” 

The audit team found six of the 28 areas audited to be “Not Effective,” five “Marginally

Effective,” and two “Indeterminate.” The audit team also initiated eight deficiency reports

regarding the software process.

The observers found the audit team used a thorough and comprehensive approach to 

performance-based auditing. However, the observation team concluded that the overall

effectiveness of the software process was “Indeterminate.” This was because the audit team

found the critical elements of Design, Implementation, and Testing of the software life cycle to

be “Not Effective” or “Indeterminate.” In addition, the software products reviewed, that have

completed the independent verification and validation process, represented a limited number of

the total number of products (approximately 28 percent) identified to date that are intended for
license application. Only two of the software products audited were Level A (complex software)

packages and the retest of legacy software (software developed before January 13, 2003) was

not yet fully implemented.

During the conduct of the audit, the observers initiated three Audit Observer Inquiries (AOIs),

OQAP-BSC-03-07-01, -02, and -03, regarding software accepted using wide tolerances, the

apparent use of a graded approach to qualify software, and the review of previous Deficiency

Reports and Correction Action Requests for adverse quality trends.

A written response to this letter and the enclosed report is not required. However, responses

are required for the AOIs 45 days from the date of issue. The staff will continue to interface

with OCRWM and follow the actions that BSC is taking to address the issues identified during

this audit and those that remain open from previous audit observations. If you have any

questions regarding this observation audit report, please contact Ted Carter of my staff at

301-415-6684.

Sincerely,

/RA/

Janet Schlueter, Chief

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

 and Safeguards

Enclosure: NRC Observation Audit Report No. OAR–03–03, 

“Observation Audit of Bechtel SAIC Company (BSC) Activities 

for the Performance-Based Audit of Software Activities

Conducted at the BSC Facilities in Las Vegas, Nevada, and at two Department

of Energy Laboratories Audit No. OQAP–BSC–03–07”

cc: See attached distribution list.
July 23, 2003

license application. Only two of the software products audited were Level A (complex software)

packages and the retest of legacy software (software developed before January 13, 2003) was

not yet fully implemented.

During the conduct of the audit, the observers initiated three Audit Observer Inquiries (AOIs),

OQAP-BSC-03-07-01, -02, and -03, regarding software accepted using wide tolerances, the

apparent use of a graded approach to qualify software, and the review of previous Deficiency

Reports and Correction Action Requests for adverse quality trends.

A written response to this letter and the enclosed report is not required. However, responses

are required for the AOIs 45 days from the date of issue. The staff will continue to interface

with OCRWM and follow the actions that BSC is taking to address the issues identified during

this audit and those that remain open from previous audit observations. If you have any

questions regarding this observation audit report, please contact Ted Carter of my staff at

301-415-6684.

Sincerely,

/RA/

Janet Schlueter, Chief

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

 and Safeguards

Enclosure: NRC Observation Audit Report No. OAR–03–03, 

“Observation Audit of Bechtel SAIC Company (BSC) Activities 

for the Performance-Based Audit of Software Activities

Conducted at the BSC Facilities in Las Vegas, Nevada, and at two

Department of Energy Laboratories Audit No. OQAP–BSC–03–07”

cc: See attached distribution list.

DISTRIBUTION:

FILE CENTER HLWB r/f DWM r/f ACNW PUBLIC OSRs

LSN CNWRA CWReamer JSchleuter KStablein LCampbell

MYoung ACampbell BSpitzberg KBrockman DDambly D. Esh

MEhnstrom DHiggs TCarter TMatula BMabrito B. Schlapper

RLatta TAhn JParrott SFlanders R. Brient

S:DWM\HLWB\THC\OAR-03-03-Software.wpd *See Previous Concurrence ML032040585

OFFICE HLWB Tech Ed HLWB HLWB HLWB

NAME T. Carter* E. Kraus* T. Matula K. Stablein J. Schlueter

DATE 7/18/03 7/3/03 7/21/03 7/22/03 7/23/03

OFFICIAL RECORD COPY
Letter to R. Milner from J. Schlueter dated ______July 23, 2003____

cc:

A. Kalt, Churchill County, NV M. Corradini, NWTRB

R. Massey, Churchill/Lander County, NV J. Treichel, Nuclear Waste Task Force

I. Navis, Clark County, NV K. Tilges, Shundahai Network

E. von Tiesenhausen, Clark County, NV M. Chu, DOE/Washington, D.C.

G. McCorkell, Esmeralda County, NV G. Runkle, DOE/Washington, D.C.

L. Fiorenzi, Eureka County, NV C. Einberg, DOE/Washington, D.C.

A. Johnson, Eureka County, NV S. Gomberg, DOE/Washington, D.C.

A. Remus, Inyo County, CA W. J. Arthur, III , DOE/ORD

M. Yarbro, Lander County, NV R. Dyer, DOE/ORD

D. Carriger, Lincoln County, NV J. Ziegler, DOE/ORD

M. Baughman, Lincoln County, NV A. Gil, DOE/ORD

L. Mathias, Mineral County, NV W. Boyle, DOE/ORD

L. Bradshaw, Nye County, NV D. Brown, DOE/OCRWM

D. Chavez, Nye County, NV S. Mellington, DOE/ORD

D. Hammermeister, Nye County, NV C. Hanlon, DOE/ORD

J. Larson, White Pine County, NV T. Gunter, DOE/ORD

J. Ray, NV Congressional Delegation S. Morris, DOE/ORD

B. J. Gerber, NV Congressional Delegation J. Mitchell, BSC

F. Roberson, NV Congressional Delegation M. Mason, BSC

T. Story, NV Congressional Delegation S. Cereghino, BSC

J. Reynoldson, NV Congressional Delegation N. Williams, BSC

L. Hunsaker, NV Congressional Delegation M. Voegele, BSC/SAIC

S. Joya, NV Congressional Delegation D. Beckman, BSC/B&A

K. Kirkeby, NV Congressional Delegation W. Briggs, Ross, Dixon & Bell

R. Loux, State of NV P. Johnson, Citizen Alert

S. Frishman, State of NV R. Holden, NCAI

S. Lynch, State of NV B. Helmer, Timbisha Shoshone Tribe

M. Paslov Thomas, Legislative Counsel Bureau R. Boland, Timbisha Shoshone Tribe
cc: (Continued)

R. Clark, EPA C. Meyers, Moapa Paiute Indian Tribe

R. Anderson, NEI R. Wilder, Fort Independence Indian Tribe

R. McCullum, NEI D. Vega, Bishop Paiute Indian Tribe

S. Kraft, NEI J. Egan, Egan & Associates, PLLC

J. Kessler, EPRI J. Leeds, Las Vegas Indian Center

D. Duncan, USGS R. M. Saulque, Benton Paiute Indian Tribe

R. Craig, USGS C. Bradley, Kaibab Band of Southern Paiutes

W. Booth, Engineering Svcs, LTD R. Joseph, Lone Pine Paiute-Shoshone Tribe

L. Lehman, T-Reg, Inc. L. Tom, Paiute Indian Tribes of Utah

S. Echols, ESQ. E. Smith, Chemehuevi Indian Tribe

M. Smurr, BNFL, Inc. V. McQueen, Sr., Ely Shoshone Tribe

J. Bococh, Big Pine Paiute Tribe of the

Owens Valley 

R. Quintero, Inter-Tribal Council of NV

 (Chairman, Walker River Paiute Tribe)

R. Mike, Duckwater Shoshone Tribe D. Eddy, Jr., Colorado River Indian Tribes

T. Kingham, GAO H. Jackson, Public Citizen

D. Feehan, GAO J. Wells, Western Shoshone National Council

E. Hiruo, Platts Nuclear Publications D. Crawford, Inter-Tribal Council of NV

G. Hernandez, Las Vegas Paiute Tribe I. Zabarte, Western Shoshone National Council

K. Finfrock, NV Congressional Delegation NRC On-Site Representatives

L. Tomero, NV Congressional Delegation
U.S. NUCLEAR REGULATORY COMMISSION 

OBSERVATION AUDIT REPORT NO. OAR–03–03, 

“OBSERVATION AUDIT OF BECHTEL SAIC COMPANY LLC (BSC) ACTIVITIES 

FOR THE PERFORMANCE BASED AUDIT OF SOFTWARE ACTIVITIES 

PERFORMED BY BSC IN LAS VEGAS, NEVADA, 

AND AT TWO U. S. DEPARTMENT OF ENERGY LABORATORIES, 

AUDIT NO. OQAP–BSC–03–07”

 /RA/ 7/18/03 /RA/ 7/18/03

Ted Carter James Firth

Projects and Engineering Section Performance Assessment Section 

High-Level Waste Branch Environmental and Performance

Division of Waste Management Assessment Branch

Division of Waste Management

Reviewed and Approved by:

 /RA/ 7/22/03

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
1

1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC), Division of Waste Management,

observed the U.S. Department of Energy’s (DOE’s) Office of Quality Assurance, performancebased audit, OQAP-BSC-03-07, on June 2-13, 2003, at the Bechtel SAIC Company LLC (BSC),

facility in Las Vegas, Nevada, and at two DOE laboratories. The objectives of this

performance-based audit were to assess: 1) software quality, including the implementation and

effectiveness of the software life-cycle processes; 2) activities that are used to manage the

acquisition, development, qualification, and use of software supporting the Yucca Mountain

Project license application (LA); and 3) BSC’s implementation of the Office of Civilian

Radioactive Waste Management’s (OCRWM’s) Quality Assurance Requirements and

Description (QARD), DOE/RW-0333P, Revision 13, regarding controlling software. The DOE

audit team assessed the critical process steps involved with the development, control, use, and

documentation of software that will be used in technical products that support the LA. The NRC

observers assessed the effectiveness of the audit team and the audit process in achieving the

audit objectives.

2.0 MANAGEMENT SUMMARY

The audit team’s goal was to assess the quality of software used to support the LA by verifying

implementation of appropriate sections of the QARD, and by reviewing software used in

completed technical products. 

The audit team’s approach involved a review of 28 areas of the software life cycle plus the

associated technical and programmatic areas supporting them. For each area reviewed, the

audit team evaluated the: 1) adequacy of QARD requirements flow-down; 2) implementation of

procedural requirements; and 3) overall effectiveness of the area. The audit team weighted

each of the 28 areas equally in determining the overall effectiveness of the software process. 

Within the 28 areas evaluated, the audit team identified eight potential deficiencies in the areas

of: 1) Technical Review Adequacy; 2) Software Classification; 3) Software Planning Activities;

4) Software Design; 5) Software Implementation; 6) Software Testing; 7) Operations and

Maintenance; and 8) Software Controls. The audit team also identified seven potential quality

observations in the areas of: 1)Testing (one observation); 2) Software Use (one observation);

and 3) Documentation (five observations). The audit team found six of the 28 areas audited to

be “Not Effective,” five “Marginally Effective”, and two “Indeterminate.” The audit team

concluded that, overall, the software process was “Marginally Effective.” 

The observers found that the audit team used a thorough and comprehensive approach to 

performance-based auditing and the observers agreed with the audit team’s findings and

observations in each of the 28 areas. However, the observers concluded that overall, the

software process was “Indeterminate” for two reasons. First, the audit team found the critical

areas of Software Design and Software Testing to be “Not Effective,” and the area of Software

Implementation to be “Indeterminate.” The observers could not agree with the audit team’s

conclusion that the software process was “Marginally Effective” while these three critical areas

were not found to be “Effective.” Second, the software products reviewed that have completed

the independent verification and validation process represented a limited number of the total

products identified to date (approximately 28 percent) that are intended for LA. Only two of the

software products audited were Level A (complex software) packages and the retesting of

legacy software (software developed before January 13, 2003) was not yet fully implemented.
2

3.0 PARTICIPANTS

DOE Audit Team Members

Marlin Horseman, Navarro Quality Services (NQS), Audit Team Leader

Sam Archuleta, NQS/Auditor

Harvey Dove, NQS/Technical Specialist

John Doyle, NQS/Auditor

Bruce Foster, NQS/Auditor

Christian Palay, NQS/Auditor

Sid Ailes, Duratec/Technical Specialist

Mario Chavez, John Hart Associates, Technical Specialist

Norm Moreau, Theseus Professional Services, Technical Specialist

NRC Observers

Ted Carter, NRC, Team Leader

James Firth, NRC, Technical Specialist

Rodney Weber, Center for Nuclear Waste Regulatory Analyses (CNWRA), QA Specialist

Mark Ehnstrom, CNWRA, QA Specialist

Randolph Folck, CNWRA, Software Specialist

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

The audit team performed the software audit by following procedure AP-18.3Q, Internal Audit

Program, and AP–16.1Q, “Management of Conditions Adverse to Quality. The audit team used

the QARD, Revision 13, and applicable QARD implementing procedures to generate the audit

checklist. The observers followed NRC Manual Chapter 2410, Conduct of Observation Audits,

July 12, 2000, while observing the audit.

4.1 Scope of the Audit

The scope of the audit included the evaluation of the 28 areas of the software life cycle

processes, as listed in Figure 1. The audit team evaluated additional programmatic control

elements associated with the software life cycle including documentation, records,

procurement, and personnel qualifications. The scope of the audit also included evaluating the

implementation of the following procedures regarding software:

 AP-SI.1Q, “Software Management”

 AP-SI.2Q, “Qualification of Level A Software”

 AP-SI.3Q, “Software Independent Verification and Validation”

 AP-SI.4Q, “Independent Verification and Validation of Legacy Code”

4.2 Conduct and Timing of the Audit

The audit team used a performance-based approach to evaluate the software process. The

audit team’s approach involved a review of 28 areas that included the basic software life cycle

elements plus the associated technical and programmatic areas supporting them. For each

area reviewed, the audit team determined the adequacy of procedures, satisfactory procedure

implementation, and the effectiveness of the software area. The audit team generated audit

checklist questions based on the process steps needed to reach the objective of each area. 

The observers found this be a thorough and comprehensive approach to a performance-based
3

auditing. DOE has not yet defined this detailed approach, but the observers suggested that it

be proceduralized.

The audit team and observers caucused at the end of each day to discuss the audit status and

any new and developing issues. The audit team met with BSC management each morning, as

appropriate, to discuss the current audit status and potential issues. The observers attended

these meetings to determine the audit team’s effectiveness in communicating issues to

interested project personnel.

The Observation Team Leader discussed the timing of audits with the Audit Team Leader and

the DOE Director, Office of Quality Assurance. It was determined that, although the sample

was taken from a limited number of products expected to be associated with the LA, some

value could be derived from a review of the products available at this time. As noted above,

additional audit or surveillance activities are recommended.

4.3 Audit Team Qualification and Independence

The observers reviewed the qualifications for the Audit Team Leader and all of the auditors and

determined that they were qualified and independent of the areas reviewed.

The audit team evaluated the 28 areas listed in Figure 1. These areas represent the software

life cycle areas plus their support technical and programmatic areas. The details of audit

activities in each area are discussed below, and the results, as determined by the auditors, are

summarized in Figure 1. 

4.4.1 General Software Qualification and Administration

The auditors reviewed documentation of training and experience for software developers,

software testers and/or software configuration management and related personnel. Documents

including transcripts, Training Requirement Matrixes, education and experience for a sample of

three independent verification and validation group members were also reviewed. The audit

team found that all documentation reviewed was acceptable.

The observers agreed with the audit team’s finding in this area. 

4.4.2 Verification and Validation

The auditors reviewed independent Verification and Validation (V&V) for STRELTSOVAADAMS.VI, Version 1.0. The audit team found that the V&V report was complete, concise, and

of sufficient detail. Comment resolution is documented in e-mails. The use of e-mails is

somewhat confusing, but was found to be adequate in the observed cases.

The procedure for independent V&V, AP-SI.3Q, provides definitions of the evaluation tasks to

be performed by independent V&V personnel. These definitions imply that independent V&V

personnel will review, in addition to the existence of documentation, the “goodness” of the

documentation and code. The auditors found a conflict between requirements contained in

procedure AP-SI.2Q, Paragraph 5.22, for the development of the validation test and those

contained in procedure AP-SI.3Q, Paragraph 3.17, for the validation test process evaluation.
4

The audit team noted that independent V&V for Level B code excludes the software design

evaluation as defined in procedure AP-SI.3Q, paragraph 3.12. For Level B software, a design

description is contained in the Software Management Report, as required by procedure APSI.1Q, paragraph 5.3.1.b.2. The audit team found that design is part of the Level B code

process and should be included in the independent V&V process. 

The observers agreed with the audit team’s findings in this area.

4.4.3 Algorithms

The auditors reviewed one Level A and seven Level B legacy codes. The audit team found

documentation pertaining to software algorithms to be limited. The documentation that is

available depends on whether the software is developed or acquired. For acquired codes, a

design document is not provided, and therefore the technical adequacy of the software, based

on a knowledge of the algorithm, cannot be determined. The auditors determined that this is

significant information that is needed for the technical review process and therefore this area

was evaluated as “not effective” for acquired codes. The auditors recommended that

information regarding the algorithms used be included in the User’s Manual.

The auditors found that software (e.g. GOLDSIM) is being treated as acquired software and is

reacquired and re-qualified when revisions are made by the supplier, even when DOE requests

and funds changes that may or may not have widespread use. When procuring software from

suppliers in the past, changes made and the evolutionary history were not captured. However,

changes are now being tracked within the software change process.

The observers agreed with the audit team’s findings in this area.

4.4.4 Technical Adequacy/Review

The audit team found that the technical review of documents to be inadequate. Evidence of

this was noted in a number of areas, including:

1. Objective evidence of reviews was inadequate for Requirements and Design Documents on

file for SZ_CONVOLUTE, V 3.0.

2. The corrective action resulting from Deficiency Report BSC(O)-02-D-0099 inadequately

addressed the technical reviews of software documentation. The team noted that

procedure AP-SI.2Q, Rev 0, ICN 0, does not adequately address review criteria before

performing document reviews, as required by the QARD. In addition, the level of specificity

is not sufficient to provide adequate review criteria and it is uncertain that the software

coordinators could adequately perform this review from a technical perspective.

3. The qualifications for the Sandia National Laboratory software coordinator were questioned. 

Questions arose related to whether the requirements for reviews to be performed by

technically competent individuals, using review criteria that consider technical adequacy,

had been met.

The audit team addressed these issues in three Deficiency Reports.

The observers agreed with the audit team’s findings in this area.
5

4.4.5 Procedures

The audit team found that the reviews conducted under the Software Management procedure

AP-SI.1Q were actually verification reviews by the software coordinator and do not replace

technical reviews. The audit team found that technical reviews are not addressed in procedure

AP-SI.1Q or the following related software management procedures: AP-SI.2Q, “Qualification of

Level A Software”; AP-SI.3Q, “Software Independent Verification and Validation”; or AP-SI.4Q,

“Independent Verification and Validation of Legacy Code.”

The audit team identified errors in numerous documents reviewed during the audit, including

errors such as 22 missing references in one document and Greek characters absent from

equations throughout another document. The errors found by the auditors in the

documentation are relevant to assessing the effectiveness of the technical reviews. In addition,

the auditors found examples of other errors in the documents (e.g., equations that are not

dimensionally correct, missing symbols, incorrect parameter values, and text errors) that would

not be expected, if thorough technical reviews had been conducted.

The observers agreed with the audit team’s findings in this area.

4.4.6 Classification

The audit team evaluated whether the process for categorization of software was leading to the

correct categorization of software and whether the appropriate documentation was identified. 

Using earlier procedures, some software was categorized as Level 1, Level 2, or Level 3. This

earlier categorization considered the importance of the software and the level of effort identified

in the Software Activity Plan, so it was possible to have critical software identified as Level 2

software, which would correspond approximately to Level B software under the current

procedures. The audit team concluded that the categorization was not effective in correctly

categorizing software. 

The observers agreed with the audit team’s findings in this area.

NOTE: The observers submitted an Audit Observer Inquiry (AOI), which addressed the

categorization process and the subsequent software management and qualification procedures. 

This AOI addressed the potential for the categorization approach to lead to grading the QA

requirements for software (i.e., greater QA being applied to Level A software than to Level B

software). This is contrary to DOE’s position, as stated by DOE on April 29, 2003, during the

Quarterly QA Meeting, that an OCRWM QA grading process will not be implemented.

4.4.7 Activity Plan

The observers did not observe audit activities in this area.

4.4.8 Life Cycle

Procedure AP-SI.2Q, “Qualification of Level A Software,” establishes the responsibilities and

processes for those activities that constitute the Level A software qualification process, provides

a detailed account of the administrative procedures, and provides an outline for the content of

various documents (i.e., Design Document (DD); Requirement Document (RD), etc.). The

procedure does not provide details on how to accomplish the various tasks described, but
6

rather their requirements. For example, paragraph 5.2.1.1 states that the design specifies the

data structures, processes, interfaces, and procedures to the level of detail necessary to plan

and execute the implementation, validation, and installation of the software project. There is

little or no guidance on what methods or techniques can be used to accomplish this task. 

Again, paragraph 5.2.2.1 states that the developer is to develop requirement-based test cases,

but does not describe how to develop these test cases, (e.g. boundary test, equivalence

partitioning, etc). As a result of these weaknesses, the audit team found implementation and

effectiveness to be marginal in this area.

The observers agreed with the audit team’s findings in this area.

4.4.9 Requirements

The audit team found that the range of applicability is not always indicated clearly in the

Requirements Documentation (RD). The auditors found subjective performance requirements

in the Software Management Report for RADPRO, V3.22. Similarly, the applicable operating

systems cannot always be traced through the software management process, including the

requirements phase, design phase, and testing phase. For example, the RD for NUFT, V3.0, is

not sufficiently detailed to understand the functional requirements. The auditors also noted that

the Validation Test Plan and Validation Test Report provided more functional details. The audit

team made two recommendations related to these subjects.

The observers agreed with the audit team’s findings in this area.

4.4.10 Design

The auditors found that the number of DDs available for review was limited. DDs were

available for only one Level A software program that was developed. Design documents are

not required for Level B software nor for acquired Level A software. The approach of

requesting changes from the vendor (e.g., Golder and Associates, Inc.) and re-qualifying the

resulting software avoided the procedural requirement for creating a DD.

The audit team reviewed a total of 12 software packages. The auditors also found that the

design baseline for principal process codes is not consistently documented. FEHM provides

module listings only, and no design documentation is available for NUFT and TOUGH2, V 1.4. 

The design baseline is split between four different packages, one of which was not available in

the project files. 

The audit team also found that the Software Management Reports for software packages

CWD, STRETSLOVA-ADAMS, and PREINFIL do not describe major components of the

software design in sufficient detail to perform the coding. Control flow, data flow, and control

logic are not adequately addressed. In addition, technical reviews of software design for FEHM

and INVIEW do not assure that relationships between design elements and system

requirements are specified to a consistent level and detail for Level B and Legacy codes. 

The audit team issued three Deficiency Reports to address these findings and made one

procedural recommendation. 

The observers agreed with the audit findings in this area.
7

4.4.11 Implementation

The audit team found that the design is not consistently documented in sufficient detail to

translate into code. The DDs (i.e., FEHM, Version 2.20) are quite detailed in describing the

technical description of the software components, but do not fully address data flow, control,

and control logic, thus making translation of the design into code difficult. Other codes

identified by the audit team with similar problems include PREINFIL and STRETSLOVAADAMS.

The Implementation phase follows the design phase and is the translation of the

requirements/design into computer code. The computer code is to adhere closely to the

documented design specification. The goal of implementation is to structure the code, define

variables and files, and provide comments in such a way so the implementation of the

requirements/design can be verified and the code can be readily maintained.

Codes reviewed by the audit team and found to be well-structured, defined, commented, and

traceable to the requirements/design included CWD, Version 2.0, and PRENINFIL,

Version 1.20, both Level B codes. However, that auditors reviewed SZ_Convolute Version 3.0,

a Level A code, and found it to contain insufficient comments with variables not fully defined,

making it difficult to verify implementation of requirements/design. The auditors found another

code, FLOW-CON, Version 1.0, a Level B code, to be heavily commented, but with a fair

amount of commented-out code, which may cause problems with future maintenance. The

auditors found discrepancies and inconsistent styles in implementation that may be attributed to

a lack of defined coding conventions or their use.

The audit team issued one Deficiency Report to address the deficiencies found in flow-down of

QARD requirements to procedures AP-SI.1Q and AP-SI.2Q. Internal coding convention,

techniques, and coding protocols are not addressed in the Deficiency Report. 

The observers agreed with the audit team’s findings in this area.
8

4.4.12 Testing

Explicit listings of the functions being validated and those not being validated (for general use

software) are provided in the validation reports. This allows the users to easily ascertain the

extent of validation rather than having to review multiple documents. The audit team evaluated

this as part of its review. The auditors observed retesting of test case number seven, for the

validation of FLAC3D, Version 2.0. The test accurately duplicated the test from the validation

test plan. The features or functional requirements tested were traced to the applicable

requirements document. In some instances, a limited set of vendor tests was repeated, and

other vendor tests may have been relied on for the validation. The auditors found that It is not

clear whether, in other cases, the tests used to validate the software encompass the full range

of its use, or that the limits within which the software has been validated (i.e., the operational

range) are clearly identified. For example, a limited set of tests were run for ANSYS, Version

5.6.2, and event elements were not tested for GOLDSIM.

The audits found that software some developers do not fully understand how much testing is

required and how much testing to document. This led to instances where there was insufficient

documentation of the tests to clearly indicate the range of acceptable performance for the

software. For example, the auditors found that the design for CWD, Version 1.0, a Level B

code, lists six equation arguments to calculate the Poisson intensity parameter, with an input

range for a thickness of 6.34 millimeters (MM) to 63.5 mm. Only one test case was

documented for 10.00 mm. Testing for CWD, Version 2.0, also included only one test case for

25.0 mm. Boundary conditions as well as invalid input values are not documented in test

cases. The range of validation for CWD, Version 2.0, and for another Level B code, StreltsovaAdams, Version 1.0, is not clearly stated. 

The Validation Test Plan for NUFT, Version 3.05, identified a test where a parameter for water

characteristics was specified. The auditors found that testing was performed using the

intended case, where the parameter specified characteristics for air, not water. This test case

was documented in the Validation Test Report and no reconciliation with the documented

Validation Test Plan. The auditors also found that, for the code BMRK 014, there was

insufficient detail in the Validation Test Plan. Inadequate detail was provided in the test cases

to allow the audit team to evaluate the test results without consulting the developer, and an

inaccurate and incomplete set of parameters for the tests was provided in the Validation Test

Plan. The auditors found that the validation testing for ANSYS, Version 5.6.2, included three

different approaches for solving a test problem, where the stress concentration was being

calculated at a hole in a plate. The three approaches led to different results and different

degrees of accuracy. A note was made in the Validation Test Report on the best approach to

use on this type of problem, but it is not clear whether this information will be considered by the

users of the software. 

The auditors determined that test documentation did not always define the full operational

range of software, include quantitative acceptance criteria, define the hardware configuration

used during testing, demonstrate that specified requirements were tested, and satisfactorily

resolve unsuccessful test results. 

The audit team wrote one Deficiency Report for the area of Software Testing and one quality

observation in this area. 

The observers agreed with the audit team’s findings in this area. 
9

The observers submitted an AOI that addresses the tolerances allowed in the validation testing,

when a determination is made about whether the software will, or will not, adequately and

correctly perform its intended functions. The observation team noted that numeric criteria

(tolerances) for the tests were not identified in the Validation Test Report (i.e., in advance of

performing the tests). The observers identified substantial variability in the tolerances identified

as being acceptable (such as ± 10 percent for GoldSim; ± 1 percent for Ashplume, Version

1.4LV, and rounding errors for Infil2Grid, Version 1.6). In some instances, the validation tests

re-run analyses provided by the vendor and the comparison is made to the numbers provided

by the vendor. There is no objective evidence that the vendor numbers have been checked or

reviewed. Consequently, the comparison is being made to unqualified analyses and numbers. 

In one instance, Golder and Associates, Inc. had submitted a revised Verification Test Report,

when an earlier version had been found to be in error. The observers noted that, for highly

flexible software, the eventual use of the software may not be within the range used for V&V

testing. The observation team recommended that this be reviewed during any future audits

addressing the resulting models or analyses that rely on highly flexible software, such as

ANSYS and GOLDSIM.

4.4.13 Operations and Maintenance

The audit team reviewed the use of codes from time of release for installation through recall for

replacement (revisions) or retirement. The audit team reviewed ten code packages for:

1) Defect reporting; 2) tracking; 3) proper application of code; 4) release, replacement, and

retirement of revised code; 5) platform testing; and 6) Documentation of tracking, user

requests, testing, and problem reporting.

The auditors found a failure to meet the requirement to submit copies of in-use test results for

TCO, TRW, DCS, and HDAS2, V 2.11, per AP-SI.1Q, Rev 3, ICN 4. Software Configuration

Management had not received copies. 

The audit team issued one Deficiency Report in this area and made one recommendation

regarding the confirmation of code integrity in comparison to baseline for codes used in the

field. 

The observers agreed with the audit team’s findings in this area.

4.4.14 Installation and Check out

The auditors verified installation and checkout of code for procedural compliance. The auditors

verified a total of six Installation Test Processes (ITPs) in Las Vegas and at Lawrence Berkeley

National Laboratory (LBNL). The auditors identified that generic operation system requirements

are referenced in the ITPs (e.g Windows, ULTRIX, and LINUX). The installation process would

be jeopardized by not fully describing the actual operating system required (e.g. Windows 98,

Windows NT, and/or UNIX and LINUX).

The auditors found that after the software leaves Software Configuration Management, there is

a disruption in the formal installation and checkout controls for that software. The Software

Users List does not always correctly identify the users of the software. The user list and

operating environment were correct in only one out of five dynamically linked libraries. These

findings highlighted a fundamental need for Software Configuration Management to become
10

more proactive in areas such as user lists, accountability, and location of software after it leaves

the Software Configuration Management.

The observers agreed with the audit team’s findings in this area.

4.4.15 Retirement

The auditors reviewed the processes for identifying, tracking, and controlling software to be

retired. The auditors reviewed procedures regarding removal from use, historical

documentation, and prevention of further use. In one instance, the software user was no longer

required to use the software program CONVERT COORDS V 1.1. The auditors recommended

that this program be retired, to prevent unintended use. The auditee answered all questions

satisfactorily, and no other observations were documented. 

The observers agreed with the audit team’s recommendation in this area.

4.4.16 Software Controls, Baseline Changes, and Configuration Management

The auditors evaluated controls placed on software. It was found that the Software

Configuration Management group issued and maintained approved software although some

inconsistencies were discovered. The auditors found that a status accounting of users and

locations is not current. Individual users and locations identified on the Software Users

Request form may not be accurate. The auditors wrote a Deficiency Report to address this

deficiency.

The auditors found that a file copy of a Software Configuration Requests for a baseline addition

at the LBNL was different and more complete than the project copy contained by Software

Configuration Management in Las Vegas. Documentation at Berkeley showed that required

actions and information had been completed and had been sent to the Software Configuration

Management Group. The audit team wrote a Deficiency Report to address this deficiency. 

The audit team also reviewed changes to existing software and found that changes to

GOLDSIM are being incorporated into new revisions. The new software revision can then be

purchased as “acquired code.” Purchasing new versions of acquired code eliminates the need

for a DD to be developed, which precludes the Software Configuration Management system

from capturing the total number of software changes. 

In addition to the two Deficiency Reports noted above, the audit team made three

recommendations: 1) Provide better coordination of identified requirements in the RD; 2) list all

changes for aquired software on a Software Configuration Control Request; and 3) correct the

CPU reference as listed in one code.

The observers agreed with the audit team’s findings in this area. 
11

4.4.17 Use

The audit team referred to three Analysis Model Reports (AMRs) to obtain references to

53 codes, from which 10 were selected for review. Each of these was reviewed for: 

1) Traceability to user requests; 2) control of installation; 3) acceptance criteria meets use

requirements; and 4) documentation of ready-for-use in Software Configuration Management

packages

The audit team made one quality observation in this area. The auditors found that the

operating system used to run DICTRA, V 2.0, and THERMA CALC, VM, was not the same as

the operating system noted in the qualified baseline. The auditors did not note any other

findings or recommendations in this area.

The observers agreed with the audit team’s findings in this area.

4.4.18 Error Reporting

If a code, a software document, or a data structure fails to meet its specified requirement(s),

this is supposed to be documented on a Software Problem Report (SPR), in accordance with

procedure AP-SI.1Q. Upon initiation of an SPR, the affected software is to be removed from

the baseline until an impact evaluation is performed and documented, including any preventive

and corrective actions. The impact analysis is to be conducted in accordance with procedures

AP-2.14Q, “Review of Technical Products and Data,” and AP-16.1Q, “Management of

Conditions Adverse to Quality.” The auditors found that, in some cases, the impact analysis

was not fully documented.

Impact analyses for the errors corrected with Service Pack 2, for GOLDSIM Version 7.51, were

available for review. This Service Pack addressed a number of changes to address problems

identified with the code, including some changes, to the display and output of results, and some

enhancements. The problems were categorized as: (1) No impact on dose; (2) no impact on

Total System Performance Assessment (TSPA); (3) TSPA has not encountered this error; or

(4) a potential impact exists and needs to be further evaluated with a model comparison. The

auditors could not determine whether objective evidence exists to indicate that an attempt was

made to reproduce the problem to determine if it was an actual software problem and not userinduced (per Section 5.10 of AP-SI.1Q, Revision 5, ICN 0), or whether the GOLDSIM was

removed from the baseline. Errors had been found in earlier versions of GOLDSIM, some of

which were broadcast to users by Golder and Associates, Inc. It could not be determined

whether these errors resulted in SPRs, Impact Assessments, or removal of the software from

the baseline while the impact analysis was being performed. Other impact analyses, such as

those from earlier versions of GOLDSIM, were not available for review. The auditors found that

the impact analysis process under procedure AP-SI.1Q, Revision 4 or earlier, did not require

that a detailed impact analysis be documented. The results for such an analysis were all that

was required.

The audit team made four recommendations in this area:

1) Review a previously issued Deficiency Report regarding timeliness of SDNs, to verify that

the corrective action will address a timeliness issue identified in this review.

2) Perform trending on SDN and Problem Reports.
12

3) Review SDN documentation generated prior to AP-SI.1Q, Rev 5, to assess the defensibility

of the impact analysis.

4) Provide direction to the user organization regarding time requirements for submittal of SDNs

and impact analyses documentation.

The observers agreed with the audit team’s findings in this area.

4.4.19 Traceability in Technical Documents

The auditors found references to software while reviewing AMRs. These were checked in the

Software Configuration Management files to verify the existence of appropriate user request

forms. The codes were also verified to exist in the software baseline. The document

information reference system was also checked to assure the appropriate references were in

the data base. The auditors found the traceability of technical documents to be effective.

The observers agreed with the audit team’s findings in this area.

4.4.20 Acquired Software

The auditors did not identify any instances where quality-affecting software (acquired Level A)

is purchased without going through a test process. Since the software was purchased as nonquality-affecting, there is no DD. The acquired software goes through a requirements

description review and an independent verification before Control Point B, by the Software

Custodian, before use. The auditors found that the process to acquire software was

acceptable. The auditors were concerned that perhaps not all requested changes to the

acquired code were being captured as quality records but did not find any evidence of missing

records. 

The observers agreed with the audit team’s findings in this area.

4.4.21 Participant Software

The audit team determined that Participant Software and Acquired Software were the same.

See Paragraph 4.5.20 for additional information.

4.4.22 Procurement

The auditors found that software was procured by the issuance of Purchase Requisitions. The

Purchase Requisitions reviewed were identified as non-quality purchases. The use of qualified

vendors was also not an issue, since acquired code was benchmarked by reviewing the

Requirements Description and performing tests on the new software as part of the software

receipt process. As discussed earlier, changes to acquired software were incorporated into

new versions of that software and purchased. The auditors found the procurement process to

be acceptable.

The observers agreed with the audit team’s findings in this area.

4.4.23 Spreadsheets
13

The observers did not observe audit activities in this area.

4.4.24 Routines and Macros

There were no audit activities in this area. The audit team deleted this item from the original

audit plan. 

4.4.25 Management Tools

The observers did not observe audit activities in this area.

4.4.26 Documentation

During the review of the DD for ASHPLUME, the auditors found 22 references in the body of

the document, but not in the References section, as required by AP-SI.1Q, Rev. 3, ICN 4, in

effect at the time that the DD was created. The auditors also reviewed ANSYS and found

unclear pen and ink changes, and changes that were made but not signed. The audit team

issued quality observations for these findings.

The audit team also found several other deficiencies related to documentation. Descriptions of

algorithms for CWD, V 2.0, were found incomplete and several parameters (e.g. Poisson

Intensity) were missing in derivations. FEHM, V 2.2, “Design Document,” had an incorrect

document identifier. The audit team issued quality observations for these findings. 

The auditors found missing files identified as not being on the CD for SZ_Convolute

Version 2.3. This problem was turned over to Software Configuration Management team for

resolution.

The observers agreed with the audit team’s findings in this area.

4.4.27 Lab Activities

The audit continued at LBNL, and Lawrence Livermore National Laboratory (LLNL). The

auditors found that personnel responsible for controlling software at both laboratories were

familiar with procedural requirements. Software Configuration Management activities

appeared to be adequate. The auditors highlighted one “best practice” during the audit at

LLNL. It consisted of a software matrix that listed the software in use and the information

specific to that piece of software. The auditors found no problems or inconsistencies specific to

any of the laboratories during the audit. 

The observers agreed with the audit team’s findings in this area.

4.4.28 Legacy Software

In December 2002 DOE made a commitment, to NRC, to retest legacy codes that were used in

technical products supporting LA (Letter from William D. Ziegler, DOE, to Janet R. Schlueter,

NRC, dated December 23, 2002). Procedure AP-SI-4Q was effective May 15, 2003, and

describes the process for the Independent V&V of legacy codes. Paragraph 5.0 of the

procedure states that this independent V&V process will “...provide a confidence level that
14

legacy codes can fulfill their intended use.” However, the auditors found that the process

described in procedure AP-SI-4Q only provides confidence that the existing test cases can be

repeated without recourse to the author/developer. There appears to be no review of the

“goodness” of the applicable test cases, just a retesting exercise. The auditors found that

testing of legacy codes in accordance with the new procedure AP-SI.4Q had not been

implemented and that a follow-up surveillance or audit activity would be required to verify the

implementation of this process. 

The observers agreed with the audit team’s findings in this area.

4.4.29 Conditions Adverse to Quality

The auditors reviewed previously generated Corrective Action Requests and Deficiency Reports

to evaluate the effectiveness of corrective actions. The auditors found that the corrective

actions for four of 16 Deficiency Reports were ineffective and the conditions were repetitive. 

The auditors also analyzed Corrective Action Requests and Deficiency Reports issued during

the period 1998 through 2002. Most conditions adverse to quality were associated with a failure

to follow procedures. 

The observers agreed with the audit team’s findings in this area.

The observers submitted an AOI in this area for the purpose of addressing corrective action

effectiveness that does not appear to be addressed by the resolution of Corrective Action

Request BSC-01-C-002. 

4.5 Summary of Overall QA Programmatic and Technical Results

The DOE audit team concluded that overall, the software process was “Marginally Effective.”

The audit team concluded this because, among other reasons, it found Software Design and

Testing Process to be “Not Effective,” and the Software Implementation to be “Indeterminate.”

These areas were a part of the 28 total evaluated during the audit of which the audit team found

six to be “Not Effective,” five “Marginally Effective,” and two “Indeterminate.” (See Figure 1) 

The DOE Audit Team Leader recommended that DOE conduct either a follow up audit or

surveillance in 3 to 4 months, when an adequate sample of software would be available to

evaluate. 

5.0 NRC STAFF FINDINGS

The observers found that the audit team used a thorough and comprehensive approach to 

performance-based auditing and the observers agreed with the audit team’s findings and

observations in each of the 28 areas. However, the observers concluded that overall, the

software process was “Indeterminate” for two reasons. First, the audit team found the critical

areas of Software Design and Software Testing to be “Not Effective,” and the area of Software

Implementation to be “Indeterminate.” The observers could not agree with the audit team’s

conclusion that the software process was “Marginally Effective” while these three critical areas

were not found to be “Effective.” Second, the software products reviewed that have completed

the independent verification and validation process represented a limited number of the total

products identified to date (approximately 28 percent) that are intended for LA. Only two of the

software products audited were Level A packages and the retesting of legacy software was not

yet fully implemented.
15

During the conduct of the audit, the observers initiated three AOIs, OQAP-BSC-03-07-01, 02, 

and 03, regarding software accepted using wide tolerances, the appearance of a graded

approach being use to qualify software, and trending of previous Deficiency Reports and

Corrective Action Requests that indicates ineffective corrective action.

During the post-audit conference, the Observation Team Leader stated that the observers

agreed with the audit team’s individual audit findings and recommendations, as presented at

the post-audit conference, but concluded that these findings were “Indeterminate.”

5.1 Statement of Observer Findings

The Observation Audit Team Leader made the following statements at the post-audit

conference, on June 13, 2003.

The observers concluded that the results of the audit are “Indeterminate.” This conclusion is

based on:

5) The limited sample size for products processed under the newly developed procedures;

6) The determination that the essential life cycle elements of Design, Implementation, and

Testing were assessed as “Not Effective” or “Indeterminate.” A higher weighting should be

assigned to these elements.

NRC’s expectations are that BSC will implement the audit team’s recommendations relative to

the technical review requirements in procedure AP-SI.4Q.

NRC (the observers) believes that the effectiveness of the Quality Management System is at

risk if prompt and effective corrective action is not taken to address deficiencies identified,

especially in the area of software testing.

The observers submitted three AOIs. The first AOI pertained to the tolerance for accepting the

results from the testing phase. The second AOI pertained to the appearance of the graded

approach being used to qualify software. The third AOI pertained to the review and evaluation

of software Deficiency Reports, to determine significance and impact related to effectiveness of

corrective actions and failure to follow procedures.

The audit team found the conduct of the audit to be productive from the start. The interactions

between the audit team and BSC staff was positive and essential in moving this audit forward in

a timely manner. The observers’ participation in the afternoon caucus was essential to their

observation process.

Although it has not yet been scheduled, NRC understands that DOE intends to perform a follow

up software audit when a better sample of software is available for review.

The observers found that, although not proceduralized, the audit process used for the

evaluation of the software life cycle areas, sub-areas, and associated programmatic elements

was found to be acceptable.

6.0 NRC AOIs
16

6.1 NRC generated three inquiries as a result of observing audit OQAP–BSC–03–07

(see

6.2 The following inquiries remain open from previous audit observations.

6.2.1 BQAP-BSC-03-02 No. 1

DOE/BSC used qualified, verification level 2 (QL-2), and unqualified data as inputs for modeling

and analysis purposes, for low risk significant applications supporting site recommendation. 

Given that unqualified Data Tracking Numbers are being used in the development of TPOs,

how will DOE/BSC assure that only qualified and verified data and software are used for high

risk significant applications supporting license application?

6.2.2 BQAP-BSC-03-02 No. 2

The audit team identified an instance where, apparently because of time and schedule

pressure, a BSC qualified checker and a BSC Quality Engineering Representative approved the

Thermal Testing Measurement Report (U0220) without reviewing all of the associated data. 

How will DOE and BSC management create an environment to assure that personnel

performing checking and quality assurance assignments will be afforded adequate time to

perform their assigned tasks as time and schedule become even more important leading up to

license application? What metric will be developed and used to assure that quality activities are

not influenced by cost and schedule?

6.2.3 LLNL-ARC-02-07 No. 1

Follow-up activities were conducted during audit activities on an AOI initiated during audit LLNLARC-02-07 at the LLNL in April 2002. The inquiry was initiated to document a procurement of

materials and non-destructive examinations supplied to LLNL. Receiving inspections could not

be completed because necessary information had not been received from the vendor. An

acceptance report completed at Livermore identified that materials analysis information had

been received, but documentation for the requested nondestructive examination was missing.

Nonconformance Report YMSCO-03-0026 was initiated in February 2003. This report has

described a disposition approach that will assure that the necessary nondestructive examination

documentation is received and accepted, thereby resolving the remaining issue. Until this

documentation is received and the nonconformance report is closed, this inquiry remains an

open issue.
SOFTWARE AUDIT RESULTS, OAR-03-03, June 2-13, 2003

Area

DeficiencyReports

Quality

Observations

Recommendations Procedure Adequacy Procedure Implementation Effectiveness 

1. General AS E2. V&V A S E 3. Algorithm X A S E* 4. Tech(review)Adequacy X IA S NE5. Software Procedures X A N/A N/A6. Classification X X A U NE7. Activity Plan X IA S NE8. Life-Cycle, General AM M9. Requirements Phase X A S* E 10. Design Phase X IA I NE11. Implementation Phase X IA U I12. Testing X X X A U NE13. Oper & Maintenance X X A S E14. Install & Checkout AS E15. Retirement X A S E16. Software Controls X X A M M17. Software Use X A M M18. Error Reporting X A S M19. Traceability/Tech Product A S E20. Acquired Software AS E21. Participant Software A S E22. Procurement AS E23. Spreadsheets AS E24. Routines & Macros NOT AUDITED25. Management Tools AS E26. Documentation X X A U M27. At the Labs N/A S E28. Legacy Software X A I I29. CAQs A U NETOTALS 8 7 24 A M M

Results Codes: A = Acceptable U = Satisfactory M = Marginal

IA = Inadequate I = Indeterminate NE = Not EffectiveS = Satisfactory E = Effective 

Figure 1. Summary of Audit Results.

17
18