UNITED STATES

NUCLEAR REGULATORY COMMISSION

WASHINGTON, D.C. 20555-0001

August 1, 2016

MEMORANDUM TO: Stephen Dembek, Acting Chief

Low Level Waste Branch

Division of Decommissioning, Uranium Recovery, 

 and Waste Programs

THRU: Christepher McKenney, Chief

Performance Assessment Branch

Division of Decommissioning, Uranium Recovery, 

 and Waste Programs

FROM: George Alexander, Systems Performance Analyst /RA/

Performance Assessment Branch

Division of Decommissioning, Uranium Recovery, 

 and Waste Programs

SUBJECT: TECHNICAL REVIEW OF QUALITY ASSURANCE 

DOCUMENTATION FOR THE CEMENTITIOUS BARRIERS 

PARTNERSHIP TOOLBOX (DOCKET NO. PROJ0734)

The U.S. Nuclear Regulatory Commission (NRC) staff performed a technical review of the 

Quality Assurance (QA) documentation related to Version 3.0 of the Cementitious Barriers 

Partnership (CBP) Toolbox (i.e., CBP Toolbox). The CBP Toolbox provides a graphical user 

interface to set up and run simulations associated with cementitious material durability. The 

main software components integrated into the CBP Toolbox are STADIUM and 

LeachXS/ORCHESTRA. The purpose of the NRC staff review was to: (1) evaluate the software 

quality assurance applied to development of the CBP Toolbox, and (2) review the technical 

basis for the main software components. In some software, the technical basis is included 

within the scope of software QA. In this NRC staff review, the technical basis was discussed in 

a stand-alone section because of the depth of the material covered.

While in some areas the QA for the CBP Toolbox was relatively complete, the NRC staff 

identified QA gaps and limitations in the documentation associated with software development 

and in the technical basis for the conceptual and mathematical models for the CBP Toolbox. 

CONTACT: George Alexander, NMSS/DUWP

 (814) 297-8385
S. Dembek - 2 -

Several of these gaps could be addressed by providing additional discussion within existing 

documentation (e.g., clarification on the capabilities and limitations of the software). However, 

several limitations would require significant resources to address, including:

• incomplete set of degradation mechanisms and limited coupling of mechanisms,

• insufficient support for modeling damage due to sulfate attack, and

• limited validation exercises

The NRC staff appreciates that many of the limitations discussed in this report are a function of 

the ongoing and challenging nature of the CBP project. The NRC staff believes that the CBP 

has the capabilities to resolve these concerns. However, without additional resources to 

complete the CBP Toolbox as originally intended, the utility of the software is somewhat limited. 

For instance, software users would need to conduct additional analyses to demonstrate that 

modeled degradation would not be increased due to degradation mechanisms that are not 

currently included in the CBP Toolbox or due to the coupling and synergism of multiple 

degradation mechanisms. 

Enclosure:

Technical Review of the Quality Assurance Documentation

 for the Cementitious Barriers Partnership Toolbox
S. Dembek - 2 -

Several of these gaps could be addressed by providing additional discussion within existing 

documentation (e.g., clarification on the capabilities and limitations of the software). However, 

several limitations would require significant resources to address, including:

• incomplete set of degradation mechanisms and limited coupling of mechanisms,

• insufficient support for modeling damage due to sulfate attack, and

• limited validation exercises

The NRC staff appreciates that many of the limitations discussed in this report are a function of 

the ongoing and challenging nature of the CBP project. The NRC staff believes that the CBP 

has the capabilities to resolve these concerns. However, without additional resources to 

complete the CBP Toolbox as originally intended, the utility of the software is somewhat limited. 

For instance, software users would need to conduct additional analyses to demonstrate that 

modeled degradation would not be increased due to degradation mechanisms that are not 

currently included in the CBP Toolbox or due to the coupling and synergism of multiple 

degradation mechanisms. 

Enclosure:

Technical Review of the Quality Assurance Documentation

 for the Cementitious Barriers Partnership Toolbox

DISTRIBUTION: DEsh JPhilip MFuhrmann

ML16196A179

OFC NMSS:TR DUWP:LA DUWP:PM DUWP:BC NMSS:TR

NAME GAlexander TMoon HFelsher CMcKenney GAlexander

DATE 07/14/2016 07/14/2016 07/15/2016 07/27/2016 08/01/2016

OFFICIAL RECORD COPY
Contents

Software Overview.................................................................................................................- 1 -

Software QA Review..............................................................................................................- 2 -

Software Quality Assurance Requirements ............................................................................- 4 -

I Software QA ...................................................................................................................- 7 -

I.1 CBP Software QA....................................................................................................- 7 -

NRC Staff Review of CBP Software QA..........................................................................- 7 -

I.2 Software QA – GoldSim GUI.................................................................................. - 12 -

NRC Staff Review of Software QA – GoldSim GUI ....................................................... - 12 -

I.3 Software QA – STADIUM....................................................................................... - 12 -

NRC Staff Review of Software QA – STADIUM ............................................................ - 13 -

I.4 Software QA - LeachXS......................................................................................... - 14 -

NRC Staff Review of LeachXS...................................................................................... - 14 -

I.5 Software QA – ORCHESTRA ................................................................................ - 15 -

NRC Staff Review of Software QA – ORCHESTRA...................................................... - 15 -

I.6 Software QA – Other Software............................................................................... - 16 -

NRC Staff Review of Software QA – Other Software .................................................... - 16 -

II Verification and Validation............................................................................................. - 18 -

Verification and Validation – General Limitations of the CBP Toolbox Software................... - 18 -

II.1 Verification and Validation – LeachXS ................................................................... - 19 -

NRC Staff Review of Verification and Validation – LeachXS......................................... - 19 -

II.2 Verification and Validation – ORCHESTRA............................................................ - 20 -

NRC Staff Review of Verification and Validation – ORCHESTRA ................................. - 21 -

II.3 Verification and Validation – STADIUM.................................................................. - 23 -

NRC Staff Review of Verification and Validation – STADIUM........................................ - 23 -

II.4 Verification and Validation – Other Software.......................................................... - 24 -

NRC Staff Review of Verification and Validation – Other Software................................ - 24 -

Conclusions .........................................................................................................................- 25 -

References ..........................................................................................................................- 26 -

Appendix: Concerns Related to Sulfate Attack for ORCHESTRA and STADIUM................. - 28 -
- 1 -

Technical Review of Quality Assurance Documentation

for the Cementitious Barriers Partnership Toolbox

Date: 07/19/16

Reviewers

George Alexander, Systems Performance Analyst, U.S. Nuclear Regulatory Commission (NRC)

David Esh, Sr. Systems Performance Analyst, NRC

Mark Fuhrmann, Geochemist, NRC

Jacob Philip, Sr. Geotechnical Engineer, NRC

Software Overview

The Cementitious Barriers Partnership (CBP) is a multi-disciplinary project supported by the 

U.S. Department of Energy (DOE) Office of Environmental Management to develop a set of 

computational tools, laboratory and field experimental data, test methods, and guidance 

documents to improve understanding and prediction of long-term structural, hydraulic, and 

chemical performance of cementitious barriers and waste forms used in nuclear applications for 

up to and greater than 100 years for operating facilities and greater than 1000 years for waste 

management. 

The CBP was initiated in 2008 and currently includes DOE-EM, Savannah River National 

Laboratory (SRNL), Vanderbilt University Department of Civil and Environmental Engineering, 

Consortium for Risk Evaluation with Stakeholder Participation, NRC, Energy Research Centre 

of The Netherlands, and SIMCO Technologies Inc. (SIMCO),

The CBP Toolbox includes software for assessing the following chemical degradation 

mechanisms of cementitious materials: chloride ingress, sulfate attack, carbonation, and 

leaching. The software consists of a Graphical User Interface (GUI) developed with GoldSim 

that is used to control inputs and settings provided to three main components: STADIUM, 

LeachXS, and ORCHESTRA. Additional supporting components (e.g. Dynamic Linked Libraries 

(DLLs)) are used to process the inputs and outputs. The collection of all the components is 

referred to as the CBP Toolbox. Figure 1 below shows the basic organization of the CBP 

Toolbox.
Figure 1. CBP Toolbox Framework (CBP-TR-2013-004-1)

Software QA Review

The software QA process can differ depending on the type of software that is developed and the 

use of the software. Different organizations may have different requirements. In addition, 

software may be acquired or developed. The software developer is responsible for 

documenting the capabilities and limitations of the software. However, during software 

acquisition, it is the responsibility of the user to ensure that the software meets the intended 

requirements. For development of technical software to be used in real-world applications, the 

major components that should be common across the variables discussed above are presented 

in Figure 2, which provides a schematic of the software QA process. NRC’s review of the CBP 

toolbox is divided into two parts: 1) software quality assurance and 2) the technical basis for the 

software components. Emphasis is placed on the software development process rather than 

the software acquisition process. It is acknowledged that the technical basis for the software

components is considered to be part of software quality assurance for some programs. The 

organization of the topics in this document is for clarity of discussion.
- 3 -

Figure 2. Software QA Process

Software QA

Software 

Development

Requirements 

Identification

Development 

Plan

Model 

Development

Conceptual 

Model

Validation

Mathematical 

Model

Verification

Software 

Acquisition

Software 

Review

Installation and 

Acceptance 

Testing

Internal Audits

External Peer 

Review

Software 

Application

User's Manual

Training

Software 

Management

Configuration 

Control and 

Release

Error 

Management

Section I

Section II
- 4 -

The following documents were reviewed:

• Vaughan, J.P. Quality Assurance Program, Cement Barriers Partnership (CBP), CBP01, SRNL, Aiken, SC, December 2008.

• Cementitious Barriers Partnership Software Quality Assurance Implementation 

Procedure, CBP-RP-2015-017, Rev. 0, November 2015.

• Smith, F.G. and G.P. Flach, Software Quality Assurance Plan: CBP Toolbox GoldSim 

Dashboard Graphical User Interface, G-SQP-G-00018, Revision 0, SRNL, Aiken, SC, 

September 2015.

• Smith, F.G. and G.P. Flach, Software Quality Assurance Plan: FloXcel, G-SQP-G00017, Revision 0, SRNL, Aiken, SC, September 2015.

• Smith, F.G., Flach, G.P., and K.G. Brown, CBP Code Integration GoldSim DLL Interface, 

CBP-TR-2010-009-2, Rev. 0, Vanderbilt University, Nashville, TN, June 2010.

• Brown, K.G., Flach, G.P., and F.G. Smith, GoldSim Interface for the CBP Software 

Toolbox, Version 2.0 User Guide, CBP-TR-2013-004-1, Rev. 0, Vanderbilt University, 

Nashville, TN, August 2013.

• Samson, E., STADIUM Calculation Core 2015b – STADIUM Quality Assurance Report, 

SIMCO Technologies, Quebec, Canada, December 2015.

• Meeussen, J.C.L., ORCHESTRA Manual and Documentation, CBP-TR-2015-00X-Y, 

Rev. C (DRAFT for CBP review), NRG, Petten, The Netherlands, December 2015.

• Meeussen, J.C.L. and K.G. Brown, 2015. ORCHESTRA Software Quality Assurance 

Documentation, CBP-TR-2015-001-1, Rev. F, December 2015.

• Seignette, P., LeachXS QA Documentation, ECN-X-15-123, Version 2.0.x DRAFT, ECN, 

Petten, The Netherlands, November 2015.

Software Quality Assurance Requirements

There are different QA requirements for different organizations and depending on the use of the 

software (e.g. safety class vs. non-safety class). The NRC staff considered a variety of sources 

of requirements when evaluating the sufficiency of the software QA for the CBP Toolbox. 

Generally the requirements are written at a high level and the implementation guidance provides 

the details which are commonly written into operating procedures. The reviewed requirements 

and implementation guidance included:

• American Society of Mechanical Engineers (ASME), NQA-1-2009, Quality Assurance 

Requirements for Nuclear Facility Applications,

• Manual 1Q, Procedure 20-1, Software Quality Assurance,

• DOE Order 414.1D, Quality Assurance,

• DOE Guide 414.1-4, Safety Software Guide for Use with 10 CFR 830 Subpart A, Quality 

Assurance Requirements,

• Technical Operating Procedure (TOP) – 018, Development and Control of Scientific and 

Engineering Software (ML050100177),

• IEEE Standard 730-2014 – Standard for Software Quality Assurance Plans,

• NRC, NUREG/BR-0167 – Software Quality Assurance Program and Guidelines,
- 5 -

• NRC, NUREG/CR-4640 – Handbook of Software Quality Assurance Techniques 

Applicable to the Nuclear Industry, and

• NRC, draft NUREG-2175 – Guidance for Conducting Technical Analyses for 10 CFR 

Part 61

The main elements of software QA addressed in these documents are software development, 

acquisition, review, management, and application. More detail (depth) is provided for software 

development because the development process can be the most complex task. Likewise, in 

Section II of this report, more detail is provided on review of the conceptual and mathematical 

models and the associated verification1 and validation activities. Comparisons of the main 

elements for select documents are shown on the next page. Many elements are consistent, 

though the terminology is different. Figure 2 is an attempt to synthesize the QA requirements 

used in this review with an emphasis on developed software. 

 1 Verification is defined in this report as the process which demonstrates that the software correctly 

performs its stated capabilities. Validation is defined as the assurance that the model embodied in a 

computer program is a correct representation of the process or system for which it is intended (adapted 

from NUREG/CR-4369, ML012750492).
- 6 -

Software QA Elements

Manual 1Q 

Software Quality Assurance

5.4.2 Requirements Phase

5.4.3 Design Phase

5.4.4 Implementation Phase

5.4.5 Test Phase

5.4.6 Installation and Acceptance Phase

5.4.7 Operations and Maintenance Phase

5.4.8 Retirement Phase 

5.5 Software Configuration Control

DOE Guide 414.1-4 Safety Software Guide

5.2.1 Software Project Management and Quality Planning

5.2.2 Software Risk Management

5.2.3 Software Configuration Management

5.2.4 Procurement and Supplier Management

5.2.5 Software Requirements Identification and Management

5.2.6 Software Design and Implementation

5.2.7 Software Safety

5.2.8 Verification and Validation

5.2.9 Problem Reporting and Corrective Action

5.2.10 Training Personnel in the Design, Development, Use, and Evaluation of Safety Software

Technical Operating Procedure (TOP) - 018

Development and Control of Scientific and Engineering Software (ML050100177)

5.2 Software Requirements Identification

5.3 Software Development Planning

5.4 Software Development

5.5 User’s Manual

5.6 Installation and Acceptance Testing

5.7 Configuration Control and Release

5.8 Software Validation

5.9 Software Change

5.10 Software Retirement
- 7 -

Section I

I. Software QA

Section I covers review of software QA with the exception of verification and validation, which 

are covered in depth in Section II.

I.1 CBP Software QA

The requirements for software QA applied to the CBP project are contained in Section 3.5 of 

CBP-01 (Vaughan, 2008). Most of the responsibility for software QA is assigned to the principal 

investigator (PI) and the PI’s organization for the relevant activity. For the CBP project, different 

individuals and organizations develop individual pieces of the overall toolbox, which are then 

combined into an integrated product. The requirements for software QA in CBP-01 include:

• review of acquired or developed software prior to use,

• review in sufficient detail (verification/validation),

• software development program,

• testing,

• independent review (if necessary),

• documentation (program ID, limitations, capabilities, intended use, source code listing), 

and

• document control (software ID, revision #, or date on all output).

The higher-level project document CBP-01 is supplemented with an implementing procedure 

CBP-RP-2015-017, Rev.0. The implementing procedure states that each PI is responsible for 

software QA and that the implementation program within their organization should adhere to 

ASME NQA-1-2009 or later and DOE Order 414.1D for Non-Safety Software. The adherence to 

NQA-1-2009 or later is more restrictive than some of the other cited QA requirements 

documents. Therefore, the elements provided in Figure 2 should apply to the CBP project, but 

because the software is classified as non-safety software the elements may be implemented 

with a graded approach. Though the CBP has software QA procedures, similar implementing 

procedures were not available for the PI’s organizations. The CBP implementing procedure 

provides more specific guidance limited to configuration control and error reporting activities. 

This procedure provides guidance on configuration control of components and bundles, 

management of software bundle releases, and problem reporting and error correction.

NRC Staff Review of CBP Software QA

Figure 3 summarizes the NRC’s overall summary of the CBP QA program. The green check 

means essentially completed or generally sufficient, the yellow dash means partially completed 

or improvement needed, whereas the red X means not completed or insufficient. These high 

level subjective ratings are essentially an average over all CBP components. For a more indepth discussion of each individual component, see the sub-sections that follow. Because an 

NRC staff gave an element a check does not mean that there are not areas of improvement. 

Likewise because NRC staff gave an element an X does not mean that it is totally deficient. 
- 8 -

The NRC staff is trying to present in a clear manner those areas that need the most effort to 

achieve a level of software QA as recommended in the implementing procedure. Because most 

of the responsibility for software QA was delegated to the PI’s organizations, there is variability 

from organization to organization.
- 9 -

Figure 3. Overview of the CBP Software QA Status
- 10 -

Requirements Identification is completed at the initiation of software development to specify 

what the software is required to perform. 

It is good practice to develop a Software Requirements Document that identifies:

• the purpose of the software,

• what the software must do, and

• development tests and acceptance criteria.

At a minimum, requirements cover areas such as software function, performance, interface with 

users, resource requirements, and installation considerations. Without a clear specification of 

what the software must accomplish, the other steps in the process, such as verification and 

testing, may be less effective and may not be complete. For example, the spatial and temporal 

scales of the problem the software is to be applied to should be identified. Identification of 

requirements can also help prevent conflicting requirements. The software requirements 

documentation is to be maintained throughout the software lifecycle. For the CBP, there was 

limited documentation of the required performance of the software.

A Software Development Plan is used to describe the activities necessary to develop or modify 

the software. A Software Development Plan may describe:

• units used in the code,

• source code and title block documenting and commenting,

• design elements,

• theory, mathematics, flow, physical models, prescribed ranges of inputs and outputs, 

projected verification and validation tests and acceptance criteria, and

• code review and documentation.

For the CBP, some software has been developed whereas other components were previously 

established and were modified. It does not appear that stand-alone development plans were 

used for the CBP but information that would typically be found in a development plan is provided 

in QA reports for some of the components. Typically, the development plan would provide a 

wide array of information on the software, such as: the theoretical basis, structure, and 

requirements for control and flow of data. The software development plan specifies the 

standards and testing that will be applied to the software. With respect to development 

planning, the completeness of the various CBP components was addressed in the individual 

sections for the components. For the CBP Toolbox, development planning was performed by 

CBP members working on the project typically at annual or more frequent meetings, though 

there is no particular document that can be referenced as the CBP development plan.

Model Development includes the description of the conceptual and mathematical models as well 

as the associated verification and validation activities. Model development documentation 

should also describe the capabilities and limitations of the model. For some software, the 

conceptual model description may not be relevant (e.g., plotting software) and the mathematical 

model may be relatively simple. 

The overall CBP is lacking a document that describes what the CBP Toolbox can, and more 

importantly, cannot do. A document for the CBP should be developed that lists the degradation 

mechanisms that are and are not included, how those mechanisms are (or are not) coupled, the 
- 11 -

spatial and temporal scales that can be addressed on a desktop computer, and a definition of all 

the input parameters and their units, and geometry and boundary conditions that can be 

evaluated. Detailed discussion of model development of the components of the CBP Toolbox is 

addressed in the individual sub-sections below.

Software Review is the process of performing installation and acceptance tests, internal audits 

and review, and external peer review. Installation and acceptance testing must be performed 

on each intended platform and operating system and the results documented. Installation and 

acceptance testing has been performed for most CBP Toolbox components, but not for each 

intended platform and operating system. In addition, the documentation does not identify, in all 

cases, on what platforms and operating systems the software can be used. In addition, the 

overall CBP Toolbox does not identify installation and acceptance tests and what platforms and 

operating systems the toolbox can be applied on. NRC staff did not receive documentation of 

internal audits or external peer review. However, because the CBP toolbox is a collaborative 

effort among different groups, an external peer review function is performed throughout the 

development process. This external peer review could be improved by formalizing the process 

and documenting the results.

Software Application is the process of using the software in an application. In terms of software 

QA, it is the products put in place to help ensure successful application of the product. 

Generally software application will include a User’s Manual and Training. Training on the CBP 

Toolbox was provided after previous releases of the CBP Toolbox. The training focused more 

on how to use the CBP Toolbox rather than the use of more complex components (i.e. 

STADIUM, LeachXS/ORCHESTRA) run from the CBP Toolbox GUI. A user’s guide for the 

GoldSim GUI for the CBP Toolbox was developed (Brown, 2013). It provides a good 

description of the GUI and how to use it, including tutorials. However, the inputs to the 

dashboard are not adequately defined in the user’s manual nor are the impact of the inputs on 

the simulation. For instance, STADIUM requires a chloride binding factor, which means that the 

user’s manual should address basic information, such as the appropriate default value, 

expected range, definition, and use of parameters.

In order for the user to properly use the CBP Toolbox, they need a better understanding of the 

inputs. In addition, a number of input parameters are unable to be changed by the user and the 

reasons why are not described for all of these affected values. The NRC staff understands that 

some values are set based on the user-selected material type, but fixing the values greatly limits 

the overall applicability of the software. The user’s guide for the CBP Toolbox should pull 

forward any necessary information from user’s manuals from the underlying partner software

about parameter definitions, uses in the model, default values, expected ranges of values, and 

any reasons why parameter values may not be adjustable.

Software Management is comprised of Configuration Control and Release and Error Reporting. 

Because the CBP Toolbox is deployed to end users, the CBP project does not have the ability 

to perform configuration control at the end-user level. The CBP project does control release of 

the software to end users by ensuring that they are registered and maintaining a record of 

registered users. The CBP implementation procedure requires that both software components 

and software bundles (e.g. “CBP Toolbox”) are uniquely identified by name, version number, 
- 12 -

and/or release date. For bundled software the information is to be explicitly shown on the GUI 

startup screen. Version 2 of the CBP Toolbox pre-dated the implementation guidance and does 

not show the version number or release date on the GUI startup screen. A directory tree is 

provided in CBP toolbox interface user’s guide (Brown, 2013).

Error Reporting is requested of end users of the CBP Toolbox. In addition, some of the CBP 

components maintain error logs that are compiled and addressed during product revision by the 

CBP. However, there does not appear to be a mechanism in place for errors in the CBP 

components to be reported and compiled by the CBP, such that CBP end users can be notified 

and determine the significance of errors in previous applications. The CBP project is supposed 

to maintain a record of problem reports and error corrections activities (CBP, 2015). NRC staff 

requested the record but did not receive it to support this review.

I.2 Software QA – GoldSim GUI

The GoldSim GUI is the mechanism through which an end user accesses the partner codes of 

the CBP Toolbox. A user’s guide for the GoldSim GUI (Version 2) was developed (Brown, 

2013) and a software QA plan was developed (Smith, 2015). Because the GoldSim software is 

a commercial product, the correct functioning of the authoring capability is demonstrated by the 

software vendor. The software quality assurance plan focuses on verification that the correct 

transfer of information occurs from the dashboard interface to the input files of the software 

components. Because of the nature of the GoldSim GUI (i.e., an input selection and transfer 

function), identification of requirements, planning the software, and model development are of 

secondary importance. The software quality assurance plan and user’s guide appropriately 

focused on software review, software application, and error reporting. 

NRC Staff Review of Software QA – GoldSim GUI

Overall the software QA applied to the GoldSim GUI (Version 2) was appropriate. The software 

quality assurance plan was clearly written and described the scope and software life cycle. The 

required and optional QA elements were clearly indicated. The types of verification tests 

performed were appropriate given the scope of the GUI software. Identification of what 

platforms and operating systems were tested could improve the documentation. In addition, 

proper transfer of outputs into the GUI could have been within the scope of the software QA 

plan. Areas for improvement include: identification of the version number of the software on the 

opening screen, defining parameter inputs (i.e., appropriate ranges, units, etc.) otherwise 

misapplication could occur. The verification tests appear to be incomplete because they do not 

evaluate whether the correct outputs have been generated and reflected in the dashboards. In 

addition, the software QA plan for the GoldSim GUI does not specify that the reviewer should 

verify that the units reflected in the dashboard (i.e., a mixture of metric and English units) are 

correct for the input files of the partner codes. That would require comparing the units in the 

dashboards to the component software user’s manuals.

I.3 Software QA – STADIUM

STADIUM is a mature product and has been used extensively in commercial application for over 

a decade. For the CBP project, a web-based application was developed to simulate concrete 

degradation cases specific to the CBP. The NRC staff reviewed a QA report for STADIUM and 

a user’s manual (Samson, 2015; SIMCO Technologies, 2015). The quality assurance report 

was focused on describing the conceptual and mathematical models and the verification and 
- 13 -

validation tests that had been performed. The user’s manual provided instructions for how to 

use STADIUM through a web-based interface. 

The STADIUM quality assurance report was primarily focused on model development 

(discussed in Section II). The STADIUM quality assurance report did not address some of the 

other broader software QA components, such as:

• Requirements identification,

• Development planning,

• Installation and acceptance testing,

• Internal audits and external peer review,

• Configuration control and management, and

• Error management.

Configuration control and management, installation testing, and acceptance testing are 

managed through use of the web-based interface. The actual simulation occurs on a remote 

computer and therefore the platform and operating system is controlled. A user’s manual was 

developed to provide operating instructions. The user’s manual did not provide parameter 

definitions, default values, and ranges.

NRC Staff Review of Software QA – STADIUM

Overall the verification and validation testing completed for STADIUM was complete and welldocumented. However, there are some areas of improvement associated with the broader 

aspects of software QA applied to STADIUM. Verification and validation is discussed in more 

detail in Section II of the report below.

Although some aspects of STADIUM are very mature, other aspects have been added more 

recently as part of the CBP project (e.g. carbonation). Therefore, while there may not be 

requirements identification and development planning completed within the CBP project for the 

mature components that is within the scope of this NRC review, there should have been for the 

recently added functionality. Without requirements identification, the verification and validation 

testing becomes more difficult. It can be difficult to verify that test cases were developed that 

demonstrate the software can adequately simulate the physical system being modeled rather

than simply verifying the aspects of the physical system the software is capable of modeling. In 

other words, it can be difficult to verify that test cases were developed that demonstrate the 

software can simulate what it needs to simulate rather than simply what it can simulate. If the 

test cases or cases very similar to the test cases were used in the development process, the 

software may simulate the test case well but fail in application. Any development testing should 

be as independent as practical from the verification and validation testing discussed in Section II 

below.

The user manual or another document should provide an installation test case(s) with results so 

the user can verify the proper execution of the software. In addition, the user manual should 

specify with what web browsers the software was verified, and discuss resource requirements. 

The STADIUM user manual should provide instructions to the user for reporting errors, and how 

registered users will be informed of errors that are internally identified.
- 14 -

QA documentation for STADIUM should describe the software review process applied and the 

records produced to demonstrate that a change or new addition to the software (e.g. 

carbonation) has been implemented correctly.

I.4 Software QA - LeachXS

The NRC staff reviewed the QA documentation for LeachXS (Seignette, 2015). The 

documentation described the purpose and approach for the software. Functionality and 

structure were described at a high level. A separate requirements identification document was 

not provided. For modifications to LeachXS, such as for the CBP Project, it could not be 

determined if development plans are used. Most of the LeachXS QA report is focused on 

verification and validation of different elements (as discussed in Section II below).

The QA documentation provides system requirements, a description of the licensing system, 

software installation instructions, and installation test problems. The documentation lists the 

name, version number, and purpose of all libraries and tools. However the QA documentation 

also states that the source code contains a few hundred files. The names of the source files

associated with a particular version should be listed in an appendix. Some of the tools listed are 

privately developed. QA would need to be provided for the privately developed tools used with 

LeachXS. The operating systems and versions of other software necessary for installation were 

provided, however it is not clear what testing was performed to verify correct operation on 

different combinations of Windows operating systems and other software. 

Although there is not a separate user’s manual, the LeachXS QA Documentation report 

provides instructions for use of the software, including installation instructions, a description of 

required computer resources, and installation test cases. However, the report does not provide 

statements about the capabilities and limitations of the software. The report does define 

parameters and their units but does not identify default values or expected ranges.

Configuration management is provided using Microsoft Team Foundation Server. Every release 

of the software is assigned a unique version and the source code is labeled that allows any 

previous version of the software to be recreated. Bugs, issues, and tasks are tracked and are 

summarized in release notes as new versions are produced. However, the user’s manual or 

other documentation does not provide instructions for how users are to report errors.

NRC Staff Review of LeachXS

Overall the verification and validation testing completed for LeachXS could be strengthened with 

additional information. The QA report clearly documented steps for users to verify the model, 

but limited information is presented within the report to verify the model. Additional information 

on model validation (e.g., how well the test methods represent potential future releases) would 

also strengthen the QA for LeachXS. Verification and validation is discussed in more detail in 

Section II below.

Requirements identification and development planning documents should be used. Without 

requirements identification, the verification and validation testing becomes more difficult. As 

discussed above with respect to STADIUM, without requirements identification, it can be difficult 

to verify that test cases were developed that demonstrate the software can simulate what it 

needs to simulate rather than what it can simulate. If the test cases or cases very similar to the 

test cases were used in the development process, the software may simulate the test case well 
- 15 -

but fail in application. Any development testing should be as independent as practical from the 

verification and validation testing discussed in Section II below.

Installation testing should be performed on different combinations of Windows operating 

systems and other software used by LeachXS. Though not all combinations need to be 

evaluated, usually the newest and oldest versions of each will be tested along with random 

selections of other combinations. If any of the random combinations fail, then a full factorial 

design is evaluated.

Software QA should be performed on internally developed software that supports LeachXS, and 

acquired software should be tested and accepted prior to use.

The LeachXS QA Documentation report contains installation instructions, a description of 

required computer resources, and installation test cases. However, this information should be 

provided in a separate user’s manual as well as statements regarding the capabilities and 

limitations of the software and instructions for how users are to report errors.

I.5 Software QA – ORCHESTRA

The NRC staff reviewed the ORCHESTRA manual and documentation (Meeussen, 2015). The 

documentation was a combination user’s manual, description of examples, and description of 

benchmark problems. Several studies including an international peer-reviewed benchmarking 

study provided strong model verification. Model validation was more limited and would benefit 

from additional case studies. Verification and validation is discussed in more detail in Section II 

below.

ORCHESTRA is somewhat different than many science and engineering models in that all 

chemical and physical model equations in ORCHESTRA are defined in text files that are read in

at run time. This approach allows the computational core to be compact and efficient. 

However, it can create some difficulties from a code verification standpoint and places additional 

QA responsibility on the user to verify that the specific CBP Toolbox application is not outside 

the bounds for which the software has been tested. 

NRC Staff Review of Software QA – ORCHESTRA

The ORCHESTRA documentation of software QA is not yet complete and not as mature as 

STADIUM and LeachXS. The NRC staff recommends that a software QA plan be developed for 

ORCHESTRA. The QA plan should address those elements in Figure 1 and the more specific 

details provided in the review of LeachXS and STADIUM. The NRC staff recommends that:

• A separate user’s manual be developed that provides installation instructions, a 

description of required computer resources, statements about the capabilities and 

limitations of the software, and installation test cases. In addition, the user’s manual 

should provide instructions for how users are to report errors. The user’s manual should 

define input parameters, default values, and expected ranges.

• Configuration control should be applied and described in QA documentation.

• Installation tests be developed. The developer should identify platforms where the 

software can be applied and provide results of tests performed to demonstrate the 

correct operation.
- 16 -

• Model requirements be identified and software developed through documented 

“requriements identification” and “development planning” processes.

A challenge associated with ORCHESTRA results from the open architecture of ORCHESTRA 

compared to the other components of the CBP Toolbox. Because the chemical and physical 

model equations are defined in text files by the end user, the burden of verification testing is 

shifted to the end user. This limitation needs to be communicated clearly to the end user. The 

developers can verify the correct functioning of the individual “building blocks” of the software. 

However, the developers cannot anticipate all the ways an end user may want to exercise the 

software and combine the components. Therefore most calculations will be unverified unless 

verification is performed by the end user. This may not be a challenge for relatively 

straightforward problems, however for systems that are more complex (e.g., systems that 

require iterative methods of solving equations), verification may be much more difficult. For 

example, the QA documentation states that use of the chemical editor GUI automates much of 

the error-prone editing, but not all of the error-prone editing. Chemical calculation templates are 

used. All of these tasks and components must be checked and verified. Implementation of a 

robust QA program by the end user will be very important for application of ORCHESTRA.

I.6 Software QA – Other Software

A variety of other software is used with the CBP Toolbox. For example, Version 2 of the CBP 

Toolbox used executable DLLs to communicate with and execute partner codes, 

forgnuplot_V1.1 for plotting, GetComputerNameA, gp440rc1_win, and Mesh2d. The current 

CBP Toolbox version retires some previous software and uses some new software (e.g. 

FloXcel).

Some of the other software had software QA plans2 developed, but not all of them. Because 

the software QA plans were developed by the same organization, they used a standard format 

and content. The software QA plans were used to perform verification testing and provide 

configuration control and management. The other software was generally used to support the 

major technical components of the CBP Toolbox. Software requirements documents were not 

developed. Though the purpose and functionality of the software was discussed in the software 

QA plans, limitations of the software were not provided. 

NRC Staff Review of Software QA – Other Software

For non-safety software it is appropriate to use graded QA. For example, if software is 

acquired, then either acceptance testing may be performed prior to using the software on the 

user’s project or comprehensive verification testing may be performed prior to using it. The 

latter approach has higher risk to the user’s project because issues may be identified at a later 

stage, but as long as testing is performed the timing is not essential. Discussion of the 

conceptual model and validation tests may not be necessary for software that is used for 

plotting, for example. In general, the NRC staff determined that the software QA plans are 

appropriate in terms of their content for the “Other” software used by the CBP project.

 2 Software quality assurance plans developed by this organization include both the planning and the 

implementation of testing (i.e. the test results).
- 17 -

All developed and acquired software supporting the CBP toolbox should have software QA 

plans. If software requirements documents are not used, the software QA plans should list the 

requirements that need to be verified. Verification testing should ensure that all required 

functionality is operating properly.
- 18 -

Section II

II. Verification and Validation

Verification and validation are arguably two of the most important tasks associated with QA of 

developed software. Verification and validation provide confidence that the software can be 

applied to solve particular problems. This section of the report identifies technical limitations 

associated with the CBP Toolbox and its supporting software.

Verification and Validation – General Limitations of the CBP Toolbox Software

Incomplete set of degradation mechanisms and limited coupling of mechanisms

The CBP has focused on the following chemical degradation mechanisms of cementitious 

materials: chloride ingress, sulfate attack, carbonation, and leaching. Although the included set 

of degradation mechanisms are necessary, they may not be sufficient to have confidence in the 

long-term projections of cementitious materials performance. Additional chemical degradation 

mechanisms (e.g., alkali-silica reaction (ASR), biodegradation) may also be relevant to the longterm performance of cementitious materials. In addition, the coupling of chemical degradation 

mechanisms with the mechanical properties of the cementitious materials could be important for 

the modeling of cementitious materials over long time frames. The CBP Toolbox user’s manual 

should clearly describe the degradation mechanisms, coupling of degradation mechanisms, and 

the boundary conditions for which the CBP Toolbox has been verified and validated. Currently, 

users would need to provide supplemental analyses to demonstrate that additional or coupled 

degradation mechanisms under reasonably anticipated future conditions would not result in 

degradation exceeding that predicted by the current CBP Toolbox models.

A recent laboratory study may provide an example of the potential risk from relying on models 

where the potential range of expected current and future conditions for cementitious materials 

has not been tested. Protière and Samson (2015) observed significant degradation of a 

simulated 150-ft diameter Savannah River Site Disposal Structure paste sample in a laboratory 

study. It is not yet clear if the observed degradation was due to an experimental artifact or if the 

laboratory results are indicative of potential ASR under field conditions. Additional studies are 

being conducted by Savannah River National Laboratory (SRNL), SIMCO, and the Savannah 

River Ecology Laboratory to determine the cause of the degradation. If these studies indicate 

that the observed laboratory degradation is due to ASR under anticipated field conditions, then 

this would demonstrate a limitation of the current CBP Toolbox.

In addition, feedback between degradation mechanisms (or coupling) could result in a 

synergistic effect on the rate and extent of degradation relative to degradation mechanisms 

acting in isolation. Although calibration data in some of the CBP Toolbox software includes data 

from studies where multiple degradation mechanisms are occurring simultaneously, the CBP 

Toolbox does not fully account for coupled degradation. 

Limited validation exercises

Additional discussion regarding the individual validation exercises for STADIUM and 

LeachXS/ORCHESTRA is provided below. In general, additional validation exercises would 

provide the necessary confidence for projections using the CBP Toolbox. The NRC staff 
- 19 -

recommends that the CBP develop additional validation test cases, including flow and transport 

of deleterious species through initially degraded cementitious materials. These validation test 

cases should include materials and conditions as similar as possible to the intended materials 

and conditions. The NRC staff appreciates that validation exercises with real-world materials 

involve boundary conditions that may not be well-characterized and may be challenging to 

account for in the CBP Toolbox software (e.g., fluctuating conditions such as temperature and 

moisture content). However, these types of validation exercises can provide confidence that the 

controlling features, events, and processes are captured and reasonably represented in CBP 

Toolbox software.

Insufficient support for modeling damage due to sulfate attack

The technical basis for long-term concrete degradation due to sulfate attack is not adequately 

supported in ORCHESTRA or STADIUM. Of the various components for modeling sulfate 

attack, the technical basis for sulfate ingress and chemical reactions/mineralogical changes 

(Maltais et al., 2004; Samson and Marchand, 2007) is well-supported. However, support is 

more limited for several assumptions related to the approach and parameterization of the 

modeling of damage due to sulfate attack. Without additional support for these assumptions, 

confidence in damage progression due to sulfate attack is similarly limited. Additional 

discussion of NRC staff’s concerns related to the modeling of sulfate attack is provided in the 

Appendix to this report. 

II.1 Verification and Validation – LeachXS

LeachXS/ORCHESTRA is a combined database and modeling system for simulating transport 

and degradation phenomena in cementitious materials (Seignette, 2015). LeachXS can be 

used to: (i) import laboratory results into a database, (ii) compare and analyze leaching test 

results for granular and monolithic materials, (iii) calculate acid/base neutralization, and (iv) 

predict laboratory or field leaching behavior for granular and monolithic materials under a range 

of conditions. The LeachXS database includes equilibrium leaching test results from pHdependence leaching tests (EPA method 1313, EN 14429, EN 14497, ISO/TS21268-4), 

dynamic leaching test results from percolation tests (EPA method 1314, PrEN 14405), and

monolith leaching tests (EPA method 1315, EN 15683), in addition to lysimeter tests and field 

measurements. 

NRC Staff Review of Verification and Validation – LeachXS

The LeachXS Documentation report by Seignette (2015) provides information on verification 

and validation for the various LeachXS software functions. Associated with the report, are 

various test cases and spreadsheets to verify that the LeachXS software correctly performs its 

stated capabilities. During this review, the NRC staff did not run LeachXS to independently 

verify the test cases. To the extent possible, it would be helpful to include results from these 

test cases within the QA report. However, the information and detailed steps are available for 

users to conduct verification exercises. Throughout the document, the term validation is often 

used with the verification cases. LeachXS has several capabilities, including predicting 

laboratory tests, where the term “validation” may be appropriate. However, with respect to 

leaching of field-emplaced cementitious materials, the term “validation” is misleading. Additional 

information is needed to provide confidence that the model adequately represents the real 

world. Several recommendations are provided below related to the use of test methods for the 

prediction of future releases.
- 20 -

LeachXS relies in large part on the results of a variety of test methods, which provide 

measurements of chemical properties, chemical reactions, release mechanisms (e.g., matrix 

diffusion, surface run-off/rinse release, dissolution) and represent a variety of environmental 

conditions. The results of leaching test methods and field data are valuable lines of evidence to 

support an understanding of future field releases. However, additional information regarding the 

use of the test methods would provide additional support, including:

• Discussion of the limitations of using test methods to represent initial field conditions 

(i.e., Do the test methods adequately represent initial conditions associated with field 

materials?). This should include discussion of any differences between laboratoryprepared and field-emplaced materials, which could result in differences in physical and 

chemical properties. Documentation should address differences between: (i) simulants 

and actual materials, (ii) laboratory sample preparation and field production, (iii) curing 

conditions, and (iv) laboratory and field boundary conditions.

• Discussion of the limitations of using laboratory test methods to represent future field 

releases (e.g., future potential changes in release rate and mechanism). This should 

include information on scaling from laboratory test methods to future field releases with 

respect to dimensionality and release rates/mechanisms. Laboratory test method results 

may not be representative of future field releases for multiple reasons, including: (i) 

different reactivity of laboratory samples versus field samples due to laboratory sample 

preparation, (ii) positive or negative feedback due to the formation of secondary phases 

that may occur in the field, but might not occur under laboratory conditions and 

timeframes, (iii) changes in hydraulic conductivity over time leading to an advectivecontrolled release versus diffusive due to shorter timeframes used in the laboratory, and 

(iv) changes in diffusive length over time due to cracking which may not occur in the 

laboratory due again to the shorter timeframes used. For reference, three recent NRC 

NUREG guidance documents (NUREG/CR-7025, NUREG/CR-7105, and NUREG/CR7199) provide guidance on conceptual models of leaching, the relationship between 

laboratory tests and field leaching, and testing protocols. 

• Discussion of the ability of LeachXS to accurately represent test method results. For 

several species, the test method leaching results provided in the validation data appear 

to vary by more than an order of magnitude from the model projections. Verification and 

validation documentation should address why this constitutes a passed test. The 

uncertainty in LeachXS should be clearly communicated to the user. 

• Information on how multiple data sets are reconciled in LeachXS, including multiple 

different test method results and/or field data. This discussion should include any 

limitations associated with the aggregation of multiple data sets.

II.2 Verification and Validation – ORCHESTRA

The LeachXS test results are coupled with equilibrium geochemical modeling using 

ORCHESTRA (Objects Representing CHEmical Speciation and TRAnsport), which is supported 

by a thermodynamic database of geochemical reactions, including dissolution/precipitation 

reactions, adsorption/desorption reactions, oxidation/reduction reactions, and solution phase 

complexation reactions. LeachXS/ORCHESTRA models cementitious materials as a network of 

interconnected cells with diffusive and advective transport of chemical species between cells, 

while maintaining chemical equilibrium between mineral phases and the pore solution. 

Currently, LeachXS/ORCHESTRA is designed to model sulfate attack, carbonation, oxidation, 
- 21 -

and leaching. The material transport properties can be subsequently updated by a damage 

calculation for modeling sulfate attack. 

NRC Staff Review of Verification and Validation – ORCHESTRA

In CBP-TR-2015-001-1, Meeussen and Brown (2015) provide several verification test cases. 

Verification Test Problems 1, 2, and 3 were relatively simple tests that evaluate a limited 

functionality of ORCHESTRA. A more comprehensive international benchmarking study (Marty 

et al., 2015) was conducted for model verification of several reactive transport models 

(TOUGHREACT, PHREEQC, CRUNCH, HYTEC, ORCHESTRA, and MIN3P-THCm). The 

benchmarking study includes the following processes: diffusional transport in saturated media 

under isothermal conditions, cation exchange reactions, and both local equilibrium and 

kinetically-controlled mineral dissolution precipitation reactions. Feedback between 

mineralogical evolution resulting in porosity changes and the subsequent changes to the 

diffusion coefficient were not considered in the benchmarking exercise. The authors noted that 

these inter-code comparisons: (i) are the most efficient method for assessing code capabilities, 

(ii) build model confidence, and (iii) demonstrate that reactive transport modeling for long-term 

performance assessment can be consistently addressed by different reactive transport models. 

Although the model results are not compared against analytical solutions, due to the complexity 

of the system geochemistry, all of the models predict very similar solute concentrations and 

mineral distributions. 

For the functions of ORCHESTRA that were evaluated in these test cases, the NRC staff 

agrees that the results provide strong model-to-model verification and that different reactive 

transport models are capable of producing consistent results. The NRC staff also agrees that 

this type of verification exercise builds confidence in the models, and this verification is 

especially useful for predictions for long times, which are difficult or impossible to validate in the 

traditional sense (i.e., by letting a physical system run for thousands of years and comparing the 

model predictions to the physical system). The results of this verification exercise should be 

evaluated against a requirements document to demonstrate that the software is correctly 

performing all of the intended functions. 

The ORCHESTRA Software QA document appropriately categorizes this study as a verification 

exercise. However, the NRC staff urges caution in using some of the discussion from Marty et 

al. (2015), which could be incorrectly interpreted by users as model validation. For example, in 

the conclusions section on page 96 of the ORCHESTRA Software QA document, the following 

quote from Marty et al. (2015) is cited:

“Critical analysis demonstrated the robustness of the simulated results regarding the 

geochemical evolution at the cement/clay(stone) interface.”

The NRC staff notes that users could incorrectly interpret that statement as addressing the 

confidence in the accuracy of the model projections. As a benchmarking study, the authors 

aligned critical model assumptions (e.g., conceptualization of the problem, thermodynamic 

database, mesh, key parameters such as the molecular diffusion coefficient, kinetic data, and 

reactive surface area). Marty et al. then showed that each of the reactive transport models 

produce similar results, providing verification of the models. However, that is the limit of the 

study by Marty et al. and the similarity in the results is only a reflection of the similarities in the 
- 22 -

underlying calculations. Each of these model assumptions contain uncertainty, as well as 

uncertainty in the conceptual model, that would not be captured or illustrated from this type of 

study. The NRC staff recommends that the CBP provide additional context for the quote from 

Marty et al. so that it is clear to users that the precision in the model results support model 

verification, but these similarities do not provide confidence in the accuracy of these model 

projections. 

In CBP-TR-2015-001-1, Meeussen and Brown (2015) provided validation tests to evaluate 

sulfate attack and carbonation. As discussed by Meeussen and Brown, the sulfate validation 

test relied on a numerical analysis developed by Sarkar et. al. (2010) to simulate degradation of 

cementitious materials under external sulfate attack, including mechanical damage and 

cracking. As discussed under General Limitations of the CBP Toolbox Software above and in 

more detail in the Appendix to this report, the NRC staff has concerns related to support for 

several assumptions related to sulfate attack. In addition, NRC staff notes that the 

ORCHESTRA software was calibrated using calcium and sulfur profiles from a SIMCO study 

after three months and validated using the profiles obtained after six months and 12 months. 

The NRC staff has two concerns regarding this approach: (i) the calibration and validation data 

are not independent and (ii) the ORCHESTRA software is calibrated and validated to include 

mechanical damage and cracking using these data. Although the SIMCO data provided calcium 

and sulfate profiles for model calibration, the test samples did not appear to have undergone 

mechanical degradation and cracking. Accordingly, this data would not be appropriate for 

calibrating or validating mechanical damage and cracking. 

For the second validation test, Meeussen and Brown (2015) compared the measured 

carbonation depth in a dome core sample from a buried tank (241-C-107) in the Hanford Waste 

Management Area C Tank Farm against the calculated carbonation depth from ORCHESTRA. 

The comparison of actual field data against model simulations is the type of study that the NRC 

staff believes is needed to provide confidence in model results. 

The NRC staff has several recommendations to provide additional confidence from this study. 

• First, the key to this analysis is the assumption of an intact disposal structure. The dome 

core sample appears to have been intact, however, observations of disposal structure 

concrete indicate that cracking is common and may control degradation. For the 

purposes of validation of diffusive transport through an intact matrix, the C-107 test case 

appears to be appropriate. However, carbonation-induced corrosion could occur in a 

much shorter period of time if advective transport occurs through cracks. For the 

purposes of long-term projections of concrete performance, additional analysis 

considering fast pathways is needed. 

• Second, more information on the selection of parameters for this validation case would 

be helpful to verify that this test problem is consistent with model validation rather than 

model calibration (i.e., to verify that parameters were not iterated to match model results 

with the observational data). 
- 23 -

• Third, the CBP should conduct a literature search to gather additional field studies for 

similar validation exercises under a range of environmental conditions. This would 

require information on the boundary conditions for the field cases and appropriate 

changes to the model would similarly be required. These types of validation studies 

could significantly enhance confidence in model projections. 

Oxidation was recently added to the carbonation model in ORCHESTRA. Figure G-5 in CBPTR-2015-001-1 illustrates a close similarity between the projected progression of the oxidation 

and carbonation fronts. However, verification and validation information included in the report 

by Meeussen and Brown (2015) is very limited. 

II.3 Verification and Validation – STADIUM

STADIUM models reactive transport for intact cementitious materials exposed to aggressive 

environmental conditions under saturated and unsaturated conditions (Samson, 2015). The 

software is designed to simulate the ingress of deleterious species into the concrete matrix, as 

well as leaching of species from the pore solution into the environment as a consequence of the 

following cementitious degradation processes: chloride ingress, sulfate attack, and carbonation. 

STADIUM is a finite-element numerical model, which separately solves the transport equations 

and chemical equilibrium reactions, as shown in Figure 6 of the QA report (Samson, 2015; 

Samson and Marchand, 2007). The transport of chemical species is described by the NernstPlanck equation and accounts for electrical coupling between ionic fluxes, chemical activity 

effects, and transport of species due to water content, gradient, and temperature effects. After 

solving the transport equation, STADIUM calculates chemical equilibrium based on the updated 

pore solution concentrations, mineral phase contents, and temperature. The assumption of 

local equilibrium between the pore solution and the solid phases is enforced at each node of the 

finite-element mesh. Solid phases are dissolved or precipitated to satisfy the condition of local 

equilibrium, which assumes that the chemical reactions are fast relative to the transport rates. 

Transport properties are then subsequently modified to reflect changes in material properties 

(e.g., tortuosity, porosity, permeability) due to the evolution of chemical conditions.

NRC Staff Review of Verification and Validation – STADIUM

The STADIUM QA Report by Samson (2015) is well written with the calculations, verification 

and validation tests, and associated references clearly documented. The QA report includes 

documentation by reference, often to papers published by members of the STADIUM team, thus 

providing a level of peer review. NRC staff also notes the maturity of the software as STADIUM 

has been tested under a range of conditions for a large variety of applications as it has been 

used on more than 150 projects and is the model specified by the U.S. Department of Defense 

to evaluate the service-life of concrete waterfront structures. 

The mathematical model is clearly described in Section 4 of the STADIUM QA report with the 

transport equations, chemical equilibrium model, and associated references well documented. 

Section 5 summarizes the numerical implementation of the STADIUM software with references 

that provide additional details and support. In Section 6, a series test cases were used to verify 

the implementation of the transport and chemical equilibrium modules. Verification included 

comparisons of model results against analytical solutions and data derived from literature. The 

results showed that the model correlates well with the various data. 
- 24 -

Section 7 of the Stadium QA report provides details of test methods that were used to determine 

porosity, diffusivity, permeability, and relative permeability, as well as tests used to validate 

modeled chloride and sulfate ingress, decalcification, and unsaturated transport. Comparison of 

the test results, such as elemental profiles in the solid, to modeled results were generally with 

good correlation and provide validation of the model. However, there was no validation of the 

damage model due to sulfate attack which may be an important process for long-term 

performance. As discussed under General Limitations of the CBP Toolbox Models above and in 

more detail in the Appendix to this report, the NRC staff has concerns related to support for 

several assumptions related to the modeling of damage due to sulfate attack.

STADIUM considers moisture transport due to capillary action, but does not consider gravitydriven flow based on the low permeability of cementitious materials. SIMCO’s ASTM test 

method results provide reasonable validation for the intrinsic permeability of intact concrete. 

However, the NRC staff is concerned that gravity-driven flow through degraded cementitious 

materials (e.g., decalcified, cracked) could be significant, especially over the long time frames of 

interest. In addition, gravity-driven flow could invalidate the assumption of local equilibrium if 

transport is fast relative to the rate of reaction. Further support should be provided to

demonstrate that gravity-driven moisture transport through cementitious materials is not risk 

significant for the long time frames of interest.

Lastly, longer-term field observations could provide users with additional information and model 

confidence. Although materials in the field are subject to complex boundary conditions, which 

may have to be abstracted in the model, comparisons of model results against field 

observations can provide additional insight. For example, comparisons with field observations 

can indicate whether the model can capture the first and second-order system behavior or if the 

model is missing real world behaviors that have a significant effect on reactive transport.

II.4 Verification and Validation – Other Software

SRNL recently developed a software module, FloXcel, for the CBP Toolbox, which is designed 

to simulate flow in degraded and fractured cementitious materials (G-SQP-G-00017; SRNL-STI2015-00190). The module contains a database of material properties for intact cementitious 

and granular materials. Using a GoldSim GUI, users can specify the fraction of the granular 

materials or the fracture aperture and spacing. Based on the user specifications, FloXcel will 

calculate effective hydraulic properties for a composite material to simulate flow through 

degraded cementitious materials. FloXcel can be used to support the LXO Percolation with 

Radial Diffusion model in the CBP Toolbox.

NRC Staff Review of Verification and Validation – Other Software

Cementitious materials often contain complex features, such as fractures and interfaces (e.g., 

cold joints, piping and tie rods extending in and out of the structures), which increase the 

likelihood of advective transport. Accordingly, the development of a module to simulate flow in 

degraded and fractured cementitious materials represents a significant advance for the CBP 

Toolbox. 
- 25 -

Verification of the FloXcel calculations were provided in G-SQP-G-00017. Validation exercises 

should be developed to demonstrate that FloXcel adequately represents flow through degraded 

cementitious materials. These tests should include comparison of model results against 

observations of flow and transport of deleterious species through a range of degraded 

cementitious materials under a range of conditions (e.g., steady state and episodic events; low, 

moderate, and high degrees of saturation). 

In addition, because transport tends to be slow relative to the relevant chemical rates of 

reaction, STADIUM and ORCHESTRA assume that a local equilibrium exists between the pore 

solution and various solid phases. This assumption may be challenged for flow and transport 

through degraded cementitious materials. Validation exercises should also include 

development of support for the assumption of local equilibrium. 

Conclusions

The NRC staff performed a technical review of the QA documentation related to Version 3.0 of 

the CBP Toolbox. The CBP Toolbox provides a GUI to set up and run simulations associated 

with cementitious material durability. The main software components integrated into the CBP 

toolbox are STADIUM and LeachXS/ORCHESTRA. The purpose of the review was to: (i) 

evaluate the software QA applied to development of the CBP Toolbox, and (ii) review the 

technical basis for the main software components. In some software, the technical basis is 

included within the scope of software quality assurance. In this review, the technical basis is 

discussed in a stand-alone section because of the depth of the material covered.

While in some areas of the CBP Toolbox the QA was relatively complete, the NRC staff 

identified QA gaps and limitations in the documentation associated with software development 

and in the technical basis for the conceptual and mathematical models for the CBP Toolbox. 

Several of these gaps could be addressed by providing additional discussion within existing 

documentation (e.g., clarification on the capabilities and limitations of the software). However, 

addressing several other limitations would require more significant resources, including:

• incomplete set of degradation mechanisms and limited coupling of mechanisms,

• insufficient support for modeling damage due to sulfate attack, and

• limited validation exercises.

The NRC staff appreciates that many of the limitations discussed in this report are a function of 

the ongoing and challenging nature of the CBP project. The NRC staff believes that the CBP 

has the capabilities to resolve these concerns. However, without additional resources to 

complete the CBP Toolbox as originally intended, the utility of the software is somewhat limited. 

For instance, software users would need to conduct additional analyses to demonstrate that 

modeled degradation would not be increased due to degradation mechanisms that are not 

currently included in the CBP Toolbox or due to the coupling and synergism of multiple 

degradation mechanisms. 
- 26 -

References

Basista M, and W. Weglewski, 2008. Micromechanical Modelling of Sulphate Corrosion in 

Concrete: Influence of Ettringite Forming Reaction, Journal of Theoretical and Applied 

Mechanics 35, 29-52.

Brown, K.G., Flach, G.P., and F.G. Smith, 2013. GoldSim Interface for the CBP Software 

Toolbox, Version 2.0 User Guide, CBP-TR-2013-004-1, Rev. 0, Vanderbilt University, Nashville, 

TN, August 2013.

Cementitious Barriers Partnership, 2015. Software Quality Assurance Implementation 

Procedure, CBP-RP-2015-017, Rev. 0, November 2015.

Ebert, W.L., 2010. Radionuclide Release from Slag and Concrete Waste Materials Part 1: 

Conceptual Models of Leaching from Complex Materials and Laboratory Test Methods. U.S. 

Nuclear Regulatory Commission report NUREG/CR-7025.

Ebert, W.L., 2011. Radionuclide Release from Slag and Concrete Waste Materials Part 2: 

Relationship between Laboratory Tests and Field Leaching. U.S. Nuclear Regulatory

Commission report NUREG/CR-7105.

Ebert, W.L., 2015. Radionuclide Release from Slag and Concrete Waste Materials Part 2: 

Testing Protocol. U.S. Nuclear Regulatory Commission report NUREG/CR-7199.

Kunter, W., 2012. Investigation of Sulfate Attack by Experimental and Thermodynamic Means. 

Thèse No 5263. École Polytechnique Fédérale de Lausanne.

Lagerblad, B., 1999. Long Term Test of Concrete Resistance Against Sulphate Attack. Materials 

Science of Concrete: Sulfate Attack Mechanisms, J. Marchand and J. P. Skalny, eds., American 

Ceramic Society, Westerbrook, Ohio, 325–336.

Kunther, W., Lothenbach, B., and K. Schrivener, 2013. On the Relevance of Volume Increase 

for the Length Changes of Mortar Bars in Sulfate Solution. Cement and Concrete Research, 46, 

23-29.

Maltais Y., Marchand J., and E. Samson, 2004. Predicting the Durability of Portland Cement 

Systems in Aggressive Environments – Laboratory Validation. Cement and Concrete Research, 

34, 1579-1589.

Marty, NCM, Bildstein, O., Blanc, P., Claret, F., Cochepin, B., Gaucher, E.C., Jacques, D., 

Lartigue, J-E, Liu, S., Mayer, K.U., Meeussen, J.C.L., Munier, I., Pointeau, I., Su, D. and C.I. 

Steefel, 2015. Benchmarks for Multicomponent Reactive Transport Across a Cement/Clay 

Interface. Computational Geosciences, Volume 19, Issue 3, 635-653.

Meeussen, J.C.L., 2015. ORCHESTRA Manual and Documentation, CBP-TR-2015-00X-Y, Rev. 

C (DRAFT for CBP review), NRG, Petten, The Netherlands, December 2015.

Meeussen, J.C.L. and K.G. Brown, 2015. ORCHESTRA Software Quality Assurance 

Documentation, CBP-TR-2015-001-1, Rev. F, December 2015.
- 27 -

Protière, Y. and E. Samson, 2015. Task 12 – Experimental Studies: SCM Paste Samples 

Exposed to Aggressive Solutions, Cementitious Barriers Partnership.

Samson E. and J. Marchand, 2007. Modeling the Transport of Ions in Unsaturated CementBased Materials, Computers and Structures 85, 1740-1756.

Samson, E. and S. Sarkar, 2009. Mechanical Damage Review. Cementitious Barriers 

Partnerships, CBP-TR-2009-002.

Samson, E., 2015. STADIUM Calculation Core 2015b - STADIUM Quality Assurance Report, 

SIMCO Technologies, Quebec, Canada, December, 2015.

Sarkar, S., Mahadevan, S., Meeussen, J.C.L., van der Sloot, H.A., and D.S. Kosson, 2010. 

Numerical Simulation of Cementitious Materials Degradation Under External Sulfate Attack,’ 

Cement and Concrete Composites, Volume 32, Issue 3, 241-252.

Seignette, P., 2015. LeachXS QA Documentation, ECN-X-15-123, Version 2.0.x DRAFT, ECN, 

Petten, The Netherlands, November 2015.

Smith, F.G. and G.P. Flach, 2015. Development and Demonstration of Material Properties 

Database and Software for the Simulation of Flow Properties in Cementitious Materials. SRNLSTI-2015-00190, SRNL, Aiken, SC, March, 2015

Smith, F.G. and G.P. Flach, 2015. Software Quality Assurance Plan: CBP Toolbox GoldSim 

Dashboard Graphical User Interface. G-SQP-G-00018, Revision 0, SRNL, Aiken, SC, 

September 2015.

Smith, F.G. and G.P. Flach, 2015. Software Quality Assurance Plan: FloXcel, G-SQP-G-00017, 

Revision 0, SRNL, Aiken, SC, September 2015.

Smith, F.G., Flach, G.P., and K.G. Brown, 2010. CBP Code Integration GoldSim DLL Interface, 

CBP-TR-2010-009-2, Rev. 0, Vanderbilt University, Nashville, TN, June 2010.

Tixier, R. and B. Mobasher, 2003. Modeling of Damage in Cement-Based Materials Subjected 

to External Sulfate Attack. II: Comparison with Experiments. Journal of Materials in Civil 

Engineering, July/August, 2003, 314-322.

U.S. Nuclear Regulatory Commission. NUREG/CR-4369, Quality Assurance (QA) Plan for 

Computer Software Supporting the U.S. Nuclear Regulatory Commission’s High Level Waste 

Management Program. Washington, DC: NRC, January, 1986.

Vaughan, J.P., 2008. Quality Assurance Program, Cement Barriers Partnership (CBP), CBP-01, 

SRNL, Aiken, SC, December 2008.
- 28 -

Appendix: Concerns Related to Sulfate Attack for ORCHESTRA and STADIUM 

The modeling of damage due to sulfate attack in ORCHESTRA and STADIUM is based on 

diffusive transport of ions and the formation of expansive mineral phases. After filling capillary 

pores in the material, these expansive phases can lead to cracking and increased transport of 

deleterious species. The NRC staff considers the diffusive transport of sulfate through intact 

cementitious materials and formation of ettringite to be well-supported. However, support is 

more limited for the basis and parameterization associated with the assumed volume change 

due to the formation of expansive mineral phases. Note that the NRC staff did not review the 

subsequent damage model components (i.e., development of strain based on those volume 

changes, relationship between strain and the cracking of cementitious materials, and the 

resultant effective diffusion for cracked cementitious materials).

First, it is not clear to the NRC staff if the volume change associated with sulfate ingress can be 

predicted by thermodynamic calculations and the formation of expansive phases in the pore 

space of cementitious materials. Kunter, Lothenbach, and Schrivener (2013) stated,

The results presented here clearly show there is no simple relation between the 

increase in volume of phases formed by the ingress of sulfate ions and the 

observed expansion.

The amount of volume increase due to the formation of ettringite, gypsum and/or 

syngenite during the interaction with sulfate solutions has no direct link to the 

amount of expansion observed in different solution or the expansion of different 

binders. While the formation of sulfate containing solids is a prerequisite for 

expansion, the supersaturation in solution and thus the force exerted by the 

forming mineral seems to be decisive.

Additional information is needed to support the use of volume increase based on equilibrium 

thermodynamic calculations as the basis for modeling damage due to sulfate attack.

Second, if support for the basis of volume increase due to calculated mineralogical changes can 

be developed, then the NRC staff has additional concerns related to model parameterization 

based on the research by Tixier and Mobasher (2003):

1. The applicability of results from Tixier and Mobasher to other cementitious systems is 

not adequately supported. Tixier and Mobasher calibrated their model based on data 

from Lagerblad (1999). Lagerblad tested six samples: three samples with different 

water-to-cement ratios for two different cement types – ordinary Portland cement and 

sulfate-resistant Portland cement. (Note that only four of these six samples have values 

reported for the fractional porosity.) These samples demonstrated that expansion is 

sensitive to water-to-cement ratios, cement type, and boundary conditions. Without 

material-specific formulations and appropriate boundary conditions, it is not clear how 

applicable the data from Tixier and Mobasher would be for predicting expansion in 

different concrete formulations and boundary conditions.

If a generalized model for different cementitious formulations cannot be determined 

based on existing data, then the CBP could conduct experiments on specific 
- 29 -

formulations to calibrate and validate the damage model for different cementitious 

materials. For materials outside of this envelope of conditions, the CBP may 

recommend users develop this information for each unique set of formulations and 

conditions.

2. The use of the fractional capillary porosity, f, as an independent parameter in 

ORCHESTRA and STADIUM is not adequately supported. Tixier and Mobasher used 

fractional porosity as a fitting parameter to match experimental data from Lagerblad. 

However, the fractional porosity is only one of several fitting functions that Tixier and 

Mobasher used to match the experimental data to the model. The other fitting 

parameters were diffusivity, tensile strength, and initial modulus of elasticity. In addition, 

the cement type (i.e., C3A content) was also critical to fitting the model. The fractional 

porosity value might only be meaningful when used in conjunction with the other fitting 

parameters and then only for the limited set of Lagerblad’s specimen. It is not clear that 

selecting a fractional porosity based on the research from Tixier and Mobasher and used 

in conjunction with separate material properties is meaningful. 

3. The fractional porosity value that is used in STADIUM and the range of values used in 

ORCHESTRA does not appear to be adequately supported. In STADIUM, the fractional 

porosity is assumed to be 0.5, based on research by Basista and Weglewski (2008). 

Basista and Weglewski refer to “experimentalists” who reported a fractional porosity of 

less than 0.5 but did not provide a reference. If Basista and Weglewski are referring to 

Tixier and Mobasher (2003), who reported values ranging from 0.05 to 0.45, then 

fractional porosity values closer to 0.5 are only appropriate for ordinary Portland cement. 

Tixier and Mobasher discussed that a fractional porosity value of 0.05 was necessary for 

fitting purposes for sulfate-resistant Portland cement (see Discussion section in the 

report by Tixier and Mobasher, 2003). The difference between a fractional porosity of 

0.5 versus 0.05 for sulfate-resistant Portland cement could significantly affect model 

results. In ORCHESTRA, the fractional porosity can be varied by the user, but limited 

information is available regarding the extent of verification and validation that has been 

conducted and the values that are appropriate for different materials and conditions. 

4. Sarkar et al. (2010) discussed the simplifying assumption that the formation of minerals 

associated with sulfate ingress are distributed homogenously throughout the cement 

matrix. Realistically, the filling of pores may occur as a moving front with sulfate ingress 

and degradation progressing in a series of spallation events. It is not clear how this 

simplifying assumption in the CBP Toolbox could impact long-term concrete degradation 

calculations.

In summary, while the sulfate attack simulations may be considered state-of-the-art, significant 

uncertainties remain. Unless the uncertainties discussed above are explicitly included in the 

simulations, the results of the simulations may be inaccurate. Additional experimentation is 

needed to develop better understanding, refine uncertain parameters, and validate the 

underlying conceptual models.