NUREG/CR-5379

PNL-7916

Vol. 2

Nuclear Plant

Service Water System

Aging Degradation Assessment

Phase II

Prepared by

D. B. Jarrell, L L Larson, R. C. Stratton, S. J. Bohn, M. L Gore

Pacific Northwest Laboratory

Operated by

Battelle Memorial Institute

Prepared for

U.S. Nuclear Regulatory Commission
r

AVAILABILITY NOTICE

Avalabirity of Reference Materials Cited In NRC Publications

Most documents cited In NRC publications wEI be available from one of the following sources:

1. The NRC Pubic Document Room, 2120 L Street. NW., Lower Level. Washington. DC 20555

2. The Superintendent of Documents. U.S. Government Printing Office. P.O. Box 37082. Washington,

DC 20013-7082

3. The NatIonal Technical Information Service. Springfield, VA 22181

Although the Dsting that follows represents the majority of documents cited In NRC publications, It Is not

Intended to be exhaustive.

Referenced documents available for Inspection and copying for a fee from the NRC Public Document Room

Include NRC correspondence and Internal NRC memoranda; NRC bulletins, circulars, Information notices.

Inspdction and Investigation notices: licensee event reports; vendor reports and correspondence: Commisalon papers: and applicant and licensee documents and correspondence.

The following documents In the NUREG series are available for purchase from the GPO Sales Program:

formal NRC staff and contractor reports, NRC-sponsored conference proceedings, International agreement

reports, grant publications, and NRC booklets and brochures. Also available are regulatory guides, NRC

regulations In the Code of Federal Regulations, and Nuclear Regulatory Commission Issuances.

Documents available from the National Technical Information Service Include NUREG-serles reports and

technical reports prepared by other Federal agencies and reports prepared by the Atomic Energy Commisslon, forerunner agency to the Nuclear Regulatory Commission.

Documents available from public and special technical libraries Include all open literature Items, such as

books. Journal articles, and transactions. Federal Register notices, Federal and State legislation, and congressional reports can usually be obtained from these libraries.

Documents such as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings are available for purchase from the organization sponsoring the publication cited.

Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the

Office of Administration, Distribution and Mail Services Section. U.S. Nuclear Regulatory Commission.

Washington. DC 20555.

Copies of industry codes and standards used In a substantive manner In the NRC regulatory process are

maintained at the NRC Library, 7920 Norfolk Avenue. Bethesda. Maryland, for use by the public. Codes and

standards are usually copyrighted and may be purchased from the originating organization or, If they are

American National Standards, from the American National Standards Institute, 1430 Broadway, New York,

NY 10018.

DISCLAIMER NOTICE

This report was prepared as an account of work sponsored by an agency of the United States Government

Neither the United States Government nor any agency thereof, oranyof their employees, makes any warranty,

expressed or Impiled, or assumes any legal liability of responsibility for any third party's use, or the results of

such use, of any Information, apparatus, product or process disclosed In this report, or represents that Its use

by such third party would not Infringe privately owned rights.
NUREG/CR-5379

PNL-7916

Vol. 2

RM, R9

Nuclear Plant

Service Water System

Aging Degradation Assessment

Phase II

Manuscript Completed: July 1992

Date Published: October 1992

Prepared by D. B. Jarrell, L L Larson, R. C. Stratton, S. J. Bohn, M. L Gore

Pacific Northwest Laboratory Richland, WA 99352

Prepared for

Division of Engineering

Office of Nuclear Regulatory Research

U.S. Nuclear Regulatory Commission

Washington, DC 20555

NRC FIN B2911
Abstract

The second phase of the aging assessment of nuclear plant service water systems (SWSs) was performed by the Pacific

Northwest Laboratory (PNL) to support the U.S. Nuclear Regulatory Commission's (NRCs) Nuclear Plant Aging

Research (NPAR) program. The SWS was selected for study because of its essential role in the mitigation of and

recovery from accident scenarios involving the potential for core-melt, and because it is subject to a variety of aging

mechanisms. The objectives of the SWS task under the NPAR program are to identify and characterize the principal

age-related degradation mechanisms relevant to this system, to assess the impact of aging degradation on operational

readiness, and to provide a methodology for the management of aging on the service water aspect of nuclear plant

safety.

The primary degradation mechanism in the SWSs, as stated in the Phase I assessment and confirmed by the analysis in

Phase I, is corrosion compounded by biologic and inorganic accumulation. It then follows that the most effective

means for mitigating degradation in these systems is to pursue appropriate programs to effectively control the water

chemistry properties when possible and to use biocidal agents where necessary.

A methodology for producing a complete root-cause analysis was developed as a result of needs Identified in the

Phase I assessment for a more formal procedure that would lend itself to a generic, standardized approach. It is recommended that this, or a similar methodology, be required as a part of the documentation for corrective maintenance

performed on the safety-related portions of SWSs to provide an accurate focus for effective management of aging.

III NUREG/CR-5379
Summary

The goal of the Service Water System (SWS) Aging Degradation Assessment task was to advance the understanding

and management of the technical safety issues relating to the aging of SWSs in operating commercial nuclear power

plants.

A Phase I interim assessment report, with the primary goal of understanding the aging process in SWSs, was completed

in June 1989 (NUREG/CR-5379). The report concluded that the principal mechanism leading to SWS degradation

and failure is corrosion compounded by biologic and inorganic accumulation. This conclusion was based on in-depth,

single plant, open system (once-through) plant information and was verified using opinions of subject experts. The

report also pointed out that existing performance measures performed in compliance with the plant technical specifications were not capable of effectively detecting aging degradation prior to failure; nor were the existing diagnostics

capable of providing adequate Identification of the stressor(s) responsible for the rootcause of degradation or failure.

The Phase II comprehensive aging assessment was performed to substantiate the Phase I conclusions, as well as to

explore suitable methods of monitoring and managing the aging of the nuclear plant SWS. The Phase I in-depth database was extended with similar data from a closed SWS type (intermediate heat exchanger), and a recirculating (spray

pond) plant to provide reasonable assurance that all common degradation mechanisms and stressors had been

reviewed. The analysis of the extended Phase II database and a review of the comprehensive NRC-AEOD assessment

of SWS degradations and failures (NUREG-1275, Vol. 3) show the Phase I conclusions, relative to corrosion being the

degradation mechanism primarily responsible for SWS failures, to be valid. This conclusion suggests that the key to

SWS aging mitigation lies in the control of service water quality and chemistry. In systems with limited chemistry control (such as in the open system), the most effective efforts to prevent aging failures are targeted toward:

* understanding the active degradation mechanisms at the specific site'

* selecting system materials to minimize the impact of the known degradation mechanisms

* using system designs that allow thorough periodic cleaning

* developing a comprehensive condition monitoring program to detect component degradation early enough to

allow planned maintenance actions.

The issues regarding management of SWS aging have received considerable attention from the U.S. Nuclear Regulatory Commission (NRC). With the implementation of the Generic Letter 89-13 on SWS problems affecting safetyrelated equipment, utilities are required to conduct component performance tests and inspections and provide chemical biofouling control. The work summarized in this and other Pacific Northwest Laboratory (PNL) reports provided

the basis for, and input to, the formulation of Generic Letter 89-13.

The final product of the Phase HI effort Is the development of a generic root-cause analysis methodology. This report

recommends that this method, or a similar root-ause analysis process having the elements described herein, be performed for all corrective maintenance required on safety-related portions of the nuclear plant SWS. Integrating the

documentation of these analyses will then provide an accurate focus for an effective, plant-specific, SWS aging mitigation program.

v NUREG/CR-5379
Acknowledgments

The authors wish to thank all of the cooperating power plants for providing knowledge, as well as data records on

operations and maintenance of their service water systems. We would also like to thank our sponsor, Dr. J. J. Burns of

the U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, for his encouragement and drive to

spread the knowledge gained through this project to other NRC components. Dr. A B. (Burt) Johnson, Jr., project

manager, provided technical direction and expertise and review of the material. 'he efforts of our editor, Nora Buel,

are also gratefully acknowledged.

vii NUREG/CR-5379
Acronym List

ACWS auxiliary cooling water system

AEOD Office for Analysis and Evaluation of Operational Data

ARCs annunciator response cards

BWR boiling-water reactor

DBA design-basis accident

DBE design-basis earthquake

ECCS emergency core cooling system

EDG emergency diesel generator

EPRI Electric Power Research Institute

ESF emergency safety feature

ESW essential service water

FDT forced draft cooling term

FSAR final safety analysis report

GDC general design criteria

HX heat exchanger

HXFR heat transfer

IPA Integrated Plant Assessment

LERs License Event Reports

LCO limiting conditions for operation

LOCA loss-of-coolant accident

MIC microbiologically-influenced corrosion

MOVs motor-operated valves

MPC maximum permissible concentration

ix ix ~~~~~~NUREGICR
MSIV mainstream isolation valve

M/U make-up (water source)

NPAR Nuclear Plant Aging Research (program)

NRC U.S. Nuclear Regulatory Commission

PNL Pacific Northwest Laboratory

PWR pressurized-water reactor

RCA root-cause analysis

RHR residual heat removal

SAR safety analysis report

SER safety evaluation report

SRP Standard Review Plan

SSW standby service water

SWS service water system

SWWG Service Water Working Group

TS technical specifications

UDC under deposit corrosion

UHS ultimate heat sink

USAR updated safety analysis report

NUREG/CR-5379 x
Contents

Abstract .......................................... iii

Summary . v

Acknowledgments . vii

Acronym List . ix

1 Introduction . . . . . ..................................... 1.1

1.1 Nuclear Plant Aging Research Program Goals 1.1

1.2 System Risk Importance .1.2

1.3 SWS Mask Development Summary .1.2

1.3.1 Understanding Aging .1.2

1.3.2 Managing Aging .1.3

1.3.3 Monitoring Aging .1.6

2 System Definition and Description .2.1

2.1 SWS Definition .2.1

2.2 SWS Descriptions .2.1

2.2.1 Open Once-Through Systems .2.1

2.2.2 Open Recirculating Systems .2.2

2.2.3 Closed Systems .2.2

2.3 SWS Summary .2.3

2.4 System Characteristics .2.4

2.4.1 Conditions and Materials .................... 2.4

2.4.2 Component Function and Process Flow Path .2.5

3 Data Acquisition and Compilation .3.1

3.1 Data Sources .3.1

3.1.1 Database Information .3.1

3.1.2 Site Visits. .. 3.1

3.1.3 Nuclear Regulatory Commission Data 3.1

3.2 Data Acquisition .3.2

3.2.1 Questionnaire Protocol .3.2

3.2.2 Site Information Resources .. 3.2

xi NUREG/CR-5379
3.3 Observation Summary .................................................................... 3.3

3.3.1 Open System Observations ........................................................... 33

3.3.2 Recirculating System Observations .................................................... 3.4

3.3.3 Closed System Observations ............... ........................................... 3.5

4 Root-Cause Methodology . ...................................................................... 41

4.1 lTsk Development . ....................................................................... 4.1

4.1.1 Safety-Related Event Reporting ....................................................... 4.1

4.1.2 Component Fahilure Root Cause ....................................................... 4.3

4.2 Development of a Generic Methodology ......................... ........................... 4.3

4.2.1 Root-Cause Process Approach ........................................................ 4.5

4.2.2 Root-Cause Analysis Methodology .................................................... 4.5

4.2.3 Basic Failure Categorization .......................................................... 4.8

4.3 Service Water System Diagnostics and Root-Cause

Analysis Case Study ....................................................................... 4.10

4.3.1 Situation Scenario .................................................................... 4.10

4.3.2 Root-Cause Issue Resolution ......................................................... 4.10

4.3.3 Categorization and Conclusions ....................................................... 4.14

5 Degradation Analysis . ......................................................................... 5.1

5.1 Source of Data . .......................................................................... 5.1

5.2 Component History Database .............................................................. 5.1

5.3 Analysis of Plant Data ..................................................................... 5.1

5.3.1 Rilure Classification ................................................................ 5.1

5.3.2 Root-Cause Analysis of Plant Data .................................................... 5.2

5.3.3 Aging Analysis ....................................................................... 5.3

5.3.4 Principal Degradation Mechanisms .................................................... 5.8

5.3.5 Control Procedures .................................................................. 5.8

6 Conclusions . .............................................................................. 6.1

7 References . .............................................................................. 7.1

Distribution ................................................................................. Distr.1

NUREG/CR-5379 xii
FIgures

1.1 Service Water System Absk Development ....................................................... 1.4

1.2 Design Bases Regulated Operation ................. ........................................... 1.5

2.1 Functional Service Water System Boundary Definition ........................................... 2.2

2.2a Open Service Water System - Once-Through System ..................... ........................ 2.3

2.2b Open Service Water System - Recirculating System ...................... ........................ 2.4

2.3 Closed Service Water System .................................................................. 2.5

2.4 Service Water Intake Structure ..................... ........................................... 2.9

4.1 U.S. Nuclear Regulatory Commission Reportable Events ........................................ 4.2

4.2 Abbreviated NPAR Program Overview .......................................................... 4.4

4.3 Analytical Approach . ....................................................................... 4.6

4.4 Generic Root-Cause Analysis Methodology .................................................... 4.7

4.5 Basic Failure Categorization Scheme .......................................................... 4.9

4.6 Simplified Service Water Flow Schematic ...................................................... 4.11

5.1 Open Service Water System Failure Rate Distribution ........................................... 5.5

5.2 Open Service Water System Component Failure Distribution ..................................... 5.5

5.3 Closed Service Water System Failure Rate Distribution .......................................... 5.6

5.4 Closed Service Water System Component Failure Distribution .................................... 5.6

5.5 Recirculating Service Water System Failure Rate Distribution .................................... 5.7

5.6 Recirculating Service Water System Component Failure Distribution .............................. 5.7

Thbles

2.1 U.S. Nuclear Plant Service Water System Summary .............................................. 2.6

4.1 Heat Exchanger Degradation Hierarchy ........................................................ 4.13

4.2 Root Solution Summary ...................................................................... 4.15

5.1 Component History Database Contents ........................................................ 5.2

5.2 Maintenance Document Categories ........................................................... 5.2

5.3 Root-Cause Analysis Certainty Assignments .................................................... 53

5.4 Service Water System Principal Aging Degradation Mechanisms .................................. 5.4

xiii NUREGICR-5379
1 Introduction

The U.S. Nuclear Regulatory Commission (NRC), Office of Nuclear Regulatory Research, Division of Engineering, is implementing the Nuclear Plant Aging Research (NPAR) Program Plan (NRC 1991) to resolve

technical safety issues related to the aging of commercial

nuclear power plants. Aging, in the context of the characteristics of a system, structure, or NPAR Program

Plan, means gradual changes in the component with

time or use as the result of one or more of the following

factors:

* natural processes during operation

* external stressors caused by storage or operation

* service wear caused by operational cycling

* excessive testing

* improper installation, application, operation or

maintenance.

This report was prepared by Pacific Northwest Laboratory (PNL);(a) it documents the second phase of the Nuclear Plant Service Miter System (SWS) Aging Degradation Assessment task of the NPAR program.

The report summarizes investigations of the safetyrelated portions of the nuclear plant SWS [often designated as essential service water (ESW) or standby service water (SSW)]. During a loss-of-coolant accident

(LOCA) or similar core-threatening, postulated accident scenario, this system is relied upon to transfer heat

from vital plant equipment, such as the residual heat removal (RHR) heat exchangers (HXs) and the emergency diesel generators (EDGs), to the ultimate heat

sink (UHS). This investigation emphasizes identifying

and characterizing the mechanisms of material and component degradation during service, and evaluating methods of inspection, surveillance, condition monitoring

and maintenance to mitigate these effects.

1.1 Nuclear Plant Aging Research

Program Goals

The specific goals of the NPAR program are:

1. Identify and characterize the primary aging mechanism(s) that could cause safety-related component

degradation.

2. Identify methods of inspection, surveillance, and

monitoring, which will ensure timely detection of

significant aging effects prior to loss of safety

function.

3. Evaluate the effectiveness of storage, maintenance,

and replacement practices in mitigating the rate and

extent of aging degradation.

This report documents the second phase of the NPAR

phased approach to system research, as stated in the

NPAR Program Plan (NRC 1991). It describes the results of Phase II for the SWS task and includes information from the following.

* available subject literature

* available generic databases

* utility machinery history

* utility system expert inputs

* commercial expertise

*. expert SWS observations.

This information is analyzed and evaluated to identify

the principal SWS aging mechanisms to provide recommendations for monitoring of aging management

programs.

(a) PNL is operated by Battelle Memorial Institute for the U.S.

Department of Energy under Contract DE-AC06-76RL0 1830.

1.1 NUREGICR-5379
Introduction

1.2 System Risk Importance

The impetus for this study originates from the significant number of documented SWS degradation-related

events that have seriously impaired the ability of this

system to carry out its intended safety function. As previously stated, the system safety function is vital to the

successful termination of many potential core-melt

scenarios. The NRC report on SWS failure and degradation, issued by the Office for Analysis and Evaluation

of Operational Data (AEOD) (Lam and Leeds 1988),

indicated significant SWS system degradation performance with a frequency of 0.4 per reactor year of operation, and a complete system failure (loss of all system

functions) of 1.5 x 10.2 per reactor year. These figures

are among the 10 highest risk factors found in an NRC

study on severe accident risks (NRC 1989). This provides clear corroboration of the risk frequency inferred

by the Phase I study.

Considering the multiple safety functions that are provided by the SWS and the confirmed frequency of partial

or complete functional degradation, the importance of

understanding, managing and monitoring the stressors

and condition of the SWS is considered essential to

minimizing core damage probability.

1.3 SWS Thsk Development Summary

The complete SWS task is depicted in Figure 1.1. The

task consists of understanding the aging phenomena

(Phase 1), managing the aging process and monitoring

for aging degradation (Phase II).

1.3.1 Understanding Aging

The Phase I effort was initiated by a review of the regulatory requirements for nuclear plant SWSs, which resulted in an understanding of the Generic Issues [GSI 51

(Hayes 1983)] and related regulatory concerns for the

subject system. The design bases for system operability

were defined from the General Design Requirements

44,45 and 46 of 10 CFR 50 (1983).

A team of engineers representing component design,

operations and maintenance, corrosion metallurgy, and

statistical analysis was assembled at PNL to perform the

assessment. This team, in conjunction with utility experts and water chemistry consultants, provided the core

of expertise necessary to produce a real-world assessment of the aging stressors that exist in the SWS

environments.

A 1987 review of available literature and database information indicated that motor-operated valve torque

switches were the source of failures in the majority of

SWS failure events. This conclusion was in conflict with

the operating experiences of the PNL team, suggesting

that corrosion was the principal degradation mechanism, and that the torque switch "malfunction* was only

a manifestation of the more basic stressor which produced valve disk to seat corrosion. 'lb resolve the issue,

the assistance of a cooperating utility was enlisted to

provide actual machinery records and personnel recollection for analysis. These accounts contradicted the

dominance of electrical component failures, indicating

that biologic and sedimentary accumulation were contributory mechanisms to the primary degradation mechanism, corrosion. This fundamental conclusion was confirmed by the analysis performed in Phase II, and forms

the premise for the remainder of the study.

The remaining Phase I conclusions were actually the

identification of specific investigative foci on 1) the obvious need for a systematic approach to performing and

documenting the root cause of a component failure

event, and 2) the instrumentation schemes that would be

necessary to detect the progress of degradation phenomena to prevent or mitigate any subsequent failure events.

A Phase I report was prepared to document the methodology and findings of the initial phase of the SWS task

(Jarrell et al 1989). The report was extensively peer

reviewed by participating NPAR laboratories and by the

Electric Power Research Institute (EPRI) Service Water

Working Group (SWWG), a task force from 36 nuclear

utilities. It was focused on resolving the source of

degradation problems encountered during SWS operation. This completed the Phase I basic understanding of

the SWS aging mechanisms.

NUREG/CR-5379 1.2
Introduction

13.2 Managing Aging

Following the focus established by Phase I, the second

phase of the SWS task (refer again to Figure 1.1)

focused on understanding the management of aging

processes from a regulatory perspective.

132.1 Regulatory Implementation Scheme

The nuclear power industry Is regulated by the NRC to

comply with a set of general design criteria (GDC) intended to ensure the protection of the public from the

potential dangers of radioactive release in the event of a

postulated plant accident. The GDCs are the underlying

basis of all nuclear plant operations and maintenance.

GDCs 44,45, and 46 (10 CFR 50 Appendix A) define

the functions and delineate the required component redundancy, inspection, and testing for the required levels

of components important to safety for nuclear plant

SWSs.

These design criteria are subsequently carried (see Figure 1.2) into the updated safety analysis report (USAR),

which is produced for each nuclear power plant to

demonstrate the manner in which the GDCs will be implemented for a specific power plant. With regard to

the SWS, the design safety specifications for each SWS

cooler are generally stated in terms of the required heat

removal rate that must be met under both LOCA and

plant emergency shutdown conditions. The adequacy of

these specifications Is then evaluated by the NRC in a

safety evaluation report (SER), which provides an adequacy judgment for each condition delineated in the

utility's USAR. These two documents, the USAR and

the SER, form the-basis for an agreed-upon envelope of

licensed operating conditions.

The actual implementation of the contractual license

agreement is embodied in the technical specifications

for each plant. These documents translate the requirements for safe operation as defined by the USAR and

approved by the SER into parametric limits for plant

operation and component (and system) surveillance

testing. Since the operational and surveillance limits are

descendants of the GDCs, operation within established

normal operating bands and test results that are within

acceptance criteria should indicate that the plant

operations and maintenance are inside the safety

envelope established by the GDCs.

As indicated by Figure 1.2, the failure of a safety-related

component to fulfill its intended function either during

normal operation or during a surveillance test indicates

a reduced ability to cope with potential accident conditions. Consequently, a timed limit on the duration of

operation under a specific failure condition is established. For component failure conditions which do not

require an immediate cessation of operation, limiting

conditions for operation (LCO) are said to exist. These

LCO time intervals are based on the increased risk of

not being able to successfully mitigate a hypothetical

core-melt scenario because of the degraded functionality

of a component or system.

As Figure 1.2 indicates, all plant operations and maintenance are guided by the GDC through application of

the regulatory structure.

13.2.2 Source of Information

The primary inputs to the Phase II effort consisted of

two additional utility visits, the AEOD report on SWS

failures (Lam and Leeds 1988), and a case study at an

operating boiling-water reactor (BWR) site. These

inputs provided the necessary information from which

to build and partially substantiate the validity of the

Phase l and II products.

The two additional cooperating utility visits provided

validated degradation and failure information to the

SWS database and allowed a single-point examination of

all types of SWS configurations for stressors which

might have otherwise gone unrecognized.

The AEOD failure and degradation compilation provided substantiating data on a more global basis. The

use of Licensee Event Reports (LERs) as input data for

root cause of failure investigations was discarded as

potentially misleading during Phase I of this task

(Murphy et al. 1984). More demanding root-cause evaluation requirements have been added since that publication, and the event compilation presented by the AEOD

correlates well with the independent conclusions of this

assessment.

1.3 NUREG/CR-5379
Introduction

IApplication Development I

I

MONITORING

Figure 1.1 Service Water System Task Development

NURE-G/CR-53791. 1.4
Introduction

Figure 1.2 Design Bases Regulated Operation

1.5 NUREG/CR-5379
Introduction

The final information source was provided through PNL

assistance to regional NRC offices in investigating SWS

events at operating BWR plants. These opportunities

not only provided additional in-depth validated failure

data, but also allowed a "field testi of the root-cause

methodology developed by this task.

13.23 Aging Management Products

wo of the three principal aging management products

(refer to Figure 1.1) are incorporated into this report:

SWS aging control mitigation factors and the failure

root-cause analysis methodology. The SWS Generic

Letter (NRC 1989) on SWS problems affecting safetyrelated equipment has been supported by this task, as

well as other PNL efforts (Daling et aL 1989, Neitzel

and Johnson 1988). Generic Letter 89-13 was published

in the Federal Register (54 FR 43209-10).

133 Monitoring Aging

The NRC's Generic Letter on SWS problems provides

an effective agent to monitor the degradation process in

SWSs. Individual plant records of component conditions and failure histories are the true indicators of the

effect of environment on the SWS. A PNL goal is to

provide continued support of individual plant monitoring efforts and to assist in collective analysis by the NRC

AEOD.

NUREGICR-5379 1.6
2 System Definition and Description

The three safety-related heat sources served by the SWS

in transferring heat loads from various sources in the

plant to the ultimate heat sink are core decay heat, decay heat removal components, and emergency power

sources. The functional definition and design description of the SWS pertaining to heat load transfer are explained in the following sections.

2.1 SWS Definition

Because of the wide variation in each plant's ultimate

heat sink and the application of multiple system design

approaches, the SWS is defined from a functional standpoint. This includes all components, their associated instrumentation, controls, electrical power, cooling and

sealing water, lubrication, and other auxiliary equipment

comprising the final heat transfer loop between the

safety-related heat sources and the ultimate heat sink.

The functional definition of an SWS is shown in Figure 2.1. The dashed boundary in the figure shows the

following range of components considered by this study:

* the intake structure, including canals or other diversion structures from the ultimate heat sink to the intake debris removal mechanism

* the pump galley and structures with all associated

water-level control devices (weirs, gates, valving,

etc.) and instrumentation

* the service water pump, shafting and motive source,

including controls, cabling and electrical

distribution system

* the piping distribution network from the pumps to

the heat exchangers, including all valving, manifolds,

instrumentation, and logic networks

* the service water side of the actual heat exchange

devices themselves

* all discharge piping, valves, and manifolds from the

heat exchangers to the outlet or discharge structure

* the discharge structure, gates, and associated

effluent channeling devices

* where applicable, the ultimate heat source

structures and components.

Only those components specified as essential to reactor

safety [Nuclear Safety Class 3 (NRC 1978b)] and designated Seismic Category I are examined in this Investigation. Seismic Category I is that category requiring that

plant structures, systems, and components be designed

to withstand a design-basis earthquake (DBE).

In addition to the other requirements, safety-related

service water cooling loops in commercial reactors are

also designed to meet the single failure criterion. That

is, redundant components are provided such that the

failure of any single active component in the SWS will

not prohibit the adequate removal of heat from any of

the safety-related loads.

2.2 SWS Descriptions

SWSs can be categorized into a number of groupings;

the grouping method used for this investigation is based

on the nature of the plant's ultimate heat sink. This

method provides a practical division of SWSs into two

broad classifications, open and closed, although the

open SWS classification Is often subdivided into two

configurations, open once-through and open recirculating. The following description of materials, environments, and configurations provides a broad knowledge

of the design and function of SWS components and does

not represent any specific BWR or pressurized-water

reactor (PWR) design.

2.2.1 Open Once-Through Systems

A simplified diagram of a typical open SWS is shown in

Figure 2.2a. This type of arrangement, often referred to

as a straight through. system, is generally characterized

by the availability of a large volume of water as an ultimate heat sink The advantage of this configuration is

its relative simplicity and resultant lower initial cost to

2.1 NUREG/CR-5379
System Definition

REACTOR PLANT

VITAL HEAT LOADS

INDICATION AND

CONTROL SIGNALS

ELECTRIC

POWER

Figure 2.1 Functional Service Water System Boundary Definition

the utility. The lack of a large-capacity intermediate

heat exchanger and no requirement for a secondary set

of component cooling water pumps, as found in the

closed-cycle configuration, make this an attractive layout because it can be installed economically and because

it has fewer components to maintain. The major disadvantage is the potential for problems associated with exposing a large number of components to a potentially

aggressive raw water environment.

2.2.2 Open Recirculating Systems

The recirculating version (Figure 2.2b) has a selfcontained ultimate heat sink, which is frequently

achieved through using a spray cooling pond or a dedicated cooling tower. The advantage of this arrangement

is twofold: 1) through settling and make-up water filtration, the water purity (turbidity) is vastly improved,

leading to significantly reduced siltation in low-flow

velocity areas; and 2) chemical control of the circulated

water is achievable, allowing a reduction in corrosion

and biofouling without the limitations imposed by environmental discharge restrictions.

2.2.3 Closed Systems

The basic distinction between open and closed systems

is the plant personnel's ability to control the coolant

chemistry that comes in contact with the system load

heat exchangers. A simplified closed system, illustrated

in Figure 2.3, resembles the open configuration of Figures 2.2a and b, but adds an intermediate heat

NUREG/CR-5379 2.2
System Definition

Figure 2.2a Open Service Water System - Once-Through System

exchanger to prevent exposure of associated component

load heat exchangers to a raw cooling water

environment

The closed system has been typically sited where corrosive or other hostile environments are known to exist

(such as an ocean salt water source). This system's open

loop is designed to be simple and redundant for on-line

maintenance, and is often provided with the means for

mitigating expected degradation stressors (e.g., thermal

back flush for biofouling control). The actual closed

loop cooling water is chemically pure and pH neutral,

and component degradation can be controlled using corrosion inhibitors.

The closed system is thus more expensive initially and

contains a larger total number of components, but is not

as susceptible to premature aging through extensive corrosion attack within the secondary loop.

23 SWS Summary

A compilation of all active commercial reactors, their

electrical power ratings, source of water used for the ultimate heat sink, and specific SWS configuration is given

in Table 2.1. This listing is primarily sorted by SWS cycle type, with plant names listed alphabetically for each

of the types. A distinction is made between a closed

pond system, which could be controlled via additive

chemicals, and an open pond, which is considered to be

too large for effective chemical control.

2.3 NUREG/CR-5379
System Definition

_ ~~~~~~~Strainer _

UHS 1 .

STOurWER S Water -

I e~pa od Inlet ( < I< I

Make-up J

Source Service Water -- ' X

r

X

*0X_ r~} Emergency

Feed Water

Underground

I- Intake Structure

Emergency 7

Cooling Pond

ACWS

.I I

H Emergency Diesel

Make-Up Pump Room Coolers

Control Room Chillers

___J - - T- - I---

; _~~~~~~~~~~~~~~~~~~~~~~~~~

RHR Heat Exchanger

Containment Spray Pump Cooler

ECCS Injection Pump Cooler

S/D HX Room Cooler

Switchgear Room Coolers

Discharge I 1 3L

l

I

Fuel Pool Heat Exchanger

Containment Cooling Coils

Emergency FW Pump Supply

Component Cooling HXs

Aux. Bldg. Room Coolers

Hydrogen Control System

.-a

Underground

Figure 2.2b Open Service Water System - Recirculating System

2.4 System Characteristics

The conditions and materials, and the component

function and process flow path of the SWS are described

in the following sections.

2.4.1 Conditions and Materials

Service water system materials are subject to a wide

range of environmental conditions depending primarily

on the nature of the ultimate heat sink (UHS). The

UHS sources of cooling or make-up water include:

* sea water -saline, high biological activity, periodic

high particulate and detritus, with a temperature

range of approximately 30"F to 85 0F

* lake water - fresh water, biologically active, low to

moderate particulate and/or detritus, temperature

range 30 0F to 95 0F

* river water - fresh to brackish water, biologically

active, low to very high particulate and/or detritus,

temperature range 30 0F to 100 0F

* coolingpond/tower - fresh water of low turbidity, low

to moderate biological activity, pH-controlled, temperature range 30"F to 95°F

Flow velocities within the system can range from

stagnant (for possibly several months at a time) to

velocities in excess of 40 ft/s at or near throttling

devices.

NUREG/CR-5379 2.4
System Definition

Figure 23 Closed Serice Water System

The predominant construction material for SWSs is carbon steeL Wide variations in materials exist from plant

to plant depending on the degree of agressiveness of the

cooling fluid. Other commonly used materials are copper and copper-based alloys, monel '(a nickel-copper alloy), and stainless steels (300 to 400 series). Copperbase alloys are widely used in heat exchanger applications because of inherently good corrosion resistance

combined with good mechanical properties, excellent

thermal conductivity, and case of soldering or brazing.

Stainless steels, particularly types 304 and 316, are used

increasingly as nickel-base replacement materials to improve corrosion resistance.

2A.2 Component Function andProcess Flow

Path

While construction arrangements, cooling water conditions, and materials used differ widely among plants, the

design of each functional area of all systems is basically the same. The following generic system description is

typical of an open plant configuration.

The intake structure admits water to the system, pro- vides for the necessary degree of debris removal, and

houses the service water pumps and their associated

25 NUREG/CR-5379
System Definition

Table 2.1 U.S. Nuclear Plant Service Water System Summary(a)

Reactor/

Ratine

Service

Water

Plant Name Parent Utility Ultimate Heat Sink Cycle

Clvrt Clfs-1,2

Cook-1

Cook-2

Fermi-2

Fort St Vrain-1

Hope Creek

Limerick-1,2

Maine Yankee

Pilgrim-1

Rancho Seco-i

River Bend-1

Salem-I

Salem-2

San Onofre-1

San Onofre-2

San Onofre-3

Shoreham

St Lucie-1,2

vaterford-3

Diablo Canyn-1

Diablo Canyn-2

Palo Vr-1,2,3

WNP-2

Byron-1,2

Vogtle-1(2)

So Tax Proj-1,2

Big Rock Point

Catawba-1,2

Clinton-1

Davis-Besse-1

Dresden-2,3

Fitzpatrick

Ginna

Kewaunee

Lasalle-1,2

McGuire-1,2

Nine Mile-1

Nine Mile-2

North Anna-1,2

Oconee-1,2,3

Palisades

(CE-845)

(W-1030)

(W-1090)

(GE-1093)

(GA-330)

(GE-1067)

(GE-1055)

(CE-790)

(GE-655)

(B&W-916)

(GE-940)

(W-1090)

(W-1115)

(W-436)

(CE-1070)

(CE-1080)

(GE-819)

(CE-810)

(CE-1165)

(W-1084)

(W-1106)

(CE-1270)

(GE-1100)

(W-1120)

(W-1125)

(W-1250)

(GE-71)

(W-1145)

(GE-955)

(B&W-880)

(GE-794)

(GE-821)

(W-470)

(W-535)

(GE-1078)

(W-1180)

(GE-610)

(GE-1080)

(W-890)

(B&W-860)

(CE-798)

Baltimore Gas & Elec

Indiana & Mich Elec

Indiana & Mich Elec

Detroit Edison Co

Colorado Public Serv

Public Service Elec

Philadelphia Elec Co

Maine Yankee Atomic

Boston Edison Co

Sacramento Muni Util

Gulf States Util

Public Serv Elec NJ

Public Serv Elec NJ

Southern CA Edison

Southern CA Edison

Southern CA Edison

Long Island Lighting

Florida P&L

Louisiana P&L

Pacific Gas & Elec

Pacific Gas & Elec

Arizona Public Serv

WA Pub PWR Supply

Commonwealth Edison

Georgia Power Co

South Texas Project

Consumers Power Co

Duke Power Co

Illinois Power Co

'Ibledo Edison Co

Commonwealth Edison

PWR Auth State of NY

Rochester Gas & Elec

Wisconsin Pub Servic

Commonwealth Edison

Duke Power Co

Niagara Mohawk Power

Niagara Mohawk Power

Virginia Elec & PWR

Duke Power Co

Consumers Power Co

Chesapeake Bay

Lake Michigan

Lake Michigan

Hyperbol/Erie

FDT/R

Hyperbol/Delawar

Hyperbol/Schukil

Back R/ATL 0

Cape Cod Bay

Hyperbol/Canal

FDT/Mississippi

Delaware R

Delaware R

Pacific Ocean

Pacific Ocean

Pacific Ocean

L I Sound

Atlantic Ocean

Mississippi R

Pacific Ocean

Pacific Ocean

Basin Filter-Well

FD/Columbia

Hyperbol/Rock

FDT/Savannah R

Reser./Colorado

Lake Michigan

FDTLake

Lake Clinton

'Ibwer/L Erie

Cooling Lake

Lake Ontario

Lake Ontario

Lake Michigan

Reservoir

Reservoir

Lake Ontario

Hyperbol/Lake MfU

Reservoir/L Anna

Reservoir

FDT/Michigan

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed

Closed Pond

Closed Pond

Closed River

Closed Well

OpenlClosed

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

NUREGICR-5379 2.6
System Definition

Tible 2.1 (Continued)

Reactor/

Rating

Service

Water

Plant Name Parent Utility Ultimate Heat Sink Cycle

Perry-1,2

Point Beach-1,2

Robinson-2

Summer

Wolf Creek

Zion-1,2

Brunswick-1,2

Crystl River-3

Millstone-1

Millstone-2

Millstone-3

Oyster Creek

Seabrook-1

Turkey Pnt-3,4

Callaway-1

Sharon Harris-1

Yankee-ROWE

ANO-1

ANO -2

Arnold

Beaver Valy-1,2

Belfont-1(2)

Braidwood-1,2

Brwns F-1,2,3

Coman Peak-1,2

Cooper

Ct Yankee

Farley-1,2

Ft Calhoun-I

Grnd Glf-1(2)

Hatch-1,2

Indian Point-2

Indian Point-3

La Crosse

Monticello

Peach Bot-2,3

Prairie L-1,2

Quad City-1,2

Sequoyah-1,2

Surry-1,2

Susquehana-1,2

TMI-1,2

Trojan

(GE-1205)

(W-497)

(W-665)

(W-900)

(W-1150)

(W-1040)

(GE-790)

(B&W-825)

(GE-660)

(CE-830)

(W-1150)

(GE-620)

(W-1150)

(W-728)

(GE-1150)

(W-900)

(W-175)

(B&W-836)

(CE-858)

(GE-538)

(W-833)

(B&W1213)

(W-1120)

(GE-1065)

(W-iiii)

(GE-778)

(W-582)

(W-860)

(CE-478)

(GE-1250)

(GE-770)

(W.-873)

(W-965)

(GE-48)

(GE-536)

(GE-1065)

(W-507)

(GE-789)

(W.1148)

(W-775)

(GE1050)

(B&W-792)

(W-1130)

Cleveland Elec Mlum

Wisconsin Elec PWR

Carolina P&L

S Carolina Electric

Kansas Gas and Elec

Commonwealth Edison

Carolina P&L

Florida Power Corp

Northeast Utilities

Northeast Utilities

Northeast Utilities

Jersey Central PWR

Public Serv Co of NH

Florida PWR & Light

Union Electric Co

Carolina P&L

Yankee Atomic Electr

Arkansas PWR & Light

Arkansas PWR & Light

Iowa Elec Light

Duquesne Light Co

TVA

Commonwealth Edison

TVA

lbxas Util Gen Co

Nebraska Public PWR

CT Yankee Atomic PWR

Alabama Power Co

Omaha Pub. PWR Dist

Mississippi PWR & Li

Georgia Power Co

Con Ed Co of NY

PWR Auth State of NY

Dairyland PWR Coop

Northern States PWR

Philadelphia Elec Co

Northern States PWR

Commonwealth Edison

TVA

Virginia Elec & PWR

Pennsylvania P&L

Metropolitan Edison

Portland General Ele

Hyperbol/L Erie

Lake Michigan

Res/Robinson

Res/Monticello

Cooling Lake

Lake Michigan

ATL 0 Outfall

Gulf of Mexico

L I Sound

L I Sound

L I Sound

Barnegat Bay

Atlantic Ocean

Canal/Biscayne Bay

Hyperbol/Missouri 

R

Hyperbol/Reservoir

Deerfield 

R

Reservoir, Ark R

Hyprbo/R M/U

FDT/R

Hyprbol/Ohio R

HyperbolUInn R

Res/Kankakee R

Combcycle Ibnn R

Res/R

Missouri 

R

Connecticut R

FDT/R

Missouri 

R

HyperboliMississippi

FDT/R

Hudson 

R

Hudson 

R

Mississippi 

R

FDT/Mississippi

FDT/Susquhanna

FDTIR

Spray Canal

Comb Cycle!Ibnn

James R

HyperboVSusquhan

.Hyperbol/Susquhan

Hyperbol/Columbia

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Lake

Open Ocean

Open Ocean

Open Ocean

Open Ocean

Open Ocean

Open Ocean

Open Ocean

Open Ocean

Open Pond

Open Pond

Open Pond

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

Open River

27 NUREG/CR-5379
System Definition

¶Nble 2.1 (Continued)

Plant Name

Vermont Yankee

Watts Bar-1,2

Reactor/

Rating

(GE-514)

(W-1170)

Parent Utility

VT Yankee Nuc PWR Co

TVA

Ultimate Heat Sink

¶Ibwers/Conn R

Hyperbol~enn R

Service

Water

Open River

Open River

(a) Information is crtracted from final safety analysis reports (ESARs) and is subject to change due to plant modifications.

FDT -forced draft cooling tower

hyperbol -hyperbolic natural draft cooling tower

R -river

L -lake

Res - reservoir

switchgear. The open system, shown in Figures 2.2a and

2.2b consists of identical bays housing bar gates, stop

logs, traveling screens, screen wash pumps, chlorine

injection equipment, and the circulating and service

water pumps. An elevation drawing, Figure 2.4, shows a

typical arrangement of these components. A separate

ventilation system for the service water pump area is

designated seismic Class I to ensure that the equipment

remains operable during a DBA and is capable of functioning afterward. The service water pumps and their

associated bays are also designated Class I seismic

equipment.

The traveling screens, in parallel arrangement, function

to remove small debris that penetrates the larger (#3-in.

opening) straining provided by the bar gates. The

screens are cleaned by an automatic high-pressure spray

system that senses differential pressure across the

screens.

Three service water pumps are typically used to provide

the necessary head for flow requirements of the various

heat exchangers in the system (from two to four pumps

per plant were observed). Normal operation means that

two of the three SWS pumps operating supply all

steady-state cooling requirements, with the third pump

in a standby condition (one out of two and three out of

four is also common). Pump motor power ranges from

approximately 200 to 600 horsepower and flow from

4000 to 10,000 gallons per minute per pump, depending

on configuration and plant size. Pumps are powered

from plant vital ac busses (with diesel backup) to ensure

a continuous electrical supply. Pump motor-winding

temperature, current, discharge pressure, and bearing

temperature instrumentation are generally available at

local and remote readout panels.

At the pump discharge, common practice is to pass the

water through a discharge strainer to ensure that any remaining particles are small enough so they will not plug

the smallest heat exchanger tube in the system. These

strainers are monitored for excess differential pressure,

which could indicate a plugging condition. The swing

check valves located just downstream of the strainers are

installed to prevent reverse flow through an idle pump

and strainer. Reverse flow short-circuits the intended

path through the heat exchanger resulting in a low flow

condition, and may cause idle pump damage.

The cross-tie valves located on the discharge header permit full operation of both service water loops with any

two SWS pumps running; these are motor-operated

valves (MOVs) powered from the plant vital bus. It is

common practice to provide for emergency make-up to

the reactor vessel or steam generators from the SWS

because it is engineered as a highly reliable water

source. Redundant headers then provide the necessary

flow distribution network to the individual cooling

loads. The discharge header then collects the system

flows and directs them back through the cooling water

flume. Some arrangements allow make-up to the emergency cooling water reservoir from the discharge header.

NUREG/CR-5379 2.8
System Definition

Figure 2A Service Water Intake Structure

The emergency cooling water source (river, cooling

pond, ocean, etc.) may take various forms depending on

local conditions, but it must meet the requirements of

Regulatory Guide 1.27 (NRC 1976) with regard to

capacity, availability, and accessibility under seismic

conditions.

2.9 NUREG/CR-5379
3 Data Acquisition and Compilation

This chapter discusses the various sources of data for the

study and the methods used to extract the information

required. It then gives an overview of the observations

made and draws some qualitative conclusions based on

this compilation.

3.1 Data Sources

The sources of component degradation and failure information documented In the Phase I investigation were

used and extended to develop the Phase II research plan.

These information sources and how they were used are

explained in the following subsections.

3.1.1 Database Information

In Phase I an attempt was made to use the available

component failure information found in several large

databases. Discrepancies between the information

found In these sources and the more intuitive knowledge

of researchers, who had experience with the operating

environment of SWSs, prompted a search for consistent

and reliable data. While we understand that a concerted

industry effort is ongoing to upgrade the quality and

quantity of information in such databases, the usefulness

of this data will be subject to question until uniform reporting requirements are fully implemented. Generic

database information was not used in the second phase

of this research.

3.1.2 Site Visits

The purpose of the site visits was to collect plant SWS

descriptive information and site personnel knowledge

and insights into system operation and degradation

phenomena. Site visits were conducted by a team of

PNL engineers at three plants, each plant one of the

three different SWS types: open (once-through), recirculating (spray pond), and closed (intermediate heat

exchanger).

3.1.3 Nuclear Regulatory Commission Data

This information consisted of 1) the previously mentioned AEOD report OperatingExperience Feedback Report: Service Water System Failures and Degradations

(Lam and Leeds 1988), 2) information available through

inspection and incident investigation reports, and

3) direct knowledge from PNL participation in SWS

investigations.

3.1.3.1 AEOD Degradation Report

This report is a compendium and analysis of operating

events involving SWS failures and degradations In

LWRs from 1980 to 1987. The analysis indicates that

the failures and degradations are multicausal and can

severely impact the safety systems that are required to

mitigate reactor accidents. The information is drawn

from LERs (documents prepared by the utilities to explain the circumstances surrounding the failure of a

safety-related component), as well as from NRC Bulletins, NRC Information Notices, and Reports to Congress on Abnormal Occurrences. Additionally, information from AEOD technical study reports, industry

component failure databases, NRC inspection reports,

and data-gathering visits to four plant sites was included.

A total of 980 events was evaluated, with 276 of these

events being judged safety-significant by the NRC staff.

3.13.2 Incident Investigation Reports

These reports evaluate nonroutine plant events or accidents considered to have a potential impact on reactor

safety, either through component failure or a more implicit scenario (managerial breakdown). This evaluation

may take the form of a fairly detailed root-cause analysis

and can therefore shed more light on occurrences than

generic data. These reports are not, however, designed

to provide information specific to aging research and

cover only a small fraction of the plant failure inventory.

For these reasons, their use in this task was somewhat

limited.

3.1 NUREG/CR-5379
Data Acquisition

3.13.3 Participation In Investigations

As a result of experience gained through this aging assessment task, PNL has developed a system-specific expertise that has been used by regional NRC staff to provide technical assistance in investigating power reactor

SWS- related problems. The problem area, component

failure, and the root cause of the system fault were

found to provide useful information and have been

added to the SWS degradation knowledge portfolio.

(See Section 4.3 for case study information.)

3.2 Data Acquisition

The success of obtaining reliable and verifiable information on component failure history and the root cause of

degradation depends on posing the appropriate questions to the appropriate personneL The knowledge

acquisition methodology developed in Phase I of this

project derived considerable validity from accessing the

information gained by plant personnel and their firsthand experience at operating and in maintaining the

SWS.

3.2.1 Questionnaire Protocol

As in Phase I, the questionnaire protocol, developed for

recording previously undocumented plant knowledge,

was used at each site to ensure complete and efficient

data gathering. Information was collected on all the

major components in the SWS, including information relating to the operation, regulation, modification, surveillance, and maintenance of the SWS.

The completed protocol includes ali the detailed plant

knowledge collected during the visit from key plant personnel, including design engineering, operating supervision, equipment operators, chemists, biologists, maintenance foremen, and maintenance personnel. This

information, more than other sources, provides considerably deeper insights into failure occurrences by taking

advantage of the inherent root-cause analysis performed

by plant personneL

3.2.2 Site Information Resources

Information pertaining to potential degradation stressors at each of the SWS types was obtained by completing the prepared protocol and by investigating and

following up on other prospective sources revealed during the site visit.

'lb accomplish the knowledge and data-gathering objectives, formal and informal meetings were held with site

personnel. Formal meetings were usually confined to

entry- and exit-level meetings with the appropriate site

manager. Informal meetings were held with small

groups of individuals from a specific department or

function, or with single individuals having special SWS

knowledge and experience.

Site personnel provided tours of selected areas of the

site by request or volunteered tours when a special

maintenance or surveillance activity was in progress.

The volunteered tours afforded an opportunity to inspect areas of the SWS which were normally closed or

inaccessible and also in many cases, an opportunity to

obtain specimens and chemical samples for further

study. Both casual and pointed questions were asked of

the tour guide and of personnel working in the tour

area, and the candid responses often provided information as well as topics for additional investigation.

The plant databases and historical files were searched,

and complete operating and maintenance histories were

obtained for individual SWS components. Where system modifications had been performed on some portion

or component of the SWS, the data on the original and

the modified configuration were reviewed, as well as

data pertaining to unique and unusual events affecting

the SWS. The events include unanticipated changes in

biofouling or water chemistry caused by environmental

changes, water hammer events, and 'one time only" SWS

cleaning activities.

Other appropriate areas were reviewed and, where applicable to a specific utility, system aging information

NUREG/CR-5379 3.2
Data Acquisition

was obtained. Areas investigated at selected utilities

included observing the SWS plant operation, witnessing

a routine SWS component maintenance activity, and observing scheduled testing.

Information obtained from these various site resources

was used to complete the protocol and to supply data for

use in the degradation analysis.

3.3 Observation Summary

While information was being collected, a series of observations was made that provide some insights into the

problems encountered in operating each type of system.

While these statements are more impressions of plant

conditions, as formed by the research team members,

rather than rigorous data, they nevertheless convey important information that could be valuable in assessing

similar situations. These observations are offered as a

collective experience of the site visit teams.

3.3.1 Open System Observations

Plant-Specfic Environment: river water with high turbidity

and organic content.

Corrosion and biofouling are the most common causes

of open SWS problems encountered by the site visit

team. 'lb maintain required component temperature

limits, many components of the open SWS system are

throttled or intermittently operated, resulting in low

flow velocities, or in many cases, completely stagnant

loops. Throttling and intermittent operation accelerates

solids deposition and various forms of corrosion in these

areas.

Stainless steels are used in an attempt to correct many

of the corrosion problems; however, not all stainless

steels are immune to the corrosive environment. Stainless steels in a stagnant water environment tend to degrade, because dissolved oxygen in water is necessary to

maintain the protective oxide layer on the pipe wall.

Valve stems made of 410 stainless steel have been corroding in this environment. By maintaining a minimum

flow of water where stainless steel is present, oxygen

depletion will be minimized and stainless steel corrosion

will be mitigated. Maintaining an adequate dissolved

oxygen content in the water also retards nickel leaching

of 90/10 copper nickel tubing commonly found in SWS

heat exchangers.

Where maintaining a minimum flow of water through

the stainless steel pipe eliminates the oxygen depletion

problem, it exacerbates carbon steel corrosion. Some

large-bore carbon steel pipe developed small leaks

caused by localized pitting. Small-bore carbon steel

pipe also developed small leaks, but more often this

condition was observed in the piping U-joints of component coolers. Carbon steel valve internal components

suffered obvious degradation due to corrosion.

The greatest number of electrical failures experienced

were found with the operators on MOVs. Most are corrosion related. External surfaces of buried piping, which

have a protective barrier coating, show evidence of corrosion but little significant degradation.

Corrosion products, primarily from carbon steel, have

caused many other less obvious problems; often these

problems are diagnosed as having causes other than

from corrosion products. Corrosion products have

caused some flow blockage in small tubed heat exchangers and have resulted in plugged instrument lines.

Often, greater torques required to operate system valves

and increased pump motor currents to maintain design

conditions are caused by the effects of corrosion

products.

In contrast to corrosion, fewer erosion problems occur

in open SWSs. With controlled low flow velocities, typically below 5 ftM, erosion is not of consequence in

piping or SWS components including heat exchangers.

Even with low flow velocities in system piping, high flow

velocities are experienced within SWS pumps and at

throttling valves. Erosion was observed on pumps with

brass impellers and required impeller replacement

about every two years. Fire water systems, using the

same water as the SWSs, experience erosive degradation

of pump impellers and the coarse screens located in the

pump suction bell.

Changes in environmental conditions are * constant

threat to increased flow blockage erosion in open, oncethrough SWSs. Any environmental event causing increases in silt intrusion into the SWS results in increased erosion.

3.3 NUREOICR-5379
Data Acquisition

In addition to corrosion, biofouling and biofouling control are challenges inherent in the design of open oncethrough SWSs. Biofouling from microorganism growth

tends to decrease heat exchanger efficiency, cause corrosion of base metal, reduce system flow, and pose the risk

of fouling as large colonies of microorganisms slough off

from piping walls.

Larger organisms, such as fresh water clams (corbicula),

also present biofouling challenges and cause corrosion,

including local erosion and pitting, similar to that observed from microorganisms. Plugging and fouling are

the most prevalent difficulties encountered. Even with

chlorination, fresh water clam removal from the SWS

bays is often required. Biofouling of an 18-in. pipe between the ultimate heat sink and the SWS required

hydro-lancing to restore the pipe to the required flow

rate.

Other environmental debris can cause mechanical

stresses exceeding those routinely expected to be found

in SWSs. For example, because of the design of open,

once-through SWSs, pieces of wood or similar debris enter the system and have been observed to lodge between

moving and stationary parts. In one instance, large

volumes of wood, leaves, and debris limited power production because the physical capacity of the traveling

screens was exceeded. Similarly, periodic fish migrations have resulted in flow reductions by overloading

traveling screen capacity.

The major source of service water pump failure is bearing degradation. 'Rending of pump vibration has been a

good predictor of bearing failure.

33.2 Recirculating System Observations

Plant-Specific Environment: arid conditions with mild

river water makeup.

Corrosion is the most common problem in recirculating

SWSs as noted by the site visit team. In terms of severity

and quantity, erosion and sediment problems are few

compared with those in open systems and generally affect limited portions of the recirculating system. Biofouling is limited to some slime and algae formations.

Corrosion, although the most common cause of problems in the recirculating SWS, is less severe and more

predictable than in other open portions of SWSs. Water

treatment products and the addition of scale and corrosion inhibitors have been effectively used to control corrosion. Chemical additions to the system to control pH

are infrequent because of the nonaggressive nature of

the make-up water source.

Before corrosion control was implemented, all smallbore carbon steel pipe suffered significant corrosion and

sediment buildup so that in some cases the pipe bore became completely blocked. Portions of the original carbon steel piping were then replaced with similar piping.

Following the change in make-up water sources and introduction of corrosion control, carbon steel pipe corrosion has been minimaL

Ring header supports for pond spray rings have sustained some pitting attacks, resulting in broad shallow

areas of slight metal removal. Magnesium anodes are

used in the ponds to protect submerged steel structures.

Underground piping inspected during an outage showed

light exterior corrosion and no significant degradation.

In other underground piping, some corrosion was evident where water seeped under the protective wrapping.

This corrosion, though evident, is minor and has not

compromised the integrity of the underground piping.

A cathodic protection system is installed on some

underground piping, but the system has not been used.

A division diesel generator heat exchanger was inspected when it was opened for maintenance. The visual

inspection revealed some corrosion and sedimentation

accumulation with a slight slime overlay. The corrosion

was uniform across all surfaces and did not appear to be

a problem. Some corrosion tubercles, up to approximately 1 in. in diameter and 1/2-in. high, were found in

flow eddy areas. No corrosion-caused thermal performance degradation had been identified in diesel generator

heat exchangers.

Sediment, which had accumulated in the spray ponds

during six years of operation, had stayed below the

allowable technical specification limit. Vacuuming

techniques were used to remove accumulated

sedimentation.

NUREGICR-5379 3A4
Data Acquisition

No significant flow blockage has occurred in heat exchangers because of sedimentation. In general, flow

reduction caused by sedimentation has been In large

cross-sectional areas of piping runs where low-fluid

velocities exist. SWS modifications to reduce or mitigate sedimentation by increasing fluid velocities have

been successful.

In general, fluid flows are slow enough that erosion is

not a problem. No significant erosion was observed in

piping or heat exchanger tubes. Some cavitation erosion

was observed on SWS discharge butterfly valves. Replacing the butterfly valves with hardened steel components is expected to mitigate the cavitation erosion.

Biofouling in the recirculating SWS Is not a significant

problem. With a spray pond temperature range of 320F

to approximately 770F and no blocides currently used in

the system, some biological activity is expected. Live

algae bloom has been noted at certain times of the year

in the spray pond bulk water. A slight slime overlay was

observed in some piping and on the surface of heat exchanger tubing. Biocidal treatment has been implemented in the pond.

Although active Asiatic dam populations are known to

exist at the river make-up water intake, no evidence of

clamns, clam debris, or clam larvae have been found in

the SWS.

After 5 years of operation, eddy current testing was

performed on three SWS heat exchangers. Results of

the examination revealed that it was not necessary to

plug tubes for any structural deficiencies.

Vibration, usually from motor operation, has caused air

line leaks in air-operated valve actuators. The leaks occur at the tubing compression fittings at the valve actuators in proximity to the motors.

A water hammer problem with the SWS was a problem

during startup. The problem has been mitigated by adding a keep-full or jockey pump and by motor-operated

valve sequencing that starts each pump against a highly

throttled discharge valve. All short- and long-term consequences from the water hammer problem are believed

eliminated by the plants experiencing the problem.

Instrumentation flow orifices, venturis, and sensing lines

have not experienced any plugging problems since replacing small-bore carbon steel pipe and adding corrosion inhibitors.

Pump motors In the SWS are routinely monitored and

surveillance tested. It has not been necessary to disassemble the motors for inspection because of adverse

monitoring results or surveillance data.

3.3.3 Closed System Observations

Plant-Specific Environment: salt water marine with high

ambient humidity.

The most common causes of closed SWS problems observed by the site visit team were corrosion and biofouling, and the greatest number of modifications to any of

the SWS types was observed with the closed system.

Most of the modifications in progress are to correct or

mitigate corrosion problems that were developing on

the raw water side of the SWS. A significant number of

modifications are planned to correct biofouling.

Two corrosion problems are evident at the intake structure: concrete pedestal cracking due to rebar corrosion

and aluminum embedment failure caused by corrosion

from the marine environment. Other indicated degradation includes long-term degradation of the structure due

to rebar corrosion, and attack from marine and groundwater chemicals on the concrete binders and cements.

The SWS pumps, having accumulated about five years of

operation and being submerged for about seven years,

were removed for routine examination. After removing

the surface oxidation coating, the pumps were found to

be in extraordinarily good condition. The pumps' pristine condition can be attributed to an effective cathodic

protection system used to control the corrosive marine

environment at the submerged pumps. However, corrosion was observed under the pump shaft sleeves during

the routine inspection, necessitating replacement of the

lower pump shaft.

A water hammer problem noted during startup was

corrected by a combination of administrative and hardware modifications. Plant personnel believe that all

3.5 3.5 ~~~~~~NUREG
Data Acquisition

short- and long-term consequences from the water hammer problem have been eliminated. Accelerated aging

and degradation effects resulting from the water hammer problem are unknown at this time.

In many of the non-safety-related piping, carbon steel

used in the drains and the intake screen wash system has

been replaced because of the effects of corrosion and biofouling. Inspection of the removed pipe spools revealed

uniform corrosion with biofouling of the inner pipe

wan. Through-wall failures have occurred where corrosion was localized; these locations are difficult to predict

from external examinations of pipe. That pipe has been

almost totally replaced with lined or stainless steel pipe.

Non-wetted valve parts have failed under stress-induced

corrosion. In some cases, corrosion has caused valve

inoperability, and in others, catastrophic valve failure.

A series of failures, due to corrosion of ductile iron

packing followers caused by the damp marine environment, valve leakage, and possibly other factors, occurred

on some SWS valves.

Butterfly valves in the SWS have failed because of corrosion between the monel shaft and the packing follower.

The packing followers were carbon steel and, as the corrosion process proceeded, carbon steel metal oxides

were formed (which are volumetrically larger than the

original carbon steel) and eventually caused binding of

the operating shaft.

The check valves were showing signs of premature wear,

primarily in the valve seat area and the check disk. New

elastomer seals were installed in the seat area and the

check disk was painted with a protectant, thus reducing

the rate of degradation to an acceptable level until the

planned modification to replace the existing check valves with aluminum-bronze valves is completed.

Valve body to valve-liner adhesion is a problem. Loss of

the valve liner integrity is not obvious during normal operation and once lost, irreparable corrosion rapidly

ensues.

The heat exchanger copper-nickel tubes were replaced

with titanium alloy tubes selected specifically to resist

salt water chemistry.

Corrosion of electrical equipment is evident but not significant, primarily because it is predictable. Electrical

equipment protected by coatings or water-tight enclosures show minimal or no visual degradation. Surface

corrosion is evident on exposed wire, cable, and electrical contacts, as well as on electrical conduits, boxes,

and enclosures. Surface corrosion is quite visible on

motor cables not protected by insulation or coatings.

No failures due to corrosion were identified in electrical

power circuits at the utility visited.

The premature failure of the rubber/elastomer expansion joints at the SWS heat exchangers is an excellent

example of air-induced aging degradation caused by

improper installation. This conclusion, reached by both

the plant's engineering staff and the component manufacturer's engineering representative, was supported by

evidence presented to the site visit team. The failure is a

series of longitudinal cracks or tears in the expansion

joint that coincide with each of the bolt holes in the

expansion joint mounting flange. The hypothesis is that

during installation an oversized torque or similar

wrench was used which, when twisted, placed a severe

stress on the elastomer material, resulting in surface

degradation. The degradation was not immediately observable but was manifested only after repeated operating cycles. Replacement by proper installation solved

the problem.

Preventing biofouling is a never-ending battle against

the marine environment because the SWSs are assaulted

by vegetation, microorganisms, barnacles, mollusks, and

other aquatic life forms. The shells of marine organisms, primarily barnacles, are continuously swept into

the heat exchangers, resulting in tube plugging. These

barnacles must be routinely removed to maintain the

flow rate required to mitigate postulated accident

conditions.

Heat treatment, chlorine injection, and manual cleaning

are common biological control processes used by the

utilities. Heat treatment (thermal backflush), with or

without chlorine, is performed on a routine basis as determined by the rate of organism growth. Chlorine is injected at regular intervals, usually several times a week.

Manual cleaning is performed during refueling outages,

or as necessary to maintain required system flow rates.

NUREG/CR-5379 3.6
Data Acquisition

A continuous cycle of SWS cleaning and inspection for

degradation is necessary, any identified degradation is

then repaired or replaced. However, long-term aging

trends can only be estimated because the cleaning/

replacement cycle is continuous. As the time between

cleaning and inspection is extended (possibly due to

system modification or process changes), the effects of

aging and degradation stressors on the SWS must be

closely monitored until the long-term trends are known

and predictable.

3.7 NUREG/CR-5379
4 Root-Cause Methodology

In Phase I of the SWS study a lack of accurate, uniform

root-cause analysis (RCA) was shown to lead to incorrect conclusions regarding the degradation mechanisms

that result in component failure. In response to this

identified need, a subtask was organized to provide a

prototype method for identifying and documenting the

root cause of component degradation and failure. This

section discusses the background and development of

this proposed methodology for performing RCA The

approach, methodology, and categorization of a generic

root-cause scheme are explained.

Section 4.1 explains the regulatory basis for the actions

being taken by the NRC in the Generic Letter on Service Water System Problems Affecting Safety-Related

Equipment (54 FR 43209-10). It also presents the impetus for the definition of a uniform methodology for

RCA. Section 42 explains the root-cause process, the

method structure, and a root-cause categorization based

on component design considerations.

4.1 Task Development

The safety-related portions of the nuclear plant SWS are

the final link in the pathway of heat dissipation to the

ultimate heat sink; for this reason, it was selected for an

NPAR system study. The SWS draws water from various sources, and depending on the plant, is composed of

a host of materials, differing configurations, and a high

variability in operating and maintenance practices.

Historically, maintenance performance involves a balance among the preventive (replace before failure), the

predictive (measure and replace before loss of capability), and corrective (repair/replace after the failure)

measures predicated principally on economic considerations, i.e., cost effectiveness. For nuclear power, however, operating safety takes on a dominant role because

of the potential impact of an accidental release of radioactivity on the general public. The safety issue, coupled

with the high cost of replacement power during plant

shutdowns, has led to an increased bias toward emphasizing preventive and predictive maintenance programs.

4.1.1 Safety-Related Event Reporting

A system for rapid reporting of significant events to the

NRC has been established and is depicted in Figure 4.1.

Like the LCOs, these events are also grouped by a perceived increase in potential core damage that could result from the event. The SWS-related events fall under

"degradation in emergency response capability' in the

one-hour category and 'conditions that could prevent

residual heat removal" in the four-hour notification

group.

For any of the events shown in Figure 4.1, an LER must

be issued by the plant within 30 days after the discovery

that such an event took place. Per Section 50.73 of

10 CFR 50, part of this report must contain the foilowing information to incorporate all the essential elements

of a failure RCA.

* a complete event scenario description

* pre-event plant conditions

* inoperable components that contributed to the

event

* cause of component failure

* failure mode, mechanism, and effect

• estimate of safety system -unavailability

* discovery method.

4.1 NUREG7CR-5379
Root-Cause Methodology

NATURAL THREAT

VALID ECCS

SIGNAL

i

SHUTDOWN

REQUIRED BY

TECH SPECS

IMPENDING NEWS

RELEASE

INN _ EMERGENCY

9 ~~~~ACTIVATION

DEGRADATION IN

EMERGENCY

RESPONSE CAPIBILITY

ONE-HOUR LER

FROM A

SHUTDOWN

MODE

ESF

ACTUATION

CONDITION THAT COULD

PREVENT:

REACTOR SHUTDOWN

RESIDUAL HEAT REMOVAL

RADIATION CONTAINMENT

ACCIDENT MITIGATION

LIQUID

RELEASE

=2X MPC

AIRBORNE

RELEASE

=2X MPC

(

Figure 4.1 U.S. Nuclear Regulatory Commission Reportable Events

NUREG/CR-5379 4.2
Root-Cause Methodology

4.1.2 Component Failure Root Cause

To better understand the search for the failure rootcause, the following definitions are provided:

* failure mode - the manner or state in which a system, structure, or component faWls. For example, a

heat exchanger can be declared inoperable (to have

failed) if it:

- does not have pressure boundary integrity (a

leak),

- restricts the flow of working fluid to less then a

required limit (a block) or,

- does not provide the required heat transfer

capacity under minimum design flow conditions

(degraded heat transfer coefficient).

* aging mechanism - specific process that gradually

changes the characteristics of a system, structure, or

component with time or use. Example: A deposition process that reduces the ability of a heat

exchanger to pass the required flow rate.

* stressor - an agent or stimulus that can produce

aging mechanisms. For the above cited deposition

process, the stressor might be high silt content in

the process stream that settles out (deposits) in low

flow areas of the heat exchanger.

Component failure root cause is thus "the process of

determining the fundamental degradation stressors

responsible for component failure, such that, if corrected, prevents recurrent failure from that mechanism

(Jarrell et al. 1989; Stratton and Jarrell 1989). This

clearly demonstrates that the utilities are required by

10 CFR 50 to perform a failure RCA for a wide range of

events that affect safety-related systems. It is the goal

of this RCA research study to provide a general framework defining the process requirements for performing

such an investigation.

The prime goal of the NPAR program is to identify

and characterize aging mechanisms that could cause

safety-related components to undergo premature or

undetected degradation. Figure 4.2 illustrates an abbreviated version of the principal products of the program. The program's investigative approach is to collect

technical information from various sources pertaining to

a specific safety-significant component or system, then

analyze it in a manner that allows correlations to be

made regarding the identity and characteristics of degradation in the component (system) of interest. A determination of the measurement accuracy and the completeness of the degradation effects on component (system) performance relative to the design criteria is then

made to ensure safety function capability. Any discrepancies found between the design criteria and the

performance measurement are then documented in a

recommendation for a revision to the regulatory

requirements.

4.2 Development of a Generic

Methodology

The need for a structure to promote a more uniform approach to RCA and reporting grew from efforts to define equipment-aging safety issues. This section traces

the evolution of that need as perceived by the Nuclear

Plant Service Vvter System Aging Degradation Assessment (Jarrell et al. 1989), and presents the logic used in

the analytical approach and the actual elements of the

RCA model derivation. The source of data most commonly used by NPAR researchers for determining the

degradation and failure mechanisms is utility-generated

data from a central data repository. Several of these

aging studies indicate that the component failures listed

in such databases are inadequately specified and/or inaccurate when analyzed for component degradation me

chanisms (Murphy et al. 1984). Since these studies-are

to be at least contributing factors in mandated utility

maintenance and surveillance requirements, the impact

of inaccurate degradation assessment can be manifested

as 1) an increase in the actual component degradation

rate, the result of repeated and unnecessary preventive

maintenance, inspection, and restoration; 2) an inefficient use of already scarce utility resources; and 3) a

contribution to an increase in actual core damage risk

because the correct degradation stressor was not

measured.

4.3 NUREG/CR-5379
Root-Cause Methodology

IDENTIFY AND CHARACTERIZE

COMPONENT DEGRADATION

STRESSORS

DEGRADATION

MEASUREMENT

< ~ADEQUACY

( RECOMMEND REGULATORY

APPLICATION GUIDELINES

MAINTENAI

SURVEIL

REQUIRE

NCE, AND PLANT T

LANCE SPECIFI

MENTS

Figure 4.2 Abbreviated NPAR Program Overview

ECHNICAL

CATIONS

NUREG/CR-5379 4.4
Root-Cause Methodology

As a direct result of discovering misleading database

information, the Nuclear Plant Service Water System

Aging Degradation Assessment in the NPAR program

(Jarrell et al 1989) has undertaken a task to establish a

methodology for conducting and documenting an RCA

investigation designed to allow a more comprehensive

and accurate diagnosis and documentation of the component degradation mechanisms. From a regulatory

perspective, this would have the effect of:

* standardizing the inputs, methods, and deliverables

of an acceptable RCA

* allowing resident inspectors to determine the

adequacy of RCA methods used at their plant.

The usefulness of a formalized component failure rootcause methodology to the utility industry will vary

widely because of the large differences in individual

plant analysis methods, level of diagnostic expertise

available, completeness of the knowledge of degradation

mechanisms, and adequacy of current documentation.

42.1 Root-Cause Process Approach

The first step (see Figure 4.3) in the root-cause process

approach was to choose and constrain a functionally significant service water component. In this case a component cooling water to service water heat exchanger was

chosen because it is vital to the system function and has

a demonstrated history of failures. Some 37 instances of

heat exchanger failures that resulted in declaring at least

portions of the SWS inoperable are documented in the

approximate seven-year span addressed in the reference

report by Lam and Leeds (1988).

To ensure that the focus is on the actual degradation

mechanisms, the AEOD database was supplemented by

visiting and examining the different SWS types at three

plants, as described in Section 3.2.2, Site Information

Sources. A total heat exchanger failure population of 82

events was then organized by the physical location and

mechanism of the failure. From this, a set of plausible

scenarios encompassing all identifiable modes of heat

exchanger failure was developed.

At this point, a root-cause failure analysis was performed by a systems engineer on this specifically

selected group of actual in-plant heat exchanger failures.

The analytical process used by the engineer to arrive at a

conclusion was explored and documented using methods

from knowledge acquisition techniques found in modelbased artificial intelligence technology. This solution

process was then abstracted and the logic used to create

the root cause of failure model shown in Figure 4.4.

A general methodology for solving the root cause of

component failure was demonstrated for this general

heat exchanger example in the Phase [ report. It is

believed this method can be expanded, at least partially

automated, and applied to other components, systems,

and system interactive failure investigations. Application of such a systematic approach to failure resolution

benefits the users by providing:

* a logical framework for assembling the elements

needed to produce a viable solution

* a method for proceeding with the analysis process

* a means of checking the reasoning invoked by the

process

* a potential for automatic documentation of the

investigation effort and its results.

4.2.2 Root-Cause Analysis Methodology

The proposed model of the root-cause analysis process

consists of these reasoning activities: fault recognition,

fault localization, fault specification, and root-cause

evaluation. As shown in Figure 4.4, these activities are

usually thought of as a two-step process, Le., fault determination (what happened) and the root-cause evaluation (why it happened). By breaking these activities

down into more discrete steps, it is noted that the processes are familiar to the utility system engineer. They

are so familiar, in fact, that most of the processes go on

almost automatically as soon as the problem is identified. This step-by-step processing does, however, have

the distinct advantage of systematically identifying and

documenting the areas where information is incomplete.

The following items are shown on Figure 4.4:

4.5 NUREG/CR-5379
Root-Cause Methodology

CONSTRAIN THE PROBLEM

TO A

MANAGEABLE SIZE

I

I I

I SIMPLE :

by v HEAT

: EXCHANGER

I

_... ....... m

, NRC

' REPORT

; ON SWS I

' FAILURES I

OBTAIN A REASONABLE

CROSS-SECTION OF KNOWN

DEGRADATION EVENTS

__I__ jr---------- ;

PROCESS :

-i PHYSICS'

i I _ _ I

EXAMINE THE COMPONENT

PHYSICAL PROCESSES ---------- a

*

CONSTRUCT "PROBABLE"

SCENARIOS FOR DEGRADATION

EVENT CLASSES

"EXPERT SOLUTION" TO

DEGRADATION SCENARIOS

WITH Al KNOWLEDGE

EXTRACTION ENGINEER

ABSTRACT THE

ROOT-CAUSE ANALYSIS

PROCESS

R jr----------

' LEAKS

BLOCKAGE

.HXFR COEF

...---------

i :

- KNOWLEDGE

REQUIREMENTS'

. I

*ROOT-CAUSE:

, METHOD '

...

Figure 4.3 Analytical Approach

NUREG/CR-5379 4.6
.. ', ~ ~ ~ ~ ~ <4- ~ ~ ~ ~ ~ ~ i mmmg, V9 

PLANT ~ STAFF ~~~~%Z LANT STAFFT

I ~~~CAUSAL

OPERATING .~POESPYISTESTING, ANALYSIS

PARAMETERS, PREDICTED BEAiO, IASEMBLY, TEST DATA ACTUAL BEHAVIOR, MAC EMERYHITOR

'CO~~~~~~~~~'A~~~~CC-C. ,(~~~(OR OSON E OSON

.~~.., ' '""*A~~~~.o'.40"o ~~0 -,, "O'0':.'" F* .LNGCETC. .~~~4~~~:>'~~~~~ '4.~~~~~.. .. DEGRADATION I~~~~~~~~~~~~~~~~

FAULT DIAGNOSIS~CC-,.. I fCO.'

z ~ ~ ~ ~........, ~~(4C$ 4444*4444440~~~~*4ROO CA SEE AL AT

Flgn GeericRoo-Caue 4.4 Aalyss Mthodlog
Root-Cause Methodology

* Fault Recognition - involves reading (instrument

data and related history), calculating, and comparing sensory and process information to determine if

a fault condition is going to occur or presently

exists. The result of this activity is the development

of component and fault knowledge, notification that

a fault condition exists, and an activation of further

evaluation. For vital plant components, this function is implemented by alarm annunciators in the

control room, while local alarms are used to indicate fault conditions of a less immediate nature.

* Fault Localization - processes a wider range of information than fault recognition. Its purpose is to isolate the fault to a specific component and possibly

to a specific part of the component This activity

may also suggest tasks to be performed for the purpose of acquiring missing information. Plant personnel are guided in this preliminary determination

by prescribed diagnostics such as annunciator response cards for most plant transient events.

* Fault Specification - integrates information and conclusions developed in the fault recognition and

localization activities to provide a complete description to the fault. It also incorporates any additional troubleshooting or research information

gathered by the plant staff as they gather pertinent

data through testing, inspection, measurement, or

history files. A complete record of what occurred

and how it occurred is contained in an adequate

fault description.

4.2.3 Basic Failure Categorization

The final step in the search for solutions to the cause of

a failure must be to examine the sources of potential

errors during the life cycle of the component Any

attempt to set up a scheme for categorizing events must

have its foundation in the specification of the design

bases of the affected system. For the SWS, this is

10 CFR 50 Appendix A, General Design Criteria 44,45

and 46. As was explained for Figure 1.2, Chapter 1, the

specific design criteria for the plant are to be found in

the safety analysis report and technical specifications at

the plant.

Following a component failure (see Figure 4.5), at least

some degree of failure specification is performed by all

plants. The relationship of the failed component's

environmental stressors to the envelope of design parameters specified by the design engineer must first be

examined. In proceeding down the right-hand path in

Figure 4.5, if the design specifications were known to be

exceeded (for example, a system pressure was taken

above its design limit), then several additional questions

remain. Does the operating or maintenance procedure

under which the component was being operated accurately reflect the design specification? If not, then

clearly an error was made when the procedure was prepared. If all the operational parameters are correctly

specified, which could be expected by properly executing

the procedure (including any transients that can be

shown to be within the design envelope), then one must

conclude that the execution of the procedure was in

error. This is because it is a given that the design specification was exceeded in this investigative branch.

If the design specification was not exceeded, then the

application of the component for this use becomes suspect. This is the case when a component variable is not

specified in the design, but unanticipated stressors result

in placing the component in an environment beyond its

capability. An example might be the service water pump

impeller that must be replaced after 18 months of operation because of severe erosion. The tacit assumption

made by the design engineer, that pure liquid would be

the working fluid, did not match the application where

the river water contained a high concentration of suspended impurities. This is a prime example of what has

been termed "component aging." Had the stressor responsible for failure been anticipated (included in the

design specification), then two options still remain. If a

manufacturing flaw can be verified, such as improper

heat treatment or improper materials, then 1) the manufacturer is culpable, or, failing all of the above, 2) the

design considerations and calculations must be reviewed

for invalid practices.

These five categories of failure--aging, design, production, execution, and procedure--then form the basis of

assigning an improvement area for preventing future

failures.

NUREG/CR-5379 4.8
AGING 

1 1 1 

DESION 

1 1 1 Lj] ~~~~~~~~~~~~~~PROCEDURE PROCEDURE PRODUCTION APPLJCATIONPEARiOEXCTO

PHENOMEMA ERR ERROR ERROR PEPRATOR ERRUTOR

0

Figure 4.5 Basic Failure Categorization Scheme0

%0~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-
Root-Cause Methodology

4.3 Service Water System Diagnostics

and Root-Cause Analysis Case Study

This section presents a case study on the evaluation of

room cooler thermal hydraulic performance and EDG

heat exchanger failure analysis, which was conducted in

support of a SWS event at a commercial nuclear power

plant. This study illustrates using the diagnostic and

root-ause methodologies developed by the NPAR SWS

task. PNL was requested to evaluate the measurement

techniques and calculation methods used in determining

the thermal-hydraulic capabilities of the safety-related

portion of the SWS, hereafter called essential service

water (ESW), at a large central station generating facility. The need for such an evaluation developed from

1) failure events in the EDG to ESW heat exchangers,

and 2) utility efforts to initiate compliance with Generic

Letter 89-13, which requires thermal performance verification of safety-related SWS components.

43.1 Situation Scenario

After approximately 2-1/2 years of commercial plant

operation, an emergency diesel expansion tank highlevel alarm was energized during routine EDG testing.

This condition indicated apparent in-leakage from the

ESW system (having a line pressure of approximately

100 psig) to the EDG engine closed cooling water system (slightly above atmospheric pressure). The Division 1 EDG was declared inoperative and, during the

subsequent LCO, the utility examined the suspect heat

exchangers and found through-wall pitting in one tube

of the dual heat exchanger units. The defective tube was

replaced with a new tube of like material (90110 Cu/Ni),

the heat exchanger was tested, and the EDG was returned to service.

Approximately one week later, a similar EDO tube failure occurred. The heat exchanger was repaired, eddy

current tested, and approximately half the tubes were

cleaned. A third failure took place two weeks later, at

which time the utility completely retubed and hydrostatically tested both the Division I EDG heat exchanger units. A consultant was called in to examine the

degraded heat exchangers and identified the degradation

mechanism as local microbiologically influenced corrosion (MIC) accelerated by the elevated thermal conditions following EDG testing.

Suspecting that a global MIC condition could exist in

the ESW system, the utility started an inspection of all

Division 1 heat exchangers (see Figure 4.6) to determine

their physical condition. This examination revealed that

the remaining Division 1 heat exchangers did not exhibit

extensive pitting corrosion. This was shown by a maximum tube wall pitting depth of less than 20% along with

moderate general corrosion and light siltation. The lack

of evidence of a widespread MIC attack was attributed

to the elevated local temperature during diesel testing,

combined with the stagnation conditions found during

standby service in the EDO heat exchangers. These conditions were thought to promote micro-organism

growth and the resulting degradation of the coppernickel tube wall.

Prior to the system-wide heat exchanger inspection, the

individual heat exchanger flow rates were measured.

This pre-inspection flow measurement was performed

using nonintrusive ultrasonic flow measurement devices

to determine fluid velocity; the resulting component

mass flow was then analytically determined. In the

ESW-served heat exchangers, as-found flows varied

from as low as approximately 20% to as high as 300% of

component design flow. Following the inspection, piping modifications were made to provide sufficient driving head to achieve design cooling water throughput in

some low-flow components which had fully opened

throttling valves. With the modifications complete, the

system flow balance was estimated, using a computer

simulation, then physically balanced to design flow

requirements using the ultrasonic flow detectors.

43.2 Root-Cause Issue Resolution

The above chronological summary provides a context

for resolving two basic root-cause issues relative to this

ESW system event:

NUREG/CR-5379 4.10
^ Mass (ok l - ) Discharge | ~n 

\ as f < Coolirg ~~~~~~~~~~~~Flow i-low

Mass§. AP Control Ro

UHS Source Chiller

Disch.

Water location Iwerter Room

FIgure 4.6 Simpifled Service Miter Fieow Schematnc A
Root-Cause Methodology

1. What was the degradation mechanism responsible

for the EDG heat exchanger tube wall failure?

2. Why were the as-found component flows reduced to

as low as 20% of the component heat exchanger

design flow rates?

The following section endeavors to answer the first of

these questions using the methods developed by this

NPAR task.

43.2.1 Emergency Diesel Generator Ihbe iall Failure

Analysis

Following the methodology outline presented in Section 4.2.2 and Figure 4.4, the main steps taken to

determine the root cause of the failure were the

following:

* Fault Recognition - A fault condition was indicated

by an annunciator alarm which identified a high

liquid level that existed in the EDG closed cooling

system head tank during EDG testing. The EDG

test was discontinued, and the alarm indication was

verified by a physical check of the sight glass on the

side of the head tank, which confirmed the abovenormal liquid level. At this point, the existence of a

pressure boundary fault was further validated by

checking other possible sources of pressurized

liquid in-leakage at interfaces to system boundaries

(make-up sources) for proper valve alignment. It

should be noted that the ESW and EDG systems are

de-energized (not pressurized and not running)

under normal'plant operating conditions.

* Fault Localization - When the boundary valve alignment checks indicated that all potential filu valves

were in their shut position, and with no previous

history of EDG-closed cooling water fill-valve failure, the most probable source of higher pressure

liquid at the EDG closed cooling boundary was the

ESW system. The decision was made to physically

open and examine the service water pressure

boundary internal to the EDG heat exchangers.

This examination subsequently revealed a throughwall localized perforation near the tube sheet in the

copper-nickel alloy tube wall. This provided a

positive location of the fault at the EDG heat

exchanger.

Fault Specification - The complete specification of a

component failure, or fault, provides all the relevant

facts that contribute to any degradation stressors

which could have an effect on the failed component.

From first principles modeling, the possible degradation mechanisms are shown in Tahble 4.1.

Due possibly to the prevalence of MIC in SWS failures,

a predilection tends to appear once the existence of

MIC-type organisms is positively identified. In this case,

a MIC expert was consulted, a positive identification of

MIC organisms was produced from an examination of

the samples, and MIC was declared the single operative

EDG mechanism by the utility root-cause investigation

team.

Figure 4.6 is a simplified drawing of one division of the

ESW and shows component flow paths and the percentage of design flow through each component prior to the

EDG heat exchanger failure event. When the ESW system components are operating (normally during EDG

testing and plant cooldown), most heat exchangers display a flow rate considerably below the design values.

The exception appears to be the failed EDG heat exchangers which have 250% to 300% of design flow.

During normal plant operation, the ESW system is in a

standby mode, i.e., the ESW pumps are deenergized and

the ESW side of the components are in a stagnated,

water-filled condition. Thus, the system had been experiencing a high flow/stagnation cyclic operation for

approximately four to five years (this system was in

operation prior to commercial operation of the plant).

Table 4.1 shows an abbreviated listing of the modeled

degradation modes, mechanisms, and stressors that were

searched for possible negative influences on heat exchanger performance. As shown by the table, of the four

possible mechanisms which could be responsible for a

breach in the heat exchanger pressure boundary, two

were shown to exist in the failed component: 1) MIC

(active during stagnant standby service), and 2) highflow-rate erosion during emergency diesel operation.

NUREG/CR-5379 4.12
Root-Cause Methodology

TAble 4.1 Heat Exchanger Degradation Hierarchy

Modes Mechanisms Sample Stressors

(that are recognized) (process to degrade the component! la possible cause for the observed degradation!

Flow Blockage

Pressure

Boundary

Penetration

Heat

7kansfer

Degradation

Biofouling

Accumulation

Foreign Debris

Corrosion

Erosion

Structural Impact

Biofouling

Accumulation

Foreign Debris

clams, mussels, kelp

high solids content, mobile corrosion products

incomplete maintenance, detached pipe coating,

detritus

galvanic action, MIC, pH, biocides

high flow velocity, suspended solids, vapor collapse

water hammer, improper maintenance

slime, algae

high solids, mobile corrosion products

oil, pesticides, fertilizer

By consulting appropriate references (Dawson et al.

1990; Uhlig and Revie 1985) the copper/nickel (90/10)

threshold for fluid-to-wall shear stress mass transfer

effects (metal removal by erosion) was determined.

With the assumption of mildly aggressive water conditions, the erosive limit was shown to be at a water velocity of approximately 7 ft/s. Calculating the tube flow

area shows that from 11 to 13 ftjs existed in the heat

exchanger tubes during ESW system operation. Thus,

by considering all available information on fluid velocities, fluid composition, and system geometry, clear

mechanisms for both erosion and MIC were shown to

exist. These mechanisms are part of a more complete

fault specification.

4.3.2.2 Causal Mechanism Analysis

A full description of the fault can now be passed to the

next task: finding a mechanistic correspondence that

exists between the root-cause hypothesis generated by

the data analysis and the behavior predicted by the degradation model.

The erosion-corrosion cycle, which existed during the

pre-balance condition, was composed of alternating

high-fluid velocity during ESW operation (13 fW/s) and

subsequent stagnation while in the standby mode. This

had the effect of thinning and removing corrosion product film, as well as some of the freshly-exposed base

metal by the high velocity fluid shear stress. Alternately,

during the stagnation conditions encountered during

standby operation, the bacteria causing the MIC phenomena were provided with the ideal environment in

which to grow, i.e., stagnant water, thermal enrichment

(a warm heat exchanger), and copper freshly exposed by

the scouring action of the high-velocity fluid. Modeling

techniques (Dawson et al. 1990) were then employed to

provide a correlation between the momentum transfer

effects and the stability of semi-protective films that

control the rate of corrosion. This analytical model was

coupled with the existing environmental conditions to

validate the conclusion that through-wall perforation of

the heat exchanger tubing would require both MIC and

erosion-corrosion to produce through-wall penetration

in the observed timeframe.

This establishes a reasonable relationship between the

observed degradation and a model of the degradation

process. The root cause of the failure is therefore attributable to the following two conditions:

4.13 NUREG/CR-5379
Root-Cause Methodology

1. excessive flow through the EDG heat exchanger

tubing, which resulted in fluid velocities capable of

removing corrosion product film and base metal

from the tube wail

2. repeated MIC attack on the cleaned tube inner wan

surface, presumably enhanced by thermal and stagnation conditions.

Actions taken by the plant were to reduce the ESW

water flow through the heat exchanger to within design

conditions (mitigates condition 1) and to pursue a program of biocide control of the MIC bacteria (designed to

control condition 2); thus, both degradation mechanisms were addressed by the corrective actions.

433 Categorization and Conclusions

Carrying this example to its conclusion, the final question to be answered requires identifying what must be

changed in the design, manufacture, implementation,

operations, or maintenance cycle of the component to

stop the degradation event from recurring.

4.33.1 Failure Categorization

The RCA of the event indicated the following: throughwallperforation of the EDG to ESWheat exchanger was

the result of MIC and erosion-corrosion in a cyclic

combination.

From this statement of cause, two specific issues must

be addressed to ensure completion of the fault

remediation:

1. MIC-pitting attack on the heat exchanger tube wall

2. erosion of the tube wall by high velocity fluid.

As stated in Section 4.1.1, efforts to avoid future failures

start with the component design process, where a clear

and obvious set of performance criteria intended for the

life of the component is evident. Using the logic diagram of Figure 4.5, we will proceed to categorize the improvement areas for each of the two identified root

causes.

MIC Attack

The root-cause specification gives a mechanism, MIC,

which was little known at the time of the design specification phase for the EDG heat exchangers. Common

design specifications list expected application environmental factors, such as maximum pressure, temperature,

fluid attributes, fluid velocity, and duty cycle consistent

with operation for a nominal 40-year design life. Thus,

since there is no mention of MIC issues in the design

specification, the answer to the design specification

being exceeded for the MIC mechanism is "No" (follow

the left-hand branch on Figure 4.5). Similarly, for the

next decision block, this causal stressor was not anticipated, leaving the condition designated as "aging." This

agrees with the definition given in the introductory section of this report.

While this condition is recognized as being outside the

original design envelope, future heat exchanger design

efforts would be expected to profit from such experiences and "design out," or at least be based on aging

mitigation using biocidal controls for bacteria shown to

be indigenous.

Erosion (High-Flow Velocity)

The design flow velocities for the EDG's heat exchangers were 4 and 5 ft/s. Measured velocities in the 11 to

13 ft/s range place the operation outside the design specifications (follow the right-hand branch on Figure 4.5).

The high ESW flow through the EDG cooler was not in

compliance with the flow balance procedure. An

application/execution error appears to have occurred.

The plant's root solution to these two degradation mechanisms was to establish a biocide program to control

the MIC organisms (mitigate future MIC attack), and to

provide new system balancing methods to accurately set

component flow rates to near-design values (eliminate

the erosion mechanism).

4.3.3.2 Summary of Conclusions

The ESW problems, their identified root causes, and the

root cause initiators are summarized by Table 4.2.

NUREG/CR-5379 4.14
Root-Cause Methodology

Table 4.2 Root Solution Summary

Problem

EDO heat exchanger

tube pressure

boundary

penetration

Root Cause

1. Microbiologically influenced

corrosion

Solution

1. Biocide injection

to control aging

Categorv

Aging Phenomena

2. High velocity

water erosion

2. Design basis

component

flow rate

Application/Execution Error that

Promoted Aging

This example case study has demonstrated links between

the design specifications of safety-significant system

components, operation of the components within the

total specification set, surveillance testing that measures

the effects of all the active degradation stressors, and the

necessity of a systematic evaluation of the root cause of

component failure. This completes the loop discussed

in Section 13 concerning the feedback mechanism required to effectively implement design-basis regulated

operation.

The importance of understanding the complete set of

operative degradation mechanisms is that it ensures, as

far as is possible, that the previously failed component is

now being operated within the specified design conditions envelope. The advantages of operations within the

envelope are that 1) historically, other similar components have survived for the duration of the design life

under the specified conditions while providing satisfactory performance, L.e, it is a proven, cost-effective manner of operation, and 2) the regulatory requirements of

the operating license are being met.

4.15 NUREG/CR-5379
5 Degradation Analysis

The Phase I degradation analysis applied a root-cause

methodology to plant-specific failure data to determine

failure mechanisms responsible for aging. Undocumented information obtained from plant personnel by

means of the questionnaire protocol was also utilized

during root-cause analysis. Analysis results Indicate that

these techniques lead to successful identification of the

primary age-related failure mechanisms that affect SWS

operability.

In Phase I, effort was concentrated on data from an

open SWS. In Phase II this same methodology was

applied to two additional types of SWSs in an effort to

ensure a complete study of all potential degradation

mechanisms that could affect SWSs.

5.2 Component History Database

Relevant Information obtained from the Phase I site

visits to a recirculating and closed SWS was entered into.

the database developed in Phase I (refer to Table 5.1).

The type of Information and level of detail on plant records from the two additional plants parallel those of the

records previously studied in Phase L The conclusion

reached in Phase I regarding plant record keeping is also

applicable to the additional data acquired during this

phase: close work with plant data has shown several

areas in which plant maintenance record keeping needs

to be improved. Specific improvements are presented in

Section 5.1 of the Phase I report (Jarrell et al. 1989).

5.1 Source of Data 5.3 Analysis of Plant Data

Data obtained in Phase I of this study were derived from

the records of a single commercial reactor with a fresh

water, open-cycle SWS and consisted of 324 plant maintenance records. These records document all SWS failures, surveillance tests, and inspections occurring over a

21-month period. The successful data-gathering techniques of Phase I (system analysis followed by formal

protocol plant visits) were repeated in Phase II to

acquire data from a closed-cycle-type SWS using saltwater, and a fresh water recirculating SWS. The result is

a database that allows cross-plant data to be compared

for common mechanism identification and failure

modes, thus leading to a more confident assessment of

all potential aging mechanisms.

During plant visits for Phase I, 554 maintenance records, covering 9 years and 10 months of operation, were

obtained from a plant using a fresh water, recirculating

SWS. A second plant with a salt water, closed-cycle

SWS provided 968 maintenance records spanning

20 years of operation, although only records dating back

through 1980 were used. These records document all

SWS failures, surveillance tests, and inspections that

took place at each plant during the time span indicated

above.

This section examines analysis of the content of the

component history database; an outline of the approach

taken to analyze and interpret the data; comparison of

cross-plant data for common tendencies and relations;

and an aging assessment of SWSs based on this limited

data, with a summary of conclusions based on that assessment. Tbst results from the Phase I report are used

to compare and substantiate the outcome of the additional SWS assessments.

5.3.1 Failure Classification

Each entry in the database was categorized based on the

type of event documented; these categories are summarized in `Ible 5.2. A Category 1 designates records that

indicate that a failure occurred. Records indicating that

a problem was discovered during a surveillance test or

an inspection are placed in Category 2. Some of these

'problems" are classed as a failure, while others simply

indicate a degraded state of the component. The third

category consists of records of surveillance tests for

which no problems were found. Category 4 records

work which was done in support of events covered by,

Categories I through 3. Design changes and resulting

diagram changes fall under Categories 5 and 6, respectively. Records indicating that work was performed,

5.1 NUREG/CR-5379
Degradation Analysis

Table 5.1 Component History Database Contents Table 5.2 Maintenance Document Categories

Database Field Information Contained Cate2orl Descriltion

Plant Name of plant document came

from

Document l3pe Iype of document data came from

(machinery history, LER, inspections report, etc.)

Document Number Individual document identification

Component Code

Component lpe

Subcomponent

Start Date

Individual component

identification

3pe of component

Specific part within component

boundary

Date problem reported, or date

document initiated

1 Fhilure(a)

2 Surveillance and/or inspection; problem

found(a)

3 Surveillance and/or inspection; no problem

found

4 Support work

5 Design change

6 Drawing change only

7 Work done; reason not stated(ab)

8 Cancelled

9 Unclear(b)

Stop Date Date installation and testing

completed

(a) Root-cause analysis carned out on these categories.

(b) Not enough information to place in any of the previous categories

Category

Comments

Failure

Root Cause

Failure Mechanism

Assigned based on reason document initiated and event outcome

Description of maintenance work

and testing

Specific failure; if it can be determined that a failure occurred

Root cause of failure, problems,

symptoms or other observations

Failure mechanism responsible for

failure

reserved for records that are incomplete and cannot be

placed in Categories 1 through & Entries in Categories

1, 2, 7 and 9 were further analyzed using a formalized

root-cause logic.

53.2 Root-Cause Analysis of Plant Data

It is essential, in the process of evaluating the effect of

aging on the plant, to be able to identify, with a high degree of certainty, the failure mode and mechanism responsible for each failure or problem, and whether this

mechanism is age-related or not. Once this has been accomplished, the effects of aging on risk may be evaluated. The root-cause analysis methodology discussed in

Section 4.2 was utilized in determining the failure mechanisms responsible for the events documented in the

component history database.

but do not indicate why the work was necessary, are

classed as Category 7. Records in Category 8 document

maintenance work that was cancelled. Category 9 is

NUREG/CR-5379 5.2
Degradation Analysis

Results of the Phase I study indicated that the information currently available on plant maintenance records is

not sufficient for assessing aging. The undocumented

detailed information used by plant personnel in reaching their conclusions about each case is needed to obtain

useful results from a root-cause analysis. Tb recover

some of this information, PNL personnel involved in the

process interviewed plant maintenance personnel before

performing the actual analysis; this additional data enabled PNL personnel to assign root causes to events that

did not otherwise have sufficient documentation to determine the root cause of the event. While these interviews improved the accuracy of the root-cause analysis

results, this type of analysis would be most effective if it

were carried out at the plant by those who actually participated in the maintenance work.

The first step in applying root-cause analysis to the plant

data was to assign an initial root cause to each event

classified as belonging to Category 1 or 2. The root

cause was then analyzed by PNL personnel who had

communicated with the maintenance personnel at the

site and assigned a certainty level, based on expert

opinion. PNL personnel who had visited the sites, confirmed or refuted the results of the root-cause analysis

from the documentation of the plant maintenance records. A certainty level, based on expert opinion, was

then assigned to each root-cause analysis, as well as to

the failure mechanism. 7able 5.3 describes how each

level of certainty was assigned.

533 Aging Analysis

The analysis process began by extracting components

from the database containing only failures associated

with aging or unknown failures and certainty values of

95% or greater (Iable 5.3, category 1). 7able 5.4

explains the types of failure mechanisms used to analyze

the SWS systems.

Figures 5.1 through 5.6 illustrate failures per year per

type of failure mechanism. The failure mechanisms per

year were determined by the year in which the work was

performed. The year chosen was based on the component's work start date; if that date was not provided, the

ending year date was used. If neither the work beginning or completion date was recorded in the records, the

failure was not entered into the graph. This loss of

information accounted for only a small percentage of

records not being displayed: open system, liess than

10%; closed system, less than 1%; open recirculating

system, approximately 0%. It can be stated that the

results will show an accurate representation of the

failure relations, given the data provided.

5.3.3.1 Open SWS Analysis

Figures 5.1 shows the number and relationship of failure

rates per year and the type of failure mechanism associated with each failure for an open SWS. Figure 5.2

shows the distribution of components to types of failures. As determined in the Phase I report, corrosion

was the largest contributor to failure, compared to other

aging mechanisms. However, it may appear from Figure 5.1 that there was a problem in 1986. It should be

noted that the data collection was taken from 1985

through 1986 and partially through 1987, thereby skewing the data to 1986, since there is a complete record of

1986 but not of 1985 or 1987. A majority of the degradation and failure events of less than 95%- cause certainty were categorized as corrosion-related, but because

of the lower certainty value, were not included in Figure 5.1. Figure 5.2 exhibits, as mentioned, corrosion as

the major contributor to failure, with biofouling and

wear being the next largest contributors totfailures.

Figure 5.2 also confirms that valves are more likely to

corrode rather than to wear or be subjected to biofouling. This figure indicates that pumps and motors

TAble 5.3 Root-Cause Analysis Certainty Assignments

Classification

1

2

3

Certainty

<95%

<75%

<50%

4

5.3 NUREG/CR-5379
Degradation Analysis

Table 5.4 Service Water System Principal Aging Degradation Mechanisms

Degradation

Mechanism Abbreviation Definition Sample SWS Manifestations

Corrosion COR (Al) Destructive attack of a metal by reaction

with its environment.

Erosion ERO (A2) Metal removal by high-speed liquid or

suspended solid impingement.

Cavitation CAV (A3) Local fluid pressure drops below

saturation producing voids that subsequently collapse, resulting in metal

removal.

Vibration VIB (A4) Low displacement, high-frequency cyclic

stress.

Fatigue FrG (A5) High frequency stress below the yield

strength of a material resulting in

failure by cracking.

Thermal THC (A6) Stress changes in a material caused by

Cycling variations in temperature.

Wear WER (A7) Removal of surface material by sliding

contact between it and another surface.

Biofouling BIO (A8) Buildup of a biological species.

rust, valve contact surface adhesion,

formation and transport of metallic

oxides, leaching (selective alloy metal

removal), through-wall-perforation

pump impeller metal removal, elbow wall

metal removal, heat exchanger tube wall

metal removal

pump impeller erosion, pump casing

erosion, valve control surface loss

fretting (e.g., baffle-tube wall thinning),

or corrosion fatigue

progressive fracture (fatigue or corrosion

fatigue) from points of high stress

concentration

pipe coating fracture, tube-to-tube sheet

break

bearing surface removal

heat exchanger tube plugging, pipe flow

area reduction, slime mold preventing

heat transfer

Accumulation

Structural

Impact

ACU (A9) Chronic inorganic solids precipitation

or radial deposition on fluid or heat

transport surfaces.

STI (AlO) A small number of cyclic loads applied

to a material at a high rate and short

duration.

bottom buildup in pipe, flow area

reduction

denting or physical deformation of

otherwise undamaged material (water

hammer, exterior impact)

pipe coating material, gasket material,

wrenches, cloth maintenance items,

detritus

Foreign

Debris

DBR (All) Rapid accumulation of material of significant size from a source outside the

component

Unknown U Less than 95% confidence in the

accuracy of the fundamental degradation stressor.

symptoms of degradation given as cause,

incomplete information, contradictory

information

NUREG/CR-5379 5.4
Degradation Analysis

120

100

I

I

-THC

U

60 r n

40

20

0 COR| 85 86 87

Calendar Year

Figure 5.1 Open Service Water System Failure Rate Distribution

80 -

60 I-

.R

16IL

E

z

E

im

0

U3

Pumps/Mtrs

Valve/Ops

Heat Exch

Pipes

Other

Stm/Scm 40

20-

0-

Al A7 A8 A9 A10 All U

Mechanism

Figure 5.2 Open Service Water System Component Failure Distribution

5.5 NUREG-5379
Degradation Analysis

*o0

2

IL

Is

z

80

60

40

20

FBIO0

-WERj

OR

79 80 81 82 83 84 85 86 87 88

Calendar Year

Figure 5.3 Closed Service Water System Failure Rate Distribution

150 1

: 100

Ia

50

{i 50

;C

* PumpslMtrs

Ea Vavt*Oft

M Heat Exch

E Pipes

o Other

E Strn*scm

0

Al A2 A7 AS A9 A10 All U

Mechanisms

Figure 5.4 Closed Service Wter System Component Failure Distribution

NUREG-5379 5.6
Degradation Analysis

I.

50

II

*1

z

.-

30 -

20 \

10

82 83 84 85 86 87 88

Calendar Year

Figure 5.5 Recirculating Service Water System Failure Rate Distribution

200 -

I

Is0

is

Pumps/Mtrs

ValvestOPs

Heat Exch

Pipes

Strn/Scm

Other/Sensors

100 -

Z-AC7Z0-

Al A2 A7 AS Ag A10 All U

Mechanisms

Figure 5.6 Recirculating Service Water System Component Failure Distribution

5.7 NUREG-5379
Degradation Analysis

are more likely to fail due to wear, and that screens and

strainers are more likely to fail because of biofouling.

The largest number of failures were of the unknown

category. Of that category the valves class contributed

approximately 70% to the number of failures.

5.3.3.2 Closed SWS Analysis

Figure 5.3 and Figure 5.4 display the failures for a closed

SWS. Figure 5.3 specifically shows failure rates per year

from pre-1980 through 1988. From 1980 to 1988 the

number of reported failures increased, with a dramatic

increase in 1988. This apparent trend is the result of

action imposed by the NRC on all nuclear power plants

to maintain a more accurate description of the maintenance work being performed.

Figure 5.4 shows that the largest failure is the result of

biofouling in the heat exchangers, with corrosion of the

valves the second largest contributor to failures. Wear

in valves, pumps, and motors also contributes a high

fraction of failures. The classes of heat exchangers,

valves and pumps, and motors create the main components that would be most likely to fail.

5.3.3.3 Recirculating SWS Analysis

Figures 5.5 and 5.6 display the failures for a recirculating

SWS. Figure 5.5 specifically shows number of failures

per year from 1982 through 1988. From 1984 to 1986

the number of failures appeared to decrease, but showed

a marked increase in 1988. As mentioned in the closed

system analysis, this trend was due to the action imposed

by the NRC on all nuclear power plants to maintain a

more accurate description of the maintenance work

being performed. The decrease from 1984 to approximately 1987 was due to the corrosion treatment program instituted by the facility.

classes--pumps and motors, valves, and sensors. The

recirculating SWS's main failure component was the

sensors. The number of unknown root causes in Figure 5.5 indicated that not enough critical information

was provided to ascertain the root cause of the failure.

5.3.4 Principal Degradation Mechanisms

The major findings in examining the three types of

SWSs are as follows:

Corrosion, fouling (biological and inorganic), and

wear were the three major failure mechanisms.

Valves and valve operators were the most likely

components to fail.

Failure of components could be correlated with a certain type of failure mechanism (e.g., valves were more

likely to fail because of corrosion, and pumps were

more likely to fail because of wear).

Failure and routine maintenance information is not

complete enough to perform an effective root-cause

analysis.

The data indicated that the recirculating SWS had the

least failures per year, assuming that the documentation was comparable to the other SWSs.

53.5 Control Procedures

A range of water chemistry control procedures for each

system type is summarized in another publication

(Johnson and Jarrell 1991). Other surveillance procedures used to counter SWS degradation are also

summarized.

The recirculating SWS's primary failure mechanism, as

shown in Figure 5.6, was corrosion. Corrosion appeared

to mainly affect sensors and valves. The unknown

category in Figure 5.6 consisted of three component

NUREG/CR-5379 5.8
6 Conclusions

In keeping with the stated goals of this report, the

Phase II SWS aging investigation produced the following conclusions:

1. The primary SWS degradation mechanism of corrosion, compounded by biological and inorganic accumulation, has been validated through the analysis of

a substantially broadened and verified component

failure database.

Discussion

monitoring for induced stressors [e g., leaching (denickelfication)J of Cu/Ni heat exchangers by a

chlorination blocide).

3. A demonstrated need for a systematic methodology

for performing and documenting an effective rootcause analysis of component failure resulted in

developing the mechanistic correspondence

approach explained in Chapter 4.

Discussion

Whereas, the Phase I report conclusion was

reached almost entirely from information

obtained from one type of SWS, Phase It

includes an in-depth review of machinery

histories from three different plant SWS types.

Industry-wide data from the NRC, contained in

the report issued by the AEOD (Lam and Leeds

1988), also support this primary conclusion.

2. A corollary to this primary conclusion is that, in

general, the most effective means of mitigating SWS

degradation is to pursue control methods that

reduce the rate of corrosion and biofouling.

Discussion

This conclusion suggests that a necessary

element to minimization of degradation in any

type of SWS is effective control of water chemistry. A range of water chemistry treatments is

available with options that apply specifically to

one or all of the SWS types. Where confirmed

biological agents are active, a biocontrol program (chemical, thermal, etc.) is an obvious

requirement. Environmental regulations can

limit the extent to which some of these solutions can be pursued, and a careful consideration of alternatives is necessary to select the

most cost-effective solution. Any changes to

the control program should be accompanied by

The formalized heat exchanger root-cause

analysis method developed during this task has

been shown to be an effective guide to performing RCA in a field environment. The methodology is generic in that, by developing degradation

mechanistic models of components other than

heat exchangers, the same technique can be

applied. It is recommended that an RCA process having the elements described in Chapter 4

be required for all corrective maintenance work

orders performed on safety-related portions of

the SWS.

4. The most reliable and complete data source available for determining age-related degradation in

nuclear power plant systems and components resides in the "corporate memory" of the engineers,

operators and maintenance personnel at the plant

Discussion

Despite vast improvements in reporting criteria

and computerized media, the root-cause failure

knowledge gained by operating and maintaining

the plant systems is not being communicated in

sufficient depth to allow confident conclusions.

Implementing the RCA methods developed for

this report and documenting the failure specification and degradation correspondence is designed to assist in filling this information/

documentation void.

6.1 NUREO/CR-5379
7 References

10 CFR 50. 1983. U.S. Nuclear Regulatory Commission, 'Domestic Licensing of Production and Utilization

Facilities.' U.S. Code of Federal Regulations.

54 FR 43209-10. October 23,1989. Generic

Letter 89-13, Office of Nuclear Regulatory Research,

U.S. Nuclear Regulatory Commission, 'Service Water

System Problems Affecting Safety-Related Equipment.'

FederalRegister.

Daling, P. M., D. L Stiles, S. A. Weakley, and

K L Johnson. 1989. Value ImpactAnalyisfor Generic

Issue 51: Improving the Reliability of Open-Cycle ServiceWaterystems. NUREG/CR-5234, PNL-6668, prepared

by Pacific Northwest Laboratory for the U.S. Nuclear

Commission, VvWshington, D.C.

Dawson, J. L, C. C. Shih, D. Gearey, and R. G. Miller.

1990. "Flow Effects on Erosion Corrosion - An Engineering Approach.* Presented at the National Association of Corrosion Engineers (NACE) Meeting, April

1990, Las Vegas, Nevada.

Hayes, P. 1983. Generic Safety Issue No. 51, Improved

Reliability of Open Service Water Systems. USNRC

memorandum from W. Minners, April 5,1983.

Jarrell, D. B., A. B. Johnson, Jr., P. W. Zimmerman, and

M. L Gore. 1989. Nuclear Plant Service Water System

Aging Degradation Assessment, Phase I. NUREG/CR5379, VoLl, PNL-6560, prepared by Pacific Northwest

Laboratory for the U.S. Nuclear Regulatory Commission, Washington, D.C.

Johnson, A. B., Jr., and D. B. Jarrell. 1991. 'Corrosion

and Corrosion Control in Nuclear Service Water Systems.' In Proceedings oftheANS Reactor Operations

Division Topical Meeting, August 1991, Bellevue,

%shington. PNL-SA-19805, Pacific Northwest

Laboratory, Richland, Washington.

Lam, P. S., and E. J. Leeds. 1988. OperatingExperience

FeedbackRepor Service Water System Failures and

Degradations. NUREG-1275, Vol. 3, Office for Analysis

and Evaluation of Operational Data, U.S. Nuclear

Regulatory Commission, Washington, D.C

Murphy, G. A., et al. 1984. Survey of Operating

Experience from LERs to IdentifyAging Trends.

NUREG/CR-3543, prepared by Oak Ridge National

Laboratory, for the US. Nuclear Regulatory Commission, Washington, D.C.

Neitzel, D. A., and K L Johnson. 1988. Technical Findings Documntfor Generic Issue S): Improving the Reliability of Open-Cycle Service-Water Systems. NUREG/

CR-5210, PNL-6623, prepared by Pacific Northwest

Laboratory for the U.S. Nuclear Regulatory Commission, Washington, D.C.

Stratton, R. C., and D. B., Jarrell. 1989. 'A KnowledgeBased System for Heat Exchanger Root-Cause Analysis.' In Proceedings of Erpen Systems Applications to the

Electric Power Industny. Electric Power Research Institute (EPRI) Orlando, Florida.

Uhhg, HF H. and R. W. Revie. 1985. Corrosion and Corrosion Control: An Introduction to Corrosion Science and

Engineering 3rd ed. John Wiley & Sons, New York.

U.S. Nuclear Regulatory Commission (NRC). 1976.

Revision to January 1976for Commen*:, Ultimate Heat

SinkforNuclearPowerPlants. NRC Regulatory

Guide 1.27, U.S. Nuclear Regulatory Commission,

%shington, D.C.

U.S. Nuclear Regulatory Commission (NRC). 1978a.

Seismic Design Classification. NRC Regulatory

Guide 1.29, U.S. Nuclear Regulatory Commission,

Washington, D.C.

7.1 NUREG/CR-5379
References

U.S. Nuclear Regulatory Commission (NRC). 1978b.

Standard Format and Content of Safety Analysis Reports

for Nuclear Power Plants. NRC Regulatory Guide 1.70,

U.S. Nuclear Regulatory Commission, Washington,

D.C.

U.S. Nuclear Regulatory Commission (NRC). 1991.

Nuclear PlantAgingResearch (NPAR) Program Plan.

NUREG-1144, Rev. 2, U.S. Nuclear Regulatory Commission, Washington, D.C.

U.S. Nuclear Regulatory Commission (NRC). 1989.

Severe Accident Risks: An Assessmentfor Five US.

NuclearPowerPlants. NUREG-1150, Vol. 1, U.S.

Nuclear Regulatory Commission, Washington, D.C.

NUREG/CR-5379 7.2
NUREG/CR-5379

PNL-7916

Vol. 2

Distribution

No. of

Copies

No. of

Copies

OFFSrDE

20 J. J. Burns (20)

U.S. Nuclear Regulatory

Commission

MS: NS217B

Vashington, DC 20555

D. A. Casada

Oak Ridge National Laboratory

MS: 8038

Oak Ridge, TN 378314038

H. L Magleby

Idaho National Engineering

Laboratory

MS: 2406

Idaho Falls, ID 83416

S. P. Nowlen

Sandia National Laboratories

Adverse Environment Safety

Assessment Division 6447

Albuquerque, NM 87185

M. Subudhi

Brookhaven National Laboratory

Building 130

Upton, Long Island, NY 11973

M. Vagins

U.S. Nuclear Regulatory

Commission

MS: NS217B

Washington, DC 20555

J. P. Vora

U.S. Nuclear Regulatory

Commission

MS: NS217B

Washington, DC 20555

ONSITE

41 Pacific Northwest Laboratory

R. P. Allen (10)

S. J. Bohn

M. L Gore

D. B. Jarrell (20)

A. B. Johnson, Jr.

L L Larson

R. C. Stratton

Publishing Coordination

Technical Report Files (5)

Distr.1
NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION I. REPORT NUMBER 12491 I~ulgnd bv NIC. Add V.J.. Swo, Pmey

NRCM 1 Xt102, Wd Addm NumbersN If aniy.

3201.3202 BIBUOGRAPHIC DATA SHEET

t"n2hy bututn Janusw UREG/CR-5379 _~ ~ ~ ~ ~~~S I~fItOP PNL-791 6

2. TITLE AND SUBTITLE Vol. 2

Nuclear Plant Service Water System Aging Degradation Assessment

3. DATE REPORT PUBLISHED

Phase 11 MONTH I YEAR

October 1992

4. FIN OR GRANT NUMBER

B2911

5. AUTHORIS) 6. TYPE OF REPORT

D.B. Jarrell, L.L. Larson, R.C. Stratton, S.J. Bohn, M.L. Gore

7. PERIOD COVERED ftiaim, Oatsi

5. PERFORMING ORGANIZATION -NAME AND ADDRESS Ita RC. DiO*V aOfIc Ad A Rio. UAS.A Ad Cn,.dAd mam wd - admLJ

Pacific Northwest Laboratory

Richland, WA 99352

8. SPONSORING ORGANIZATION-NAME AND ADDRESS WfRC. ip sme .3w. II r,.DmMNZrCA'iOfl. Orfkorgovqi.N US MwReguibrworCnwnhlion,

Division of Engineerina

Office of Nuclear Regulatory Research

U.S. Nuclear Regulatory Commission

Washington, DC 20555

10. SUPPLEMENTARY NOTES

11. ABSTRACT 200 mds rbWai

The second phase of the aging assessment of nuclear plant service water systems (SWSs) was performed by the Pacific Northwest Laboratory (PNL) to support the U.S. Nuclear Regulatory Commission's (NRC's) Nuclear Plant Aging Research (NPAR)

program. The SWS was selected for study because of its essential role in the mitigation of and recovery from accident scenarios

involving the potential for core-melt, and because It Is subject to a variety of aging mechanisms. The objectives of the SWS task

under the NPAR program are to identify and characterize the principal age-related degradation mechanisms relevant to this

system, to assess the impact of aging degradation on operational readiness, and to provide a methodology for the management

of aging on the service water aspect of nuclear plant safety.

The primary degradation mechanism in the SWSs, as stated in the Phase I assessment and confirmed by the analysis in Phase II,

is corrosion compounded by biologic and inorganic accumulation. It then follows that the most effective means for mitigating

degradation in these systems is to pursue appropriate programs to effectively control the water chemistry properties when possible and to use blocidal agents where necessary.

A methodology for producing a complete root-cause analysis was developed as a result of needs Identified in the Phase I assessment for a more formal procedure that would lend itself to a generic, standardized approach. It is recommended that this, or a

similar methodology, be required as a part of the documentation for corrective maintenance performed on the safety-related

Portions of SWSs to provide an accurate focus for effective manauement of amine.

12. KEY WORDOS/ESCRIPTORS ftiveri wuen W Ad Ad 13. AVAIN ILITY STATEMENT

Nuclear Power Production Unlimited

System Aging 14. SECURITY CLASSIFICATION

Service Water System _ _

Component Degradation -Unlimited

Root Cause Analysis- InR"Werl

fatilure lechfntsis- Unlimited

15. NUMBER OF PAGES

16. PRICE

NRC FORM 335 249)
Federal Recycling Program
NvKEfG/u*-bT/9'/ Vol. 2 - NNULhA PeAm4i SXVL bI WAI'M 1 IJISMAtI M1Nti IUftAlAA11'N ASSESSMENU Ot:rOBER 1992

UNITED STATES

NUCLEAR REGULATORY COMMISSION

WASHINGTON, D.C. 20555-0001

FIRST CLASS MAIL

POSTAGE AND FEES PAID

USNRC

PERMIT NO. G-67

OFFICIAL BUSINESS

PENALTY FOR PRIVATE USE, $300