- 1 -

Operating Experience Smart Sample (OpESS) FY 2009-02

ANegative Trend and Recurring Events Involving Feedwater Systems@

Note: This is a public NRC document. 

Highlighted hyperlinked documents should have active links. 

OBJECTIVE: 

To support NRC inspector review of feedwater concerns by providing both main feedwater and 

related backup systems (auxiliary / emergency feedwater or RCIC/HPCI/CI) inspection guidance 

and references to recent operating experience (OpE) information related to various feedwater 

issues. This OpESS provides follow-up for NRC Information Notice 2008-13: Main Feedwater 

System Issues and Related 2007 Reactor Trip Data. 

NOTE: This OpESS is a voluntary inspection sample performed under the baseline program. When 

performed it should be documented in an inspection report under the baseline inspection procedure 

selected as described in the REPORTING INSPECTION RESULTS / TIME CHARGES / 

ADDITIONAL ISSUES section for this OpESS.

BACKGROUND: 

Based on the recent negative trend information and several feedwater related issues and concerns, 

the OpE clearinghouse and NRR management directed development of this Operating Experience 

Smart Sample (OpESS) to supply operating experience and focus inspector attention in this area. 

NRR staff review of recent operating experience related to main feedwater identified recurring 

events and a negative trend issue, [see NRC Information Notice 2008-13: Main Feedwater 

System Issues and Related 2007 Reactor Trip Data ]. The staff identified a negative trend 

related to percentage of 2007 reactor scams caused by feedwater related events. Specifically, 37% 

of all scrams were initiated as a result of feedwater related events (typically, feedwater related 

scrams caused between 15 and 25% of all scrams). 

A review of more than 200 NRC main feedwater related findings generally involved either the failure 

to perform proper preventive maintenance/ maintenance errors or corrective action / maintenance 

errors more than half (~53%) of the time. 

NRR OpE staff also reviewed the backup systems Technical Specification and /or safety-related 

feedwater system inspection findings and noted 17 - “greater than green” findings. Auxiliary 

feedwater pumps (at PWR units) were the primary contributor in 14 of 17 - “greater than green” 

findings. 

Additionally, the staff noted potential negative trends in the area of main feedwater system scrams 

and transients being caused by instrumentation and control (I&C) issues. Specific component area 

of concern issues were often focused on main feed regulating valve issues. Additionally, there may 

a growing area of concern related to digital I&C related issues. 

An event that occurred at the Perry site (a BWR unit) on November 28, 2008 also highlights the 

complications that can arise from these digital I&C related issues and feedwater systems. 
- 2 -

Additional background documents with links describing this Perry event and the Special Inspection 

review of this event are provided in the source documents section below. 

SOURCE DOCUMENTS (with active links): 

1) NRC Information Notice 2008-13: Main Feedwater System Issues and Related 2007 

Reactor Trip Data -available in ADAMS ML ML080880115 

2) The Perry SIT (on digital I&C feedwater trip) Inspection Report can be viewed at ADAMS: 

ML080280499

NOTE: Baseline inspection procedures are also considered source documents and are to be used 

for performing an OpESS. These may include inspection procedures used for various feedwater 

system alignment inspections, feedwater surveillance testing, post maintenance testing or PI&R 

corrective action follow-up inspections related to various feedwater issues. Links to these base line 

procedures are provided in the inspection guidance section below. Inspection hours charged for 

this OpESS are to the baseline inspection procedure used.

CORNERSTONES: Initiating Events (50%), Mitigating Systems (50%) 

APPLICABILITY: All licensed operating commercial nuclear reactors. This OpESS provides additional 

information related to events involving Main Feedwater systems for all plants and Technical 

Specification / safety-related backup feedwater systems: (auxiliary feedwater/emergency feedwater for 

PWRs) or (RCIC/ Isolation Condenser and HPI systems for BWRs). 

INSPECTION GUIDANCE: 

$ IP-71111.04, AEquipment Alignment” 

$ IP-71111.15 “Operability Evaluations”

$ IP-71111.19 “Post Maintenance Testing”

$ IP-71111.22 “Surveillance Testing”

$ IP-71152 “Problem Identification and Resolution” 

The above procedures provide baseline inspection program attachments that are used to support 

various feedwater inspections and follow-up to feedwater related issues. Review, as applicable – 

(ideally during IP-71111.04, “an Equipment Alignment,” for the site specific “safety-related” feedwater 

systems). The additional feedwater inspection guidance is described below and in the OpESS 

Attachment. 

1) Review the Background and Source Document Sections of this OpESS and the related documents 

as needed to obtain a general understanding of some of the various feedwater related concerns (i.e., 

purpose is for inspectors to detect issues on the non-safety main feedwater systems prior to selfrevealing transient events). Additionally, it is also desired that this OpESS assist NRC inspectors in 

detecting issues earlier for important to safety “backup” feedwater related systems, to ensure those 

system readiness, and thereby help improve plant safety and reduce risk and the number of “greater 

than green” findings in these areas. Review any licensee corrective action response to: NRC 

Information Notice 2008-13: Main Feedwater System Issues and Related 2007 Reactor Trip Data.

a.) An additional recent (10/22/2008) issue is described in Event Notice (EN) 44593 Pilgrim, “RCIC 

Inoperability Due to Flow Controller Components with Age Well Beyond Recommended 
- 3 -

Replacement Age.” In this EN 44593, Pilgrim site stated several of the capacitors installed in the 

controller were noted to be between 21 to 30 years of age. Industry recommended replacement interval 

for the capacitors is typically between 7 to 10 years of age. PNPS engineering review in conjunction 

with Entergy fleet consultation concluded that there is no definitive technical bases to provide a 

reasonable expectation that the RCIC flow controller function can be assured throughout its mission 

time due to the capacitor aging concern. 

Keep these earlier findings and issues in mind and review these items as needed for similar issues/ 

concerns at your site. 

2) During equipment alignment walkdowns, surveillance testing, post maintenance testing and 

corrective action followup inspections, as applicable, inspect for similar issues to those noted in the 

Background Documents and the Source Documents (i.e., similar findings). 

3) Review feed regulation valve issues and related licensee related maintenance and corrective 

actions related to these valves should they occur to verify that necessary corrective actions 

commensurate with the significance of the issue have been identified and implemented by the 

licensee. 

4) (a) For sites with specific NUREG “risk-based inspection guides” listed on the following page 

review this guidance for AFW / EFW systems (PWRs) or HPCI systems (BWRs) provided for 

“your specific site” as described in the “risk based inspection guide” NUREG. 

NOTE: USE THESE NUREGS WITH CAUTION as this is “guidance material only.” Realize that 

these NUREGS were developed in the early 1990’s and may be out of date regarding systems 

lineups (etc.) and may be in need of revision due to subsequent plant modifications or other 

changes that may have been implemented since these documents were initially developed. 

However, the inspection techniques and methodologies derived from these “risk bases inspection 

guides” should still provide useful inspection insight. 

(b) If your “specific” site is not represented by any of these linked NUREG “risk based inspection 

guides” then review a similar site’s (sister plant/peer group plant) to understand the techniques 

demonstrated by this “risk based inspection guidance” material which should still provide useful 

information for inspectors to review and consider during their own site inspections. 
- 4 -

Review the appropriate site specific risk-based NUREG inspections guidance for insights, as 

applicable. NOTE: USE THESE NUREGS WITH CAUTION as this is “guidance material only.” 

Realize that these NUREGS were developed in the early 1990’s and may be out of date. 

For PWR Site(s) /”Risk Based Inspection Guides” (for AFW/EFW systems)–NUREG/CR #

Arkansas Nuclear One (ANO) - Unit 2 / NUREG/CR-5828 

Beaver Valley / NUREG/CR-5835 

Braidwood and - 

Byron Sites / NUREG/CR-4427 

Callaway/ NUREG/CR-6357 

Catawba/ NUREG/CR-5827 

Comanche Peak/ NUREG/CR-5831 

Cook, D.C./ NUREG/CR-5832 

Crystal River-3/ NUREG/CR-5467 

Diablo Canyon Unit 1/ NUREG/-5616 

Farley, J.M./ NUREG/CR-5617 

Fort Calhoun/ NUREG/CR-5834 

Ginna/ NUREG/CR-5764 

Kewaunee/ NUREG/CR-5821 

McGuire/ NUREG/CR-5830 

North Anna/ NUREG/CR-6837 

Palo Verde/ NUREG/CR-5836 

Point Beach/ NUREG/CR-5898 

Prairie Island Units 1 and 2/ NUREG/CR-5839 

Robinson, H.B./ NUREG/CR-5833 

Saint Lucie Unit-1 / NUREG/CR-5896 

Salem/ NUREG/CR-5761 

San Onofre Unit-2/ NUREG/CR-5766 

South Texas Project/ NUREG/CR-5897 

Summer, V.C./ NUREG/CR-5838 

Three Mile Island Unit 1/ NUREG/CR-5488 

Turkey Point/ NUREG/CR-5633 

For BWR Sites/ Risk Based Inspection Guides (for HPCI systems) – NUREG/CR #

Browns Ferry/ NUREG/CR-6022 

Dresden Units 2 & 3 / NUREG/CR-5933 

Fermi, Enrico - Unit-2/ NUREG/CR-5959 

Hatch/ NUREG/CR-6014 

Hope Creek/ NUREG/CR-5923 

Pilgrim/ NUREG/CR-5924 

Quad Cities Units 1 and 2/ NUREG/CR-5934 

Susquehanna / NUREG/CR-5932 

Note: This completes the OpESS inspection, however, other feedwater inspection guidance for 

consideration by inspectors is also provided in the “ATTACHMENT” to OpESS FY2009-02 Negative 

Trend and Recurring Events Involving feedwater systems.” 
- 5 -

REPORTING INSPECTION RESULTS / TIME CHARGES / ADDITIONAL ISSUES: 

Document any inspection result findings, as applicable, in an integrated inspection report (i.e., 

quarterly inspection report/ PI&R or CDBI report) and reference the title/ OpESS number (example: 

AReview of Operating Experience Smart Sample: OpESS FY2009-02,A Negative trend and 

Recurring Events Involving feedwater systems.@ If no findings are identified document 

completion of the OpESS using the AOpESS number/ title@ under the applicable inspection 

attachment (i.e., 1R04,) stating that no findings of significance were identified. 

Inspection time for this OpESS is to be charged to the normal baseline procedure under which it is 

being documented (along with any routine preparation and documentation charge times). 
- 6 -

ATTACHMENT

(OpESS FY2009-02 “Negative trend and Recurring Events Involving Feedwater Systems 

(additional inspection guidance for consideration)

1) Consider establishing a site specific “safety-related” feedwater or HPCI inspection notebook (as 

applicable) that retains important reference information and guidance for use related to their site 

specific AFW/ EFW (for PWRs) and RCIC / Isolation Condenser/ HPCI (for BWR) system inspections, 

operability issue reviews, etc. This handbook can be used during new resident training and during 

turnover at the site. Consider verification and updating, as necessary, of the NUREG risk based 

inspection guides for your specific site; or based on a sister/ Peer group similar plant site listed under 

the Inspection Guidance Item 4 above of this OPESS. NOTE: USE THESE NUREGS WITH CAUTION

as this is “guidance material only.” Realize that these NUREGS were developed in the early 1990’s

and may be out of date. 

The below information provides some general feedwater inspection guidance also for consideration 

in a feedwater system notebook development. 

2) Prior to performing baseline inspections on the feedwater systems, inspectors should consider 

reviewing the licensee CAP documents (perform a search of the licensee CAP for feedwater system 

related issues) to look for negative feedwater system trends or specific CAP issues that should be 

examined further. Review Maintenance Rule 10 CFR50.65 documentation [any (a1) status corrective 

actions status, etc). Look for incomplete actions, inadequate or untimely corrective actions related to 

feedwater issues. 

3) Prior to performing baseline inspections on the feedwater systems, inspectors should consider 

reviewing outstanding (open) work orders, work history / maintenance records, equipment deficiencies, 

and any temporary system modifications existing on the feedwater system. Review these to ensure 

they have been appropriately entered into the licensee’s CAP or other appropriate program and have 

been resolved or are being resolved in a timely manner, appropriate to their level of safety significance. 

4) Review typical operational parameters based on the licensee procedures /vendor guidance/ for 

safety-related feedwater design basis and Technical Specification surveillance requirements for the 

applicable site feedwater system to ensure that it is operable. Review current and past feedwater 

operating data recorded during testing including applicable system engineer trend analysis data, as 

necessary. 

5) To ensure safety-related feedwater system readiness for operation, inspect during routine plant 

status walkdowns, or during other base line inspections related to the feedwater system the following 

items, as necessary (refer to IP-71111.04). Some of these may need to be verified by actual 

observation during licensee operator rounds, or licensee conducted operational testing or surveillance 

test runs. These feedwater system operability type items typically include, but are not limited to, the 

following types of operational parameters (which may include minimum/maximum and or “expected”/ 

normal values), If values are in question ensure licensee addresses them appropriately: 

a.) Air pressure for air controlled valves properly set 

b.) Various Tank levels properly maintained: 

 i.e., Condensate storage tank level (PWR) or RCIC Storage tank level (BWR) 

c.) Pump lubricating oil - proper levels 

d.) Cooling supply alignments for pumps 
- 7 -

e.) Cooling water or keep-warm oil temperatures properly maintained 

f.) TDEFW / RCIC Governor Settings properly maintained 

g.) Turbine Overspeed trip level (in the correct run position) 

Additional turbine driven pump note: A risk data review obtained from a Research Report – 

“Component Performance Study -- Turbine Driven Pumps (TDP) 1998-2006,” identified some 

statistically significant increasing and decreasing trends. This item provides the figures mentioned 

below. 

Increasing trends included: Standby systems, industry-wide TDP Failure To Run (FTR) > 1Hour 

trend. (See Figure 3 in link above) and Frequency (events per reactor year) of FTR > 1Hour events, 

standby TDPs. (see Figure 14 in link above) 

These two trends are actually the same increase due to Failure To Run > 1Hour events. There were 

five Failure To Run > 1Hour events in the standby TDPs. Four of the five events occurred during in 

the time period FY 2004 to FY 2006. In addition, the run hours associated with Failure To Run > 1 

Hour for the standby systems decreased during these same years. 

Statistically significant decreasing trends were identified from this TDP study for the following: 

 · Pooled AFW, HPCI, and RCIC TDP Unavailability (UA) trend. (see Figure 6, in link above) 

 · Frequency (events per reactor year) of start demands, standby TDPs. (see Figure 9,) 

 · Standby TDP run hours per reactor critical year of run ≤ 1Hour hours. (see Figure 10 in 

 link above) 

h.) Check for unusual noises during system operation (that may indicate signs of mechanical trouble) 

and ensure these are investigated and documented appropriately by the licensee. 

i.) Check for unusual smells that may indicate leaks, over-heating, and for any smoke or burned 

discolored paint, etc. that may indicate problems. 

j.) Review the room for proper transient combustible and fire ignition control (i.e., hot work controls). 

k.) Check the TDEFW pump / RCIC/ HPCI pumps for fluid/air system or exhaust system leaks (cooling 

system, steam system, lube oil system, exhaust system, air system and ensure the licensee is aware of 

and captures these in the appropriate response program (i.e., captured in a CAP or work 

order/equipment deficiency programs, as applicable). 

l.) Verify the steam driven pumps shuts down properly in accordance with licensee procedure 

(observing for any abnormal conditions, such as excessive vibration, etc.). 

 

m.) Review operator logs and procedures during and following surveillance testing (to ensure they are 

followed and that they are properly conducted and the necessary operating data is correct. Pay 

particular attention to any notes of comments recorded in the procedure(s) by operators that may 

indicate abnormal conditions exist (verify necessary actions such as work orders/ corrective action 

documents were generated, in response to these notes). 

n.) Discuss concerns of feedwater system performance with the appropriate personnel to gain their 

insights into potential issues, consider discussions with various: Operator(s), Maintenance personnel or 

System Engineer(s). 
- 8 -

t.) Review available trending reports comparing operating logs over extended time that may indicate an 

impending or slowly developing issue or negative trend. 

6) During maintenance activities review foreign material exclusion (FME) controls to ensure licensee is 

maintaining system cleanliness (especially during extended or refueling outages). 

 

7) Periodically observe and/or review, as applicable, feedwater related In-Service Testing (IST) 

vibration data results and discuss this vibration trend analysis with the appropriate engineer(s). 

 

8) Consider for review any feedwater related permanent plant modifications (using IP-71111.17) or 

any related temporary modifications (using IP-71111.23), and review proper use of temporary 

scaffolding in these areas. 

9) Review, as necessary, any feedwater system troubleshooting activities to ensure licensee 

properly identifies the root cause (when needed) under the corrective action process. Diagnostic 

aids and reference materials that may assist the inspectors in this review may include such items 

as: Piping and Instrumentation diagrams (P&IDs), equipment drawings, system descriptions, 

maintenance/work history records, previous operating logs, CAP data base, vendor diagrams/ 

drawings, manufacturer maintenance manuals including; troubleshooting charts, parts lists, and 

operating experience data including INPO EPIX data, vendor owner’s group alert lists, etc. Also 

observe /conduct discussions between and with the system engineer(s), maintenance engineer(s), 

vendor service representatives and various troubleshooting team members and supervisors, as 

necessary. 

10) Periodically review any feedwater system health reports (maintenance rule reports, etc), on-line 

computer monitoring programs, or other unique diagnostic tools licensees use for feedwater 

systems for any diagnostic or performance trending insights they may provide in following up on 

feedwater related issues. 

11) During and/or following painting in the feedwater system pump room(s) verify painting activities 

do not adversely impact the feedwater system operability. 

12) Periodically review various feedwater support systems and their capability to support feedwater 

system operability (i.e., ventilation (heating ventilation and air conditioning), service water or other 

cooling systems, pump room sumps and drains/ flood control, fire suppression systems, and 

various electrical power supplies to feedwater system controls and protections (including alternate 

shutdown functions, as applicable). 

13) Periodically review specific electrical components and circuits that affect the feedwater systems 

operation - (i.e., the specific testing of sequence logic controls and relay testing, SI start initiator 

relays, load shed logic controls and relay testing, as applicable for safety related systems). During 

surveillance testing of these relays, and other specific components that impact feedwater operation 

periodically select these surveillances for review (under IP-71111.22, Surveillance Testing and / or 

IP-71111.20, Refueling and Other Outage Activities, etc.). 

14) Inspectors should considering developing and retaining other feedwater system checklists and 

guidance information (updating the risk based inspection guides as discussed in Step 1, as 

necessary), for inspecting their site specific feedwater (AFW/EFW or HPCI) systems and for 
- 9 -

subsequent site turnover information, training of new inspectors, etc., as applicable. 

-END-