February 21, 2007

EA-06-296

Randall K. Edington, Senior 

 Vice President Nuclear

Mail Station 7602

Arizona Public Service Company

P.O. Box 52034

Phoenix, AZ 85072-2034

SUBJECT: FINAL SIGNIFICANCE DETERMINATION FOR A WHITE FINDING AND

NOTICE OF VIOLATION - NRC SPECIAL INSPECTION REPORT

05000528/2006-012; 05000529/2006-012; 05000530/2006-012 - PALO VERDE

NUCLEAR GENERATING STATION

Dear Mr. Edington:

The purpose of this letter is to provide you the final results of our significance determination of

the preliminary White findings identified in the subject inspection report. The NRC’s risk

informed conclusion is that the violations of NRC requirements discussed in this letter are best

characterized as a single White finding, that is, a finding with low to moderate increased

importance to safety. Our rationale for this conclusion is discussed in the remainder of this

letter.

NRC Inspection Report 05000528; 05000529; 05000530/2006-012 (ML063400561) described

the results of the special inspection conducted in response to the failures of the Unit 3, Train A

Emergency Diesel Generator (EDG A). This report identified findings involving failures of EDG

A to perform its safety function when tested on July 25 and September 22, 2006. These

findings also represented two violations of NRC requirements. The first violation involved a

failure to develop adequate instructions for corrective maintenance of the EDG A K-1 relay. The

second violation involved the failure to identify and correct the cause of erratic K-1 relay

operation before installation in the EDG A voltage excitation circuit on July 26, 2006. The K-1

relay is a key component, which must operate properly in order for the EDG’s mechanical output

to be converted and routed as electricity for safety components during certain types of

accidents. These two violations resulted in EDG A not being able to perform its safety function

between September 4 and September 22, 2006. 

The NRC’s preliminary assessment of the safety significance of these inspection findings, which

is documented in Attachment 3 of NRC Inspection Report 05000528; 05000529;

05000530/2006-012, resulted in an upper bound increase in core damage frequency (CDF) of

8.5E-6/year, or White and close to the White/Yellow threshold for safety significance. Our

preliminary assessment assumed the loss of offsite power (LOOP) initiating event frequency

and EDG non-recovery/repair probabilities, as described in NUREG/CR-6890, “Reevaluation of

Station Blackout Risk at Nuclear Power Plants, Analysis of Loss of Offsite Power Events: 1986-
-2-

2004.” The NRC assessment also quantified the risk associated with the 40-day period prior to

the failure of EDG A, in which there was a higher probability of failure of the K-1 relay. Finally,

we assumed a gas turbine generator (GTG) 1 hour recovery value of 0.1 and a 2-hour battery

depletion time for station blackout (SBO) sequences. 

At the request of the Arizona Public Service Company (APS), a Regulatory Conference was

held on January 16, 2007, to discuss APS’s position on the accuracy of, corrective actions for,

and safety significance of the findings. In APS’s letter dated January 9, 2007 (ML070180148),

APS accepted the findings described in NRC Inspection Report 05000528; 05000529;

05000530/2006-012 and indicated that APS had no substantive disagreement with the facts, as

documented in the report. 

During the Regulatory Conference, APS reiterated its view, which was provided to NRC staff on

January 12, 2007, that the violations were Green, which is of very low safety significance. The

APS determination was made based on some key assumptions, including that EDG A was

unavailable for 18 days between September 4 and 22, 2006; using a LOOP frequency based on

historic events in the Western United States; adding a K-1 relay recovery action to the Palo

Verde probabilistic risk assessment (PRA) model; using a GTG 1 hour recovery value of 0.16;

and completing an analysis which extended the service life of safety and non-safety related

batteries from 3 to 7 hours. The extended battery service life was used in the Palo Verde PRA

model to: (1) extend the operating time of the auxiliary feedwater system, and (2) increase the

probability of recovery actions for starting the GTGs located at the Palo Verde water

reclamation facility, as well as, recovery of offsite power or recovery of the EDGs. Using these

assumptions, APS stated that the increase in CDF was approximately 3.1E-7/year (Green). The

January 16, 2007, Regulatory Conference meeting summary, dated January 26, 2007

(ML070260550), includes a copy of the APS presentation. 

During the Regulatory Conference, APS also asserted that adding the risk associated with the

40-day period, in which there was a higher probability of failure of EDG A, to the risk from the 18

days when the EDG was known to be unavailable was, in essence, a double counting of the risk

impact. For our final analysis, the NRC only assessed the 18 day period of EDG A

unavailability.

APS presented information regarding how long the safety related and non-safety related

batteries would function under SBO conditions. The duration of battery operation is a key

assumption in assessing the risk of this condition because the longer the batteries function, the

higher the probability that electrical power from other sources could be restored before core

damage occurs. The design life of the batteries was two hours. The APS PRA model assumed

that the safety related batteries would last 3 hours based on generic industry guidance for PRA

models. During the Regulatory Conference, APS presented information from an engineering

analysis which extended the battery service life to a best estimate value of 7 hours. While the

NRC staff determined that there may be some uncertainty associated with aspects of the APS

analysis (e.g., the technical basis of the newly analyzed load profile), the NRC staff concluded

that APS provided sufficient information to justify a more realistic estimate of battery duration. 

As a result, the NRC used a 7-hour best estimate of battery duration in the final analysis. 

The NRC determined that APS did not provide a sufficient basis to demonstrate that the NRC

staff used overly conservative LOOP frequency and EDG non-recovery/repair probability

assumptions. The information APS provided did not sufficiently demonstrate that recovery of
-3-

the EDG A K-1 relay failure would have been more rapid or probable than that expressed by the

time-weighted recovery curves used in the NRC’s PRA model for Palo Verde. In particular, the

NRC’s review of the APS risk analysis identified significant uncertainties in the risk estimates

and analysis methods which were inconsistent with Regulatory Guide 1.200, "An Approach for

Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk

Informed Activities,” and the associated American Society of Mechanical Engineers PRA

standard. APS’s credit for the EDG A K-1 relay repair relied on appropriate maintenance

activities being performed which conflicts with the cause of the underlying performance

deficiency (i.e., the failure to perform adequate maintenance on a risk-significant component). 

Therefore, the NRC determined that the standard values provided in NUREG/CR-6890, which

are based on industry EDG experience and provide a more robust statistical basis for

completing EDG repairs, should be used. 

The NRC determined that the higher LOOP frequency is justified on the basis of the actual

operating history at Palo Verde. A LOOP event occurred in June of 2004 when the operational

age of the three Palo Verde units was less than 20 years. 

During the Regulatory Conference, APS presented information to support the use of a nominal

recovery value of 0.16 for starting and aligning the GTGs to the safety related switchgear within

1 hour. The NRC questioned the validity of the nominal recovery value given recent

performance deficiencies involving starting and aligning the GTGs to a safety related electrical

bus. Specifically, in October 2005, the NRC raised concerns associated with testing of the

GTGs and the amount of time required to align a GTG to a safety related electrical bus. In May

2006, operations personnel failed to properly align the GTGs to the Unit 3 Train A safety bus

resulting in a loss of power and automatic start of EDG A. The underlying causes involved the

adequacy of procedures for aligning the GTGs and training of water reclamation and operations

personnel. APS completed corrective actions for these issues in November 2006. In December

2006, APS identified additional concerns associated with the time required for water reclamation

personnel to start and align the GTGs. 

In February 2007, following the Regulatory Conference, APS completed a timing study of the

starting and electrical alignment of the GTGs by four different operating crews during a

simulation of the implementation of the SBO emergency operating procedure. The timing study

revealed that none of the crews could align the GTGs to the electrical safety bus within 1 hour. 

The underlying factors associated with the delay involved work processes, procedure adequacy,

training, and the complexity of the task. 

Given the problems associated with GTG operation, the NRC determined that an increase in the

1 hour recovery value for the GTG was appropriate. Using the NRC’s method for assessing

performance shaping factors, the NRC determined that the best estimate value in the model for

recovery of the GTG within 1 hour was between a lower bound of 0.23 and an upper bound of

0.6. Using the NUREG/CR-6890 assumptions, a 7 hour battery life, 18 days of EDG A

unavailability, and a 0.23 GTG recovery value, the APS model yielded a delta CDF of

approximately 1.1E-6/year. With a GTG recovery value of 0.6, the APS model yielded a delta

CDF of approximately 1.8E-6/year. These values closely aligned with the results obtained from

the NRC’s analysis. 
-4-

After careful consideration of the information provided at the Regulatory Conference, the

information provided in your letter of January 9, 2007, and the information provided in your

subsequent letter of January 24, 2007 (ML0703900400), the NRC determined that the best

approximation of the safety significance of the EDG A failures is in the range of 1.1E-6/year to

1.8E-6/year. Therefore, the NRC has concluded that this issue is appropriately characterized

as White, that is, a finding with low to moderate increased importance to safety. 

The actual safety significance of these findings could have been much greater. In response to

the degraded spray pond issues described in NRC Inspection Report 05000528; 05000529;

05000530/2006-011 (ML062710607), APS implemented corrective actions to increase the

monthly EDG testing frequency to approximately once every two weeks to monitor the rate of

fouling of the EDG heat exchangers. If it had not been for the increased testing of the EDGs,

then EDG A would not have been able to perform its safety function for as long as 38 days

before the discovery of the latent failure through Technical Specification required surveillance

testing.

You have 30 calendar days from the date of this letter to appeal the staff’s determination of

significance for the White finding. Such appeals will be considered to have merit only if they

meet the criteria in NRC Inspection Manual Chapter 0609, “Significance Determination

Process,” Attachment 2, “Process for Appealing NRC Characterization of Inspection findings

(SDP Appeal Process).”

As previously discussed, the NRC determined that the failure to establish appropriate

instructions for performing corrective maintenance activities on a K-1 relay and the failure to

identify and correct the cause of erratic K-1 relay operation prior to installation in the EDG A

voltage regulator circuit and returning EDG A to service represent violations of 10 CFR Part 50,

Appendix B, Criterion V, “Instructions, Procedures, and Drawings,” and 10 CFR Part 50,

Appendix B, Criterion XVI, “Corrective Action,” respectively. The violations are cited in the

enclosed Notice of Violation (Notice). In accordance with the NRC Enforcement Policy, the

Notice is considered escalated enforcement action because it is associated with a White finding.

APS’s January 9, 2007, letter also indicated that the root causes associated with the EDG A

failures involved: (1) inconsistent equipment root cause of failure program management,

resulting in varying degrees of document quality, (2) no formal problem solving and

troubleshooting process in place to establish evaluation consistency, (3) inconsistent

management reinforcement of the equipment root cause of failure application methodology

resulting in inconsistent evaluation quality, and (4) inconsistent application of a problem solving

methodology, as evidenced by varying degrees of rigor in root cause of failure determinations. 

APS’s corrective actions included, in part: (1) planning to replace all EDG automatic voltage

regulators, including replacement of the K-1 relays, with a different design during the next

refueling outage for each unit, (2) assessing and improving the equipment root cause of failure

analysis program, (3) developing a troubleshooting and problem solving process to be used by

operations, maintenance, and engineering, (4) establishing a quality checklist for engineering

product review board and corrective action review board assessments of engineering root

cause of failure evaluations, and (5) training selected personnel on the new troubleshooting and

problem solving process. On the basis of the information provided in the January 9, 2007 letter,

the NRC has determined that no additional response to the enclosed Notice is required. 
-5-

The NRC notes that the root causes for the EDG A failures are similar to the causes associated

with other recent inspection findings, and to findings documented in NRC inspection reports and

assessments since 2004. For example, troubleshooting and problem solving processes lacked

the technical rigor necessary to ensure deficiencies were properly identified and resolved. The

NRC’s determination that station personnel were not providing adequate technical rigor during

the evaluation of degraded or failed structures, systems, and components was a significant

factor in the NRC’s decision to hold open the Yellow finding associated with voiding of the

emergency core cooling system that was documented in our letters dated January 27, 2006

(ML060300193), and October 11, 2006 (ML062840601). Additionally, the lack of technical rigor

by personnel contributed to the failure of the spray pond chemistry controls and the resultant

fouling of safety-related heat exchangers supplied by the spray pond as documented in our

December 22, 2006 letter (ML063600175). As stated in our December 22, 2006 letter, these

performance deficiencies indicate that your performance improvement initiatives have not been

effective. Therefore, the NRC will continue to monitor your progress in implementing

performance improvement initiatives to address the area of technical rigor during future

inspections.

In addition, we will use the NRC Action Matrix to determine the most appropriate NRC response

and any increase in NRC oversight, or actions you need to take in response to the most recent

performance deficiencies. We will notify you by separate correspondence of that determination. 

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its

enclosure, and your response (if any) will be made available electronically for public inspection

in the NRC Public Document Room or from the NRC’s document system (ADAMS), accessible

from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html. To the extent possible,

your response should not include any personal privacy, proprietary, or safeguards information

so that it can be made available to the public without redaction. The NRC also includes

significant enforcement actions on its Web site at www.nrc.gov; select What We Do,

Enforcement, then Significant Enforcement Actions.

Sincerely,

/RA/

Bruce S. Mallett

Regional Administrator

Docket Nos. 50-528; 50-529; 50-530

License Nos. NPF-41; NPF-51; NPF-74

Enclosure: Notice of Violation

cc:

Steve Olea

Arizona Corporation Commission

1200 W. Washington Street

Phoenix, AZ 85007
-6-

Douglas K. Porter, Senior Counsel

Southern California Edison Company

Law Department, Generation Resources

P.O. Box 800

Rosemead, CA 91770

Chairman

Maricopa County Board of Supervisors

301 W. Jefferson, 10th Floor

Phoenix, AZ 85003

Aubrey V. Godwin, Director

Arizona Radiation Regulatory Agency

4814 South 40 Street

Phoenix, AZ 85040

Jeffrey T. Weikert

Assistant General Counsel

El Paso Electric Company

Mail Location 167

123 W. Mills

El Paso, TX 79901

John W. Schumann

Los Angeles Department of Water & Power

Southern California Public Power Authority

P.O. Box 51111, Room 1255-C

Los Angeles, CA 90051-0100

John Taylor

Public Service Company of New Mexico

2401 Aztec NE, MS Z110

Albuquerque, NM 87107-4224

Geoffrey M. Cook

Southern California Edison Company

5000 Pacific Coast Hwy, Bldg. N50

San Clemente, CA 92672

Robert Henry

Salt River Project

6504 East Thomas Road

Scottsdale, AZ 85251
-7-

Brian Almon

Public Utility Commission

William B. Travis Building

P.O. Box 13326

1701 North Congress Avenue

Austin, TX 78701-3326

Karen O'Regan

Environmental Program Manager

City of Phoenix

Office of Environmental Programs

200 West Washington Street

Phoenix, AZ 85003 

Matthew Benac

Assistant Vice President

Nuclear & Generation Services

El Paso Electric Company

340 East Palm Lane, Suite 310

Phoenix, AZ 85004
DISTRIBUTION: 

RIDSSECYMAILCENTER

RIDSOCAMAILCENTER

RIDSEDOMAILCENTER

RIDSOEMAILCENTER

RIDSOGCMAILCENTER

RIDSNRROD

RIDSNRRADIP 

RIDSOPAMAIL

RIDSOIMAILCENTER

RIDSOIGMAILCENTER

RIDSOCFOMAILCENTER

RIDSRGN1MAILCENTER

RIDSRGN2MAILCENTER

RIDSRGN3MAILCENTER

OEWEB

RIDSNRRDIPMIIPB

ROPreports

via e-mail:

Gwynn - TPG Mallett - BSM1 Fuller - KSF

R4ALLEGE Maier - WAM Howell - ATH

Vegel - AXV Chamberlain - DDC Caniano - RJC1

Pruett - TWP Powers - DAP Werner - GEW

Sitek - MAS3 Dricks - VLD Herrera - MSH3

Warnick - GXW2 Solorio - DLS2 Carpenter - CAC

OEMAIL Starkey, OE - DRS Pascarelli - RJP3 

Haire - MSH2 Vasquez - GMV Cullison - DGC

PV Site Secretary - PRC

SUNSI Review Completed: TWP ADAMS: x Yes G No Initials: TWP

x Publicly Available GNon-Publicly Available GSensitive x Non-Sensitive

C:\FileNet\ML070530206.wpd

PBD PBA D:DRS D:DRP ACES

TPruett MHay DChamberlain ATHowell MHaire

/RA/ /RA/ /RA/ /RA/ /RA/

2/1/07 2/1/07 2/7/07 2/12/07 2/12/07

OE NRR DRA RA

C Carpenter BPascarelli PGwynn BMallett

per D Starkey /RA/ E BSM for /RA/

2/21/07 2/21/07 2/21/07 2/21/07

OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax
NOTICE OF VIOLATION 

Arizona Public Service Company Docket No. 50-530

Palo Verde Nuclear Generating Station License No. NPF-74

EA-06-296

During an NRC inspection completed November 30, 2006, violations of NRC requirements were

identified. In accordance with the NRC Enforcement Policy, the violations are listed below:

A. 10 CFR Part 50, Appendix B, Criterion V, “Instructions, Procedures, and Drawings,”

states, in part, that activities affecting quality shall be prescribed by documented

instructions, procedures, or drawings, of a type appropriate to the circumstances and

shall be accomplished in accordance with these instructions, procedures, or drawings. 

Contrary to the above, between July 25 and September 22, 2006, activities affecting

quality were not prescribed by documented instructions appropriate to the

circumstances. Specifically, the licensee failed to develop appropriate instructions or

procedures for corrective maintenance activities on the Unit 3, Train A Emergency

Diesel Generator K-1 relay. This resulted in the emergency diesel generator being

inoperable between September 4 and September 22, 2006. 

B. 10 CFR Part 50, Appendix B, Criterion XVI, “Corrective Action,” states, in part, that for

significant conditions adverse to quality, measures shall assure that the cause of the

condition is determined and corrective action taken to preclude repetition. 

Contrary to the above, on July 26, 2006, the licensee failed to assure that the cause of a

significant condition adverse to quality was determined and that corrective action was

taken to preclude repetition. Specifically, the licensee did not identify and correct the

cause of the erratic Unit 3, Train A Emergency Diesel Generator K-1 relay operation

prior to installation of the relay on July 26, 2006. This resulted in the emergency diesel

generator being inoperable between September 4 and September 22, 2006. 

These violations are associated with a White SDP finding.

The NRC has concluded that information regarding the reason for the violations, the corrective

actions taken and planned to correct the violation and prevent recurrence and the date when full

compliance will be achieved is already adequately addressed on the docket in your letter dated

January 9, 2007 (ML070180148). However, you are required to submit a written statement or

explanation pursuant to10 CFR 2.201, if the description therein does not accurately reflect your

corrective actions or your position. In that case, or if you choose to respond, clearly mark your

response as a "Reply to a Notice of Violation," include the EA number, and send it to the U.S.

Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001

with a copy to the Regional Administrator, Region IV, and a copy to the NRC Resident Inspector

at the facility that is the subject of this Notice, within 30 days of the date of the letter transmitting

this Notice of Violation (Notice).

Enclosure 
-2-

If you choose to respond, your response will be made available electronically for public

inspection in the NRC Public Document Room or from the NRC’s document system (ADAMS),

accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html. Therefore, to

the extent possible, the response should not include any personal privacy, proprietary, or

safeguards information so that it can be made available to the Public without redaction.

In accordance with 10 CFR 19.11, you may be required to post this Notice within two working

days.

Dated this 21st day of February 2007