## i

## TABLE OF CONTENTS

I. E XECUTIVE 

### S UMMARY



## ....................................................................................... - 1

- A. S YNOPSIS OF THE 

### D ISTURBANCE AND 

### S YSTEM 

### R ECOVERY



## ................................. - 1

- B. M AP OF 

### A FFECTED 

### A REA AND 

### K EY 

### F ACILITIES 

### I NVOLVED IN THE 

### E VENT



## ............ - 3

- C. K EY 

### F INDINGS



### , C AUSES



### , AND 

### R ECOMMENDATIONS



## .......................................... - 5

-

II. I NTRODUCTION



## ............................................................................................ - 10

- A. I NQUIRY 

### P ROCESS



## ..................................................................................... - 10

- B. S YSTEM 

### O VERVIEW



## .................................................................................... - 15

-

III. S EQUENCE 

### O F 

### E VENTS



## .................................................................................. - 23

- A. P HASE 

### 1: P RE 

### -D ISTURBANCE 

### C ONDITIONS



## ................................................... - 24

- B. P HASE 

### 2: T RIP OF THE 

### H ASSAYAMPA 

### -N ORTH 

### G ILA 

### 500 K 

### V L INE



## .................... - 30

- C. P HASE 

### 3: T RIP OF THE 

### C OACHELLA 

### V ALLEY 

### 230/92 K 

### V T RANSFORMER AND 

### V

OLTAGE 

### D EPRESSION



## ............................................................................... - 36

- D. P HASE 

### 4: T RIP OF 

### R AMON 

### 230/92 K 

### V T RANSFORMER AND 

### C OLLAPSE OF 

### IID’ S 

### N ORTHERN 

### 92 K 

### V S YSTEM



## ......................................................................... - 40

- E. P HASE 

### 5: Y UMA 

### L OAD 

### P OCKET 

### S EPARATES FROM 

### IID AND 

### WALC



## .................. - 44

- F. P HASE 

### 6: H IGH 

### -S PEED 

### C ASCADE



### , O PERATION OF THE 

### SONGS S EPARATION 

### S CHEME AND 

### I SLANDING OF 

### S AN 

### D IEGO



### , IID, CFE, AND 

### Y UMA



## ....................... - 47

- G. P HASE 

### 7: C OLLAPSE OF THE 

### S AN 

### D IEGO 

### /CFE/Y UMA 

### I SLAND



## ......................... - 53

- H. S YSTEM 

### R ESTORATION



## ............................................................................... - 60

-

IV. C AUSES



### , F INDINGS



### , AND R ECOMMENDATIONS



## ................................................ - 63

- A. P LANNING



## ................................................................................................ - 63

- 1. N EXT 

### -D AY 

### P LANNING



## ............................................................................ - 63

- 2. S EASONAL 

### P LANNING



## ............................................................................. - 72

- 3. N EAR



### - AND 

### L ONG 

### -T ERM 

### P LANNING



## ......................................................... - 79

- B. S ITUATIONAL 

### A WARENESS



## .......................................................................... - 85

- C. S YSTEM 

### A NALYSIS



## ..................................................................................... - 96

- 1. C ONSIDERATION OF 

### BES E QUIPMENT



## ....................................................... - 96

- 2. IROL D ERIVATIONS



## ............................................................................... - 97

- 3. I MPACT OF 

### P ROTECTION 

### S YSTEMS ON 

### E VENT



## ........................................... - 100

- 4. A NGULAR 

### S EPARATION



## .......................................................................... - 110

-

A PPENDICES 

### 114



## i

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# I. E XECUTIVE 

# S UMMARY



# A. Synopsis of the Disturbance and System Recovery

On the afternoon of September 8, 20 11, an 11-minute system disturbance occurred in the Pacific Southwest, leading to cascading outages and leaving

approximately 2.7 million customers without power. 1 

### The outages affected parts of Arizona, Southern California, and Baja Californ ia, Mexico. All of the San Diego area lost power, with nearly one-and-a-half million cu stomers losing power, some for up to 12

hours. The disturbance occurred near rush hour, on a business day, snarling traffic for

hours. Schools and businesses closed, some flights and public transportation were disrupted, water and sewage pumping stations lost power, and beaches were closed due to sewage spills. Millions went withou t air conditioning on a hot day.

The loss of a single 500 kilovolt (kV) 2 

### transmission line initiated the event, but was not the sole cause of the widespread outa ges. The system is de signed, and should be operated, to withstand the loss of a single line, even one as large as 500 kV. The affected line—Arizona Public Service’s (APS) Hass ayampa-N. Gila 500 kV line (H-NG)—is a segment of the Southwest Power Link (SWP L), a major transmission corridor that transports power in an east-west direction, from generators in Arizona, through the service territory of Imperial Irrigation District (IID), into the San Diego area. It had tripped on multiple occasions, as recently as July 7, 2011, without causing cascading

outages.

With the SWPL’s major east-west corridor broken by the loss of H-NG, power flows instantaneously redistributed througho ut the system, increasing flows through lower voltage systems to the north of the SWPL, as power continued to flow into San Diego on a hot day during hours of peak demand. Combined with lower than peak

1 “Customers” are not the same as “people” in utility parlance. The term customer generally refers to a single

meter, whether at a residence, an apartment building, or a factory. Thus, a single customer could represent one

or more persons, and a single person could be two customers, for example, if the same utility se rved both an individual’s residence and his small business. Estimates of “people” affected by blackouts generally are prepared by increasing the customer numbers by a multiplier, often two or three.

2 A list of acronyms used in this report is included in Appendix A.



## - 1 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

generation levels in San Diego and Mexico, 3 

### this instantaneous redistribution of power flows created sizeable voltage deviations an d equipment overloads to the north of the SWPL. Significant overloadin g occurred on three of IID’ s 230/92 kV transformers

located at the Coachella Vall ey (CV) and Ramon substations, as well as on Western

Electricity Coordinating Council (WECC) Path 44, 4 

### located south of the San Onofre Nuclear Generating Station (SONGS) in Southern California.

The flow redistributions, voltage deviat ions, and resulting overloads had a ripple

effect, as transformers, transmission lines, an d generating units trippe d offline, initiating automatic load shedding throug hout the region in a relatively short time span. Just

seconds before the blackout, Pa th 44 carried all flows into the San Diego area as well as parts of Arizona and Mexico. Eventually, the excessive loading on Path 44 initiated an intertie separation scheme at SONGS, desi gned to separate SDG&E from SCE. The SONGS separation scheme sepa rated SDG&E from Path 44, led to the loss of the SONGS nuclear units, and eventually resulted in the complete blackout of San Diego and Comisión Federal de Electricidad’s (CFE) Baja California Control Area. During the 11 minutes of the event, the WECC Reliability Coordinator (WECC RC) issued no directives and only limited mitigating actions were take n by the Transmission Operators (TOPs) of

the affected areas.

As a result of the cascading outages stemming from this event, customers in the

SDG&E, IID, APS, Western Area Power Ad ministration-Lower Colorado (WALC), and CFE territories lost power, some for multip le hours extending into the next day. Specifically,

 SDG&E lost 4,293 Megawatts (MW) of firm load, affecting approximately 1.4 million customers.

 CFE lost 2,150 MW of net firm load, affe cting approximately 1.1 million customers. 5

 IID lost 929 MW of firm load, affecting approximately 146,000 customers.

3 Total summer peak generation for San Diego Gas and Electric’s (SDG&E) territory and Comisión Federal de

Electricidad’s (CFE) Baja California Control Area is 5,774 MW. On September 8, 2011, the total generation for

SDG&E and CFE’s Baja California Control Area was 4,168, a difference of 1,606 MW.

4 Path 44 is one of 81 Rated Paths in the WECC region. A Rated Path is composed of “an individual transmission

line or a combination of parallel transmission lines.” WECC 2011 Path Rating Catalog, January 2011, at item 1

‐ i. Path

44, also referred to as “South of SONGS,” is an aggregati on of five 230 kV lines that delivers power in a north

‐ south direction from the Southern California Edison (SCE) footprint in the Los Angeles area into the SDG&E footprint.

5 CFE is Mexico’s state

‐ owned utility. Only its Baja California Control Area was affected on September 8, 2011. The inquiry is particularly grateful to CFE for it s willingness to share data and information to assist the inquiry in developing the most accurate conclusions and recommendations.



## - 2 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

 APS lost 389 MW of firm load, affe cting approximately 70,000 customers.

 WALC lost 74 MW of firm load, 64 MW of which affected APS’s customers. The remaining 10 MW affected 5 WALC customers.



### After the blackout, the affected entities promptly instituted their respective

restoration processes. 6 

### All of the affected entities had access to power from their own or

neighboring systems and, therefore, did not need to use “black start” plans. 7 

### Although there were some delays in the restoration process due to communication and coordination issues between entities, the process was generally effective. SDG&E took 12

hours to restore 100% of its load, and CFE took 10 hours to restore 100% of its load. IID, APS, and WALC restored power to 100% of their customers in approximately 6 hours. The affected entities also worked to restore generators and transmission lines that tripped during the event. IID and APS rest ored generation—333 MW for IID and 76 MW

for APS—in 5 hours. Meanwhile, CFE restor ed 1,915 MW of tripped generation in 56 hours; SDG&E restored 2,229 MW of tripped generation in 39 hours; and SCE restored 2,428 MW of tripped generation in 87 hour s. IID restored its 230 kV transmission system in 12 hours and its 161 kV system in 9 hours; APS restored H-NG in 2 hours; SDG&E restored its 230 kV system in 12 hours; WALC restored its 161 kV system in 1.5 hours; and CFE restored its 230 kV system in 13 hours and its 115 kV system in 10 hours.



# B. Map of Affected Area and Key Facilities Involved in the Event

The following map, showing the areas affected by the September 8th event and the key facilities involved during the event, can be used as a reference throughout the

report:

6 The term “affected entities” in this report refers to TOPs and Balancing Authorities (BAs) that were affected by

the event. The affected entities include SDG&E, IID, APS, WALC, SCE, CFE, and the California Independent System Operator (CAISO).

7 Black start plans work to energize systems using internal generation to get from shutdown to operating

condition without assistance from the Bulk Electric System (BES).



## - 3 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 4 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# C. Key Findings, Causes, and Recommendations 8



### The September 8, 2011, event showed that the system was not being operated in a

secure N-1 state. 9 

### This failure stemmed primarily fr om weaknesses in two broad areas— operations planning and real-time situationa l awareness—which, if done properly, would have allowed system operators to proactively operate the system in a secure N-1 state during normal system conditions and to restor e the system to a secure N-1 state as soon as possible, but no longer than 30 minutes. Without adequate planning and situational awareness, entities responsible for operat ing and overseeing the transmission system could not ensure reliable oper ations within System Operating Limits (SOLs) or prevent

cascading outages in the event of a single contingency. 10 

### As demonstrated in Appendix C, inadequate situational awareness and planning were also identified as causes of the 2003 blackout that affected an estimated 50 million people in the United States and

Canada.

The inquiry also identified other underlying factors that contributed to the event, including: (1) not identifying and studying the impact on Bulk-Power System (BPS) 11

8 While this section highlights the most significant causes, findings, and recommendations, the report details the complete list of findings, causes, and recommendations in section IV. In addition, for ease of reference all of the findings and recommendations are summarized in table format in Appendix B.

9 The North American Electric Reliability Corporation’s (NERC) mandatory Reliability Standards applicable to the

BES require that the BES be operated so that it generally remains in a reliable condition, without instability,

uncontrolled separation or cascading, even with the occurrence of any single contingency, such as the loss of a generator, transformer, or tran smission line. This is commonly known as the “N

‐ 1 criterion.” N

‐ 1 contingency planning allows entities to identify potential N

‐ 1 contingencies before they occur and to adopt mitigating

measures, as necessary, to prevent instability, uncontrolled separation, or cascading. As the Federal Energy Regulatory Commission (Commission) stated in Order No. 693 with regard to contingency plan ning, “a single contingency consists of a failure of a single element that faithfully duplicates what will happen in the actual system. Such an approach is necessary to ensure that planning will produce results that will enhance the reliability of that system. Thus, if the system is designed such tha t failure of a single element removes from

service multiple elements in order to isolate the faulted element, then that is what should be simulated to assess system performance.” Mandatory Reliability Standards for the Bulk Power System , Order No. 693, FERC Stats. & Regs. ¶ 31,242, at P 1716 (2007), or der on reh’g, Mandatory Reliability Standards for the Bulk

‐ Power System , 120

FERC ¶ 61,053 (Order No. 693

‐ A) (2007).

10 A contingency is the unexpected failure of an electrical system component.

11 The BPS is defined by Section 215(a) (1) of the Federal Power Act as “facilities and control systems necessary for operating an interconnected electric energy transmission network (or any portion thereof), and electric

energy from generating facilities needed to maintain transmission system reliability.” The meaning of BPS and

BES differ somewhat an d, thus, this report uses each term in its proper context. With respect to reliability, the

Commission has jurisdiction over all users, owners, and operators of the BPS. In Order No. 693 at P 75, the

Commission adopted, at least for an initial period, the BES definition as the threshold for application of the NE RC Reliability Standards. Thus, this report uses BES when referring to entities’ specific facilities or elements that are subject to the Reliability Standards, but BPS when discussing the overall reliability impact. On January 25, 2012,



## - 5 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

reliability of sub-100 kV faciliti es in planning and operations; 12 

### (2) the failure to recognize Interconnection Reliability Op erating Limits (IROLs) in the Western

Interconnection; 13 

### (3) not studying and coordinating the effect of protection systems,

including Remedial Action Schemes (RASs), during plausible contingency scenarios; 14 

### and (4) not providing effective tools and oper ating instructions for use when reclosing

lines with large phase angle differe nces across the reclosing breakers. 15



### With regard to operations planning, some of the affected entities’ seasonal, nextday, and real-time studies do not adequately consider: (1) operations of facilities in external networks, including the status of tr ansmission facilities, expected generation output, and load forecasts; (2) external cont ingencies that could impact their systems or internal contingencies that could impact thei r neighbors’ systems; and (3) the impact on BPS reliability of internal and external sub-100 kV facilities. As a result, these entities’ operations studies did not accurately predict th e impact of the loss of APS’s H-NG or the

loss of IID’s three 230/92 kV transformers. If the affected entities had more accurately predicted the impact of these losses prior to the event, these entities could have taken appropriate pre-contingency measures, such as dispatching additional generation to mitigate overloads and prevent cascading outages.

To improve operations planning in the WECC region, this report makes several

recommendations designed to ensure that TOPs and BAs, 16 

### as appropriate: (1) obtain information on the operations of neighbor ing BAs and TOPs, including transmission outages, generation outages and schedules, load forecasts, and scheduled interchanges;

NERC filed a petition with the Commission for approval of a revised definition of the BES. The proposed definition

of BES would cover all elements operated at 100 kV or higher, with a list of specific inclusions and exclusions.

Transformers with the primary terminal and at least one secondary terminal operated at 100 kV or higher are on the list of spec ific inclusions. See North American Electric Reliability Corp., Docket No. RM12

‐ 6

‐ 000. This report takes no position on the petition.

12 This report does not attempt to define the limits of which sub

‐ 100 kV facilities impact BPS reliability. Certainly,

many facilities below 100 kV do not impact BPS reliability. The sub

‐ 100 kV facilities in this event affected the BPS

because they were in parallel to significant transmission corridors.

13 This report recommends that WECC RC should work with TOPs to consider whether any SOLs in the Western

Interconnection constitute IROLs. As part of this effort, WECC RC should: (1) work with affected TOPs to consider whether Path 44 and H

‐ NG should be recognized as IROLs; and (2) validate existing SOL s and ensure that they take into account all transmission and generation facilities and protection systems that impact BPS reliability.

14 This failure caused the derived SOLs on H

‐ NG and Path 44 to be invalid on the day of the event.

15 As discussed in more detail in connection with Finding and Recommendation 27 below, when a line trips, the phase angle at one end of the line may be much larger than the phase angle at the other end. If the difference

between the two angles is too great, reclosing the line could ca use damage to generators or even system instability.

16 See “Reliability Responsibilities” section at page 16 below.



## - 6 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

(2) identify and plan for external contin gencies that could impact their systems and internal contingencies that could impact their neighbors’ systems; and (3) consider facilities operated at less than 100 kV that could impact BPS reliab ility. This effort should include a coordinated review of planni ng studies to ensure that operation of the

affected Rated Paths will not result in the loss of non-conseq uential load, system instability, or cascading outa ges, with voltage and therma l limits within applicable ratings for N-1 contingencies originating from within or outside an entity’s footprint.

The September 8th event also exposed en tities’ lack of adequate real-time

situational awareness of conditions an d contingencies throughout the Western Interconnection. For example, many entities’ real-time tools, such as State Estimator and Real-Time Contingency Analysis (RTCA), are restricted by models that do not accurately or fully reflect facilities and operations of external systems to ensure operation of the BPS in a secure N-1 state. Also, some entities’ real-time tools are not adequate or operational to alert operator s to significant conditions or potential contingencies on their systems or neighboring systems. The lack of adequate situational awareness limits entities’ ability to identi fy and plan for the next most critical contingency to prevent instabil ity, uncontrolled separation, or cascading outages. If

some of the affected entities had been aware of real-time external conditions and run (or reviewed) studies on the conditions prior to the onset of the event, they would have been better prepared for the impacts when the event started and may have avoided the cascading that occurred.

To improve situational awareness in the WECC region, this report makes several recommendations: (1) expand entities’ extern al visibility in their models through, for example, more complete data sharing; (2) improve the use of real-time tools to ensure the constant monitoring of potential internal or external contingencies that could affect reliable operations; and (3) improve communi cations among entities to help maintain

situational awareness. In addition, TOPs shou ld review their real-time monitoring tools,

such as State Estimator and RTCA, to ensure th at such tools represent critical facilities needed for the reliable operation of the BPS. These improvements will enable system operators to utilize real-time operating tools to proactively operate the system in a secure

N-1 state.

In addition to the planning and situatio nal awareness issues, several other factors contributed to the September 8th event. For example, WECC RC and affected entities do not consistently recognize the adverse impact that sub-100 kV facilities can have on BPS reliability. The prevailing SOLs should have included the effects of facilities that had not been identified and classified as part of the BES, as well as the effects of critical facilities such as Special Protection Systems (SPSs) an d the SONGS separation scheme. Relevant



## - 7 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

to the event, these entities di d not consider IID’s 92 kV network and facilities, including the CV and Ramon 230/92 kV transformers, as part of the BES, despite some previous studies indicating their impact on the BPS due to the fact they were electrically in

parallel with higher-voltage facilities. 17 

### If these facilities had been designated as part of the BES, or otherwise incorporated into pl anning and operations studies and actively monitored and alarmed in RTCA systems, the ca scading outages may have been avoided. Accordingly, the inquiry makes a recommendat ion to ensure that facilities that can impact BPS reliability, regardless of voltage level, are considered for classification as part of the BES and/or studied as part of entities’ planning in various time horizons.

The inquiry also found some significant issues with protection system settings and coordination. For example, IID used conservative overload relay trip settings on its CV transformers. The relays were set to trip at 127% of the transformers’ normal rating, which is just above the transformers’ emergency rating (110% of normal rating). Such a narrow margin between the emergency rating an d overload trip setting resulted in the facilities being automatically removed from service without providing operators enough time to mitigate the overloads. As a result of these settings, both CV transformers tripped within 40 seconds of H-NG tripping, initiating cascading outages. To avoid a similar problem in the future, the inquiry recommends that IID and other Transmission Owners (TOs) review their transformers’ over load protection relay settings. A good guideline for protective rela y settings is Reliability Stan dard PRC-023-1 R1.11, which states that relays be “set to allow the transforme r to be operated at an overload level of at least 150% of the maximum applicable nameplate rating, or 115% of the highest operator established emergency transformer rating, whichever is greater.” TOPs should also plan to take proper pre-contingency mitigation measures with due consideration for the applicable emergency ratings and overload protection settings (MW and time delay) before a facility loads to its relay trip point and is automatically removed from service.

The SONGS separation scheme’s operation provides another example of the lack of studies on, and coordination of, protection systems. This scheme, classified by SCE as

a “Safety Net,” 18 

### had a significant impact on BPS reliability, separating SDG&E from

17 See, e.g ., CFE’s Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19.

18 A Safety Net protection system protects the power system from unexpected, low

‐ probability events that are outside the normal planning criteria, but which may lead to a complete system collapse. Safety Nets operate to

minimize the severity of the event and attempt to prevent a system collapse or cascading outages. A S afety Net

is typically intended to handle severe disturbances resulting from extreme, though perhaps not well

‐ defined,

events. A Safety Net is subject to review by the WECC Remedial Action Scheme Reliability Subcommittee if

unintended operation would result in cascading or other performance standard violations. WECC Guideline:

Remedial Action Scheme Cla ssification, February 9, 2009.



## - 8 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

SCE, resulting in the loss of both SONGS nuclear generators, and blacking out SDG&E

and CFE. Nevertheless, none of the affected entities, including SCE, as the owner and operator of the scheme, studied its impact on BPS reliability. Th e September 8th event shows that all protection syst ems and separation schemes, including Safety Nets, RASs, and SPSs, should be studied and coordinated pe riodically to understand their impact on BPS reliability to ensure their operation, inadvertent operation, or misoperation does not

have unintended or undesirable effects.



## - 9 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# II. I NTRODUCTION



# A. Inquiry Process

On September 9, 2011, the Commission an d NERC jointly announced an inquiry to determine the causes of the outages and make recommendations for preventing such events in the future. The purpose of the inquiry was not to determine whether there may have been violations of applic able regulations, requirements , or standards subject to the Commission’s jurisdiction. Thus, while this report describes conduct which may warrant

future investigations un der Part 1b of the Commission’s regulations, 19 

### or actions by

NERC under its Compliance Moni toring and Enforcement Program, 20 

### it draws no

conclusions about whethe r violations occurred.

The inquiry was composed of smaller te ams with particular subject-matter expertise, primarily from Commission and NE RC professional staff, each of which conducted rigorous analyses of a key issue or issues involved in the event. Those teams and their primary responsibilities were as follows:

 

### Sequence of Events – developed a precise and accurate sequence of events (SOE) to provide a foundation for root cause analysis , computer model simulations, and other analytical aspects of the inquiry.

 

### System Modeling and Simulation – developed an accurate system modeling case, benchmarked the case to actual conditions at critic al times, replicated system conditions leading up to and during the outage, and simulate d alternate “what if” scenarios.

 

### Root Cause and Human Performance Analysis – performed in a systematic evaluation of the root causes and contributing factors an d identified areas requiring further inquiry.

 

### Operations Tools, Supervisory Control and Data Acquisition (SCADA)/Energy Management System (EMS), Communications, and Operations Planning – considered all aspects of the blackout related to operator and reliability coordinator knowledge of system condit ions, actions or inactions, and communications, particularly the observability of the electric system and effectiveness of operational reliability assessment tools.

 

### Frequency/Area Control Error (ACE) Analysis – reviewed potential frequency anomalies related to the blackout, and analyzed underfrequency generator, load, and tie line tripping.

19 18 C.F.R. Part 1b (2011).

20 NERC Compliance Monitoring and Enforcement Program, Appendix 4C to the NERC Rules of Procedure, January 31, 2012.



## - 10 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

 

### System Planning, Design, and Studies – analyzed factors used in setting SOLs and actual limits in effect on the day of the blackout, determined whether those limits were exceeded, and analyzed the extent to which actual system conditions varied from the assumptions used in setting the SOLs.

 

### Transmission and Generation Performance, Protection, Control, Maintenance, and Damage – analyzed the causes of automatic facility operations and generator trips, analyzed transmission and generati on facility maintenance practices, and identified equipment damage.

 

### Restoration Review – reviewed the appropriateness and effectiveness of the restoration plans implemented, as well as the effectiveness of the coordination of these plans among the affected entities and WECC RC.



### Each team not only examined its own subject area to determine what may have

contributed to the event, but also co nsidered lessons learned and potential recommendations for preventing such events in the future.

The inquiry devoted substantial time an d resources to determine and study the causes of the event and develop meaningful recommendations with the goal of preventing similar events in the future. The team’s analyses were extensive, involving the review of high-quality data from variou s reliability entities in the WECC region and simulations of the event using sophisticate d computer models. Described below in summary form are the primary steps the inquiry took to complete its analysis.



## Data Gathering

The inquiry received and reviewed more than 20 gigabytes of data from approximately 500 data requests sent to entities in and around the affected areas. On September 19, 2011, the inquiry also began site visits with various entities involved in the outages, including entities with responsi bility for balancing load and generation, transmission operation, and reliability coordi nation. During the site visits, the inquiry

toured control centers, conducted dozens of interviews and depositions, and viewed equipment involved in the event. These vi sits and depositions allowed the inquiry to learn about control room operations and prac tices, system status and conditions on the day of the event, operating procedures, planning, operations, and real-time tools, and restoration planning and procedures. The in quiry also conducted dozens of follow-up meetings and issued follow-up data requests.

Of particular use to the inquiry were phasor measurement unit (PMU) records. PMUs are complex, multi-functional, high re solution recording devices installed widely throughout the Western Interconnection pursua nt to a voluntary WECC-wide initiative. PMUs provide continuous, high-speed (30 scans per second) records of system conditions, including frequenc y, voltage, and phase angle relations. The continuous



## - 11 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

nature of the data available through the PMUs, as well as their wide distribution throughout the power system, proved especial ly valuable to the inquiry in forming an accurate picture of the SOE and state of the system at particular points in time throughout the disturbance.



## SOE Methodology

More than 100 notable events occurred in less than 11 minutes on September 8,2011. The inquiry’s SOE team established a precise and accurate sequence of outagerelated events to form a critical building bl ock for the other parts of the inquiry. It provided, for example, a foundation for the root cause analysis, computer-based simulations, and other event analyses. Althou gh entities time-stamp much of the data related to specific events, their time-stamping methodolog ies vary, and not all of the time-stamps were synchronized to the Nation al Institute of Standards and Technology (NIST) standard clock in Bould er, Colorado. Validating the precise timing of specific events became a time-consuming, importan t, and sometimes difficult task. The availability of global po sitioning system (GPS)-time synchronized PMU data on frequency, voltage, and related power angles ma de this task much easier than in previous blackout inquiries and investigations.

To develop the SOE, the SOE team started by resolving discrepancies between the multiple sources of data, sign convention in consistencies, and incorrect data. The SOE team then developed an events database starting with all known events and times. Initial sources for the development of the data base included prelimin ary reports filed by the affected entities as well as initial responses to data requests. The team then examined each record in the database to verify event times using available SCADA and PMU data. As the frequency, line flow, or vo ltage data suggested that additional events might have occurred on the system, the team added other possible events and verified them through additional data requests.

The SOE team developed multiple iteratio ns of an SOE narrative document based

on the database and the available SCADA and PMU data. Some iterations of the SOE narrative required that more data be requ ested of affected entities, and ultimately multiple data requests were sent to each entity. After the team completed the SOE narrative, the inquiry’s Modeling and Simula tion team verified the SOE using power flow, voltage stability, and dynamic stability analyses.



## - 12 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Powe r Flow and Dynamics Analysis

The inquiry’s Modeling and Simulation team, after validating the SOE, considered several “what if” scenarios. The Modeling and Simulation team’s work is described in more detail in Appendix D. Power flow analyses study power systems under quasi-steady-state co nditions by matching load and generation to obtain voltage magnitude and angle at each bus and the real and reactive power flowing through each transmission facility. Dynamic stability anal yses study the impact of disturbances on frequency, voltage, and rotor angle stability, and determine whether transients in the power system are stable, thus allowing the powe r system to return to a quasi-steady-state

operating condition following a disturbance. 21



### As the first step in performing power flow and dynamic stability analyses, the Modeling and Simulation team developed an d benchmarked a modeling case of system conditions prior to the event. The team started with the WECC heavy summer base case and made adjustments based on State Estimato r snapshots, EMS data, actual generation and schedules, PMU data, and a base case prepared by a separate team (led by CAISO) that studied the event. The team further adjusted and benchmarked the base case using SCADA and PMU data to match the system conditions for the entire event. The team

devoted considerable time and effort to resolving discrepancies between the various

sources of data to best cali brate the modeling case to actual measured data. As illustrated by Figure 1 , on the next page, and described in more detail in Appendix D, the Modeling and Simulation team achieved a si gnificant degree of accuracy. This figure compares Path 44 flows simulated by the Mode ling and Simulation team to actual Path

44 PMU data.

After developing and benchmarking a va lid case, the Modeling and Simulation team simulated the entire SOE using both po wer flow and dynamic simulations. This replication of the SOE established the vali dity of the model and enabled meaningful simulation of several alternative scenarios, developed to answer “what if” questions regarding the event. For example, the inquiry considered what would have happened if some of the affected entities had dispatched generation at certain locations during the event, if overload relays had been set at differe nt levels, or if RASs, Safety Nets, or other SPSs had been designed or operated differently.

21 Transient stability refers to the ability of synchronous generators to move to a new quasi

‐ steady

‐ state operating point while remaining synchronized after the system experiences a disturbance.



## - 13 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Figure 1: Comparison of Actual and Simulated Path 44 Flows



## Outreach S essions

After developing a list of preliminary fi ndings and recommendations, the inquiry conducted outreach meetings with various industry associations and groups, including

CAISO, WECC, the American Public Power Association (APPA), the North American Transmission Forum (NATF), the Edison Elec tric Institute (EEI), the National Rural Electric Cooperative Association (NRECA), an d representatives from Regional Entities (REs), Regional Transmission Organizations, and Independent System Operators. Teammembers shared the inquiry’s preliminary findings and recommendations on a nonpublic basis with members of these organization s to obtain feedback and, with respect to the recommendations, input as to their prac ticality and feasibility. The inquiry considered the feedback and input provided by these organizations and incorporated much of it into the findings and reco mmendations included in this report.



## - 14 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# B. System Overview

This subsection provides an overview of: (1) the West ern Interconnection and its position in the North American electric grid ; (2) the reliability entities responsible for operating the grid; (3) a description of the affected entities; and (4) a discussion of the

interconnected nature of these entities.



## The Western Interconnection and Its Position in the North American

Electric Grid

NERC shares its mission of ensuring the reliability of the BPS in North America with eight REs through a seri es of delegation of author ity agreements. WECC is the designated RE responsi ble for coordinating and promotin g BPS reliability in the Western Interconnection. In its capacity as the RE, WECC monitors and enforces compliance with Reliability Standards by the users, ow ners, and operators of the BPS. WECC also functions as an Interconnection-wide planning facilitator, aiding in transmission and resource integration planning at the request of its members, as well as a provider of data, analysis, and studies related to transmi ssion planning and reliability issues.

The WECC region extends from Canada to Mexico. It includes the provinces of Alberta and British Columbia, the northern port ion of Baja California, Mexico, the states of Washington, Oregon, California, Idaho, Nevada, Utah, Arizona, Colorado, Wyoming, and portions of Montana, South Dakota, New Mexico, and Texas. See Figure 2 , on the next page. The WECC region is nearly 1.8 mi llion square miles in size, has over 126,000 miles of transmission, and serves a population of 78 million. WECC contains 37 BAs and 53 TOPs. Due to the diverse characteristics of this extensive region, WECC encounters unique challenges in day-to-day coordination of its intercon nected system. WECC is tied

to the Eastern Interconnection through a number of high-voltage direct current

trans mission ties.

WECC also operates two RC offices that provide situational awareness and realtime monitoring of the entire Western In terconnection. WECC RC was an affected entity, and will be discussed with other affected entities below.



## - 15 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Figure 2: Map of WECC Region



## Reliability Responsibilities

NERC categorizes the entiti es responsible for planning and operating the BPS in a reliable manner into multiple functional entity types. The NERC functional entity types most relevant to this event are BAs, TOs, TOPs, Generator Operators (GOPs), Planning Coordinators (PCs), Transmission Planners (TPs), and RC s. These functions

are described in more detail in NE RC’s Reliability Functional Model. 22 

### Some of the affected entities conduct multiple reliability functions.



##  Balancing Authority

The BA integrates resource plans ahead of time, maintains in real time the balance of electricity resources (generation and interchange) and electricity demand or load within its footprint, and supports the In terconnection frequency in real time. There

22 NERC Reliability Functional Model, Version 5, http://www.nerc.com/files/Functio nal_Mo del_V5_Final_2009Dec1.pdf

.



## - 16 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

are 37 BAs in the WECC footprint. The following five BAs were affected by the event:

APS, IID, WALC, CAISO, and CFE.



##  Transmission Owner, Transmission Operator and Generator Operator

The TO owns and maintains transmission facilities. The TOP is responsible for the real-time operation of the transmission assets under its purview. The TOP has the authority to take corrective actions to ensure that its area operates reliably. The TOP performs reliability analyses, including seasonal and next-day planning and RTCA, and coordinates its analyses and operations with neighboring BAs and TOPs to achieve reliable operations. It also develops contingency plans, operates within established SOLs, and monitors operations of the transmissi on facilities within its area. There are 53 TOPs in the WECC region. The following seven TOPs were affected by the event: APS, IID, WALC, CAISO, CFE, SDG&E, and SCE. The GOP operates generating unit(s) and performs the functions of su pplying energy and other services required to support reliable system operations, such as prov iding regulation and reserve capacity.



##  Planning Coordinator

The PC is responsible for coordinating an d integrating transmission facility and

service plans, resource plans, and protection systems. 23



##  Transmission Planner

The TP is responsible for developing a long-term (generally one year and beyond) plan for the reliability of th e interconnected bulk transmission systems within its portion of the Planning Coordinator Area.



##  Reliability Coordinator

The RC and TOP have similar roles, bu t different scopes. The TOP directly maintains reliability for its own defined area. The RC is the “highest level of authority” according to NERC, and maintains reliability for the Interconnection as a whole. Thus, the RC is expected to have a “wide-area” view of the entire Interconnection, beyond what any single TOP could observe, to ensure operation within IROLs.

The RC oversees both transmission an d balancing operatio ns, and it has the authority to direct other functional entities to take certain actions to ensure reliable

23 PCs are the same as Planning Authorities (PAs) with respect to NERC registration and the Reliability Standards.



## - 17 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

operation. The RC, for example, may direct a TOP to take whatever action is necessary

to ensure that IROLs are not exceeded. 24 

### The RC performs reliability analyses including next-day planning and RTCA for the Intercon nection, but these studies are not intended

to substitute for TOPs’ studies of their own areas. Other responsi bilities of the RC include responding to requests from TOPs to assist in mitigating equipment overloads. The RC also coordinates with TOPs on system restoration plans, contingency plans, and reliability-rela ted services.



## Descriptions of Affected Entities

The following entities were affected by the September 8th event:



##  WECC RC

In its capacity as the RC, WECC is the hi ghest level of authority responsible for the reliable operation of the BPS in the Western Interconnection. WECC RC oversees the operation of the Western Interconnection in real time, receiving data from entities throughout the entire Interconnection, and providing high-level situational awareness for the entire system. WECC RC can direct the entities it oversees to take certain actions in order to preserve system reliability. Although WECC is both an RE and an RC, these two functions are organizat ionally separated.



##  Imperial Irrigation District

IID, which encompasses the Imperial Valley, the eastern part of Coachella Valley in Riverside County, and a small portion of San Diego County, in California, owns and operates generation, transmission , and distribution facilities in its service area to provide comprehensive electric service to its customer s. Thus, IID is a vertically integrated utility. IID’s generati on consists of hydroelectric units on the All-American Canal as well as oil-, nuclear-, coal-, and ga s-fired generation facilities, wi th a total net capability of 514 MW. IID purchases power from other electr ic utilities to meet its peak demands in summer, which can exceed 990 MW. IID’s transmission system consists of approximately 1,400 miles of 500, 230, 161, and 92 kV lines, as well as 26 transmission substations. Among other NERC registrations, IID is a TOP, BA, and TP responsible for resource and transmission planning, load balancing, and frequency support for its footprint.

24 For example, IRO

‐ 005

‐ 1 R.5 requires that “[e]ach [RC] shall identify the cause of any potential or actual SOL or

IROL violations. The [RC] shall initiate the control action or emergency procedure to relieve the potential or actual IROL violation without delay, and no longer than 30 minutes. The [RC] sha ll be able to utilize all resources, including load shedding, to address an IROL violation.”



## - 18 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



##  Arizona Public Service

APS is a vertically integrat ed utility that serves a 50 ,000 square mile territory spanning 11 of Arizona’s 15 counties. Amon g other NERC registrations, APS is the TOP and BA for its territory. APS engages in bo th marketing and grid operation functions, which are separated. APS owns and operates transmission facilities at the 500 (including H-NG), 345, 230, 115, and 69 kV levels, and owns approximately 6,300 MW of installed generation capacity. APS’s 2011 pe ak load was 7,087 MW.



##  Western Area Power Administration

– Lower Colorado

WALC is one of the four entities constituting the Western Area Power Administration, a federal power marketer within the United States Department of Energy. WALC operates in Arizona, Southe rn California, Colorado, Utah, New Mexico, and Nevada, and is registered with NERC as a BA, TOP, and PC for its footprint. As a net exporter of energy, WALC’s territory has over 6,200 MW of generation, serving at most 2,100 MW of peak load. A majority of WALC’s generation is federal hydroelectric facilities, with the balance consisting of thermal generation owned and operated by independent power producers. WALC also operates an extensive transmission network within its footprint, and is interconnected with APS, SCE, and nine other balancing

areas.



##  San Onofre Nuclear Generating Station

SONGS is a two-unit nucl ear generation facility capable of producing

approximately 2,200 MW of power, and is located north of San Diego. 25 

### SONGS produces approximately 19% of the power used by SCE customers and 25% of the power used by SDG&E customers. SONGS is jointly owned by SCE (78.21%), SDG&E (20%), and the City of Riverside (1.79%). SCE, as TO and GO, is responsi ble for ensuring the safe and reliable operation of SONGS within the grid.



##  California Independent System Operator

CAISO runs the primary market for wholesale electric power and open-access transmission in California, and manages the high-voltage transmission lines that make

25 SONGS is currently in the midst of an extended outage. According to a March 2012 press release by CAISO, if

both SONGS units remain offline for the summer, “San Diego and portions of the Los Angeles Basin may face local reliability challenges.” http://www.caiso.com/Documents/SummerGridOutlook Complicated

‐ PossibleExtendedOutage

‐ NuclearPowerPlant.pdf.



## - 19 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

up approximately 80% of California’s power grid. CAISO operates its market through day-ahead and hour-ahead markets, as well as scheduling power in real time as necessary. Among other registrations, CAISO is PC and BA for most of California, including the city of San Diego. It also acts as TOP for several entities within its footprint, including SDG&E and SCE. CAISO likewise engages in modeling and planning

functions in order to ensure long-term grid reliability, as well as identifying infrastructure upgrades necessary for grid function.



##  San Diego Gas and Electric

SDG&E is a utility that serves both electr icity and natural gas to its customers in San Diego County and a portion of southern Orange County, and is the primary utility for the city of San Diego. SDG&E owns re latively little generation—approximately 600 MW—although generation owned by others in its footprint brings the total generation capacity of the area above 3,350 MW. Peak load for the area can exceed 4,500 MW in the summer. SDG&E also operates an extensive high-voltage transmission network at

the 500, 230, and 138 kV levels. SDG&E, operating as a TOP within CAISO’s BA footprint, has delegated part of its resp onsibilities as a TOP to CAISO.



##  Comisión Federal de Electricidad

– Baja California Control Area

CFE is the only electric utility in Me xico, servicing up to 98% of the total population. CFE’s Baja California Control Area is not connected to the rest of Mexico’s electric grid but is connected to the Wester n Interconnection. CFE’s Baja California

Control Area covers the northwest corner of Mexico, including the cities of Tijuana,

Rosarito, Tecate, Ensenada, Mexicali, and San Luis Rio Colorado. CFE’s Baja California Control Area operates transmission systems at the 230, 161, 115, and 69 kV levels, and owns 2,039 MW of gross generating capacity and the rights to a 489 MW independent power producer within the Baja California Co ntrol Area. CFE’s Baja California Control Area had a net peak load of 2,184 MW for su mmer 2010. CFE’s Baja California Control Area is connected at the 230 kV level with SDG&E through two transmission lines on

WECC Path 45. CFE functions as the TO, TOP, and BA for its Baja California Control Area under the oversight of WECC RC. For th e remainder of this report, “CFE” refers only to its Baja California Control Area.



##  Southern California Edison

SCE is a large investor-owned utility which provides electr icity in central, coastal, and southern California. SCE is a wholly-o wned subsidiary of Ed ison International, which is also based in California. Among ot her NERC registrations, SCE operates as a



## - 20 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

TOP within CAISO’s BA footprint, and has delegated part of its resp onsibilities as a TOP to CAISO. SCE is also registered as TP, an d is responsible for the reliability assessments of the SONGS separation scheme. SCE owns 5,490 circuit miles of transmission lines, including 500, 230, and 161 kV lines. SC E also operates a subt ransmission system of

7,079 circuit miles at the 115, 66, 55, and 33 kV levels. Of the affected entities, SCE is interconnected with APS, IID, and SDG&E at various transmission voltage levels. SCE owns over 5,600 MW of generation, includin g a majority share in SONGS, and its peak load exceeds 22,000 MW. Along with SONGS st aff, SCE is responsible for the safe and reliable operation of the nuclear facility.



## Interconnected Operations

The September 8th event exemplifies the interconnected operations of three parallel transmission corridors through wh ich power flows into the area where the blackout occurred. Typically, BAs, thro ugh dispatch, balance the flows on these

corridors so that no one corri dor experiences overloads in an N-1 situation, but this did not happen on September 8th.

The first transmission corridor consists of the 500 kV H-NG, which is one of several transmission lines forming Path 49 (“ East of River”). Along with two 500 kV lines, one from North Gila to Imperial Va lley and another from Imperial Valley to Miguel, they form the SWPL. The majority of the SWPL is geographically parallel to the United States-Mexico border. The SWPL meets the SDG&E and IID systems at the Imperial Valley substation. This is shown as the “H-NG Corridor” on Figure 3 , on the

next page.

The second corridor is Path 44, also kn own as “South of SONGS,” operated by

CAISO. This corridor includes the five 230 kV lines in th e northernmost part of the SDG&E system that connect SDG&E with SCE at SONGS.

The third transmission corridor, shown as the “S Corridor” on Figure 3, consists of lower voltage (230, 161 and 92 kV) faciliti es operated by IID and WALC in parallel

with those of SCE, SDG&E, and APS. The only major interconnection between IID and SDG&E is through the 230 kV “S” Line, wh ich connects the SDG&E/IID jointly-owned Imperial Valley Substation (operated by SDG&E) to IID’s El Centro Switching Station.

The S Line interconnects the southern IID system with SDG&E and APS at Imperial Valley, which is also a terminus for the SWPL segment from Miguel and the SWPL segment from North Gila. WALC is connected to the SCE system and the rest of the Western Interconnection by 161 kV ties at Blythe, to IID by the 161 kV tie between



## - 21 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

WALC’s Knob and IID’s Pilot Knob substations, and to APS by a 69 kV tie via Gila at

North Gila.

The eastern end of the SWPL, which terminates at APS’s Hassayampa hub, is

connected to SCE via a 500 kV line that co nnects APS’s Palo Verde and SCE’s Devers substations. The northern IID system is co nnected to SCE’s Devers substation via a 230

kV transmission line that connects from Devers to IID’s CV substation. These connections, along with SDG&E’s connection to SCE via Path 44’s terminus at SONGS,

make the SWPL, Path 44, and IID’s and WALC’s systems operate as electrically parallel

transmission corridors. 26 

### The following simplified diagram illustrates the

interconnected nature of these three parallel corridors. Red lines represent 500 kV, blue lines represent 230 kV, and green lines represent 161 kV.

Figure 3: Three Parallel Corridors

26 Power transfers from APS to SDG&E and CFE generally flow across the SWPL, but, due to parallel path flows,

also known as loop flows, some of the power transfers flow through IID’s and WALC’s systems. Loop flow refers

to power flow along any transmission paths that are in parallel with the mo st direct geographic or contract path.



## - 22 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# III. S EQUENCE 

# O F 

# E VENTS 27



### The 11 minutes of the disturbance are divide d into seven phases, as highlighted in Figure 4 , on the next page. This figure displays the progressive loading of the five 230 kV tie lines from SCE north of San Diego that form Path 44. This section describes how the loss of various elements during an 11-minute period combined to exceed the 8,000 amp setting on the SONGS separation scheme. After sustained loading on Path 44 above 8,000 amps, the SONGS separation scheme operated. Once the SONGS separation scheme operated, San Diego and IID, CFE, and Yuma, Arizona, blacked out in less than

30 seconds. This section is divided into subsections for each phase, including the key events during the phase, their causes and effects, and, where relevant, what the affected

entities knew and did not know as the events were unfolding. Each section begins with a brief summary. A final subsection describe s restoration efforts after the blackout.

A set of graphics is included at the end of each phase to demonstrate the effect of the events during the phase. The first graphic in each set depicts the aggregate loading

in amps on the five South of SONGS lines. 28 

### The bottom portion of the graphic shows all of the phases, while the majority of the graphic shows an expanded view of the phase being discussed. The second graphic in each set represents the loading on key facilities after each phase. The third graphic in ea ch set shows how power flows redistributed through Arizona, Southern California, and Mexi co after each phase. Phases 6 and 7 have multiple power flow graphics. Phases 1 and 7 include only the second and third type of graphics.

27 All times are in Pacific Daylight Time (PDT) unless otherwise noted. Times are listed to millisecond (three

decimal places) or tenth

‐ of

‐ second (decimal place) accuracy when possible. If milliseconds or tenth

‐ of

‐ seconds

are not listed, the event is reconciled to the nearest second.

28 Path 44 flows (complex power in volt amperes, current in amps) were calculated from SONGS PMU data.

Those readings differ somewhat from disturbance monitoring equipment that was unavailable until completion of

the inquiry’s analysis. The differences are explained by variances in how some minor auxiliary loads are measured

and in measurement equipment tolerances.



## - 23 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

The following figure shows all seven phases of the disturbance.

Trip of Hassayampa

– North Gila 500 kV

Line Trip of Coachella Valley 230/92 kV

Transformers

Trip of Pilot

Knob – El

Centro 161 kV

Line

South of SONGS

Separation Trip of Pilot Knob 161/92 kV & Yucca 161/69 kV

Transformers Trip or Gila 161/69 kV

Transformers &

Yuma Cogen

Trip of Ramon

230/92 kV

Transformer

2 1 3 4 5 6 7

Disturbance Phases 

### Figure 4: Seven Phases of the Disturbance



# A. Phase 1: Pre

‐ Disturbance Conditions

Phase 1 Summary:

 Timing: September 8, 2011, before H-NG trips at 15:27:39

 A hot, shoulder season day with some generation and transmission maintenance outages

 Relatively high loading on some ke y facilities: H-NG at 78% of its normal rating, CV transformers at 83%

 44 minutes before loss of H-NG, IID’s RTCA result s showed that the N-1 contingency loss of the first CV transformer would result in an overload of the second transformer above its trip point

 An APS technician skipped a critical step in isol ating the series capacitor bank at the North Gila substation



### September 8, 2011, was a relatively no rmal, hot day in Arizona, Southern California, and Baja California, Mexico, with heavy power imports into Southern California from Arizona. In fact, imports into Southern Californ ia were approximately 2,750 MW, just below the import limit of 2, 850 MW. September is generally considered a “shoulder” season, when demand is lower than peak seasons and generation and transmission maintenance outages are scheduled. By September 8th, entities throughout the WECC region, including some of the affected entities, had begun



## - 24 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

generation and transmission outages for maintenance pu rposes. For example, on September 8th maintenance ou tages included over 600 MW of generation in Baja

California 29 

### and two 230 kV transmission lines in SDG&E’s territory. However, there were no major forced outages or major plan ned transmission outages that would result

in a reduction of the SOLs in the area.



##  Pre

‐ Disturbance Conditions in IID

Despite September being considered a shoulder month, temperatures in IID’s

service territory reached 115 degrees on September 8th. 30 

### IID’s load headed toward near-peak levels of more than 900 MW, which required it to dispatch local combustion turbine generation in accordance with established operating procedures. Prior to the event, loading on IID’s CV transformers reached approximately 125 megavolt amperes (MVA) per transformer, which is approximately 83% of the transformers’ normal limit. Loading on IID’s Ramon transformer was 153 MVA, which is approximately 68% of its

normal limit.

IID’s S Line ties IID to SDG&E, and through SDG&E, to generation in Mexico at

La Rosita. It also ties CFE and IID, through SDG&E’s La Rosita international transmission line. Before the event, IID was importing power on the S Line, and thus power was flowing northward from the jointly owned Imperial Valley substation to IID’s

El Centro substation. Flows on the S Line wo uld reverse multiple times during the event. When power flowed on the S Line from south to north, the implication was that IID was supplied radially through SDG&E. Throughout the event, as power flowed from north to south, the implication was that flows inte nded for SDG&E and/or CFE were moving through IID’s system. Eventually, in Phase 6, south to north flows on the S Line would activate a RAS that would ultimately trip more than 400 MW of generation at La Rosita and the S Line, thereby worsening the loading on Path 44.

Forty-four minutes prior to the loss of H-NG on September 8, 2011, IID’s RTCA results showed that the N-1 contingency loss of the first CV transformer would result in

an overload of the second transformer above its trip point. The IID operator was not actively monitoring the RTCA re sults and, therefore, was not alerted to the need to take any corrective actions. At the time of th e event, IID operators did not keep the RTCA

29 The generation was known as Termoelectrica de Mexicali, and will be hereafter referred to as “TDM.” It is also

shown as “TDM” on the Map of Affected Entities.

30 According to IID, the temperature in El Centro, California reached 115 degrees on September 8, 2011.



## - 25 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

display visible, and RTCA alarms were not audible. By reducing loading on the CV transformers at this pre-event stage, the operator could have mitigated the severe effects on the transformers that resulted when H-NG tripped. Since the event, IID has required, and now requires, its operators to have RTCA results displayed at al l times. The loading on IID’s CV transformers was pivotal to this event. Loading on the CV transformers is influenced by: (1) the pre-cont ingency flow on H-NG; (2) load and generation in IID’s 92 kV network; (3) flow on the S Line; and (4) to a lesser extent, generation connected to the Imperial Valley substation. See Figure 5, below.

800 1000 1200

1400

1600 1800

40% 50% 60% 70% 80% 90% 100% 110% 120% 130% 140% 150% 160% 170% 180% 190% 200%0501001502002503003504004505005506006507007508008509009501000 Pre

‐ Contingency Hassaympa

‐ North Gila

Line Loading

(MW)TransformerRating(%)

All 92kV Generation Level (MW) Post

‐ Contingency Coachella Valley Transformer Loading

All 92kV Generation and Normal Rating

Event Conditions Below normal rating Below emergency rating Below trip zone Trip Zone 

### Figure 5: Post-Contingency CV Transformer Loading Based on All IID 92 kV Generation



##  Pre

‐ Disturbance Conditions in CFE

At 15:07 CFE’s Presidente Juarez Unit 11 tripped, which required CFE to activate its Baja California Control Area contingency reserves to restore its ACE. At 15:15 PDT

CFE returned its ACE to where it had been before the unit tripped. Although still complying with the spinning reserve requ irements, CFE was short on non-spinning reserve, with all of its available resources in use or already deployed.



## - 26 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



##  Pre

‐ Disturbance Focus of WECC RC

Prior to the event, WECC RC operators were monitoring unscheduled flow on several paths in Northern California. WE CC RC did not view any of the scheduled

transmission or generation outa ges as significant. As illustrated by the chart below, two minutes before the event (at 15:25), major pa ths in the blackout area were operating below their Path ratings:

Major Paths in the Blackout Area Established Path Ratings/Flow Limits Path Loadings in MW and %

500 kV H-NG (Part of Corridor 1 into blackout area) 1,800 MW 31 

### 1,397 MW 78%

Path 44 (Corridor 2 into blackout area) 2,200 MW 32 

### 1,302 MW 59%

230 kV S Line (Part of Corridor 3 into blackout area) 239 MW 90 MW 38%

SDG&E Import SOL 2,850 MW 2,539 MW 89%

SDG&E to CFE Path 45 800 MW S-N; 408 MW N-S 241 MW N-S 60%



##  Pre

‐ Disturbance Conditions in APS

APS manages H-NG, a segment of the SWPL . At 13:57:46, the series capacitors 33



### at APS’s North Gila substation were automatically bypassed due to phase imbalance protection. APS sent a substation techni cian to perform switching to isolate the capacitor bank. The technician was experien ced in switching capa citor banks, having performed switching approximat ely a dozen times. APS also had a written switching order for the specific H-NG series capacito r bank at North Gila. After the APS system operator and the technician verified that they were working from the same switching order, the operator read steps 6 through 16 of the switching order to the technician. The

31 The limit of H

‐ NG is a portion of the rating of Path 49. The inquiry determined that the limit is approximately 1,800 MW.

32 With one segment of the SWPL out, the limit increases to 2,500 MW.

33 A series capacitor is a power system device that is connected in series with a transmission line. It increases the

transfer capability of the line by reducing the voltage drop across the line and by increasing the reactive power injection into the line to compensate for the reactive power consumption. In simple terms, a 50% series compensated line means it has the equivalent of 50% of the el ectric distance (or impedance) of the otherwise uncompensated line.



## - 27 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

technician repeated each step after the operator read it, and the operator verified the technician had correctly understood the step. The technician then put a hash mark beside each of steps 6 through 16 to indicate that he was to perform those steps. The technician did not begin to perform any of steps 6 through 16 unti l after all steps had been verified with the system operator.

The technician successfully performed step 6, verifying that the capacitor breaker was closed, placing it in “local ” and tagging it out with “do not operate” tags. However, because he was preoccupied with obtaining a ssistance from a maintenance crew to hang

grounds 34 

### for a later step, he accidentally wrote the time that he had completed step 6 on the line for step 8. For several minu tes, he had multiple conversations about obtaining assistance to hang the grounds. He then looked back at the switching order to see what step should be perf ormed next. His mistake in writing the time for step 6 on

the line for step 8 caused him to pick up with step 9, rather than step 7. 35 

### Thus, he skipped two steps, one of them the crucial step (step 8) of cl osing a line sw itch to place H-NG in parallel with the series capacitor bank. This step woul d bypass the capacitor bank, resulting in almost zero voltage across the bank and virtually zero current through the bank. Because he skipped step 8, when he began to crank open the hand-operated

disconnect switch to isolate the capacitor bank, it began arcing under load. 36 

### He could not manage to toggle the gearing on the switch to enable its closure, so he stayed under the arcing 500 kV line, determined to crank open the switch far enough to break the arc, thereby preventing additional damage to the equipment. Figure 6 , on the next page, is a schematic of the APS series capacitor bank, showing steps seven through nine.

34 Grounds are temporary protective connections that are run from conductive parts of lines, structures, and equipment, to earth or some other grounding system that substitutes for earth. If the isolated equipment is accidentally energized, grounds are intended to: (1) limit the voltage rise at the worksite to a safe value; and (2) provid e a pathway for fault current to flow, thereby allowing upstream protective devices to trip.

35 In human performance analysis, this is known as a “place keeping” error, by failing to physically mark steps as they are completed.

36 An electric arc is a luminous discharge of current that is formed when a strong current jumps a gap in a circuit.



## - 28 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Phase 1 Graphics 37 

### Figure 6: APS Series Capacitor Bank

Pre-disturbance (000) Step 9 (Open) Steps 7, 8 (Cl ose)

37 For the dial graphics shown here, green indicates available capacity on the facility, red indicates that the facility

is fully loaded to its normal limit, blue indicates the amount by which the facility is overloaded, and gray indicates

that the facility has tripped or load has been lost. For the power flow graphics

, black borders indicate islanding,

and gray areas bounded by black are those where load was lost.



## - 29 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

15:27:00



# B. Phase 2: Trip of the Hassayampa

‐ North Gila 500 kV Line

Phase 2 Summary:

 Timing: 15:27:39 to 15:28:16, just before CV transformer No. 2 trips

 H-NG trips due to fault; APS operators believ e they will restore it quickly and tell WECC RC

 H-NG flow redistributed to Path 44 (84% increase in flow), IID, and WALC systems

 CV transformers immediately overloaded above their relay setting

 At end of Phase 2, loading on Path 44 at 5,9 00 out of 8,000 amps needed to initiate SONGS separation scheme



### At 15:27:39, the arc that had developed on each phase of the disconnect switch lengthened as the switch continued to open, to the point where two phases came into contact. This caused H-NG to trip to clear th is phase-to-phase (A to C) fault. The highspeed protection system correctly detected the fault and tripped the line in 2.6 cycles (43 milliseconds). After discussion with the te chnician, APS operators erroneously believed that they could return the line to service in approximately minutes, even though they had no situational awareness of a large phase angle difference caused by the outage. More time would have been needed to redispatch generation to reduce the phase angle

difference to the allowed valu e. APS system operators in formed CAISO, Salt River Project (SRP), and WECC RC that the line would be reclosed quickly, even though they were unaware that this was not possible because of the large phase angle difference that



## - 30 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

existed between Hassayampa and North Gila. Th e inquiry’s simulation indicates that the post-contingency angular difference was be yond the allowed North Gila synch-check relay reclosing angle setting of 60 degrees, and there would not have been adequate generation for redispatch to reduce the ph ase angle difference to within the allowed value. APS operators were only able to se e the angular difference on EMS displays after isolating the North Gila capacitor bank an d re-energizing H-NG from the Hassayampa substation (before closing at North Gila).

H-NG, which has a flow limit of 1,800 MW 38 

### with a 30 minute emergency rating of 2,431 MW, was carrying 1,391 MW flowing from east to west along the SWPL at the time of the trip. As a result of the line tr ip, flows redistributed across the remaining lines into the San Diego, Imperial Valley, and Yuma areas. The IID and WALC systems, located between the two parallel high voltage Paths, were forced to carry approximately 23% of the flow that had initially been carried by H-NG. The majority of the flow

diverted to Path 44, as discussed below.

Immediately after the loss of H-NG, the load ing on both of IID’s CV transformers increased to 130% of their normal rating an d 118.5% of their emergency rating. The time overcurrent relays on the CV transformers picked up because the current flow was above the overcurrent relay setting, and began timing according to their very inverse 39 

### time delay. The CV transformers would both trip within 40 seconds of the loss of H-NG. At the same time, loading on IID’s Ramon 230/92 kV transformer increased to 94% of its normal rating and 85% of its emergency rati ng. Three seconds after the loss of H-NG, SCADA metering for the CV transformer bank s stopped recording accurate readings due to remote terminal unit (R TU) exceeding maximum scale. IID and WECC RC no longer had accurate information about or situational awareness of the loading on these important transformers.

IID also experienced increased loading on several of its 161 kV lines immediately after the loss of H-NG: Blythe-Niland and Knob-Pilot Knob loading increased by 49% and 55%, respectively. Flows on IID’s S Line reversed from south to north (SDG&E to

38 See footnote 31, supra

.

39 “Very inverse” describes the time/current characteristic of the relays’ time delay which is inversely proportional to the current magnitude sensed by the relay. That is, the greater the current, the less time before

the relay will trip.



## - 31 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

IID) to north to south (IID to SDG&E) during this phase of the event, indicating that flows intended for SDG&E were being routed through IID’s 161 and 92 kV systems.While IID was aware of the flow changes on the S Line, it was unable to see the loss of HNG in real time.

Flows on WALC’s Gila 161/69 kV transfor mers increased from approximately 12

MVA to 60 MVA, still well be low their normal limits of 75 MVA each, but indicative of the sudden increase in flows on WALC’s syste m just after the loss of H-NG. WALC also experienced significant voltage drops on its 161 kV system, particular ly at Blythe (6.9% drop) and Kofa (6.7% drop) substations, due to the increased flows on that system.

The loss of H-NG interrupted the southern 500 kV path into San Diego. The majority of the flow diverted to the northern entry to SDG&E, Path 44. Flow on Path 44 increased by approximately 84%, from 1,293 MW to 2,362 MW. This flow equates to a tie current of 5,900 amps relative to the 8,000 amps required to initiate the SONGS separation scheme.

Because so much of the flow on H-NG was intended for San Diego, the inquiry considered whether increasing internal generation in SDG&E’s area would have avoided

the cascading outages. 40 

### Figure 7 , on the next page, illustrates post-contingency loading on the CV transformers based on pre-contingency loading on H-NG and the generation level at IID’s and SDG&E’s jointly owned Imperial Valley substation. The red area on the graph indicates the large zone in which loading below H-NG’s 1,800 MW

SOL would load the CV transformers above thei r trip point. This area demonstrates the non-secure N-1 operating point of the CV tr ansformers. It shows that the operating conditions that would reduce the loading on the transformer are: increased generation at Imperial Valley, reduced flow on H-NG before it tripped, or both. For example, the graph indicates that for the sa me amount of transfer on H-NG, additional generators connected at Imperial Valley would reduce the post-conti ngency loading on the CV

transformers.

40 The inquiry’s analysis is not intended to suggest specific generation adjustments that could have been made by specific entities on September 8, 2011, but rather to show the extent to which the affected entities are interdependent.



## - 32 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

In general, adding generation in San Dieg o, CFE, or Imperial Valley and backing down generation in APS’s system (east of Pa th 49) would reduce the loading on IID’s 92 kV system for the loss of H-NG. For example, an additional 600 MW of generation at Imperial Valley and a reduction of generati on in APS’s system by the same amount would have reduced the pre-contingency lo ading on H-NG by 20% and improved the post-contingency voltage in WALC’s Blythe area by approximately 4%. Under this condition, the loading on the CV transf ormers for the loss of H-NG would be approximately 111% of their normal rating (166 MVA), well below their trip setting of

127%. This is a further demonstration of the importance of including all facilities when deriving SOLs.

After seeing the alarm for the loss of H-NG, the WECC RC operator promptly called the line’s operator, APS. APS told WECC RC it could get H-NG restored within minutes. While WECC RC was monitoring Rated Pa ths, it took no action specific to Path 44, believing it would take fi ve or ten minutes for APS to restore H-NG. As the entire event took only 11 minutes, WECC RC did not is sue any directives in connection with the

loss of H-NG.

Shortly after H-NG tripped, at 15:27:49, one of the combustion turbines at CFE’s Central La Rosita substation tripped while pr oducing 156 MW. This trip may have been 800

1000

1200

1400 1600 1800

40% 50% 60% 70% 80% 90% 100% 110% 120% 130% 140% 150% 160% 170% 180% 190% 200%

Pre

‐ Contingency Hassaympa

‐ North Gila

Line Loading

(MW)TransformerRating(%)

Imperial Valley Generation Level (MW) Post

‐ Contingency Coachella Valley Transformer Loading Imperial Valley Generation and Normal Rating

Event Conditions Below normal rating Below emergency rating Below trip zone Trip Zone 

### Figure 7 : Post-Contingency CV Transformer Loading Based on Imperial Valley Generation



## - 33 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

triggered by transients 41 

### caused by the initial fault at No rth Gila and subsequent trip of

H-NG. Loss of this unit further increased the flow on Path 44, raising the current to 6,200 amps out of the 8,000 needed to initiate the SONGS separation scheme. However, the La Rosita trip alone was not significant in causing the cascading that

followed. 42 

### CFE was also unaware in real time th at H-NG had tripped. After losing the

Central La Rosita unit, CFE was unable to re cover its ACE with its own resources, and at 15:30, it requested 158 MW of emergency assistance from CAISO for the remainder of

the hour.

Phase 2 Graphics

Hassyampa –

N. Gila 500

kV line trip CCM Unit 1 generator trip 

### South of SONGS

– Calculated Phase Current

41 See footnote 21. CFE stated that the trip was triggered by transients.

42 The Modeling and Simulation team conducted a “what if” simulation and determined that, even without the

inadvertent tripping of 160 MW of generation at La Rosita, the overloads and ensuing blackout would still have

occurred.



## - 34 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

15:27:39 – The Hassayampa- North Gila 500 kV line tripped.

15:27:40



## - 35 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# C. Phase 3: Trip of the Coachella Valley 230/92 kV Transformer and Voltage Depression

Phase 3 Summary:

 Timing: 15:28:16, when CV transformer bank No. 2 tripped, to just before 15:32:10, when Ramon transformer tripped

 Both CV transformers tripped with in 40 seconds of H-NG tripping

 IID knew losing both CV transformers would overload Ramon transformer and S Line connecting it with SDG&E

 Severe low voltage in WALC’s 161 kV system

 At end of Phase 3, loading on Path 44 at 6,700 amps out of 8,000 needed to initiate SONGS separation scheme



### At 15:28:16, less than a minute after H- NG tripped, IID’s CV transformer bank No. 2 tripped on the 230 kV side. The CV overload protection relays detected an overload immediately after H-NG was lost. The overloads were caused by through-flows on IID’s 92 and 161 kV systems which parallel APS’s 500 kV system. The normal ratings

for these transformers are 150 MVA, but im mediately after H-NG tripped, each CV transformer was carrying more than 191 MVA. The relays were set to trip at approximately 127% 43 

### of the transformers’ normal ratings, or 191.2 MVA at nominal voltage. The inverse time relays took 37.5 se conds to trip bank No. 2 and 38.2 seconds to trip bank No. 1. Thus, CV bank No. 1 tripped only 677 milliseconds after bank No. 2, again on the 230 kV side. Although the primar y winding or high side voltages of the CV

transformers are 230 kV, the banks were not considered as elements of the BES because their secondary winding or low side voltages ar e below 100 kV. As discussed in detail in

Section IV, because these transformers and the underlying 92 kV system were not classified as elements of the BES, IID, neighboring TOPs, and WECC RC did not assess the impact of critical external contingencies on overloading the CV banks, the effect of losing the CV banks and the subsequent impa ct on the Ramon bank, and, finally their overall adverse effect on BPS reliability.

IID was aware of the potential for local casc ading if the CV transformers tripped. IID’s next-day plan for Sept ember 8, 2011, which was not based on updated studies,

indicated that if both CV transformers tripped, 44 

### the Ramon 230/92 kV transformer would trip and the S Line tie with SDG&E would overload to 109% of its normal rating. The next-day plan also indicated that this ov erloading, in turn, would result in tripping

43 IID’s transformer protection philosophy specifies trip settings at 120% of normal ratings. IID chose the closest

available relay tap, which was approximately 127% of the normal rating.

44 This contingency scenario had nothing to do with H

‐ NG tripping. IID’s studies did not show any effect on the

CV banks resulting from the loss of H

‐ NG.



## - 36 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

generation because the S Line RAS trips genera tion supplied to Imperial Valley when the

S Line loads to 108% of its no rmal rating. IID’s next-day mitigation plan for loss of the CV transformers required starting turbines at Coachella and Niland and asking CAISO to redispatch generation to reliev e the S Line. This was a post-contingency mitigation plan. But after the event, IID’s operator admitted that if the CV transformers tripped on overload, he would have “very little time to mi tigate the Ramon [transformer], if at all.” Even the quickest-starting turbines take about 10 minutes to start and ramp to full load, but IID effectively had only four minutes be fore the Ramon transformer would trip, after

the loss of the CV transformers.

The loss of the CV banks caused flows on the S Line between SDG&E and IID to again reverse direction. Because its load ex ceeded internal generation, IID began pulling power from SCE through SDG&E due to the lo ss of key facilities in IID’s northern system. The tripping of the second CV ba nk also open-ended the Coachella ValleyRamon 230 kV “KS” Line (at CV), which was carrying about 41 MVA. This further increased loading on the Mirage-Ramon 230 kV line and through-flow from IID’s 230 kV collector system through Devers, but had little effect on the overall disturbance. By 15:31:35, IID’s operators switched in 92 kV capacitor banks at Avenue 42, Avenue 58, and Highline due to low voltage.

The loss of IID’s two CV transformers caus ed the aggregate current on Path 44 to increase from 6,200 amps to 6,600 of the 8,000 amps necessary to trigger the SONGS separation scheme. However, by the end of this Phase aggregate Path 44 current reached 6,700 amps.

The loss of the CV banks caused a severe voltage depression on the WALC 161 kV system south of Blythe. During this period, lo ads in that area (largely irrigation pumps) were highly susceptible to mo tor stalling, which can create additional reactive demand and exacerbate transmission loading, both of which contribute to additional voltage

decline. See Figure 8 , on the next page. At 15:28:18, the Blythe 161 kV bus alarmed at

142 kV (0.882 per unit). 45 

### WALC continued to experience severe low voltage on its 161 kV system until the S Line tripped at 15:38:02.4.

45 Other alarms and low voltage readings followed throughout WALC’s system one to nine seconds later, including the Parker

‐ Kofa 161 kV line, which alarmed for overload at 169 MVA (167 MVA rating); Kofa 161 kV bus voltage recorded at 143 kV (0.888 per unit); Knob 161 kV bus voltage recorded at 142 kV (0 .882 per unit); Parker

161 kV bus voltage recorded at 149 kV (0.925 per unit); Gila and Goldmine 161 kV bus voltages recorded at 144 kV (0.894 per unit); and Parker 230 kV bus voltage recorded at 222 (0.965 per unit).



## - 37 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Trip of Hassayampa

– North Gila 500 kV

Line El Centro – Pilot

Knob 161 kV Line Trip

Trip of Co achella Valley 230/92 kV

Transformers Yucca 161/69 kV

Transformers 1 and

2 Trip

Trip of Over 400

MW in Northern

IID 92 kV Load

Pocket Ramon 161/92 kV

Transformer Trip 

### Figure 8: Blythe 161kV Voltage



### On September 8, 2011, CAISO had partial vi sibility of IID’s system, but could not see that the CV banks had tripped. Prior to the event CAISO and IID had been working together to increase their mutual visibility and those efforts are continuing. Currently, CAISO receives loading data from the 230 kV side of the CV transformers.

Despite the fact that it did not consider the CV banks to be part of the BES, WECC RC does observe much of IID’s 92 kV system in real time, incl uding the CV banks. The WECC RC operator did notice the CV tran sformers trip, but he was focused on when

APS would return H-NG to service.



## - 38 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Phase 3 Graphics

South of SONGS – Calculated Phase Current

Coachella Valley

230/92 kV

transformers trip



### 15:28:17 – Two Coachella Valley 230/92 kV transformers and the Coachella Valley Ramon 230 kV “KS” line tripped. (030)



## - 39 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

15:28:18



# D. Phase 4: Trip of Ramon 230/92 kV Transformer and Collapse of IID’s

Northern 92 kV System

Phase 4 Summary:

 Timing: 15:32:10 to just before 15:35:40

 IID’s Ramon 230/92 transformer tripped at 15:32:10 , was set for 207% of its normal rating instead of its design setting of 120%, wh ich allowed it to last approximat ely four minutes longer than CV transformers

 IID experienced undervoltage load shedding, genera tion and transmission line loss in its 92 kV system

 Path 44 loading increased from approximately 6,700 amps, to as high as 7,800 amps, and ended at around 7,200 amps (out of 8,000 needed to initiate the SONGS separation scheme)



### At 15:32:10.621, less than five minute s after the trip of H-NG, IID’s Ramon 230/92 kV transformer tripped on the 92 kV side. The normal rating for this transformer was 225 MVA, and its relays were set to trip above 207% of its normal rating, or 466 MVA. Before it tripped, the SCADA metering for the Ramon bank had stopped recording accurate readings due to RTUs exceeding maximum scale, just as for the CV banks. Following the loss of the CV transformers, the inverse time relays took less than four minutes to trip the Ramon transformer. IID had intended to set the Ramon transformer to trip at 120% of its normal rating. Had it been set at this level, the Ramon transformer would have tripped almost immediately after the loss of the CV transformers, approximately four minutes earlie r than the time of its actual trip. IID

believed that the Ramon transformer would ov erload beyond the trip point upon the loss of both CV transformers. Its next-day plan noted, “the Ramon Bank #1 transformer will overload and relay out of service because th e overcurrent settings are set to trip at



## - 40 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

120%.” IID’s next-day plan re lied on a post-contingency oper ating philosophy of starting the Coachella Gas Turbines to mitigate overloads following the loss of the CV transformers, but the plan was unrealistic as IID would not have had time to start any additional generation between the loss of th e CV transformer banks and the loss of the

Ramon transformer.

Within less than one second after the lo ss of the Ramon transformer, automatic distribution undervoltage prot ection in IID’s northern 92 kV system began tripping

distribution feeders and sheddi ng load. From 15:32:11 to 15 :33:46, 444 MW of IID’s load tripped, with nearly half of the load be ing shed within 10 seconds of the Ramon transformer tripping. As illustrated in Figure 9 , below, the severe voltage depression following the loss of the Ramon transformer appears to have prom pted a local voltage collapse within IID’s northern 92 kV system , evidenced by both the steep drop-off in voltage as well as a sharp rise in reac tive power flow due to motor stalling.

Ramon 161/92 kV

Transformer Trip Trip of Over 400

MW in Northern

IID 92 kV Load

Pocket

Over

‐ Voltage Trip

of 92 kV System

Capacitors 

### Figure 9: 92kV Voltage (per unit) at Avenue 58



## - 41 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

The loss of IID’s northern resources an d subsequent system response caused IID to lose multiple generators connected to its 92 kV system, including IID’s Niland Gas Turbine 2 (generating 45 MW), IID’s CV Gas Turbine 4 (generating 20 MW), independent power producer Colm ac’s unit (generating 46 MW ), and IID’s Drop 4 Unit 2 Hydro Generator (generating 10.3 MW).

IID also began losing transmission line s. The Blythe-Niland 161 kV “F” Line, which saw increased loading during Phase 2, tripped at 15:32: 13 (approximately 3 seconds after loss of the Ramon banks). Its no rmal rating was 165 MVA, and it was set to trip at 129% of the normal rating (212 MVA at nominal voltage) with a 3-second time

delay. 46 

### The Niland-CV 161 kV “N” Line, carrying 83 MVA, tripped approximately 2

seconds later at 15:32:15.29 due to Zone 3 distance protection. 47



### In WALC’s territory, the Blythe-Goldmine-Knob and Parker-Kofa 161 kV lines overloaded approximately four seconds after the Ramon transformer tripped, at 15:32:14, but did not trip. Th ese lines each had a normal rating of 167 MVA, but were

loaded to 177 MVA. Power flows redistribute d through the Parker and Blythe areas after IID lost the Blythe-Niland line. WALC took some actions in an attempt to arrest the voltage depression it was experiencing, including a directive to start hydropower generation units Parker 3 and 4 for voltage suppo rt at 15:34:07. At the time, Parker area voltage was at 150 kV (0.932 per unit). WALC also switched in shunt capacitors on the 69 kV system at Gila and Kofa. At the time, voltage at Gila was at 65.5 kV (0.906 per unit) while Kofa was at 59 kV (0.86 per unit).

CAISO attempted to bring on generati on through its exceptional dispatch 48



### process to bring Path 44 back within its limi t of 2,500 MW, anticipa ting that it had 30 minutes to do so. At 15:35, it dispatched the Larkspur No. 2 peaking unit (rated 50 MW) within San Diego, which has a 20-minute start-up time. Also at this time, APS began taking steps to restore H-NG by completing the bypass of the series capacitor bank.

46 Based on the last available SCADA scan before the line tripped, the voltage at Blythe was at 123.1 kV (0.765 per unit) and the line was loaded to 172 MVA. Based on these measurements, the line was carrying 807 amps at the

last time recorded; the relay was set to trip wit h a 3

‐ second time delay at 762 amps.

47 A distance relay is a relay that compares observed voltage and current on a line and operates when that ratio is

below its preset value. Zone 3 relays are typically set to protect against faults that are more than one substation

away from the observed line as backup protection. An appropriate ti me delay should be set in the relay to give

the remote station relays the opportunity to operate and isolate the minimum amount of equipment necessary to clear the fault. A common issue with the application of Zone 3 relays is that they can restrict the loading on transmission lines (e.g. the N Li ne) during abnormal system conditions like those present on September 8th.

48 CAISO’s exceptional dispatch process involves calling on generators outside of the market automated dispatch process.



## - 42 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

During Phase 4, aggregate loading on the South of SONGS 230 kV transmission lines increased from approximat ely 6,700 amps to as high as 7,800 amps. The loading settled around 7,200 amps and remained there for the rest of Phase 4.

Phase 4 Graphics

South of SONGS

– Calculated Phase Current

Ramon 230/92 kV

transformer trip Multiple line,

generator and load trips Voltage collapse in pocket, followed by load tripping



### Time: 15:32:10

– The Ramon 230/92 kV transformer tripped and IID shed 444 MW of load. (110)



## - 43 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## 15: 32:35 Time: 15:32:35



# E. Phase 5: Yuma Load Pocket Separates from IID and WALC

Phase 5 Summary:

 Timing: 15:35:40 to just before 15:37:55

 The Gila and Yucca transformers tripped, isolating the Yuma load pocket to a single tie with SDG&E

 Path 44 loading increased from 7,200 to 7,400 amps after Gila transformer tripped, and ended at 7,800 amps after loss of the Yucca transformers and YCA generator (very close to the 8,000 amps needed to initiate the SONGS separation scheme)



### At 15:35:40, approximately eight minute s after H-NG tripped, WALC’s Gila 161/69 kV transformers tripped due to time-overcurrent protection. The two

transformers are each rated 75 MVA, but the 69 kV bus section that connects the transformers to the rest of the 69 kV substation is rated 1,200 amps (143 MVA at nominal voltage), and the overcurrent protection is set accordingly at 1,200 amps. The bus was carrying 1,312 amps at the time of the trip.

One minute later, at 15:36:40, the Yucca 161/69 kV transformers 1 and 2 tripped when their common 69 kV breaker tripped due to overload protection. Bank No. 1 is owned by IID and is rated 73 MVA, and bank No. 2 is owned by APS and is rated 75 MVA. The IID Yucca generator and four out of the six APS combustion turbines connected to APS’s 69 kV system were offline at the time of the event, as was the IID GT21 combustion turbine on the 161 KV side. These generators may have supported load in the area had they been in service. Almo st immediately, the Pilot Knob breaker on the

Pilot Knob-Yucca 161 kV “AX” transmission li ne, which is effectivel y the 161 kV breaker



## - 44 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

for the Yucca 161/69 kV transformers, received a direct transfer trip from the Yucca transformer overload protection, thereby tripping the AX Line. As a result of the loss of the Yucca and Gila transformers, the Yuma load pocket was isolated to only one tie to the SDG&E system, causing loading on each N. Gila 500/69 kV transformer bank to increase

from 57 MVA to 164 MVA.

Less than one second after the Yucca tr ansformers and AX Line tripped, at 15:35:40, the Yuma Cogeneration Associates (YCA) combined cycle plant on the Yuma 69 kV system tripped. The combustion turbine is rated at 35 MW and the heat recovery unit is rated at 17 MW, totaling 52 MW. It appears that both units were fully loaded at the time of the trip. The cause of the trip is unknown, but the loss of the YCA unit hastened the collapse of the Yuma load pocket.

Approximately one minute later, at 15:37:41, a common 161 kV breaker tripped

IID’s Pilot Knob 161/92 kV transformers Nos. 2 and 5 for No. 2 overload protection. The overload protection was set to trip the banks at 121% of the normal rating (37.5 MVA at nominal voltage).

At WALC’s request, between 15:36:48 an d 15:36:52, SCE directed Metropolitan Water District operators to drop 80 MW of pumping load attached to the Gene substation (near Parker) to improve 230 kV voltage support at Parker in an attempt to arrest declining voltages.

As it had done during Phase 4, CAISO ordered exceptional dispatch to bring Path 44 below its 2,500 MW limit. At 15:36:00, CAISO called SCE and ordered an exceptional dispatch of Larkspur Peaking Unit No. 1 (r ated 50 MW), and Kear ny GT2 and GT3 (each rated 59 MW) to go to full load. The Larkspur unit takes 20 minutes to start, and the Kearny units are 10-minute “quick start” peaking generators. All of these units were

offline at the time, and they were unable to come online before the system collapsed. 49



### The tripping of the Gila 161/69 kV transformers caused the aggregate loading on Path 44 to increase from approximately 7,2 00 amps to approximately 7,400 amps, out of the 8,000 amps necessary to in itiate the SONGS separation sc heme. After the loss of the Yucca 161/69 kV transformers, the YCA pl ant, and the Pilot Knob 161/92 kV transformers, the loading further increased to approximately 7,800 amps.

49 Larkspur generation is connected to the SDG&E 69 kV system south of Otay Mesa, and Kearny generation is

connected to the SDG&E 69 kV system in northern San Diego.



## - 45 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Phase 5 Graphics

South of SONGS

– Calculated Phase Current

Gila 161/69 kV

transformers trip YCA generating

units trip

Yucca 161/69 kV

transformers trip Pilot Knob 161/92 kV

transformers trip



### Time: 15:35:31

– Yuma Cogen (YCA) tripped. (180)



## 15: 37:42

- 46 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Time: 15:37:42



# F. Phase 6: High

‐ Speed Cascade, Operation of the SONGS Separation

Scheme and Islanding of San Diego, IID, CFE, and Yuma

Phase 6 summary:

 Timing: 15:37:55 to 15:38:21.2

 IID’s El Centro-Pilot Knob line tripped, forcing a ll of IID’s southern 92 kV system to draw from SDG&E via the S Line

 S Line RAS operates, tripping generation at Imperial Valley and worsening the loading on Path 44

 S Line RAS trips S Line, isolating IID from SDG&E

 Path 44 exceeds trip point of 8,000 amps, to as high as 9,500 amps

 SONGS separation scheme operates and creates SDG&E/CFE/Yuma island



### When the El Centro-Pilot Knob 161 kV li ne tripped at 15:37:55 (10 minutes after loss of H-NG), it isolated the southern IID 92 kV system onto a sing le transmission line from SDG&E: the S Line. Forcing all of th e remaining load in IID to draw through the SDG&E system pushed the aggregate current on Path 44 to 8,400 amps, well above the trip point of 8,000 amps. If the aggregat e current on Path 44 remained above 8,000



## - 47 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

amps, the definite minimum time relay 50 

### would initiate the SONGS separation scheme to separate SDG&E from SCE at SONGS.

IID’s El Centro-Pilot Knob 161 kV line open-ended at El Centro when a 161 kV

breaker at El Centro tripped on Zone 3 relay protection 51 

### with a one second delay. The apparent impedance detected on the Zone 3 relay at El Centro was hovering near its trip zone immediately following the Pilot Knob 161/92 kV transformer trips (12 seconds earlier), but did not cross into the Zone 3 tripping region until this time.

By this time in the event, the South of SONGS lines were San Diego’s only source of critical imported generation, and were also keeping IID and CFE’s Baja California Control Area from going dark. If the aggr egate current was brought below 8,000 amps,

the blackout could have been avoided, but at this point no operator action could have occurred quickly enough to save the Sout h of SONGS Path. Had there been formal operating procedures that recognized the need to promptly shed load as the aggregate current approached 8,000, and had operators been trained on the 8,000 amp set point, it is possible that operation of the SONGS separation scheme could have been averted by

earlier control actions.

Milliseconds after the loss of IID’s El Centro-Pilot Knob 161 kV line, at

15:37:55.890, NextEra’s Buck Boulevard comb ustion turbine generator tripped due to

operation of SCE’s Blythe Energy RA S, dropping 128 MW of generation. 52 

### This was caused by a reduction of counter-flows on th e Julian Hinds-Mirage 230 kV line that had been created by heavy flows from the Julian Hinds-Eagle Mountain 230 kV line feeding toward the WALC 161 kV system to support th e heavy north to south 161 kV flows toward Pilot Knob. When the El Centro-Pilot Knob 161 kV line tripped, those counter-flows disappeared, initiating the RA S operation. The Buck Boule vard heat recovery unit ramped down by 82 MW over the next few minutes. The Buck Boulevard combined cycle plant was generating 409 MW (535 MW rati ng) at the time the combustion turbine tripped. Tripping the Buck Boulevard gene rator did not increase loading on Path 44,

because it is not located south of Path 44.

50 A definite minimum time relay can operate in one of two ways. When current reaches a certain value, the relay will operate with a definite time delay that reflects the relay’s fastest operating time. Before the relay

reaches that value, the time for the relay to operate is inversely proportional to its obser ved current magnitude

. During the event, the relay operated while following the latter characteristic.

51 See footnote 47, supra

.

52 The Blythe Energy RAS, among other functions, trips generation owned by NextEra to protect the Julian Hinds

‐ Mirage 230 kV line from overloading with east to west flows for a potential loss of the Julian Hinds

‐ Eagle

Mountain 230 kV line. Buck Boulevard is connected to SCE’s 230 kV system in the Blythe are a.



## - 48 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Just three seconds after the loss of IID’s El Centro-Pilot Knob 161 kV line, at 15:37.58.2, the S Line RAS at Imperial Valley Substation initiated the tripping of two combined cycle generators at Central La Rosita in Mexico. The S Line RAS currently protects El Centro’s 161/92 kV transformer No. 2 by initially tripping a combination of CLR II generators when the flow on the S Li ne exceeds 269 MW flowing northward from

SDG&E into IID. Two combustion turbines we re loaded to 152 MW (193.5 MW rating), and 153 MW (193.5 MW rating), respectively , and the associated steam heat recovery unit (which also tripped following loss of the turbines) was loaded to 127 MW (159.3 rating), totaling 432 MW of generation.

Loss of the CLR II generation drove the South of SONGS flows from about 8,400 amps to about 9,500 amps, which remained above the 8,000 amp setting of the SONGS separation scheme. The inquiry’s simulation showed that had the S Line tripped without the S Line RAS tripping the CLR II generation , the flow on Path 44 would have fallen below 8,000 amps to settle at an estimated 7,730 amps, and the SONGS separation

scheme might not have operated. 53



### Approximately four seconds after the S Line RAS tripped the CLR II generators, at 15:38:02.4 the S Line RAS tripped the S Li ne itself due to flow above 289 MW toward IID from SDG&E. Tripping of this line crea ted an IID island. IID reported that from

15:37:59 to 15:40:24, 507.85 MW of load tri pped on its system, mostly in the southern 92 kV system.

The tripping of the S Line meant that IID was no longer pulling power from SDG&E and CFE through Path 44, so the aggregate Path 44 flows decreased from approximately 9,500 amps to approximately 8,700 amps, but were still above the 8,000 amps required to trigger the SONGS separation scheme.

At 15:38:21.2, not quite 11 minutes after H-NG tripped, the SONGS separation scheme operated, reconfiguring the SONGS 23 0 kV switchyard and isolating the SONGS generators onto the SCE system to the north. This reconfiguration effectively separated

53 The inquiry’s simulation showed that if the S Line RAS tripped only the S Line, IID’s system would still have collapsed, but San Diego and the Yuma load pocket would likely have survived. Voltages would have remained acceptable, and the 230 kV system around SONGS may have experienced minor overloads. While this would have

resulted in a lar ge phase angle difference on H

‐ NG, the fact that the SONGS separation scheme would not have operated would have allowed time for system operators to make the load and generation changes necessary to reduce the phase angle difference.

Had the S Line RAS not operat ed at all, or only operated to trip the CLR II generators, Path 44 flows would have

settled above the 8,000 amp threshold and thus the SONGS separation scheme would still have operated.



## - 49 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

all five South of SONGS 230 kV transmission lines from the SONGS units and the SCE system, and separated SDG&E from the rest of the Western Interconnection. Operation of the SONGS separation scheme created an island consisting of the SDG&E system, the remaining Yuma-area load connected through the 500 kV system from Miguel to North Gila, and CFE’s Baja Ca lifornia Control Area.

September 8, 2011, was the first time that the SONGS separation scheme had ever activated, and its effects on neighboring systems had not been studied. Although this sequence of events has focused on how the loss of elements combined over the 11 minutes to exceed the 8,000 amp SONGS separa tion scheme trigger, in real time, no entity was monitoring that limit or reco gnized the potential consequences of its

operation.

WECC RC, responsible for the reliable oper ation of the BPS, and with having a wide area view of the BPS, did not have any alarm that would al ert operators before operation of the separation scheme. Althou gh WECC RC operators were monitoring the Path limit on Path 44, they were not watchi ng the aggregate flows with respect to the SONGS separation scheme trigger. WECC RC operators noticed the five South of SONGS breakers open after the scheme had already operated.

CAISO, the TOP for SDG&E and SCE, did not have any alarms specifically tied to the operation of the SONGS separation scheme either. CAISO only has alarms for when Path 44 exceeds its Path rating, but had no ability to monitor the SONGS separation scheme, set at 3,100 MW (8,000 amps). After the loss of H-NG, which caused Path 44 to exceed its Path rating, CAISO operators were primarily concerned with returning flows on Path 44 to below the Path rating of 2, 500 MW, but believed they had 30 minutes to do so. Unlike Path ratings, the separation scheme would not allow CAISO operators 30

minutes to reduce flows on Path 44. CA ISO did attempt to di spatch additional generation within SDG&E to reduce flows on Path 44. The other method to reduce flows would have been to manually shed load in SDG&E in time to prevent operation of the SONGS separation scheme. SDG&E estimates th at it could have shed approximately 240

MW in between two and two-and-a-half minutes. However, SDG&E was never

instructed to shed load and was un aware of the need to shed load.



## - 50 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Phase 6 Graphics

South of SONGS

– Calculated Phase Current

El Centro – Pilot

Knob 161 kV line trip Imperial Valley

– El Centro 230 kV

“S” line trip CLR generating

units trip SONGS separation



## 15:38:02.4

15.38.21.2 Time: 15:37:55 – The El Centro-Pilot Knob 161 kV line tripped. (230)



## - 51 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Time: 15:37:55.110

Time: 15:38:02.4

Time: 15:38:21.2



## - 52 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# G. Phase 7: Collapse of the San Diego/CFE/Yuma Island

Phase 7 Summary:

 Timing: Just after 15:38:21.2 to 15:38:38

 Underfrequency Load Shedding (UFLS) was not able to prevent the SDG&E/CFE/Yuma island from collapsing

 SONGS nuclear units shut down even though they remained connected to the SCE side of the SONGS separation scheme



### During phase 7 of the event the SDG&E/CFE/Yuma island broke into three separate islands, all of which collapsed due to an imbalance between generation and demand, resulting in severe underfrequency which tripped both loads and generation.

The SDG&E/CFE/Yuma island created by operation of the SONGS separation scheme had a significant imbalance between generation and load from the beginning. As a result, the frequency in the island rapidly declined. By less than a second after the SONGS separation scheme activated (15:38:2 2), the UFLS programs of SDG&E, APS, and CFE had all began activating within the island. Figures 10 and 11, below show the frequency within the island as it collapses. All steps of the UFLS systems activated and system frequency in the island briefly stalle d at approximately 57.2 hertz (Hz). CFE’s UFLS analysis showed 512 MW of load shed by 15:38:21.901.

However, the same analysis showed that three CFE generators, totaling 459 MW, tripped offline beginning at 15:38:21.905, pa rtially negating CFE’s UFLS actions. In addition, a number of smaller generators, totaling about 130 MW, tripped only 0.5

seconds later while CFE was still connected to SDG&E and while SDG&E’s UFLS

program was still working to shed load. 54 

### See Figure 11, below. The net effect of CFE’s UFLS actions and generator trips—512 MW shed by UFLS and 590 MW of tripped generation—was that CFE’s imports from SDG&E increased from approximately 440 MW to approximately 520 MW. This worsen ed CFE’s system conditions and increased

the stress on SDG&E before SDG&E’s unde rfrequency separation protection systems opened the ties between CFE and SDG&E. SDG&E also had three generators with underfrequency protection that operated at 57.3 Hz, above the frequency at which the system leveled out. Due to these early generation losses , the frequency continued to decline below 57 Hz, which was the underfrequen cy setting for the majority of generators

in the island. Thus, the island blacked ou t, shortly after separating into three subislands.

54 The fact that several generators tripped during load shedding suggests that CFE may benefit from analyzing

whether its UFLS program and generator underfrequency protection systems are coordinated.



## - 53 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

SDG&E, CFE, and APS UFLSs Operation

56.50 57.00 57.50 58.00 58.50 59.00 59.50 60.00 60.50

21.0 21.2 21.4 21.6 21.8 22.0 22.2 22.4 22.6 22.8 23.0 23.2

Time 15:38 (s) 1

4 3 2

5 6

7 8 9

SDG&E's Block 9 Load shed (3080 MW) complete. Frequency is at ~57.25 Hz SDG&E

UFLS

Blocks' pick-up points 

### Separation

6 Blocks of CFE Load shed (512 MW) complete. Frequency is at ~57.4 Hz 5 Blocks of APS-Yuma Load shed (209 MW) complete. Frequency is at ~57.4 Hz CFE Generato r (150 MW) Trips

Two CFE Generators (309 MW) Trip Smaller

CFE

Generators

(130 MW) Trip 

### Figure 10: Frequency, Voltage in the SDG&E/Yuma/CFE Island

Figure 11: Frequency Performance in the SDG&E/Yuma/CFE Island



## - 54 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

The CFE island separated from SDG&E after their only two remaining ties tripped in rapid succession. At 15:38:22.2, the Otay Mesa-Tijuana 230 kV transmission

line open-ended at Tijuana in CFE’s terri tory due to underfrequency protection. 55 

### Less

than a second later, at 15:38:23.13, the Im perial Valley-La Rosita 230 kV transmission line open-ended at Imperial Valley in SDG&E’s territory by underfrequency

protection. 56 

### According to CFE, its UFLS program was not designed for the o a SDG&E/CFE/Yuma island, but for the oper ation of a “southern WECC island.” peration of

The Yu ma island separated from SDG&E at 15:38:23.12, when the Imperial Valley-North Gila 500 kV transmission line tripped by underfrequen cy protection. APS’s UFLS operated on 26 out of the 28 feeders in the Yuma area prior to the loss of the local Yucca steam generators that were on line . However, there was insufficient local generation to stabilize the load pocket in Yuma. At 15:38:38, the Yuma island internal units tripped on underfrequency protection.

At about the same time that it separated from CFE and APS’s Yuma pocket, SDG&E lost four generating units, to taling 570 MW, due to the generators’

underfrequency protection. 57



### Although the SONGS generators remained connected to the SCE side of the switchyard at SONGS, at about 15:38:27.5 , or approximately six seconds after the SONGS separation scheme initiated, the SONGS turbines both experienced a brief acceleration in speed and tripped due to turb ine control logic. At the same time, local system frequency at SONGS was observed to sp ike from 59.974 Hz to 61.203 Hz. After the initial impulse caused by the system sepa ration, the frequency in the main body of the Western Interconnection peaked near 60.170 Hz. This can be seen on Figures 12 and 13, on the next page. The turbine trip init iated a reactor shutdown, and the units began coasting down. A little more than a seco nd later, at 15:38:28.963, SONGS Unit 3 electrically disconnected from the system, an d less than three seconds after the reactors

55 The Tijuana end opened instantaneously. Subsequently, the Otay Mesa end of the line in SDG&E’s territory opened at 15:38:23.044 by underfrequency protection (with 1

‐ second delay).

56 The line’s underfrequency setting was 57.9 Hz, with 1

‐ second delay. The instantaneous underfrequency protection scheme at La Rosita in CFE’s territory failed to operate due to a bad fuse connection.

57 At 15:38:23.000, the Palomar Energy Center combustion turbines CT1 and CT2 tripped on underfrequency,

followed by the heat recovery unit ST at 15:38:23.07 (all set to trip at 57.3 Hz with a 750 millisecond time delay).

CT1 was loaded to 160 MW, CT2 was loaded to 165 MW, and ST was load ed to 195 MW at the time of the trips. It is

believed that additional unit Goal Line LP, generating 50 MW, tripped around the same time due to a 58 Hz frequency with a 1

‐ second time delay.



## - 55 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

shut down, at 15:38:30.209, SONGS Unit 2 el ectrically disconnected from the system. Loss of the 2,300 MW of SONGS’ generation e ffectively reduced the loss of load for the main body of the Western Interconnection from a 3,400 MW loss to a net 1,100 MW load loss. This made the recovery from the resulting overfrequency event much easier. The SONGS generators did not lose offsite powe r because the SONGS sw itchyard was still connected to the SCE system.

Figure 12 : Frequency Excursion in WECC Interconnection Immediately after the SONGS

Figure 13: SONGS Generation Trips and Auxiliary Loads Transfer to 230 kV Bus



## - 56 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

By 15:38:38, the SDG&E, CFE and Yuma islands had all collapsed, leaving approximately 2.7 million customers without power.

Phase 7 Graphics

Time: 15:38:30 – The South of SONGS Separation Scheme operates and both SONGS units tripped. (300)



## - 57 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Yuma Separates (Time: 15.38.23.12)

CFE Separates (Time: 15.38.23.13)



## - 58 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

SDG&E, CFE, and Yuma Blackout ( by 15.38.30)



## - 59 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# H. System Restoration

None of the affected entities needed to implement black start plans because they all were able to access sources of power from their own or a neighbor’s system that was still energized. The restoration process generally proceeded as expected, and some entities restored load more quickly than they had expected. The following charts indicate how long it took the affected entities to fully restore their lost load, generation,

and transmission.

LOAD RESTORATION EFFORTS

Entities Demand Interrupted (MW) Time Until Demand Fully Restored Date Restored 100 % Demand Restored (hrs) Number of Customers Affected

SDG&E 4,293 03:23 9/9/11 12 1.4 Million

CFE 2,205 01:37 9/9/11 10 1.1 Million

IID 929 21:40 9/8/11 6 146,000

APS 389 21:12 9/8/11 6 70,000

WALC 74 22:23 9/8/11 6.5 5 58

58 The majority of WALC’s lost load (64 MW) affected APS customers. SCE lost 117 fringe load customers who

were served by the SDG&E system.



## - 60 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

GENERATION RESTORATION EFFORTS

Entities Generation Lost (MW) Time Generation Restored Date Restored Generation Restored (hrs)

SCE 2,428 06:33 9/12/11 87

SDG&E 59 

### 2,229 06:20 9/10/11 39

CFE 1,915 23:43 9/10/11 56

IID 333 20:42 9/8/11 5

APS 76 20:37 9/8/11 5

TRANSMISSION RESTORATION EFFORTS

Entities Final Transmission Restored (kV) Time Transmission Restored Date Restored Transmission Restored (hrs)

IID 230

161 03:37

00:31 9/9/11

9/9/11 12

9

SDG&E 500

230 17:36

03:47 9/8/11

9/9/11 2

12

APS 500 16:51 9/8/11 1.5

WALC 161 17:09 9/8/11 1.5 60



### CFE 230

115 04:03

01:58 9/9/11

9/9/11 12.5

10

59 According to SDG&E, after restoring the SDG&E transmission systems, CAISO took over restoring SDG&E’s generation.

60 This represents the time it took WALC to restore its 161/69 kV Gila transformers, however, none of WALC’s

transmission lines were lost in the outage.



## - 61 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

WECC RC could have taken a more active role in coordinating the restoration efforts. WECC RC has the largest area of vi sibility and more advanced real-time study tools than the TOPs. During a multi-system restoration, issues are likely to arise between neighboring BAs and TOPs that may re quire either a neutral decision maker, or rapid technical analysis of un planned system conditions. WECC RC is uniquely situated to provide such assistance. WECC RC shou ld clarify its role, and the real-time information it can provide, in emergency situ ations like a multi-system restoration. WECC RC should also specifically addres s the issue of coordination among other functional entities (like BAs and TOPs) in its operating area, ou tlining the areas of responsibility during system rest oration and other emergencies.

The inquiry reviewed recordings and other data about restoration which disclosed the following incidents that coul d have benefitted from better WECC RC

coordination and assistance in real time:

 A 30-minute debate occurred between SCE, which was attempting to provide cranking power to SDG&E to restore SDG&E’s system, and the SONGS operators, about the conditions necessary for resetting the SONGS separation scheme lockout relay.

 Recordings showed a lack of clarity among WE CC RC, CAISO, and SDG&E about responsibilities for restoration efforts. Among other things, this resulted in a SONGS operator making a unilateral decision to open a circuit breaker on the line responsible for restoring power to SDG&E’s system, leaving the line in a less reliable configuration (connected to a single bus).



## - 62 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# IV. C AUSES



# , F INDINGS



# , A ND 

# R ECOMMENDATIONS



# A. Planning

Next

‐ Day Planning

 Background

TOPs are required to perform next-day studies to identify and plan for potential limitations on their system in the day-ahead timeframe, and to coordinate these studies

with their neighboring TOPs. 61 

### These studies provide a proactive mechanism to ensure that the system can be operated reliably and allow time to develop effective operating

solutions. 62 

### These solutions include, among other things, effective control actions needed to return the system to a secure state in anticipated normal and contingency system conditions. The development of th ese plans in the day-ahead timeframe is critical because it would be nearly impossibl e, due to the complexity of the BPS, for control room operators to return the system to a secure operating state under stressed conditions without effective action plans developed in advance. The adequacy of nextday studies depends on how extensively and a ccurately facilities and next-day system conditions are incorporated into the models used for the studie s. This includes consideration of a reasonably accurate, current, and complete list of external contingencies that could impact a TOP’s system as well as internal contingencies that could impact external SOLs. Consistency of study inputs among all TOPs and BAs is also critical for reliable operation.

The inquiry found that the affected TOPs’ and BAs’ procedures for conducting next-day studies and models used in these st udies vary considerably. As explained more fully below, APS does not conduct next-day st udies, relying, instead, on two sets of

studies, conducted on a seasonal and annual basis, that consider a list of possible, predetermined contingency scenarios and provid e plans to mitigate the contingencies if violated. Meanwhile, IID has a policy of conducting next-day studies each day, but between April and October of 2011, it failed to perform the required studies on a daily basis. All other affected TOPs conduct next-day studies, but they use models that do not

61 See NERC Reliability Standard TOP

‐ 002

‐ 2b R11.

62 See, e.g ., NERC Reliability Standard TOP

‐ 002

‐ 2b (“Current operations plans and procedures are essential to be prepared for reliable operations, including response for unplanned events.”).



## - 63 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

adequately reflect next-day operations of facilities in networks external to them. These TOPs’ next-day studies also do not consider a full list of internal and external contingencies that could impose limitation s on their daily operations or external operations. Moreover, most of these TOPs’ ne xt-day studies do not consider the impact

of sub-100 kV facilities on BPS reliability, such as the impact of IID’s CV transformers.

WECC RC is the highest level of authority responsible for reliab le operation of the BPS in the Western Interconnection, with the authority to prevent or mitigate emergency operating conditions in the next-day and re al-time timeframes. As such, WECC RC also conducts next-day studies for the entire We stern Interconnection and builds its model from the previous day’s peak State Estimator case, which includes all facilities operated

at 100 kV and above and some sub-100 kV facilities. WECC RC then incorporates forecast information, which typically includ es transmission outages as provided by TOPs, generation outages or derates of 50 MW or greater as provided by TOPs, as well as load forecasts, expected net interchange, and unit commitment forecast data from BAs. While WECC RC has a more extensive representation of facilities throughout the WECC footprint in its model than any individual TO P, it does not necessarily monitor or alarm for certain lower voltage facilities and faci lities deemed non-BES that can impact BPS reliability. Moreover, because some of the forecasted information can change betweenthe time the TOPs and BAs provide it to WECC RC and the time WECC RC runs its nextday studies, WECC RC’s next-day studies might not accurately reflect next-day

operations.

The September 8th event exposed four weaknesses with the foregoing procedures for conducting next-day studies in WECC’s region. These weaknesses are detailed in the following four findings. A co mmon theme prevails in all four findings: the affected entities do not accurately account for external next-day operating conditions or potential external contingencies that could impact their systems.

Finding 1 Failure to Conduct and Share Next-Day Studies:

 Not all of the affected TOPs conduct next-day studies or share them with their neighbors and WECC RC. As a result of failing to exchange studies, on September 8, 2011 TOPs were not aler ted to contingencies on neighboring systems that could impact their internal system and the need to plan for such contingencies.



### Recommendation 1:

 All TOPs should conduct next-day studies and share the results with neighboring TOPs and the RC (before the next day) to ensure that all contingencies that could impact the BPS are studied.



## - 64 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Failure to Conduct Next-Day Studies

APS does not conduct next-day studies. Inst ead, it relies on two sets of studies, conducted on a seasonal and annual basis, for its daily operations. First, APS uses its

summer and winter seasonal studies fo r the non-WECC Rated Paths within its transmission system. APS performs these stud ies on a model that it builds from the WECC heavy summer base case. In a coordinated effort with other entities in Arizona, it updates this WECC base case with anticipate d loads and resources from the state. APS then adds a detailed representation of the entire state’s network, including its own subtransmission system down to the 12 kV di stribution system, to finalize the summer

model. To create its winter model, APS mo difies the summer model with winter peak conditions throughout Arizona.

Once these summer and winter models are complete, APS studies a set of predetermined contingencies, and relies on the results to determine the response of its transmission system to single and multiple contingencies during peak load conditions with planned outages modeled. The studies’ list of contingencie s is based on past studies, operating experience, and engineer ing judgment. The st udies also establish mitigating measures for contingencies that do not meet loading or voltage guidelines.

Second, APS relies on a single manual, de veloped annually, as a guide for its daily operations on four Rated Paths within its system . This manual is the result of studies of possible, predetermined contingencies on Ra ted Paths. The resu lts and operational

instructions in this manual are based on seasonal models that APS develops in coordination with four WECC regional study groups, led by CAISO. CAISO first sends a base case to each study group to update with topology changes for the upcoming season. Individual members of each study group also update the model with details from their systems. CAISO then incorporates all of the updates and stresses key Paths in California before sending the model back to the study gr oups. APS uses this model as a starting point to study the four Rated Paths in its sys tem. APS analyzes the resulting peak-load model using a predetermined set of single and double contingency events that are focused primarily on high-voltage transmission outages to determine required actions to secure the system for the next most critical N-1 event. 63 

### The manual directs APS to rerate relevant Path(s) and id entifies necessary mitigating measures as long as the contingency (or multiple contingency) scenario is included in the manual. The manual, however, may not include a part icular contingency (or multiple contingency) scenario, or

63 APS’s manual covers only 500 kV and 345 kV facilities, and nothing lower.



## - 65 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

may not accurately reflect the internal an d external system topology for the day in question, resulting in the potential for unforeseen circumstances.

Thus, APS uses seasonal studies for no n-Rated Paths and the manual for Rated Paths as tools in the day-ahead timeframe, without any additional analysis to validate that the tools remain valid for the next day’ s specific configuration and operation, such

as transmission or generation outages ex ternal to APS’s footprint that were not anticipated at the time the base seasonal st udy was performed. APS maintains that these tools are sufficient for day-ahead purposes because they include the most severe contingencies identified in its system. This viewpoint overlooks th e purpose of next-day studies—to plan for next-day operations in light of conditions that change daily . By relying on tools based on studies conducted on a seasonal and annual basis, APS cannot account for all plausible daily scenarios. For typical days that fall within the boundaries of the underlying studies and analysis, APS’ s tools may be viable. For atypical days where conditions fall outside the studied boundaries, however, this approach may not be adequate. For example, September 8, 2011, was an atypical day not contemplated by

APS’s manual, as the manual di d not account for various genera tion outages in effect for

maintenance.

Between April and October 2011, IID also did not consistently perform adequate next-day analyses for each day. Although II D had a policy of cond ucting separate nextday analyses for each new day, it failed to consistently perform the required analyses. Specifically, IID produced a document each new day showing various changes in weather, load and generation forecasts, plan ned facility outages, potential contingency violations, or mitigation measures for id entified contingencies, but did not always perform the underlying power fl ow studies for each day between April and October 2011. On average, between April 2011 and October 2011 IID actually performed a study no more than two times per week. For the other da ys, IID simply referenced past studies. For example, it appears that IID did no t perform a separate, updated study for September 8, 2011, because the powerflow st udy case provided for this day does not match the contingency results included in the daily operations guide for the day. In other words, it appears that for September 8, 2011, IID simply changed the forecasted data without actually performing the next-day study. Instead, IID referenced a previous study. The referenced study, however, was not valid because it did not match the load and generation dispatch data for the day, and there were differences in projected overloads reported as potential contingencies. IID’s next-day studies were purportedly reviewed by IID for accuracy, but these discrepancies were not identified. IID discovered this issue during the course of the inquir y and is in the process of implementing corrective actions to ensure accurate next-day analyses are completed in the future.



## - 66 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Finally, the inquiry heard on more than one occasion from TOPs, including APS, that WECC RC was responsible for conducting next-day studies or that WECC RC should conduct next-day studies that TOPs are cu rrently responsible for conducting. WECC RC’s next-day studies for the entire Interconnection, however, are not intended to substitute for the TOPs’ next-day studies of their own systems.

Failure to Effectively Share and Coordinate Next-Day Studies

In addition to finding that not all entiti es conduct next-day studies, the inquiry found problems with sharing and coordination among the affected TOPs that do conduct

such studies. The affected TOPs do not cons istently share their st udies with neighboring TOPs, BAs, and the RC. TOPs generally pr ovide studies to WECC RC only if the RC identifies an issue in its study and specifically asks to review a TOP’s study. In addition, WECC RC’s method of sharing its next-day stud ies with other entities is not effective. Specifically, WECC RC’s practice is to shar e the results of its next-day studies when

conditions warrant, or when it rece ives a request for a study result. 64 

### WECC RC posts on a secure Internet portal a list of limitati ons or SOLs identified by its next-day studies

for individual TOPs and BAs to view, but it is up to TOPs and BAs to access this list. Also, this list contains only issues that WECC RC deems significant and does not include basic, next-day operating conditio ns, such as scheduled outages.

One example of the adverse consequences of these sharing and coordination issues relates to the 600-plus MW of TDM generation that was o ffline for maintenance on September 8th. The TDM generation outage was included in WECC RC’s and CAISO’s next-day studies, and posted on CA ISO’s website, but not incorporated into

other entities’ next-day models and studies. 65 

### WECC RC receives outage information from TOPs and BAs through its Coordinated Ou tage System (COS). While TOPs and BAs submit their own information into COS, they cannot access information submitted by

others. IID could have benefitted from know ledge of the TDM outages. The TDM units radially connect to the Imperial Valley subs tation, jointly owned by IID and SDG&E. If the TDM units had been online, they could ha ve mitigated northern IID overloads on the

64 See WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/W ECC RC Operations Planning.pdf

.

65 CAISO knew about the outages because the TDM units participate in the CAISO market. CAISO posts daily outage unit status reports on its public website that provide the best available data at the date and time of the report, for generation units that participate in CAISO’s market. These outages are posted at http://www.caiso.com/market/Pages/ OutageManagement/UnitStatus.aspx

. In CAISO’s archives, the TDM units

are shown on outage on September 7 and 8, at a minimum. Dispatch details, however, are not included. WECC

RC receives CAISO’s outage unit status reports daily by email and was aware of the outages. However, IID and

APS did not know about the TDM outages.



## - 67 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

CV and Ramon transformers that resulted when H-NG tripped. If IID had learned about these outages from WECC RC or CAISO, it co uld have incorporated the outages in the day-ahead timeframe and dispatched additi onal generation, or taken other control actions, to compensate for the overloads on it s system caused by having these generators offline and the H-NG tripping.



## The September 8th event illustrates th at conducting next-day studies and sharing the results of such studies are crit ical to allow TOPs to identify and plan for potential contingencies.

Finding 2 Lack of Updated External Networks in Next-Day Study Models:

 When conducting next-day studies, some affected TOPs use models for external networks that are not updated to reflect next-day operating conditions external to their systems, such as generation schedules and transmission outages. As a result, these TOPs’ next-day studies do not adequately predict the impact of exte rnal contingencies on their systems or internal contingencies on external systems.



### Recommendation 2:

 TOPs and BAs should ensure that their next-day studies are updated to reflect next-day operating conditions external to their systems, such as generation and transmission outages and scheduled interchanges, which can significantly impact the operation of their systems. TOPs and BAs should take the necessary steps, such as executing nondisclosure agreements, to allow the free exchange of next-day operations data between operating entities. Also, RCs should review the procedures in the region for coordinating next-day studies, ensure adequate data exchange among BAs and TOPs, and facilitate the next-day studies of BAs and TOPs.



### As a starting point for their next-day studies, the affected TOPs use models from either a TOP’s seasonal base case or the pr evious day’s EMS model, if available. The seasonal base case represents next-day operating conditions internal to the TOPs’ systems, but leaves external networks exac tly as they were represented in the WECC seasonal base case. The affected TOPs’ EMS models sometimes include only one or two buses outside each TOP’s internal footprint. Thus, neither type of day-ahead model contains actual day-ahead forecasts of system conditions external to each TOP’s system. For example, leading into September 8th, the affected TOPs had limited knowledge of

the current status of transm ission facilities, expected generation output, and load predictions outside their footprints. Cons equently, their next-day studies could not adequately predict the impact of external co ntingencies on their systems or of internal contingencies on external systems.



## - 68 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

IID’s next-day study for September 8th illustrates the adverse effects of not accounting for external next-day planned operations. IID used the WECC heavy summer

seasonal base case to model external conditio ns for its next-day stud y for September 8th.

This base case reflects that most external generation is online to meet summer peak loads. A heavy summer base case does not accurately represent a shoulder season day like September 8th. By Sept ember, both generation and transmission maintenance had

started.

For example, on September 8th TDM generator units in Mexico, totaling more than 600 MW, were offline for maintenance. These units are external to IID and radially connect to IID’s jointly owned Imperial Valley substation. When online, this generation can help to mitigate overloads on the CV and Ramon transformers in IID’s system. Because IID relied on a heavy summer seasonal model for external networks and did not incorporate any updates about the TDM generati on, its next-day study did not reflect the maintenance outage of these units. With the TDM generation incorrectly represented as being online, IID’s next-day study did not co rrectly identify how much the loss of H-NG

would overload IID’s transformers in its 92 kV system. In fact, IID’s next-day study for September 8, 2011, did not show that th e loss of H-NG would overload the CV

transformers to their trip point. 66 

### If IID had learned about the TDM outages (whether

from CAISO’s website or BY some other meth od) and incorporated the information into its model, it could have dispatched addition al generation, adjusted load, or taken other

control actions before the loss of H-NG to mitigate such overloading.

As mentioned above, WECC RC receives next-day data from the entities through interfaces such as the COS. WECC RC is well-situated to facilitate data-sharing among the 37 BAs and 53 TOPs in the WECC footprint. Given the large number of BAs and TOPs in the WECC region, some of which are relatively small in size and resources,

central coordination and facili tation may be necessary to en sure that all BAs and TOPs

accurately reflect next-day operating conditions external to their system. 67 

### WECC RC has been working to facilitate data sharing by drafting and circulating a universal

66 The heavy summer base case has more than 1,000 MW more generation in the affected area than was available on September 8, 2011. In addition to not representing the offline generation, IID’s study, by relying on the heavy summer base case, did not accurately reflect the flow on H

‐ NG. The heavy summe r base case shows

flow on H

‐ NG as 1,118 MW, while actual flow on H

‐ NG at the time of the trip was 1,391 MW.

67 Under current WECC RC procedures, the RC only shares the results of its operational planning analyses if the

results indicate the need for specific operational actions to prevent or mitigate an instance of exceeding an operating limit. WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/W ECC RC Operations Planning.pdf

.



## - 69 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

nondisclosure agreement. As this report was being finalized, less than 30 of the

approximately 100 discrete entities wi thin WECC had signed the agreement. 68



### Finding 3 Sub-100 kV Facilities Not Adequately Considered in Next-Day Studies:

 In conducting next-day studies, some affected TOPs focus primarily on the TOPs’ internal SOLs and the need to stay within established Rated Path limits, without adequate consideration of some lower voltage facilities. As a result, these TOPs risk overlooking facilities that may become overloaded and impact the reliability of the BPS. Similarly, the RC does not study sub100 kV facilities that impact BPS reli ability unless it has specifically been alerted to issues with such facilities by individual TOPs or the RC has otherwise identified a particular sub-1 00 kV facility as affecting the BPS.



### Recommendation 3:

 TOPs and RCs should ensure that thei r next-day studies include all internal and external facilities (including those below 100 kV) that can impact BPS reliability.



### The September 8th event showed that some sub-100 kV facilities can have significant impacts on BPS reliab ility, such as causing instab ility or cascading outages. Yet, it appears that these facilities are not adequately considered in the day-ahead timeframe. For example, IID’s 92 kV networ k runs parallel to two major transmission Paths: (1) Path 44, which connects to the SW PL via the Palo Verde-Devers 500 kV line (part of Path 49) and runs to the north of IID; and (2) the SWPL, which runs to the south of IID. Given the parallel natu re of its system, IID’s 92 kV system is forced to carry a significant portion of any ea st-west power flows whenever segments of Path 44 or the

SWPL are out of service.

Because none of the affected TOPs, beside s IID, considered IID’s 92 kV network in their next-day studies, they were not aware how their internal contingencies could

affect IID’s 92 kV network, or how an overload on IID’s 92 kV network could affect their systems. For example, APS does not routin ely study IID’s lower voltage facilities, including the CV and Ramon transformers, in the day-ahead timeframe. APS uses seasonal studies and its operations manual as its tools in the day-ahead timeframe. While the model used for the seasonal studies physically has IID’s 92 kV network represented, neither the model nor the operations manual are used to consider the next day’s specific configuration and operation, su ch as transmission or generation outages external to APS’s footprint that were not an ticipated at the time the seasonal study and

68 The agreement does address market concerns by requiring entities who participate in data

‐ sharing to respect

the separation of market and operations functions.



## - 70 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

manual were updated. As a result, APS was not able to predict what occurred on IID’s system—increased flows and overloading on its 92 and 161 kV transformers and transmission lines—when H-NG tripped offline. Similarly, affected TOPs other than IID do not consider in their day-ahead pla nning how the loss of the CV and Ramon transformers, leading to the S Line RAS operation, could adversely affect their internal systems. Accordingly, TOPs should revise th eir next-day study practices to account for all facilities, including those operated belo w 100 kV, that impact BPS reliability.

WECC RC also did not adequately consider sub-100 kV facilities not identified as BES that can have significant impacts on BP S reliability. While WECC RC does model IID’s CV transformers in its next-day studies, prior to September 8, 2011, it did not “flag”

them in its studies fo r active monitoring. 69 

### This means that WECC RC had data showing

that the transformers would overload under certain conditions, but the overloads were not identified by alarms to be seen by RC operators. WECC RC did not actively monitor the CV transformers in its next-day studies because they are below 100 kV and IID had not alerted WECC RC to any issues that woul d warrant monitoring of the transformers. Given the CV transformers’ impact on BPS re liability, WECC RC should actively monitor

these transformers. 70



### Finding 4 Flawed Process for Estimati ng Scheduled Interchanges:

 WECC RC’s process for estimating schedu led interchanges is not adequate to ensure that such values are accurately re flected in its next-day studies. As a result, its next-day studies may not a ccurately predict actual power flows and contingency overloads.



### Recommendation 4:

 WECC RC should improve its process for predicting interchanges in the dayahead timeframe.



### Interchanges are energy transfers that cross BA Areas. Interchanges can affect flows across transmission systems, so forecast ing accurate interchanges is important in the day-ahead timeframe to plan for potential overloading. WECC RC’s process for estimating scheduled interchanges is not adequate to ensure that the scheduled interchanges incorporated into its next-day studies are accurate. Under this process, by 10:00 AM each day BAs provide WECC RC with all interchanges they have approved for

69 To aid in effectively and efficiently processing and analyzing reliability data for the entire Western Interconnection, WECC RC has the option of flagging a subset of facilities for active monitoring in its studies. It

has since updated this feature to flag the CV transformers for monitoring.

70 WECC RC has implemented new procedures since September 8, 2011, to monitor RTCA results for the CV

transformers.



## - 71 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

the next day. The BAs typically submit this information once per day without any subsequent updates. WECC RC then vali dates these scheduled interchanges by comparing the values with what the BAs pr ovided the prior day and with what WECC RC’s state estimator observed in the prior days and weeks.

The accuracy of interchange data in WECC RC’s next-day studies could be improved by allowing for updates closer to real time. BAs’ interchange data are likely to change after their 10:00 AM submittal to WECC RC. Some BAs have automated systems, which send updates of interchange data to WECC RC. Most BAs submit the data manually, only once at 10:00 AM. Incl usion of a process or requirement for BAs to update their scheduled interchanges after th eir 10:00 AM submission would increase the

likelihood of accura te interchange data.

The accuracy of interchange data af fected WECC RC’s next-day study for September 8, 2011. Specifically, the sche duled interchanges reflected in WECC RC’s next-day study for September 8, 2011, were no t sufficiently accurate to predict that IID’s

CV 230/92 kV transformers would overload to their trip point upon the loss of H-NG. After the event, WECC RC ran its next-day study using actual interchanges, and found that the CV transformers would overload beyo nd their tripping thre shold upon the loss

of H-NG. If WECC RC had used more accura te net interchange data and flagged the CV transformers for monitoring, it could have le arned of the issues with these transformers

and alerted IID or issued directives for contro l actions to mitigate th e situation, such as increasing generation or shedding load.



## Seasonal Planning

 Background

Following a set of disturbances in the Western Interconnection during the

summer of 1996, WECC establis hed a new seasonal planning structure design ed to avert system-wide disturbances while maximizing the commercial avai lability of transmission capacity. This new structure involved the cr eation of the Operating Transfer Capability Policy Committee (OTCPC). The purpose of the OTCPC was to provide coordinated standard development and determination of seasonal Operating Transfer Capabilities (OTCs), or Operating Transfer Limits, 71 

### within the Western Interconnection. 72

71 OTCs are now known as SOLs.

72 The OTCPC itself was abolished and replaced with a new structure in June 2011; however, planning for the

seasonal period in which the blackout occurred was performed under the OTCPC structure, so the inquiry’s analysis focused on the OTCPC structure.



## - 72 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Among other things, the OTCPC was design ed to be responsible for determining

which transmission Paths should be studie d, facilitating OTC dispute resolution, ensuring that seasonal stud ies maintain consistent stan dards and methodologies, and approving seasonal studies of OTC limits. To that end, the OTCPC was charged with reviewing and approving study plans and te chnical simulation results; developing policies and procedures addressing seasonal OTCs; establishing working groups such as subregional study groups and the Operating Procedures Review Group; addressing OTC seams issues between subregions; an d providing technical guidance.

The seasonal study plans that are reviewed and approved by the OTCPC were created by a set of four subregional study groups (sometimes referred to as SRSGs or simply subregions). There were four gr oups: (1) the California/Mexico Operations Study Subcommittee (OSS); (2) the Northwest Operational Planning Study Group (NOPSG); (3) the Rocky Mountain Subreg ional Study Group (RMSG); and (4) the Southwest Area Study Group (SASG). The affected entities were members of two of these groups: the OSS (CAISO, SDG&E, SCE, CFE, and IID) and the SASG (APS, WALC).

On an annual basis, each subregional study group reviewed the Paths in its subregion to determine which Paths should be studied and the system conditions under which they should be studied. Then, season ally, the four subregio nal study group chairs

submitted their recommendations of which Pa ths to study to the OTCPC for review and approval. Following OTCPC’s approval, the st udies were performed in accordance with the OTC study process. This process began with establishment of an initial “base case” by WECC staff, with input from representati ves of each subregional group. The “base case” is a computer model of projected or star ting power system conditions for a specific

point in time. For the 2010-2011 planni ng year, five base cases were used. 73 

### Once the comments from the four subregional representatives were incorporated, the final cases

were made available via WECC’s web site for adjustment and modification by subregional members in order to forecast expe cted seasonal conditions on the system. The subregional members performed their own se asonal studies, and then met to discuss the results. A subregional seasonal planning case was produced on this basis, but no further studies were performed. Subregional seasonal cases were shared among the four subregions via liaisons from the other subr egions. No comprehensive WECC-wide Path rating study was prepared on the basis of the four subregional studies.

73 These included low summer load, high summer load, low winter load, high winter load, and high spring load.



## - 73 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

In addition to, and apart from, the seas onal planning studies just described, TOPs also conduct their own seasonal studies focusing on their own internal networks. These internal studies follow a different process from the seasonal Path rating studies, though both begin with the WECC base case. Internal seasonal studies, however, are not aggregated or reviewed at the subregional level. Instead, TOPs generally replace the

information from the WECC base case with mo re accurate and granular detail for their own areas only. Once updated, the TOPs perform contingency analyses for their own internal purposes. They then share with their neighbors the results of these operational studies, which typically contain only the default data from the WECC base case for everything outside of their own areas.

The inquiry identified a number of issues relating to both types of seasonal planning by the affected entities. These issues impaired the accuracy and effectiveness of the seasonal studies by excluding, in variou s ways, pertinent issues and information that

should have been taken into consideration.

Finding 5 Lack of Coordination in Seasonal Planning Process:

 The seasonal planning process in the WECC region lacks effective coordination. Specifically, the four WECC subregions do not adequately integrate and coordinate studies across the subregions, and no single entity is responsible for ensuring a thorough seasonal planning process. Instead of conducting a full contingency analysis ba sed on all of the subregions’ studies, the subregions rely on experience and engineering judgment in choosing which contingencies to discuss. As a result, individual TOPs may not identify contingencies in one subregion that may affect TOPs in the same or another subregion.



### Recommendation 5:

 WECC RE should ensure better integration and coordination of the various subregions’ seasonal studies for the entire WECC system. To ensure a thorough seasonal planning process, at a minimum, WECC RE should require a full contingency analysis of the entire WECC system, using one integrated seasonal study, and should identify and eliminate gaps between subregional studies. Individual TOPs should also conduct a full contingency analysis to identify contingencies outs ide their own systems that can impact the reliability of the BPS within their system and should share their seasonal studies with TOPs shown to affect or be affected by their contingencies.



### No comprehensive WECC-wide seasonal stud ies are performed. With respect to seasonal Path rating studies, a representative or leader from each subregion adapts the WECC base case on the basis of input from subregional members, and then makes these

revised cases available to the other subregional members for review, comment, and approval. The subregional leader then conducts the seasonal studies concentrating only on the rated Paths in the subregion. The results of the seasonal Path rating studies are



## - 74 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

shared and discussed first among the subreg ion’s members, and then with the other subregions, but neither WECC RE nor the OTCPC performs or mandates any further

seasonal studies, and no new WECC-wide se asonal study is performed to reflect the input of all of the subregions. Instead, representatives of the subregional groups gather informally to discuss the results of their se asonal studies and rely on experience and engineering judgment to identify and resolve any issues.

The events of September 8, 2011, illustrate that this process is not adequate: the tripping of one line in a rated Path—H-NG, which is part of Path 49—ultimately led to the tripping of other lines in other rated Pa ths, including Paths 44 and 45. Focusing exclusively on Path ratings—and solely on a subregional basis—ignores network facilities that can impact rated Paths (and vice-versa) and does not account for the interrelationships of Paths and other fa cilities across WECC’s subregions.

With respect to the internal seasonal st udies, there is even less coordination. TOPs generally perform internal seasonal stud ies using models that include detailed data for their own system, but defaul t to WECC base case data, which may not be sufficiently detailed or updated, for everything else. TOPs perform contingency analysis for their own internal areas using this model. No study is done to identify the impact of external contingencies on the TOP’s system, or the impa ct of the TOP’s internal contingencies on the SOLs of other TOPs. TOPs provide the re sults of their internal seasonal studies to neighboring TOPs for informat ional purposes, after which those TOPs may or may not provide comments.

In all, this situation indicates that the TOPs’ internal seasonal planning studies are too heavily reliant upon the assumptions underlying an d reflected in a single WECC base case, and do not consider and study impacts of variations from that base case.

The September 8th event demonstrated one example where better integration of seasonal studies across two subregions is need ed. When H-NG (part of Path 49) tripped, approximately 12% of the flow from that line, which is loca ted in the SASG subregion,

was transferred across IID’s 230/92kV transf ormers, via the IID 92kV local network to the southern IID 161 kV network, which are all in the OSS subregion. This additional flow on IID’s CV transformers ultimately resulted in cascading outages and impacted Paths 44 and 45. The affected entities were unaware of this potential inter-Path impact, because the SASG and OSS studies had not been jointly considered. Moreover, since the subregional studies concentrate only on Pa th ratings, this flow transfer was not apparent. If the seasonal studies of SASG and OSS had been better coordinated and more rigorously analyzed, the potential for the loss of H-NG to overload IID’s 92 kV

network could have been identified and mitigation plans developed.



## - 75 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Finding 6 External and Lower-Voltage Facilities Not Adequately Considered in Seasonal Planning Process:

 Seasonal planning studies do not adequately consider all facilities that may affect BPS reliability, including external facilities and lower-voltage facilities.



### Recommendation 6:

 TOPs should expand the focus of thei r seasonal planning to include external facilities and internal and external sub-100 kV facilities that impact BPS reliability.



### As noted above, TOPs performing subr egional Path rating studies do not sufficiently account for the impact of facilities external to their subregion, or facilities within their subregion that are not part of a rated Path. Moreover, no WECC-wide Path rating study is performed to harmonize and analyze the impact of one subregion on the rest of the subregions.

The problem with this approach is illustrated in the example cited above: The tripping of a part of one rated Path, H-NG, whic h is part of Path 49, led to the tripping of portions of other rated Paths. The mechan ism whereby these other trips were triggered was the transfer of flow across low-voltage (bel ow 100 kV) facilities that were located in a different subregion. Under the approach to Pa th rating studies in place at the time, it would have been impossible for WECC RE or TOPs to anticipate and study this possibility, because it occurred across subregions, indirectly, via lower-voltage facilities. Even if seasonal Path rating studies had been performed ac ross subregions, these studies would not have anticipated this possibility, unless they also took into account lowervoltage facilities, which they presently do not.

The internal seasonal planning studies of the various TOPs are subject to similar omissions, although these studies encompass more than just the ra ted Paths and contain more detail than the Path rating studies. The practices of individual TOPs differ, but none contains sufficient detail and accuracy with respect to facilities outside their own footprints, as well as lower-voltage facilities . IID, for example, has explained that it “does not identify or study components outside of the II D territory below 100 kV for impacts on the BPS reliability in its territory,” nor does it “identify or study components inside of the IID territory below 100 kV for impacts on the BPS reliability outside of its territory.”

Similarly, while CAISO studies in its seasonal planning process “all of the transmission components that it operates, so me of which are below 100 kV,” it has also



## - 76 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

acknowledged that it “does not have the necessary information to accurately study transmission components below 100 kV outside of its territory to determine if they have an impact on the BPS reliability in [CAISO’s] service territory.”

The events of September 8, 2011, demonstrate that sub-100 kV facilities in parallel with BPS systems can have a significant effect on BPS reliability. The loss of HNG caused the overloading and tripping of bo th 230/92 kV transformers at CV, which in

turn caused another sub-100 kV transformer to trip at Ramon, which led to the cascading outages discussed in detail above. This possibility was not studied as part of the seasonal studies by any of the TOPs, other than IID, because the CV transformers’ secondary windings are below 100 kV. The seasonal studies conducted by affected TOPs, other than IID, did not study the impact of the CV transformers. If the CV transformer contingency overloads had been identified as limiting elements in the seasonal plans, the cascading outages might have been avoided or lessened by having pre-contingency mitigation in place, such as increasing generation on IID’s 92 kV system.

Finding 7 Failure to Study Multiple Load Levels:

 TOPs do not always run their individual seasonal planning studies based on the multiple WECC base cases (heavy and light load summer, heavy and light load winter, and heavy spring), but, instead, may focus on only one load level. As a result, contingencies that occur during the shoulder seasons (or other load levels not studied) might be missed.



### Recommendation 7:

 TOPs should expand the cases on whic h they run their individual planning studies to include multiple base cases, as well as generation maintenance outages and dispatch scenarios during high load shoulder periods.



### WECC created five base cases for the 20 10-2011 season— heavy and light load summer, heavy and light load winter, an d heavy spring—intended to capture the spectrum of possible loading configurations at different times of the year. The inquiry

found that some of the affected TOPs deemed it unnecessary to run individual planning studies based on the multiple WECC base case s. Instead, these TOPs identified some subset of these base cases that they conclude d were most relevant to their concerns and ran studies based on only that subset of base cases. Some TOPs employed only one base case—the heavy load summer base case—for planning the season during which the September 8, 2011 blackout occurred. By limi ting the run of planning studies to a small

subset of base cases, TOPs restrict th eir ability to anticipate and respond to contingencies arising in the context of load leve ls that vary significantly from those in the subset of base cases upon which their studies were predicated.



## - 77 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

As noted above, September 8, 2011 was a very hot day in the region, and scheduled flows in the IID footprint were ne ar record peaks. The high demand on September 8th was indeed similar to what would have been modeled in a heavy load summer seasonal study. The generation pi cture, however, was very different. By September 8, 2011 ge neration maintenance—which is not typically scheduled for summer peak days—had begun. The “heavy peak” summer study base cases that were actually used for September 8 th



### therefore had built into them the incorrect assumption

that there would be minimal maintenance—i.e. , that most generation would be on line—

and thus did not account for the normal re sumption of facility maintenance in the

shoulder season.

If IID’s seasonal studies had assumed ev en a modest decrease in the available generation, they might have enabled IID to anticipate and prevent the events that occurred on its system. IID was unaware of the TDM maintenance outages, but if it had

conducted a shoulder season st udy, it might have been operating in a mode that more accurately reflected actual op erating conditions on that day and could have potentially avoided the overloading of CV transformers to the tripping point. This lack of awareness illustrates the risks of not separately modeli ng the shoulder months such as September, when facility maintenance has begun but demand could remain or become very high. During these times, generation to serve load may come from other areas, changing flow patterns from those that typically occur on a normal summer peak day in which most generation is on line.

Finding 8 Not Sharing Overload Relay Trip Settings:

 In the seasonal planning process, at least one TOP did not share with neighboring TOPs overload relay tr ip settings on transformers and transmission lines that impacted external BPS systems.



### Recommendation 8:

 TOPs should include in the information they share during the seasonal planning process the overload relay trip settings on transformers and transmission lines that impact the BPS, and separately identify those that have overload trip settings below 150% of their normal rating, or below 115% of the highest emergency rating, whiche ver of these two values is greater.



### As discussed in greater detail below, the relay trip settings of IID’s CV 230/92 kV transformers were set very low, just abov e the facilities’ emergency rating. These settings effectively meant that IID’s system op erators had very little time to respond to the overload resulting from the loss of H-NG beyond emergency ratings and could not rely on post-contingency mitigation. If IID’ s neighbors had been aware of the relay trip



## - 78 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

settings on these transformers when preparin g their seasonal studies, they would have been able to plan for the possibi lity of the CV transformers tripping at a lower trip point.

As a general matter, TOPs should be aware of the relay trip settings of facilities in neighboring areas that have the potential to impact portions of the BPS within their own areas, regardless of whether or not those facili ties have been defined as, or deemed to be,

BES facilities. This concern is particularly acute where the overload trip points of the facility in question are set below 150% of their normal rating, or below 115% of their emergency rating, because, as discussed below, such settings sharply limit the amount of time available for operators to implement po st-contingency mitigation measures. These settings require that all entities that could be affected are aware and able to implement

pre-contingency mitigation.



## Near

‐ and Long

‐ Term Planning

 Background

TPs and PCs conduct near- and long-ter m studies to ensure their systems are planned for reliable operatio n under normal operating conditions. In addition, the system facilities must remain stable in the event of single and multiple contingency scenarios. Near-term studies consider potent ial contingencies one to five years past the study date, and long-term studies consider po tential contingencies si x to ten years past the study date. The near- and long-term planning process in the WECC region involves a coordinated effort among individual TPs and PC s at the local level, Subregional Planning

Groups (SPGs) 74 

### at the regional level, and WECC RE at the Interconnection-wide level. It is a multi-step process, performed annually.

First, TPs and PCs submit data about thei r internal networks to their respective SPG for each horizon year studied (i.e., ye ars one through ten). These data include forecasted load levels and facilities projected to be in or out of service. Also, these data assume peak load conditions and, thus, reflects that most internal generation is online. Second, SPGs add information to these data based on their broad knowledge of planning projects and reliability issues within their respective regi ons. For example, an SPG

74 There are five SPGs in the WECC region, each representing a specific area and composed of various members

and stakeholders, including individual owners and operators of transmission networks, representatives of local government agencies, and independent developers. SPGs allow for the joint consideration of issues among individual members. APS, IID, and WALC are members of We stConnect, which performs the SPG function in the

Southwest region. SDG&E and SCE are members of CAISO, which performs the SPG function in parts of California.

The SPGs are involved in near ‐ and long

‐ term planning only and are unrelated to the SRSGs, discussed above,

which deal with season al planning.



## - 79 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

might add data for a particular horizon ye ar based on its knowledge of a merchant generator’s desire to connect to the grid. SPGs also consider future projects needed for reliability and the effect of environmental regulations on the future operation of generator units. Third, SPGs merge all of thei r members’ cases to create a regional case. Fourth, WECC RE merges the various regional ca ses from all the SPGs to create the base case for each horizon year. WECC RE makes th ese cases available on its website for TPs, PCs, and SPGs to access. Finally, TPs and PCs add their own subtransmission facilities to the base cases to run their near- and long-term studies. TPs and PCs typically choose a list of contingencies to study based on past experience and engineering judgment.

As discussed below, this mu lti-step process has severa l shortcomings, which left the affected entities unprepared for the September 8th event.

Finding 9 Gaps in Near- and Long-Term Planning Process:

 Gaps exist in WECC RE’s, TPs’ and PCs’ processes for conducting near- and long-term planning studies, resulting in a lack of consideration for: (1) critical system conditions; (2) the impa ct of elements operated at less than 100 kV on BPS reliability; and (3) the interaction of protection systems, including RASs. As a consequence, th e affected entities did not identify during the planning process that the lo ss of a single 500 kV transmission line could potentially cause cascading outages. Planning studies conducted between 2006 and 2011 should have id entified the critical conditions that existed on September 8th and proposed appropriate mitigation strategies.



### Recommendation 9:

 WECC RE should take actions to mitigate these and any other identified gaps in the procedures for conducting near- and long-term planning studies. The September 8th event and other major events should be used to identify shortcomings when developing valid ca ses over the planning horizon and to identify flaws in the existing planni ng structure. WECC RE should then propose changes to improve the perf ormance of planning studies on a subregional- and Interconnection-wide basis and ensure a coordinated review of TPs’ and PCs’ studies. TOPs, TPs and PCs should develop study cases that cover critical system cond itions over the planning horizon; consider the benefits and potential adverse effects of all protection systems, including RASs, Safety Nets (such as the SONGS separation scheme), and overload protection schemes; study the interaction of RASs and Safety Nets; and consider the impact of elements op erated at less than 100 kV on BPS reliability.



### The affected entities’ near- and long-term planning studies for horizon year 2011 (i.e., the studies conducted in 2001 throug h 2010) did not identify that the loss of a single 500 kV line in APS’s territory would cause cascading outages across the territories of SDG&E, CFE, IID, and WALC. Several gaps in the near- and long-term planning process contributed to these omissions. Firs t, TPs and PCs submit peak load data to



## - 80 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

WECC for incorporation into the base case and, thus, the data assume that most internal generation is online to meet peak conditio ns. As a result, the models for 2011 did not contain accurate, realistic representations of online generation. Running studies under the assumption that most generation is on line provided an unrealistic portrayal of system transfers on the day of the event.

Indeed, system transfers following the loss of H-NG were higher than the transfers seen in the base case used for near- and long-term studies. Significant flows

from H-NG transferred across IID’s and WALC ’s systems and onto Path 44. Flow on Path 44 increased by approximately 84% following the loss of the line. These large system transfers went undetected in near- and long-term studies, and the affected entities were not alerted to the need to plan for these critical system conditions. To avoid this problem in the future, TPs and PCs should study more generation dispatch scenarios to provide a more realistic projection of system transfers following contingencies.

Second, TPs and PCs do not run a full list of external contingencies during the near- and long-term planning process. Instead, they rely on experience and engineering judgment, focusing on previously identified contingencies. This can be particularly problematic in today’s operating environment in which the nature and limitations of the system are rapidly changing. For example, as part of its near- and long-term planning IID studied potential contingencies on four WECC Rated Paths, but did not study the

loss of H-NG. As a result, II D was not prepared for the effe ct on its system when that line tripped. Also, while IID’s CV 230/92 kV transformers are included in the base case,

some of the affected TPs and PCs did not stud y the potential loss of these facilities. By not considering a complete list of external contingencies that coul d impact their systems, TPs’ and PCs’ studies for horizon year 2011 we re not sufficient to identify and plan for the impact of external contingencies on thei r internal systems or internal contingencies on neighboring systems.

Third, TPs and PCs do not study external subtransmission facilities in the nearand long-term planning proce ss. Individual TPs and PCs add their own subtransmission

facilities after the base case has been cr eated by WECC RE, but do not add external subtransmission equipment. If external su btransmission systems were included in the base case, entities could identify the parallel flow on such lower-voltage systems that can result from transmission contingency outage s. This consideration is particularly important for lower voltage systems that para llel external high voltage systems. For example, when APS’s H-NG tripped, approximately 12% of its flow transferred to IID’s 92 kV system. This increased flow and overloading on IID’s system had a ripple effect, causing cascading outages throughout neighb oring territories. Because the affected entities did not study extern al subtransmission systems in their near- and long-term



## - 81 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

studies, they did not identify the potential for overloading on IID’s 92 kV system or the impact on their systems from this overloading.

Fourth, TPs and PCs do not sufficiently study the interaction of protection systems in external networks in their near - and long-term planning studies. For example, some of the affected TPs and PC s did not study the interaction between the overload protection on IID’s three 230/92 kV transformers, or betw een the protection on these transformers and the S Line RAS. Based on the pre-event conditions, the loss of one CV transformer would automatically re sult in the loss of the second, followed automatically by the loss of th e Ramon transformer, which in turn, would result in either voltage collapse and load shedding, or overloading on the S Line. The S Line RAS is designed to mitigate overloads by tripping generation in Mexico that supplies power to IID. However, operation in this manner only served to further overload IID and WALC facilities and exacerbate system conditions on the day of the ev ent. The affected entities should have studied the interaction of these schemes to prepare for the impacts on their

systems.

Finding 10 Benchmarking WECC Dynamic Models:

 The inquiry obtained a very good correlation between the simulations and the actual event until the SONGS sepa ration scheme activated. After activation of the scheme, however, neither the tripping of the SONGS units nor the system collapse of SDG&E and CFE could be detected using WECC dynamic models because some of the elem ents of the event are not explicitly included in those models. Sample simulations of the islanded region showed that by adding known details from the actual event, including UFLS programs and automatic capacitor switching, the simulation and event become more closely aligned following activation of the SONGS separation scheme.



### Recommendation 10:

 WECC dynamic models should be benchm arked by TPs against actual data from the September 8th event to improv e their conformity to actual system performance. In particular, improv ements to model performance from validation would be helpful in analy sis of under and/or over frequency events in the Western Interconnection and the stability of islanding scenarios in the SDG&E and CFE areas.



### The inquiry simulated the dynamic system response of the September 8th event from prior to the loss of H-NG through the separation of Path 44 and the unsuccessful islanding of SDG&E and CFE. The team ob tained very good correlation between the

simulation model and the actual event until the SONGS separation scheme activated. However, neither the tripping of the SONGS units nor the system collapse of SDG&E and



## - 82 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

CFE could be predicted using existing WECC dynamic models entities use to perform near- and long-term planning.

This inability to use the existing system models to reproduce the actual event is also evident in the post-event analysis that was prepared by SDG&E on the effectiveness

of UFLS programs following the September 8th event. 75 

### The SDG&E post-event analysis shows that the UFLS performance should have prevented the SDG&E syste from frequency collapse, similar to the “as is” results shown in Figure 14 , below. However, the SDG&E analysis does not expl ain why the simulation results are so different than the actual system responses—i. e., successful islanding operation versus m

system collapse.

F igure 14: Actual and Simulated Frequency at Miguel 500 kV Bus



### ity studies and adding details from the actual event, including UF LS performance, PMU The inquiry’s Modeling and Simulation te am was able to obtain a simulation more closely aligned with actual measured pe rformance by performing several sensitiv

75 Preliminary Analysis of SDG&E Off

‐ Nominal UFLS Program Effectiveness Following September 8, 2011 Pacific

Southwest Event, Performed by SDG&E, December 7, 2011.



## - 83 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

data, and generation tripped in CFE’s an d SDG&E’s territories. For example, one sensitivity study (referred to here as “Test 3”) simulated approximately:

a) 3,080 MW of UFLS in SDG&E 1.3 seconds afte r Path 44 tripped (compared to 2,760 MW in “as-is” case) b) 520 MW of UFLS in CFE after Path 44 tripped, but prior to SDG&E separation from CFE/APS (compared to 900 MW mo deled in “as-is” case) c) 589 MW of generation tripped in CFE after Path 44 tripped, but prior to SDG&E separation from CFE/APS (compared to zero in “as-is” case) d) 1,000 MW of generation trip ped in SDG&E immediately af ter SDG&E separated from CFE/APS (compared to zero in “as-is” case)



### Figure 15 , below, shows results of “Test 3.” As can be seen, this simulation more closely follows the actual event than the “as-is” model used in Figure 14.

Figure 15: Miguel Frequency Actual and Simulated for “Test 3”



### The simulation studies explain the ineffe ctiveness of the UFLS program, despite up to 75% of SDG&E load that was shed wi thin 1.3 seconds of the SONGS separation scheme operating. The simulation analysis confirmed findings in the inquiry’s SOE that the frequency collapse was ca used by generation trips an d UFLS misoperations within CFE shortly after Path 44’s separation, follo wed by additional generation trips within SDG&E around the time it separated from CFE/APS.



## - 84 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# B. Situational Awareness

Background

TOPs, BAs, and RCs have system operators who constantly monitor their

networks to maintain situat ional awareness of system co nditions, identify potential system disturbances, and institute mitigating measures, as necessary. The affected entities utilize a range of tools to perform these functions. All of the entities use SCADA systems as their main monitoring tool. SC ADA systems typically consist of a central computer that receives information from various RTUs and intelligent electronic devices (IEDs), located throughout the system. SC ADA systems provide control center operators with real-time measurements of system cond itions and can send alarms to signal a problem.

Most of the affected entities also use se veral other tools to study and analyze the information received from their SCADA system s. Two of the most important tools are State Estimator and RTCA. State Estimator gathers the available measurements from the SCADA system and calculat es estimated real-time valu es for the whole system.

RTCA then takes the information from State Estimator and studies “what if” scenarios. For example, RTCA determines the potential effects of losing a specific facility, such as a generator, transmission line, or transformer, on the rest of the system. In addition to studying the effects of various contingencies, RTCA can prioritize contingencies. It can also provide mitigating actions and send alar ms (visual and/or audible) to operators to alert them to potential contingencies.

While most of the affected entities have and use these tools, the inquiry identified several concerns with entities’ ability to ad equately monitor, identify, and plan for the next most critical contingency in real time . Several areas for improvement are described in the findings below.

PMUs did not play a role in observing th e September 8th event in real time, but may prove increasingly important in situational awareness. Of the affected entities,

CAISO, SCE, and APS are equipped with PMUs. PMUs are widely distributed throughout WECC as the result of a WECC- wide initiative known as the Western Interconnection Synchrophasor Program (WISP). Their high sampling speed (up to 30 samples per second) and excellent GPS-based time synchronization offer new granularity in information about voltage phase angles and other grid conditions. PMUs are expected to be used to identify and monitor for grid stress, grid robustness, dangerous oscillations, frequency instability, voltage instability, and reliabilit y margins. While not



## - 85 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

yet sufficiently integrated to have been used by the affected entities in their control rooms on September 8th, as discussed earlier, PMU data proved valuable in constructing the sequence of events and other post-event analysis.

Finding 11 Lack of Real-Time External Visibility:

 Affected TOPs have limited real-time visibility outside their systems, typically monitoring only one external bu s. As a result, they lack adequate situational awareness of external contingencies that could impact their systems. They also may not fully understand how internal contingencies could affect SOLs in their neighbors’ systems.



### Recommendation 11:

 TOPs should engage in more real-time data sharing to increase their visibility and situational awareness of external contingencies that could impact the reliability of their systems. They should obtain sufficient data to monitor significant external facilities in real time, especially those that are known to have a direct bearing on the reliability of their system, and properly assess the impact of internal contingencies on the SOLs of other TOPs. In addition, TOPs should review their real-time monitoring tools, such as State Estimator and RTCA, to ensure that such tools represent critical facilities needed for th e reliable operation of the BPS.



### Although all of the affected TOPs use SC ADA to monitor their own systems, some TOPs’ situational awareness is hindered by their limited visibility into neighboring systems. Some of the affected TOPs’ real-time external visibility is limited to one or two buses outside their systems. The September 8, 2011, even t demonstrated that more expansive visibility into neighboring systems is necessary for these TOPs to maintain

situational awareness of external conditions and contingencies that could impact their systems and internal conditions and continge ncies that could impact their neighbors’ systems. During the 11-minute time span of the September 8th event, entities observed changes in flows into their systems, but we re unable to understand the cause or significance of these changes and lacked suffici ent time to take corrective actions. If

affected entities had seen and run studies base d on real-time external conditions prior to the event, they could have been better prepared to redispatch generation or take other control actions and deal with the impacts when the event started.

IID, for example, is adjacent to APS, an d the changes in flows on APS’s system, especially on its 500 kV lines, can affect the flows on IID’s system and vice versa. Yet, IID’s visibility into APS’s system is limite d to information about the tie line between them. In fact, IID’s visibility into all of its neighbors is limited to one or two buses



## - 86 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

outside its system. 76 

### As a result, IID did not learn in real-time that H-NG tripped. IID also did not understand prior to the event how changes in flows or the loss of H-NG would affect its system. Immediately after H- NG tripped, IID observed loading on its CV transformers escalate rapidly, but it had not been prepared for this escalation.

If IID had greater visibility into APS’s system and IID had an equivalent on its

RTCA that modeled the external network usin g APS’s real-time data instead of pseudogenerators modeled at the end of each tie line, IID’s RT CA could have more accurately studied the results of a post-contingency loss of H-NG on its system before it occurred. After seeing the more accurate RTCA resu lts, IID could have initiated appropriate control actions before H-NG tripped. Also, having real-time status of the H-NG would have better prepared IID to deal with the effects of its loss in real time.

In addition to IID not havi ng adequate situational aw areness of APS’s system, the

affected TOPs and BAs external to IID were no t aware in real time of the effect of the post-contingency loss of IID’s three 230/92 kV transformers on their systems. Losses of the CV and Ramon transformers can cause SOL violations on neighboring systems. Indeed, on September 8th, these transformer outages had a significant ripple effect and led to the cascading nature of the event. Yet, entities outside IID’s footprint were not prepared for these outages and, except for WECC RC, were unaware of the outages in

real time because of a lack of adequate visibility into IID’ s system. For example, at the time of the event, CAISO’s visibility into IID’ s system stopped at the tie line into IID’s El

Centro station.

The September 8th event exposed the negative consequences of TOPs having limited external visibility into neighboring s ystems. Providing TOPs with the ability to observe and model external system conditions and events on a continuous real-time basis will allow them to study and plan fo r the impact of external conditions and contingencies before it is too late to react, as was the case on September 8th.

76 IID has made efforts, even before the September 8th event, to receive more data points from adjacent utilities

and is currently continuing this effort with all of its neighbors.



## - 87 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Finding 12 Inadequate Real-Time Tools:

 Affected TOPs’ real-time tools are not adequate or, in one case, operational to provide the situational awareness ne cessary to identify contingencies and reliably operate their systems.



### Recommendation 12:

 TOPs should take measures to ensure that their real-time tools are adequate, operational, and run frequently enou gh to provide their operators the situational awareness necessary to identify and plan for contingencies and reliably operate their systems.



### Although many of the affected TOPs have and use real-time tools such as State Estimator and RTCA, some of the tools are not adequate or operational to provide the situational awareness necessary to effectivel y monitor and operate th eir systems. Also, some TOPs run or view these tools infrequently, while others run RTCA, for example, every five minutes.

The alarming function on IID’s RTCA provid es an example of a real-time tool that does not adequately maximize situational aw areness capabilities. IID’s RTCA does not provide operators with any audible alarms or pop-up visual alerts when an overload is predicted to occur. Instead, IID’s RTCA uses color codes on a display that the operator must call up manually to learn of significant potential contingencies. For example, IID’s RTCA might show that on the next contingency, a specific element will become overloaded. However, as currently designed, the operator must go to the specific page

related to this element to view this result. The result will be color coded on this page, but

this code does not function as an alarm.

This design feature of IID’s RTCA had negative consequences on the day of the event. Forty-four minutes prior to the loss of H-NG, IID’s RTCA results showed that the N-1 contingency loss of the first CV transf ormer would result in overloading of the second CV transformer to its tripping point. If IID had taken action at this precontingency stage, it could have avoided the loss of both transforme rs. The IID operator, however, did not view the appropriate RTCA display and, therefore, was not alerted to the need to take action. If the operator had reviewed the RTCA results and taken necessary corrective actions, he could have re lieved loading on the transformers at this



## - 88 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

pre-event stage, and thus mitigated the severe effects on the CV transformers that resulted when H-NG tripped. 77



### One affected entity, APS, has State Esti mator and RTCA capability, but neither tool is operational. As a result, APS has limited capability to monitor and operate its system to withstand potential real-time cont ingencies. Instead of using RTCA, APS relies on a set of previously studied contingencies and pre-determined plans to mitigate them. These studies are included in a manual that is created annually and usually updated several times a year. 78 

### By relying on pre-determined studies, APS cannot account and prepare for all potential continge ncy scenarios in real time. RTCA would provide APS with a more realistic analysis of its next potential co ntingency because the RTCA analysis is based on real-time conditions, as meas ured by State Estimator. Without RTCA, APS operators are not fully prepared to identify and plan for the next most critical contingency on its system.

RTCA would have allowed APS operators to study the impact of the loss of its HNG. Although APS could have studied this contingency in its manual and seasonal

studies, it could not have studied it based on real-time operating conditions that only State Estimator can provide. For example, APS’s manual and seasonal studies did not study the loss of H-NG together with the mu ltiple generator outages that existed on the

day of the event. 79 

### As a result, APS was unprepared for the actual consequences of losing H-NG on September 8, 2011, includ ing overloads on IID’s 92 kV system and

potential difficulty reclosing H-NG due to large phase angle differences. 80

77 Since the event, IID has initiated changes to its RTCA program. First, it is working with a vendor to install an audible alarm feature. Second, IID has instructed its operators to constantly leave the RTCA result display screen

on, rather than periodically calling it up.

78 APS can also ask WECC RC or an APS engineer for a current

‐ day study, but it usually relies on its manual for operations. APS also relies on WECC RC to notify it of any major post

‐ contingency issues detected by WECC RC’s

RTCA results, but WECC RC might not consistently and promptly notify indivi dual TOPs of all major issues.

79 APS has indicated that it has had difficulty obtaining generator outage information from other BAs due to

market and/or tariff concerns.

80 Prior to the event, APS had been working with a vendor to build its RTCA capability and, since the event; it has

accelerated its efforts to make RTCA operational.



## - 89 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Finding 13 Reliance on Post-Conting ency Mitigation Plans:

 One affected TOP operated in an unsecured N-1 state on September 8, 2011, when it relied on post-contingency mitigation plans for its internal contingencies and subsequent overload and tripping, while assuming there would be sufficient time to mitigate the contingencies. Post-contingency mitigation plans are not viable under all circumstances, such as when equipment trips on overload relay prot ection that preven ts operators from taking timely control actions. If this TOP had used pre-contingency measures on September 8th, such as dispatching additional generation, to mitigate first contingency emergency overloads for its internal contingencies, the cascading outages that were triggered by the loss of H-NG might have been avoided with the prevailing system conditions on September 8, 2011.



### Recommendation 13:

 TOPs should review existing operatin g processes and procedures to ensure that post-contingency mitigation plans reflect the time necessary to take mitigating actions, including control acti ons, to return the system to a secure N-1 state as soon as possible but no longer than 30 minutes following a single contingency. As part of this review, TO Ps should consider the effect of relays that automatically isolate facilities without providing operators sufficient time to take mitigating measures.



### Before September 8, 2011, II D consistently relied on po st-contingency mitigation plans, rather than proactively responding on a pre-contingency basis, for RTCA results showing that the N-1 loss of one CV transf ormer would result in overloading on the second CV transformer. Post-contingency plans can work to prevent a second contingency as long as operators have sufficient time to take mitigating actions. Postcontingency mitigation is not an appropriat e choice for the CV transformers, which are set to trip by overload protection relays with out allowing operators enough time to take mitigating actions. Specifically, the transformers’ overload protection scheme is set with a thin margin between the emergency rating an d the relay trip point. The normal rating of the transformers is 150 MVA, the emergency rating is 165 MVA, and the relay trip point is set at 190.5 MVA, or 127% of the normal rating. Thus, when the transformers reach their emergency rating, operators may have the mistaken belief that they have sufficient time to take mitigating actions, when , in fact, the operators will have very little time before the transformers will trip offlin e, because they will soon reach the relay trip setting. As shown below, pre-contingency mitigation measures are necessary when operators are faced with setti ngs that leave such little margin between the emergency rating and overload trip point.

On multiple days during the summer of 20 11, IID’s RTCA results showed that an N-1 contingency tripping of one of the CV tr ansformers would result in overloading on



## - 90 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

the second transformer. IID continued to operate in this state on multiple days without taking any pre-contingency mitigating acti ons. For example, IID did not dispatch additional generation on a pre-contingenc y basis to control the loading on one CV transformer to prevent overloading on the second CV transformer. There were potentially severe consequences of not taking pre-contingency actions. Specifically, IID’s next-day study for September 8t h detailed that the loss of both CV transformers would overload: (1) IID’s Ramon transformer to its trip point; and (2) the S Line, which, in turn, would cause the S Line RAS to trip gene ration in Mexico that supplies power to the Imperial Valley substation. In short, on multiple days in summer 2011, IID’s RTCA

results showed that the loss of one CV transformer would overload the second transformer, and IID’s next-day study reveal ed the cascading outages that would stem

from the loss of both transformers. Yet, II D did not institute pre-contingency mitigating measures, such as dispatching additional generation.

Instead, IID relied on post-contingency pl ans. On most days in summer 2011, the level of overloading on the CV transformers gave IID just enough time to successfully use a post-contingency mitigation plan to st art generation after the loss of the first

transformer to avoid the loss of the second tr ansformer. However, on at least two days observed by the inquiry, a post-contingency plan would not allow the operator enough time to implement necessary procedures to mi tigate the problem. On those two days, the loading on both CV transformers was high en ough that only pre-contingency mitigation measures could have prevented the loss of th e second transformer upon the loss of the first. On the first of those two days, IID was simply fortunate that the N-1 contingency

loss of the first transformer never occurred. The second of the two days was September

8, 2011.

Forty-four minutes prior to the loss of H-NG, IID’s RTCA results showed that the N-1 contingency loss of the first CV transf ormer would result in overloading of the second transformer to approximately 139% of its normal rating—leading to the loss of the transformer by relay action. If IID had taken action at this pre-contingency stage, IID might have been able to avoid the loss of both transformers. 81 

### After H-NG tripped, the relays took less than 40 seconds to trip both CV transformers. Operators had no time to mitigate the overloads before the transformers were removed from service.

81 The inquiry understands that the IID operator did not see these RTCA results and, thus, would not have known

of the need for pre

‐ contingency mitigating measures. There is no indication, however, that IID would have used

pre

‐ contingency measures regardless of the results. IID consistently relied on post

‐ contingency measures.



## - 91 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Finding 14 WECC RC Staffing Concerns:

 WECC RC staffs a total of four operators at any one time to meet the functional requirements of an RC, including continuous monitoring, conducting studies, and giving direct ives. The September 8th event raises concerns that WECC RC’s staffing is not adequate to respond to emergency conditions.



### Recommendation 14:

 WECC RC should evaluate the effectivene ss of its staffing level, training and tools. Based on the results of this evaluation, it should determine what actions are necessary to perform its fu nctions appropriately as the RC and address any identified deficiencies.



### WECC RC performs its reliability coordina tion functions through two offices. Although each office is capable of monitoring the entire Interconne ction, during normal operations the offices have primary responsibility for monitoring different parts of the

Western Interconnection. WECC RC’s Vanc ouver, Washington, office is primarily responsible for monitoring the Pacific Northwes t (excluding PacifiCorp East), California, and CFE’s territory in Mexico. WECC RC’s Loveland, Colorado, office is primarily responsible for monitoring the Desert Southwest area, Rocky Mountain area, PacifiCorp’s East area, Sierra Pacific Power Company’s area, IID’s area, and the Los Angeles intermountain area. Each office staffs two on-shift operators at all times. Each center dedicates an operator to the real-time desk (real-time operator) and the other operator to the study desk (study desk operator).

The real-time operator’s primary responsi bilities include monitoring limits and operating parameters, identifying exceedan ces, evaluating mitigation plans, and directing corrective actions. The study desk operator’s primary resp onsibilities include monitoring expected post-contingency condit ions to identify potential exceedances, evaluating actions being taken, and directing corrective action as necessary. The study desk operator also reviews WECC RC’s next-d ay study for accuracy , conducts real-time studies to evaluate system conditions, and mo nitors EMS applications, such as RTCA, to identify any performance issues and request co rrective actions, as necessary. The realtime operator and study desk operator also have some joint responsibilities, including reporting events that impact the BPS, iden tifying events or system conditions that require notification to adjacent RCs, and monitoring and testing primary and backup internal communication systems. Throug h these responsibilities, WECC RC is responsible for the reliable operation of the BPS in the WECC footprint, and it has theultimate authority to prevent or mitigate emergency operating situations in both nextday and real-time timeframes.



## - 92 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

In addition, WECC RC is responsible for providing information to the entities in its footprint, including the 53 TOPs and 37 BA s. Some of this information is provided over the telephone. During the event, in addition to performing the many RC functions they are responsible for performing, the RC operators had to answer phone calls providing or seeking information on the disturbance.

Given WECC RC’s responsibility and authority, four total oper ators—two in each regional office—might not be sufficient to e ffectively perform its fu nction, particularly during emergency conditions. Several examples from the September 8th event highlight

this concern.

First, after the loss of H-NG, many alar ms began sounding in WECC RC’s control rooms, as voltage dropped and facilities over loaded. With so many alarms sounding in an emergency situation, the real-time operat or had a difficult time prioritizing which alarms to monitor. WECC RC has eight unique categories, or “buckets,” of alarms within its EMS applications, grouped according to importance. Buckets 1 and 2 contain the highest priority alarms. Bucket 1 includes all 500 and 345 kV circuit breaker status changes, frequency and Path violations, status of generators greater than 50 MW and

associated circuit breakers, and critical bus voltages. Bucket 2 includes all 220/230 kV circuit breaker status changes and automatic voltage regulator status. 82 

### Buckets 3 through 8 include lesser priority items, such as RAS status changes, non-critical bus voltages, and circuit breaker status change s below 220 kV. Operators receive audible alarms for buckets 1 and 2 and typically leave bucket 1’s display on the screen constantly and use one other screen to display all othe r buckets. It is a constant process to continually monitor the alarms, even during normal operating conditions, and it might not be possible for one real-time operator to keep track of and prioritize multiple alarms sounding at once. Also, both operators had numerous phone calls to field from entities throughout the affected areas, reporting and requesting information. Overburdening the real-time operator in this way could undermin e his or her ability to perform the critical functions of monitoring system conditions an d directing necessary corrective actions. Accordingly, WECC RC should consider whet her additional operators are necessary to adequately perform these functions.

A second indication that the current RC staffing levels might not be sufficient came during the September 8th event when the study desk operator had to abandon his duties in order to provide support to the re al-time operator by fielding phone calls and

82 The CV 230/92 kV transformers are included in bucket 2.



## - 93 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

monitoring conditions. On this day, the RC operators were able to call for an engineer to conduct some studies. Because the September 8th event occurred during the afternoon, an engineer was available. Finding an engine er to substitute for th e study desk operator may not always be so easy. Late at night an d early in the morning, no engineers are on duty. That the study desk operator needed to leave his responsibilities to support the real-time operator may indicate that one real -time operator and one study desk operator per office might not be sufficient to fulfill WECC’s reliability coordination functions.

Alternatively, additional training and en hanced tools may enable an entity to accomplish more with the sa me number of personnel. While the inquiry observed a sampling of WECC RC’s tools to be adequate during its site vi sit, WECC RC is in the best position to identify the combination of addition al staff, enhanced tools, or training that best addresses the concerns identified by this report.

Finding 15 Failure to Notify WECC RC and Neighboring TOPs Upon Losing RTCA:

 On September 8, 2011, at least one affected TOP lost the ability to conduct RTCA more than 30 minutes prior to and throughout the course of the event due to the failure of its State Estimator to converge. The entity did not notify WECC RC or any of its neighboring TO Ps, preventing this entity from regaining situational awareness.



### Recommendation 15:

 TOPs should ensure procedures and trai ning are in place to notify WECC RC and neighboring TOPs and BAs promptly after losing RTCA capabilities.



### When entities temporarily lose their RTCA capability due to technical issues, they

become blind to the next most severe contin gency on their system, and they do not know what pre-contingency measures might be necessary. Thus, when they lose RTCA, they must take immediate action to try to regain their situational awareness. For example, after losing RTCA an entity should contact WECC RC, so the RC can monitor the entity’s system and inform it of any significant issues . In such instances, the RC should also notify neighboring entities of any major cont ingencies that could impact their systems.

Between 13:59 and the start of the event on September 8, 2011, WALC lost its RTCA when its State Estimator stopped solving. 83 As a result, WALC lost its ability to identify and study post-contingency violatio ns and to take pre-contingency mitigating measures, as necessary. When it lost it s RTCA, WALC should have contacted WECC RC

and asked it to monitor WALC’s area. WECC RC could have then notified WALC

83 By not solving, or converging, the State Estimator stopped providing estimated values for the system.



## - 94 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

regarding any significant problems and could ha ve also contacted WALC ’s neighbors if it

learned of any SOLs in WALC 

## that were impacting the neighbors’ systems

. 84 

### Prior to the event on September 8, 2011, WALC experienced several post-contingency SOL violations, but, without its RTCA capabili ty, remained unaware of them. WECC RC’s

RTCA results showed these violations. WALC , however, did not notify WECC RC when it

lost RTCA and, thus, WECC RC was unawar e that it should notify WALC of the violations. An entity should never be operating in an unknown state, as WALC was when

it lacked functional RTCA and State Estimator, and did not ask any other entity to assist

it with situational awareness.

Finding 16 Discrepancies Between RTCA and Planning Models:

 WECC’s model used by TOPs to conduct RTCA studies is not consistent with WECC’s planning model and produces conflicting solutions.



### Recommendation 16:

 WECC should ensure consistencies in model parameters between its planning model and its RTCA model and should review all model parameters on a consistent basis to make sure discrepancies do not occur.



### The usefulness of RTCA study results and other real-time studies depend on the

models used in the studies. Inaccurate models jeopardize the accuracy of studies, as well as entities’ ability to respond appropriately to potential contingenc ies identified by the studies. The inquiry’s simulation of the September 8th event discovered that a discrepancy exists between WECC RC’s mode l used to conduct RTCA studies and the model used for WECC’s planning studies. Specifically, the impedance of IID’s CV transformers differed by a factor of two between the WECC models. WECC’s planning model has an impedance of 0.1 per unit , while WECC RC’s RTCA model has an impedance of 0.05 per unit. This difference re sulted in an error of approximately 16% in the RTCA model compared to the planning model with resp ect to loading on the CV

transformers.

Although the inquiry did not perform a comprehensive comparison of all parameters in WECC’s various models, this discrepancy between the RTCA and planning models on such important facilities calls into question the validity of other parameters in

WECC’s models.

84 While not at issue in this event, the RC should also notify TOPs if it loses its RTCA, so that TOPs know that the

RC is not able to observe their systems.



## - 95 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



# I. System Analysis

Consideration of BES Equipment

 Background

The BES is generally defined as all facili ties operating at voltages above 100 kV, although certain sub-100 kV facilities with a significant impact on the BPS may be considered a part of the BES. Each RE currently determines its specific procedure for determining what is or is not BES. If a fa cility is not considered BES, relevant TOPs, BAs, and RCs may not study and mode l the impact of that facility.

Finding 17 Impact of Sub-100 kV Facilities on BPS Reliability:

 WECC RC and affected TOPs and BAs do not consistently recognize the adverse impact sub-100 kV facilities can have on BPS reliability. As a result, sub-100 kV facilities might not be designated as part of the BES, which can leave entities unable to address the reli ability impact they can have in the planning and operations time horizons . If, prior to September 8, 2011, certain sub-100 kV facilities had been designated as part of the BES and, as a result, were incorporated into the TOPs’ and RC’s planning and operations studies, or otherwise had been incorp orated into these studies, cascading outages may have been avoided on the day of the event.



### Recommendation 17:

 WECC, as the RE, should lead other entities, including TOPs and BAs, to ensure that all facilities that can adversely impact BPS reliability are either designated as part of the BES or othe rwise incorporated into planning and operations studies and actively moni tored and alarmed in RTCA systems.



### WECC RC, as well as TOPs and BAs impacted by the event, did not consider IID’s 92 kV network and facilities (including the CV and Ramon transformers) as BES

elements. IID did not reconsider whether the CV and Ramon transformers should be

studied like BES facilities even after a draf t study sponsored by CFE (and shared with IID) suggested the existence of a through-flow issue between the 500 kV substations at Devers and Imperial Valley, adversely impact ing IID’s 92 kV network (including the CV

and Ramon transformers) during continge ncies on BPS systems, including H-NG. 85



### Because the Reliability Standards apply to BES facilities, if the CV transformers had been considered BES facilities, IID would have been required to study the impact they could

85 See CFE’s Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19.



## - 96 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

have on BPS reliability. 86 

### Also, WECC RC and the affected TOPs would likely have included the facilities in their studies and been aware of the impact the loss of H-NG would have on IID’s 92 kV system, as well as the impact various trips within IID’s 92 kV system would have on the rest of the BPS. The inquiry determined that, during the event, approximately 12% (168 MW) of the original flow on H-NG was transferred through IID’s 92 kV system, making the 92 kV system part of a bulk power path as well as a significant looped transmission facility . The cascading outages that resulted from the loss of H-NG demonstrated the significan t potential for IID’s 92 kV system, including the CV transformers, to impact BPS reliability.



## IROL Derivations

 Background

In order to ensure the reliable operation of the BPS, entities are required to identify and plan for IROLs, which are SOLs that, if violated, ca n cause instability, uncontrolled separation , and cascading outages. Once an IROL is identified, system operators are then required to create plans to mitigate the impact of exceeding such a limit to maintain system reliability.

Finding 18 Failure to Establish Valid SOLs and Identify IROLs:

 The cascading nature of the event that led to uncontrolled separation of San Diego, IID, Yuma, and CFE indicates that an IROL was violated on September 8, 2011, even though WECC RC did not recognize any IROLs in existence on that day. In addition, the established SOL of 2,200 MW on Path 44 and 1,800 MW on H-NG are invalid for the present infrastructure, as demonstrated by the event.



### Recommendation 18.1:

 WECC RC should recognize that IROLs do exist on its system and, thus, should study IROLs in the day-ahead timeframe and monitor potential IROL exceedances in real-time.

86 See, e.g ., NERC Reliability Standard TOP

‐ 002

‐ 2b R11 (TOPs “shall perform seasonal, next

‐ day, and current

‐ day

Bulk Electric System studies to determine SOLs”).



## - 97 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Recommendation 18.2:

 WECC RC should work with TOPs to consider whether any SOLs in the Western Interconnection constitute IROLs. As part of this effort, WECC RC should: (1) work with affected TOPs to consider whether Path 44 and H-NG should be recognized as IROLs; and (2) validate existing SOLs, and ensure that they take into account all transmission and generation facilities and protection systems that impact BPS reliability.



### The NERC Glossary defines an IROL as an SOL that, if violated, could expose a widespread area of the BPS to instability, uncontrolled separation, or cascading outages that adversely impact the reli ability of the BPS. Each IROL is associated with a maximum time limit (Tv) that the IROL ca n be exceeded before the risk to the Interconnection or another RC area becomes gr eater than acceptable. The time limit can vary, but any IROL’s Tv must be less than or equal to 30 minutes. 87

For this event, the loss of H-NG should ha ve been associated with an IROL with a Tv for this N-1 contingency of essentially 0 minutes, because the cascading from the loss of H-NG began within seconds. However, neither WECC RC nor any of the affected entities have previously identified this IROL. The WECC region historically has maintained an operating philos ophy of not recognizing IROLs. 88 Instead, entities in the

W ECC region believe that as long as they operate within the conditions they have studied, they will not face the risk of IROL s and will not need to calculate IROLs. The September 8th event undermines this philosophy.

Prior to the event, the WECC system was su pplying loads in the various balancing authority areas in the range of 85-95% of their recorded peak loads. The power flows on all the Paths in the WECC region were belo w their maximum ratings and voltages were within acceptable levels. In particular, th e two major transmission corridors into the blackout area, namely Path 44 and H-NG, we re loaded respectively to 1,302 MW and

87 As defined by the NERC Glossary of Terms, an IROL’s Tv is “[t]he maximum time that an [IROL] can be violated

before the risk to the interconnection or other Reliability Coordinator Area(s) becomes greater than acceptable.

Each [IROL’s] Tv shall be less than or equal to 30 minutes.” NERC Glossary of Te rms, February 8, 2012, at 26.

88 As described by WECC in a February 16, 2012 Webinar on its SOL Methodology revision, “ The WECC operating philosophy is to operate only in conditions that have been studied. Therefore, under these normal operating conditions, there are never IROL conditions (only SOLs). An IROL condition may be created by the occurrence of

one or more unanticipated con tingencies. When this occurs, under WECC Reliability Standards, bulk electric system operators are required to resolve the IROL condition within 20 minutes (stability) or 30 minutes (thermal).” http://www.wecc.biz/awareness/Reliability/Documents /SOL Met hodology Presentation 02.16.2012.pdf (emphasis in original).



## - 98 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

1,372 MW. Compared to their maximum SOL ratings of 2,200 MW and 1,800 MW, these loadings represent 59% and 78% of their maximum ratings—well within current limits. Path 44 and H-NG ratings of 2,200 MW and 1,800 MW may be invalid for the present infrastructure because cascading outages due to a single contingency occurred at loadings well below the SOL ratings.

During the 11-minute disturbance, the single contingency of the sudden loss of HNG resulted in a series of cascading outages, with multiple elements exceeding their applicable ratings and leading to a widespread blackout of the area.

Accordingly, WECC RC should lead all relevant TOPs in the blackout area to study and report on the appropriateness of identifying Path 44 and H-NG as IROL paths. WECC RC should similarly assess transfer Paths outside this blackout area to ensure that there are no other similar reliability issues in the Western Interconnection. Existing operating processes and procedures should be reviewed to ensure corrective control capabilities are provided to system operators to enable them to return the system to a secure N-1 state as soon as possible, but no longer than 30 minutes following a single

contingency.

WECC RC has a proposed new SOL Method ology document (current effective date of June 4, 2012), which acknowledges the need to establish IROLs, and the RC’s

responsibility to monitor IROLs. 89 

### It recognizes that “Stability SOLs may qualify as IROLs depending on the potential consequences of exceeding the limit and the impact on BES reliability. WECC RC makes this dete rmination by collaborating with TOPs to understand the nature of the stability SOL, understanding the conditions that result in the establishment of the stability SOL, an d determining the BES impacts of exceeding

the stability SOL.” 90 

### WECC RC also has a proposed multi-step proce ss for determining whether thermal or voltage SOLs are IROLs. In general, WECC RC will look at whether potential IROLs cause “Widespread Adverse System Impacts,” or “potential cascading.” “Widespread Adverse System Impacts” is define d as “loading of three or more additional BES Facilities beyond 125% of their applic able emergency thermal Facility Rating, or

[t]hree or more additional BE S Facilities with bus voltages experiencing voltages less

than 90%.” 91 “Potential cascading” is defined as “when studies indicate that a

89 See WECC System Operating Limits Methodology for the Operations Horizon, Version 6.1, available at http://www.wecc.biz/awareness/Reliability/ WECC RC Operating Procedures/W ECC FAC 011

‐ EFFECTIVE DATE 6

‐ 4

‐

2012 SOL Methodology for the Operations Horizon.pdf

.

90 Id. at 5.

91 Id. at 6.



## - 99 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

contingency results in severe lo ading on a Facility, triggering a chain reaction of Facility disconnection by relay action, equipment failure, or forced immediate manual disconnection of the Facility (for example, public safety concerns, or no time for the operator to implement mitigation actions).” 92



## Impact of Protection Systems on Event

 Protection System Coordination

When an abnormal system condition is detected on the BPS, relay protection systems operate to isolate the problem while causing minimum distur bance to the power system. This requires the relay to be se lective in determining which elements to interrupt. The only method of obtaining th is selectivity is to perform coordination studies. The inquiry discovered that two TO s did not properly coordinate a protection system and a third TO implemented a prot ection scheme without performing any

coordination studies at all. This lack of coordination of protection systems resulted in circuits unnecessarily being interrupted, which had an undesirable effect on BPS reliability during the September 8th event.

Finding 19 Lack of Coordination of the S Line RAS:

 Several TOs and TOPs did not proper ly coordinate a RAS by: (1) not performing coordination studies with the overload protection schemes on the facilities that the S Line RAS is designed to protect; and (2) not assessing the impact of setting relays to trip generation sources and a 230 kV transmission tie line prior to the operation of a single 161/92 kV transformer’s overload protection. As a result, BES facilities were isolated in excess of those needed to maintain reli ability, with adverse impact on BPS reliability.



### Recommendation 19:

 The TOs and TOPs responsible for design and coordination of the S Line RAS should revisit its design basis and protection settings to ensure coordination with other protection systems in order to prevent adverse impact to the BPS, premature operation, and excessive isolation of facilities. TOs and TOPs should share any changes to the S Line RAS with TPs and PCs so that they can accurately reflect the S Line RAS when planning.



### Operation of the S Line RAS isolates facilities beyond what is necessary to ensure reliability. The S Line RAS is a directiona l overload scheme, located at the Imperial Valley substation, which is jointly owned by SDG&E and IID. The S Line RAS was originally implemen ted to protect the sole 230/161 kV transformer at El Centro from

92 Id.



## - 100 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

overloads due to increased flow on the S Line. 93 

### At the time, this was the only transfer point from the 230 kV line to the 161 kV sys tem, and subsequently the 92 kV system, in IID’s southern area. After implementing this RAS, IID has since installed a 230/92 kV transformer at El Centro, providing another path from the 230 kV system to the lower voltage networks.

IID’s current intention for the S Line RAS is to reduce loading on the S Line by tripping generation and, if insufficient to re duce flow, tripping the S Line at Imperial Valley Substation before transformer overload protection operates to trip the 161/92 kV transformer at El Centro. Tripping the S Line before allowing the El Centro 161/92 kV transformer’s overload protection to take acti on effectively results in the removal of the 230 kV source at the El Centro substation, which normally feeds a 230/92 kV

transformer and a 230/161 kV transformer. Thus, the design of the S Line RAS intentionally isolates networke d BES facilities to mitigate an overload on a non-BES facility (El Centro 161/92 kV transformer) to support reliability of the local system.

While this action alone does not constitute miscoordination, proper coordination of a RAS should take into account, through syst em studies, the potential impact on BPS reliability, including potentia l interaction with other RASs and protection systems.

During the September 8th event, the S Line RAS operated as designed, in that it tripped when it reached the settings that IID had prescribed. However, if one considers the purpose of the S Line RAS, which was to protect the El Centro transformer from overloads, the S Line RAS operated long before it was needed. At the time that the S Line RAS operated, the El Centro 161/92 transformer was only loaded to 38% of its normal rating, and its overload trip point is 178% of its normal rating. Thus, the El

Centro 161/92 transformer could have carried at least four times as much load before the transformer’s overload protection system wo uld have operated. Even though the El Centro transformer that the S Line RAS was designed to protect was nowhere near overloading, the S Line RAS tripped important generation and a 230 kV line. This calls into question the coordination of the S Line RAS with the transformer overload protection systems at El Centro.

IID provided SDG&E with the S Line RAS settings to implement. IID did not perform any studies to coordinate the S Line RAS with IID’s protection systems. SDG&E did some studies to verify that the RAS c oordinated with SDG&E’s protection systems.

There is no indication that the S Line RAS was coordinated with IID’s transformer overload protection at the El Centro station at which the S Line terminates. At a

93 The S Line RAS also serves as secondary protection for other IID facilities if a RAS on the Imperial Valley to Miguel 500 kV line fails to operate.



## - 101 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

minimum, IID, SDG&E and CAISO (as the TO P for SDG&E) should work together to ensure the proper coordina tion of the S Line RAS.

To make matters worse, during the September 8th event, San Diego was relying on generation at Imperial Valley from the south when the S Line RAS tripped that generation. Loss of the Imperial Valley gene ration caused San Diego to pull even more power from the north, increasing the loading on Path 44 and causing the SONGS separation scheme to further exceed its trip point. If not trippe d by the S Line RAS, generation at Imperial Valley could have helped SDG&E survive after the operation of the SONGS separation scheme. The inquiry’ s simulation showed that, had the S Line RAS tripped only the S Line without trippi ng the generation, the SONGS separation

scheme would not have operated, and only IID would have lost power. 94



### Finding 20 Lack of Coordination of the SONGS Separation Scheme:

 SCE did not coordinate the SONGS separa tion scheme with other protection systems, including protection and turb ine control systems on the two SONGS generators. As a result, SCE did not realize that Units 2 and 3 at SONGS would trip after operation of the separation scheme.



### Recommendation 20:

 SCE should ensure that the SONGS sepa ration scheme is coordinated with other protection schemes, such as th e generation protection and turbine control systems on the units at SONGS and UFLS schemes.



### SCE, the TO and TOP of the SONGS se paration scheme, did not perform any protection system coordination studies for the separation scheme it implemented at SONGS. The scheme is intended to isolate five 230 kV lines simultaneously if its preset value is exceeded for a sustained period. If SCE had coordinated the separation scheme with other protection and generation control systems at SONGS, it may have recognized the potential for the operation of the SONG S separation scheme to cause the SONGS generators to trip. Coordination in this context requires system studies to assess the impact of operation of the RAS on the powe r system, including potential interaction with other RASs and protection systems, such as UFLS schemes.

In addition to the consequences at SONGS itself, the lack of coordination of the systems means that, when the scheme operates , the system enters an unknown state. During the event, the operation of the protec tion scheme significantly contributed to the

94 See footnote 53.



## - 102 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

blackout of SDG&E, CFE, and Yuma—an e ffect neither coordinated nor adequately studied prior to the event. The inquiry’s simulation indicates that SDG&E, CFE and, Yuma would not have been blacked out if the SONGS separation scheme had not operated, with limited impact to the rest of the Western Interconnection.

Finding 21 Effect of SONGS Separation Scheme on SONGS Units:

 The SONGS units tripped due to their turbine control systems detecting unacceptable acceleration following op eration of the SONGS separation scheme.



### Recommendation 21:

 GOs and GOPs should evaluate the sens itivity of the acceleration control functions in turbine control systems to verify that transient perturbations or fault conditions in the transmission system resulting in unit acceleration will not result in unit trip without allowing time for protective devices to clear the fault on the transmission system.



### When the SONGS separation scheme operated, turbines at SONGS began to accelerate in excess of their control system se tting causing both units to trip offline. The tripping of the SONGS units in this manner raises questions about the sensitivity of the turbine control system’s settings. The units ar e expected to withstand severe faults on the transmission system and allow the tran smission protection systems to operate without the generators tripping offline. The coordination required for this protection is not a traditional relay-to-relay coordination; rather, the setting for the acceleration function should be coordinated with capabi lities of the turbine and with the system response anticipated following operation of transmission protection systems for faults under various system conditions. The setting should also be coordinated with the system response following operation of the SONGS se paration scheme. Had the turbine control system acceleration function been coordinated in this manner, the trip of the units may

have been avoided.



## - 103 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Pro tection System Studies

Finding 22 Lack of Review and Studying Impact of SPSs:

 Although WECC equates SPSs with RASs , prior to October 1, 2011, WECC’s definition of RAS excluded many protec tion systems that would be included within NERC’s definition of SPS. As a result, WECC did not review and assess all NERC-defined SPSs in its region, and WECC’s TOPs did not perform the required review and asse ssment of all NERC-defined SPSs in their areas.



### Recommendation 22:

 WECC RE, along with TOs, GOs, and Distribution Providers (DPs), should periodically review the purpose and impa ct of RASs, including Safety Nets and Local Area Protection Schemes, to ensure they are properly classified, are still necessary, serve their intended purposes, are coordinated properly with other protection systems, and do not have unintended consequences on reliability. WECC RE and the approp riate TOPs should promptly conduct these reviews for the SONGS separation scheme and the S Line RAS.



### The NERC definition of an SPS conclude s with “Also called Remedial Action

Scheme.” 95 

### This implies that all SPSs are RASs and vice versa, but prior to October 1,

2011, the WECC region did not equate SPSs with RASs. 96 

### WECC created four classifications of protection systems that fall under the NERC definition of SPS, and, instead of including all of these classifi cations in the RAS definition, WECC only identified a subset of those protection systems as RASs. Safety Nets, Wide Area Protection Systems (WAPS), and Local Area Protection Systems (LAPS) were excluded from the WECC definition of a RAS even though they are SPSs as defined by NERC.

For example, SCE did not study the impact of the SONGS separation scheme on BPS reliability because it believed, by classifyin g this scheme as a Safety Net, that it was not required to be studied. SCE also did not submit the separation scheme to WECC for review by the Remedial Action Sche me Reliability Subcommittee (RASRS). The inquiry determined that the SONGS separation sche me is indeed an SPS/RAS as defined by NERC, because it altered the BPS configuratio n by separating Path 44 and redistributing generation in the absence of any faulted equipment. WECC, SDG&E, and SCE did not study the impact that the SONGS separation scheme could have on BPS reliability and, thus, were unaware of its severe impact on the BPS when the scheme operated: blacking out SDG&E and CFE and leading to the loss of the SONGS generators.

95 NERC Glossary of Terms, February 8, 2012, at 46.

96 On October 1, 2011, WECC revised its definition of RAS to include Safety Nets and Local Area Protection

Schemes.



## - 104 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Another protection system that did not get the necessary scrutiny due to WECC’s

narrow definition of RAS was the S Line RAS. The S Line is a 230 kV transmission line that serves as a major tie between SDG&E & IID. It runs from IID’s and SDG&E’s jointly owned Imperial Valley station on one end to II D’s El Centro station on the other. The S

Line RAS, as IID and SDG&E called it, was clas sified as a LAPS by WECC, which called it the “S Line Scheme.” Thus, the RAS received no periodic assessments. Like the SONGS scheme, the S Line RAS appears to be a SPS/ RAS as defined by NERC, because it is an automatic protection system that took action other than isolating a faulted facility by tripping generation in Mexico for loading on a tie line between SDG&E and IID.

The S Line RAS was implemented for two reasons: (1) to protect IID’s system from overload during an N-2 event at SDG&E’s Miguel substation; and (2) to protect

IID’s lone 230/161kV transformer at El Ce ntro from overloads due to generation additions at Imperial Valley substation. The inquiry questions whether the scheme is still necessary, as both of the concerns that originally trigge red installation of the S Line RAS have been mitigated. IID added a new transformer bank at El Centro, mitigating

the concern for overloads on the 230/161kV transformer. Also, reconfigurations at Miguel along with the modifications to a RA S at Miguel have mitigated the concern of adverse effects on IID’s system as a result of an N-2 event at Miguel. Since LAPSs are not periodically reviewed, the arguably outdat ed S Line RAS was still active during the September 8th event

, and its operation contributed to II D’s uncontrolled separation and the operation of the SONGS separation scheme by tripping over 400 MW of generation before the S Line itself tripped. At a minimum, SDG&E, IID and CAISO should participate in the review of the S Line RAS.

The SPSs that operated during the event suggest that WECC’s previous exclusion

of certain NERC-defined SPSs from WECC’s RAS definition had an adverse impact on BPS reliability.

Finding 23 Effect of Inadvertent Operatio n of SONGS Separation Scheme on BPS Reliability:

 The inquiry’s simulation of the event shows that the inadvertent operation of the SONGS separation scheme under normal system operations could lead to a voltage collapse and blackout in the SDG&E areas under certain high load conditions.



## - 105 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Recommendation 23:

 CAISO and SCE should promptly verify that the inadvertent operation of the SONGS separation scheme does not pose an unacceptable risk to BPS reliability. Until this verification ca n be completed, they should consider all actions to minimize this risk, up to and including temporarily removing the SONGS separation scheme from service.



### The inquiry conducted a simulation to evaluate what would happen if the SONGS separation scheme inadvertently operated during normal system operations (e.g., in the absence of any outages, overloads, or SOL vi olations). Based on this simulation, the inquiry determined that under certain high load conditions, the operation of the scheme could result in voltage collapse and a blacko ut in SDG&E’s and CFE’s territories. The inquiry conducted a voltage stability study us ing a Power-Voltage (P-V) curve to estimate the amount of SDG&E load that could reliably be supplied after an inadvertent operation of the SONGS separation scheme. The P-V curve below in Figure 16 demonstrates that such operation would lead to a voltage collapse and a blackout in the SDG&E and CFE territories under certain high load conditions.

400 420 440 460 480 500 520 540 560 580

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000 7500 8000NorthGilaVoltage(kV)

San Diego Load 

### PV Curve for North Gila 500

‐ kV Bus

North Gila 500

‐ kV Base Case North Gila 500

‐ kV Path 44 Open 

### Figure 16: PV Curve for North Gila 500 kV Bus



### Specif ically, the system is most likely to collapse when the SDG&E load exceeds

3,500 MW. In 2010, SDG&E’s load ex ceeded this amount for 851 hours, 97 

### meaning that

97 SDG&E Annual Electric Balancing Authority Area and Planning, FERC Form No. 714 (2010).



## - 106 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

the system was exposed to a potential blackout for approximately 10% of the year. This shows the potential risk to BPS reliability duri ng normal system operations as a result of the inadvertent operation of the SONGS separation scheme. Accordingly, given the lack of studies done on the scheme, the inquiry recommends that the inadvertent operation of the SONGS separation scheme be reviewed promptly to ensure it does not pose an unacceptable risk to BPS reliability. Until this verification can be completed, CAISO and

SCE should consider all actions needed to minimize this risk, up to and including temporarily removing the scheme from service.

Moreover, if SCE and CAISO were to deci de to temporarily remove the scheme from the service, the inquiry does not believe that BPS reliability would be jeopardized. Indeed, inquiry simulations conducted for the day of the event show that if the scheme had not operated, the system, with the exception of collapses in the IID and Yuma areas,

would have stabilized with minor overload s in the area around SONGS, acceptable voltages in the SDG&E area, and sufficient re active margins in the critical portion of SCE’s system.

Finding 24 Not Recognizing Relay Settings When Establishing SOLs:

 An affected TO did not properly establish the SOL for two transformers, as the SOL did not recognize that the most limiting elements (protective relays) were set to trip below the established emergency rating. As a result, the transformers tripped prior to the facili ties being loaded to their emergency ratings during the restoration proce ss, which delayed the restoration of power to the Yuma load pocket.



### Recommendation 24:

 TOs should reevaluate their facility ratings methodologies and implementation of the methodologies to ensure that their ratings are equal to the most limiting piece of equipment, including relay settings . No relay settings should be set below a facility’s emergency rating. When the relay setting is determined to be the most limiting piece of equipment, consideration should be given to reviewing the setting to ensure that it does not unnecessarily restrict the transmission loadability.



### TOs are required to designate and share their facilities’ SOLs. An SOL is the value that satisfies the most limiting element of a facility beyond which the system cannot operate reliably. The inquiry’s relay lo adability calculations show that APS failed to properly establish the SOL for two of it s 500/69 kV transformers in North Gila, because the transformers’ relay loadability or load limit was actually set below their emergency ratings. A facility cannot operate above its relay load limit, as operation in



## - 107 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

excess of a load limit results in the facility being removed from service. Thus, these settings prevented the TOP from taking advantage of the short term emergency ratings identified by the transformers’ SOLs. These settings resulted in difficulties restoring power to the Yuma load pocket, as operators believed they could lo ad the transformers up to their emergency rating. Instead, th e transformers tripped below the emergency rating, delaying the restoration of power to Yuma.

If the SOL derivation had considered th e transformer relay load limit, the TO could have (1) provided an SOL that accurately reflected the relay load limit so the system operator could have limited the tr ansformer loading appropriately, or (2) reviewed the relay load limit to determine whether it unnecessarily limited the transformer loadability, and if so, raised the transformer relay setting threshold above the transformer emergency rating while coordinating the setting with the transformer short-time thermal capability.

Load-Responsive Phase Protection Sy stems Set Too Close to Normal or Emergency Ratings

BES facilities at a minimum are required to have normal and emergency ratings. The normal rating is a continuous rating or a ra ting that a facility can be operated to on a daily basis that specifies the amount of elec trical loading a facility can support. The emergency rating specifies the level of electrical loading a facility can support for a finite period of time. Operating a facility beyond its normal and/or emergency rating for an extended period of time will expose certain equipment in th at facility to the risk of thermal damage. In order to prevent therma l damage to facilities, some TOs implement overload protection systems that are designed to automatically isolat e the facilities if operated beyond their emergency rating.

A problem arises when overload protection systems are set in close proximity to a facility’s normal or emergency ratings. Se tting the overload protection close to the normal or emergency ratings restricts facility loading and prevents operators from having sufficient time to take remedial action to mitigate an overload before the facility is

automatically isolated by th e overload protection system. 98 

### As the Commission stated in Order No. 733, “manual mitigation of thermal overloads is best left to system operators, who can take appropriate actions to support Reliable Op eration of the Bu lk-

98 NERC Reliability Standard PRC

‐ 023

‐ 1 R1.11 provides the following guidance on setting of overload protection systems on transformers: “Set the relays to allow the transformer to be operated at an overload level of at least 150% of the maximum applicable nameplate rating, or 115% of the highest operator established emergency transformer rating, whichever is greater.”



## - 108 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Power System.” 99 

### Protective relay settings limi ted transmission loadability with extremely conservative overload protection settings, resulting in cascading outages during the September 8th event. These settings resulted in faciliti es being automatically removed from service by relays before operat ors had an opportunity to take remedia l

action.

etween Overload Re lay Protection Settings and Finding 25 Margin B Em ergency Rating:

he protective relays tripped the transformers following an N-1 contingency.



## c mmendation 25:

perator established emergency transformer rating, whichever is greater.



### r

een in

-NG, and IID operators might have had time to take

actions to prevent cascading. 100 

###  Some affected TOs set overload relay protection settings on transformers just above the transformers’ emergency rating, resulting in facilities being automatically removed from service before TOPs have sufficient time to take control actions to mitigate the resulting overloads. One TO in particular set its transformers’ overload protection schemes with such narrow margins between the emergency ratings and the r elay trip settings that t



## R e o

 TOs should review their transformers’ overload protection relay settings with their TOPs to ensure appropriate margins between relay settings and emergency ratings developed by TOPs. For example, TOs could consider using the settings of Reliability Standard PRC-023-1 R.1.11 even for those transformers not classified as BES. PRC-023-1 R.1.11 requires relays to be set to allow the transformer to be oper ated at an overload level of at least 150% of the maximum applicable namep late rating, or 115% of the highest

o



### Relay loadability calculations indicate that the relay settings on a number of

transmission facilities limited transmission loadability to slightly above the emergency rating. For example, the relays on IID’s CV transformers were set to trip at 127% of their normal rating. The parallel CV transformers were loaded to 130%, which was above thei 127% overload relay trip point, immediately after the loss of H-NG. Both transformers tripped less than 40 seconds later. If the tr ansformers’ overload trip point had b

accordance with PRC-023-1 R.1.11, the trip point would not have been exceeded immediately after the loss of the H

99 Transmission Relay Loadability Reliability Standard

, 130 FERC ¶ 61,221, at P 212 (2010).

100 IID originally used conservative settings because the CV transformers are rare, expensive, load

‐ serving

transformers. IID has indicated, however, that it will increase the overload relay settings on the CV transformers

to 150% of their normal rating, and will relocate an additional 230/92 kV transformer from another substation to

CV.



## - 109 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

During the September 8th event, IID was unaware that the overload relay se for the Ramon 230/92 kV transformer had been mistakenly set at 207% of its normal rating. IID intended the Ramon transformer to have been set to trip at 120% of its normal rating. After the even t, IID reduced the Ramon transformer’s trip setting 207% to 120%, making it more likely to trip during high-loading conditions or con similar to those that precipitated the bl ackout, decreasing the opportunity for its

operators tting

from

ditions

to take mitigating actions during such conditions. This setting actually creased the risk of future cascading outages like the one which occurred on September in

8, 2011.

F inding 26 Relay Settings and Proximity to Emergency Ratings:

 Some TOs set relays to isolate facilities for loading conditions s lightly above eir thirty minute emergency ratings. As a result, several transmission ers tripped within seconds of exceeding their emergency nsufficient time to mitigate overloads.

to cannot be raised allow more time for TOPs to take manual action, TOPs must ensure that



### ,

tigated the cascade, the applied settings evertheless do not leave operators sufficient ti me to take mitigating steps to prevent or

ences of future events.



## Angula th lines and transform ratings, leaving TOPs i



### R ecommendation 26:

 TOs should evaluate load responsive relays on transmission lines and transformers to determine if the setting s can be raised to provide more time for TOPs to take manual action to mi tigate overloads that are within the short-time thermal capability of the eq uipment instead of allowing relays prematurely isolate the transmission lines. If the settings

to the settings are taken into account in developing facility ratings and that automatic isolation does not result in cascading outages.



### In addition to the problematic protection settings of the CV transformers, which precipitated the cascade, the inquiry discover ed that several other facilities, including a

number of IID’s 161 kV transmission lines and two of WALC’s 16 1/69 kV transformers had relay protection settings which were only slightly above those facilities’ emergency ratings. These conservative settings severe ly limited TOPs’ response time before the facilities were isolated, preventing the operat ors from taking effective mitigating action against the cascade. While the inquiry did not determine whether less conservative relay settings on these other facilities would have mi

n ameliorate the consequ



## r Separation

When a transmission line trips or goes out of service, the phase angle will generally increase between its two terminal points. When angle differences become large, facilities connected to the system ca n lose synchronization, causing the system to



## - 110 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

become unstable. Also, if the phase angle is too large, closing the line breaker back into service with a large an gle difference may result in damage to nearby generator tu shafts, and the resulting power swings and osci llations could lead to system instability or collapse. To enable successful reclosing, studies should be run to determine the maximum phase angle difference allowable for a line to be closed back in and safeguards be put into place to prevent reclosure with excessive pha rbine

se angle difference. Should the phase a gle difference exceed the established limit, generation or load must be adjusted n

to reduce it to the level that allows the line to be closed.

F inding 27 Phase Angle Difference Following Loss of Transmission Line:

 A TOP did not have tools in place to determine the pha se angle difference etween the two terminals of its 500 kV line after the line tripped. Yet, it nd another TOP that the line would be restored quickly, ave been accomplished.

g plans lso train effectively respond to phase angle differences. These plans should be developed based on the seasonal and next-day contingency



### n

APS

th sides of North Gila was dispatched or load reductions in the are est of North Gila were implemented to reduce the difference of the voltage phase a b inform ed the RC a when, in fact, this could not h



### R ecommendation 27:

 TOPs should have: (1) the tools necessary to determine phase angle differences following the loss of lines; and (2) mitigation and operatin for reclosing lines with large phase angle differences. TOPs should a operators to

analyses that address the angular di fferences across opened system elements.



### The inquiry’s simulation shows that afte r H-NG tripped, the voltage phase angle between the two terminals increased from 20 degrees to approximately 72 degrees. O the day of the event, APS’s synchro-check relay was set at 60 degrees, 101 

### meaning would not have been able to reclose H-NG until it reduced the phase angle difference from 72 to 60 degrees, or changed the rela y setting to allow the breaker to close. Specifically, the 60 degree setting would no t have allowed APS to reclose H-NG until appropriate generation on bo

as w ngle to 60 degrees.

101 Based on additional studies, APS has since determined the maximum settings on its synchro

‐ check relay at

North Gila to allow a maximum phase angle difference of 75 degrees to reclose a line. To add margin, APS has implemented the relay setting at 70 degrees.



## - 111 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Although APS operators are trained to effectively respond to phase angle

differences, 102 

### APS currently lacks the tools ne cessary to determ ine phase angle differences following the loss of a transmi ssion line until the line is reenergized. 103 

### Th training, therefore, does little good if the operators cannot determine whether a phase angle difference exists in the first place. Generally, APS operators can monitor phas angles through SCADA, but in order to receive and review this data, the transmission line must be energized. After H-NG trippe d, and prior to reenergizing the line, for example, APS had no way to know if the line could be reclosed within the permissive 60 degree setting of its synchro-check relay. It lacked situational awareness of the phase angle difference e

e

. Yet, APS informed WECC RC and CAISO that it beli eved the line could be reclosed quickly, when, in fact, this could not have been done due to the phase angle

differen

to incr ease

ired

nce. Here, for exampl , APS likely could not have reclosed the line quickly, even had it known the phase a

at

h es. As generation is dispatched to its maximum output in the vicinity of the two ce. 104



### To avoid a similar situation in the future , TOPs should ensure that they have adequate tools to determine phase angles af ter the loss of transmission lines. For example, they can install PMUs throughout their system, as AP S plans to do, their situational awareness of phase angles. Moreover, TOPs should ensure that their operators are trained to respond to ph ase angle differences by, for example, redispatching generation. In addition, TOPs should not underestimate the time requ to reclose a line, particularly without first knowing the phase angle differe

e ngle difference, given system conditions on the day of the event.

Indeed, the inquiry conducted a series of power flow simulations and found th significant amounts of generation redispat ch were needed to close the phase angle

difference. Figure 17 , on the next page, shows the relationship between the voltage phase angle of H-NG as generation is redispatched between California and Arizona. The dispatched approach adjusts the available generation near est the Hassayampa and Nort

Gila bus

102 APS provides its certified operators with two training classes, Power System Dynamics and Dynamics of

isturbances, both of which address power angles and their ramifications. In addition, APS provides its new operator trainees with training on power angles. D

103 APS plans to expand its use of PMUs to enable it to determine phase angle differences even without a line being energized. Through the PMU data, APS would be able to determine voltage and angle measurements on live buses in its substations, through which it could calculate phase angle differences.

104 APS did not intentionally mislead WECC RC and CAISO with this statement. Rather, it did not expect that

there would have been such a large phase angle difference, as it had not previously experienced such a difference. Moreover, APS determined that the line was not damaged and, thus, it did not believe there would be

any is sues closing the line.



## - 112 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

stations, oth er generators farther out are adju sted to effect the change in voltage phase angles.

Figure 17: Phase Angle of H-NG vs. Generation Shift



### T he blue line in Figure 17 illustrates th at with the particular conditions of the Septem ds

r to i.e., to within the permissive 60 degree setting of the synchro-check re lay). The green line shows that more generation—more than twice as much—must be redispatched if units are chosen in Northern California to close the angl e between Hassayampa and North Gila.

While system operators could redispatch generation from available spinning

reserves or commit units in the Southern and/or Northern California area, it is questionable how quickly 1,800 MW could be dispatched. ber 8th event, approximately 1,800 MW needed to be redispatched on both en of H-NG (and close to the terminals, in So uthern California and Arizona) in orde close the voltage phase angle from 72 degrees to 60 degrees (



## - 113 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Appendix A: List of Ac ronyms Used in Report

ACE Area Control Error

APS Arizona Public Service

BA Balancing Authority

BES Bulk Electric System

BPS Bulk-Power System

CAISO California Independent System Operator, Inc.

CFE Comisión Federal de Electricidad

CV Coachella Valley

EMS Energy Management System

GO Generator Owner

GOP Generator Operator

H-NG APS’s Hassayampa-North Gila 500 kV transmission line

IEEE Institute of Electrical and Electronics Engineers

IID Imperial Irrigation District

IROL Interconnection Reli ability Operating Limit

kV Kilovolt

LAPS Local Area Protection System

MVA Megavolt-ampere

MW Megawatt

NERC North American Electric Reliability Corporation

OSS California/Mexico Operations Study Subcommittee

OTC Operating Transfer Capabilities

OTCPC Operating Transfer Capability Policy Committee

PC Planning Coordinator

PMU Phasor Measurement Unit

RAS Remedial Action Scheme

RC Reliability Coordinator

RE Regional Entity

RTCA Real-Time Contingency Analysis

SASG Southwest Area Study Group

SCADA Supervisory Control and Data Acquisition

SCE Southern California Edison

SDG&E San Diego Gas & Electric

SE State Estimator

SOE Sequence of Events

SOL System Operating Limit

SONGS San Onofre Nuclear Generating Station

SPS Special Protection System



## - 114 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

SRP Salt River Power

SWPL Southwest Power Link

TO Transmission Owner

TOP Transmission Operator

TP Transmission Planner

UFLS Underfrequency Load Shedding

VAR Volt-Ampere Reactive

WALC Western Area Power Administration – Lower Colorado

WAPS Wide Area Protection System

WECC Western Electricity Coordinating Council

YCA Yuma Cogeneration Associates



## - 115 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Appendix B: Table of Findings and Recommendations

The following table provides a complete list of findings and corresponding

recommendations, each of which are discussed in detail at Section IV of the

report.

NEXT-DAY PLANNING

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 1 – Failure to Conduct and Share Next-Day Studies : Not all ofthe affected TOPs conduct nextday studies or share them with their neighbors and WECC RC. As a result of failing to exchange studies, on September 8, 2011 TOPs were not alerted to contingencies on neighboring systems that could impact their internal system and the need to plan for such contingencies. Recommendation 1 : All TOPs should conduct next-day studies and share the results with neighboring TOPs and the RC (before the next day) to ensure that all contingencies that could impact the BPS are studied. TOPs

Finding 2 – Lack of Updated External Networks in Next-Day Study Models : When conducting next-day studies, some affected TOPs use models for external networks that are not updated to reflect next-day operating conditions external to their systems, such as generation schedules and transmission outages. As a result, these TOPs’ next-day studies do not adequately predict the impact of external contingencies on their systems or internal contingencies on external systems. Recommendation 2 : TOPs and BAs should ensure that their next-day studies are updated to reflect next-day operating conditions external to their systems, such as generation and transmission outages and scheduled interchanges, which can significantly impact the operation of their systems. TOPs and BAs should take the necessary steps, such as executing nondisclosure agreements, to allow the free exchange of next-day operations data between operating entities. Also, RCs should review the procedures in the region for coordinating next-day studies, ensure adequate data exchange among BAs and TOPs, and facilitate the next-day studies of BAs and TOPs. TOPs, BAs, RCs

Finding 3 –Sub-100 kV Facilities Not Adequately Considered in Next-Day Studies : In conducting next-day studies, some affected TOPs focus primarily on the TOPs’ internal SOLs and the need to stay within established Rated Path limits, without adequate consideration of some lower voltage facilities. As a result, these TOPs risk overlooking facilities that may become overloaded and impact the reliability of the BPS. Similarly, the RC does not study sub-100 kV facilities that impact BPS reliability unless it has Recommendation 3 : TOPs and RCs should ensure that their next-day studies include all internal and external facilities (including those below 100 kV) that can impact BPS reliability. TOPs, RCs



## - 116 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

specifically been alerted to issues with such facilities by individual TOPs or the RC has otherwise identified a particular sub-100 kV facility as affecting the BPS. Finding 4 – Flawed Process for Estimating Scheduled Interchanges : WECC RC’s process for estimating scheduled interchanges is not adequate to ensure that such values are accurately reflected in its next-day studies. As a result, its next-day studies may not accurately predict actual power flows and contingency overloads. Recommendation 4 : WECC RC should improve its process for predicting interchanges in the day-ahead timeframe. WECC RC



## SEASONAL PLANNING

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 5 – Lack of Coordination in Seasonal Planning Process : The seasonal planning process in the WECC region lacks effective coordination. Specifically, the four WECC subregions do not adequately integrate and coordinate studies across the subregions, and no single entity is responsible for ensuring a thorough seasonal planning process. Instead of conducting a full contingency analysis based on all of the subregions’ studies, the subregions rely on experience and engineering judgment in choosing which contingencies to discuss. As a result, individual TOPs may not identify contingencies in one subregion that may affect TOPs in the same or another subregion. Recommendation 5 : WECC RE should ensure better integration and coordination of the various subregions’ seasonal studies for the entire WECC system. To ensure a thorough seasonal planning process, at a minimum, WECC RE should require a full contingency analysis of the entire WECC system, using one integrated seasonal study, and should identify and eliminate gaps between subregional studies. Individual TOPs should also conduct a full contingency analysis to identify contingencies outside their own systems that can impact the reliability of the BPS within their system and should share their seasonal studies with TOPs shown to affect or be affected by their contingencies. WECC RE, TOPs

Finding 6 –External and LowerVoltage Facilities Not Adequately Considered in Seasonal Planning Process : Seasonal planning studies do not adequately consider all facilities that may affect BPS reliability, including external facilities and lower-voltage facilities. Recommendation 6 : TOPs should expand the focus of their seasonal planning to include external facilities and internal and external sub-100 kV facilities that impact BPS reliability. TOPs

Finding 7 – Failure to Study Multiple Load Levels : TOPs do not always run their individual seasonal planning studies based on the multiple WECC base cases Recommendation 7 : TOPs should expand the cases on which they run their individual planning studies to include multiple base cases, as well as generation maintenance outages and dispatch scenarios during high TOPs



## - 117 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

(heavy and light load summer, heavy and light load winter, and heavy spring), but, instead, may focus on only one load level. As a result, contingencies that occur during the shoulder seasons (or other load levels not studied) might be missed. load shoulder periods.

Finding 8 – Not Sharing Overload Relay Trip Settings : In the seasonal planning process, at least one TOP did not share with neighboring TOPs overload relay trip settings on transformers and transmission lines that impacted external BPS systems. Recommendation 8 : TOPs should include in the information they share during the seasonal planning process the overload relay trip settings on transformers and transmission lines that impact the BPS, and separately identify those that have overload trip settings below 150% of their normal rating, or below 115% of the highest emergency rating, whichever of these two values is greater. TOPs



## NEAR- AND LONG-TERM PLANNING

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 9 – Gaps in Near- and Long-Term Planning Process : Gaps exist in WECC RE’s, TPs’ and PCs’ processes for conducting near- and long-term planning studies, resulting in a lack of consideration for: (1) critical system conditions; (2) the impact of elements operated at less than 100 kV on BPS reliability; and (3) the interaction of protection systems. As a consequence, the affected entities did not identify during the planning process that the loss of a single 500 kV transmission line could potentially cause cascading outages. Planning studies conducted between 2006 and 2011 should have identified the critical condit ions that existed on September 8th and proposed appropriate mitigation strategies. Recommendation 9 : WECC RE should take actions to mitigate these and any other identified gaps in the procedures for conducting near- and long-term planning studies. The September 8th event and other major events should be used to identify shortcomings when developing valid cases over the planning horizon and to identify flaws in the existing planning structure. WECC RE should then propose changes to improve the performance of planning studies on a subregional- and Interconnection-wide basis and ensure a coordinated review of TPs’ and PCs’ studies. TOPs, TPs and PCs should develop study cases that cover critical system conditions over the planning horizon; consider the benefits and potential adverse effects of all protection systems, including RASs, Safety Nets (such as the SONGS separation scheme), and overload protection schemes; study the interaction of RASs and Safety Nets; and consider the impact of elements operated at less than 100 kV on BPS reliability. WECC RE, TOPs, TPs, PCs

Finding 10 – Benchmarking WECC Dynamic Models : The inquiry obtained a very good correlation between the simulations and the actual event until the SONGS separation scheme activated. After activation of the scheme, however, neither the tripping of the SONGS units nor the system collapse of SDG&E and CFE could be detected using WECC dynamic models because Recommendation 10 : WECC dynamic models should be benchmarked by TPs against actual data from the September 8th event to improve their conformity to actual system performance. In particular, improvements to model performance from validation would be helpful in analysis of under and/or over frequency events in the Western Interconnection and the stability of islanding scenarios in the SDG&E and CFE areas. TPs



## - 118 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

some of the elements of the event are not explicitly included in those models. Sample simulations of the islanded region showed that by adding known details from the actual event, including UFLS programs and automatic capacitor switching, the simulation and event become more closely aligned following activation of the SONGS separation scheme. 

## SITUATIONAL AWARENESS

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 11 – Lack of Real-Time External Visibility : Affected TOPs have limited real-time visibility outside their systems, typically monitoring only one external bus. As a result, they lack adequate situational awareness of external contingencies that could impact their systems. They also may not fully understand how internal contingencies could affect SOLs in their neighbors’ systems. Recommendation 11 : TOPs should engage in more real-time data sharing to increase their visibility and situational awareness of external contingencies that could impact the reliability of their systems. They should obtain sufficient data to monitor significant external facilities in real time, especially those that are known to have a direct bearing on the reliability of their system, and properly assess the impact of internal contingencies on the SOLs of other TOPs. In addition, TOPs should review their real-time monitoring tools, such as State Estimator and RTCA, to ensure that such tools represent critical facilities needed for the reliable operation of the BPS. TOPs

Finding 12 – Inadequate RealTime Tools : Affected TOPs’ realtime tools are not adequate or, in one case, operational to provide the situational awareness necessary to identify contingencies and reliably operate their systems. Recommendation 12 : TOPs should take measures to ensure that their real-time tools are adequate, operational, and run frequently enough to provide their operators the situational awareness necessary to identify and plan for continge ncies and reliably operate their systems. TOPs

Finding 13 – Reliance on PostContingency Mitigation Plans : One affected TOP operated in an unsecured N-1 state on September8, 2011, when it relied on postcontingency mitigation plans for its internal contingencies and subsequent overload and tripping, while assuming there would be sufficient time to mitigate the contingencies. Post-contingency mitigation plans are not viable under all circumstances, such as when equipment trips on overload relay protection that prevents operators from taking timely control actions. If this TOP had used pre-contingency measures on Recommendation 13 : TOPs should review existing operating processes and procedures to ensure that post-contingency mitigation plans reflect the time necessary to take mitigating actions, including control actions, to return the system to a secure N-1 state as soon as possible but no longer than 30 minutes following a single contingency. As part of this review, TOPs should consider the effect of relays that automatically isolate facilities without providing operators sufficient time to take mitigating measures. TOPs



## - 119 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

September 8th, such as dispatching additional generation, to mitigate first contingency emergency overloads for its internal contingencies, the cascading outages that were triggered by the loss of H-NG might have been avoided with the prevailing system conditions on September 8, 2011.

Finding 14 – WECC RC Staffing Concerns : WECC RC staffs a total of four operators at any one time to meet the functional requirements of an RC, including continuous monitoring, conducting studies, and giving directives. The September 8th event raises concerns that WECC RC’s staffing is not adequate to respond to emergency conditions. Recommendation 14 : WECC RC should evaluate the effectiveness of its staffing level, training and tools. Based on the results of this evaluation, it should determine what actions are necessary to perform its functions appropriately as the RC and address any identified deficiencies. WECC RC

Finding 15 – Failure to Notify WECC RC and Neighboring TOPs Upon Losing RTCA : On September 8, 2011, at least one affected TOP lost the ability to conduct RTCA more than 30 minutes prior to and throughout the course of the event due to the failure of its State Estimator to converge. The entity did not notify WECC RC or any of its neighboring TOPs, preventing this entity from regaining situational awareness. Recommendation 15 : TOPs should ensure procedures and training are in place to notify WECC RC and neighboring TOPs and BAs promptly after losing RTCA capabilities. TOPs

Finding 16 – Discrepancies Between RTCA and Planning Models : WECC’s model used by TOPs to conduct RTCA studies is not consistent with WECC’s planning model and produces conflicting solutions. Recommendation 16 : WECC should ensure consistencies in model parameters between its planning model and its RTCA model and should review all model parameters on a consistent basis to make sure discrepancies do

not occur. WECC



## CONSIDERATION OF BES EQUIPMENT

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 17 – Impact of Sub-100 kV Facilities on BPS Reliability : WECC RC and affected TOPs and BAs do not consistently recognize the adverse impact sub-100 kV facilities can have on BPS reliability. As a result, sub-100 kV facilities might not be designated Recommendation 17 : WECC, as the RE should lead other entities, including TOPs and BAs, to ensure that all facilities that can adversely impact BPS reliability are either designated as part of the BES or otherwise incorporated into planning and operations studies and actively monitored and alarmed in RTCA systems. WECC RE, TOPs, BAs



## - 120 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

as part of the BES, which can leave entities unable to address the reliability impact they can have in the planning and operations time horizons. If, prior to September 8, 2011, certain sub-100 kV facilities had been designated as part of the BES and, as a result, were incorporated into the TOPs’ and RC’s planning and operations studies, or otherwise had been incorporated into these studies, cascading outages may have been avoided on the day of the event.



## IROL DERIVATIONS

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 18 – Failure to Establish Valid SOLs and Identify IROLs : The cascading nature of the event that led to uncontrolled separation of San Diego, IID, Yuma, and CFE indicates that an IROL was violated on September 8, 2011, even though WECC RC did not recognize any IROLs in existence on that day. In addition, the established SOL of 2,200 MW on Path 44 and 1,800 MW on H-NG are invalid for the present infrastructure, as demonstrated by the event. Recommendation 18.1 : WECC RC should recognize that IROLs do exist on its systemand, thus, should study IROLs in the dayahead timeframe and monitor potential IROL exceedances in real-time.

Recommendation 18.2 : WECC RC should work with TOPs to consider whether any SOLs in the Western Interconnection constitute IROLs. As part of this effort, WECC RC should: (1) work with affected TOPs to consider whether Path 44 and H-NG should be recognized as IROLs; and (2) validate existing SOLs, and ensure that they take into account all transmission and generation facilities and protection systems that impact BPS reliability. WECC RC, TOPs



## PROTECTION SYSTEMS

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 19 – Lack of Coordination of the S Line RAS : Several TOs and TOPs did not properly coordinate a RAS by: (1) not performing coordination studies with the overload protection schemes on the facilities that the S Line RAS is designed to protect; and (2) not assessing the impact of setting relays to trip generation sources and a 230 kV transmission tie line prior to the operation of a single 161/92 kV transformer’s overload protection. As a result, BES facilities were isolated in excess of those needed to maintain reliability, with adverse impact on Recommendation 19 : The TOs and TOPs responsible for design and coordination of the S Line RAS should revisit its design basis and protection settings to ensure coordination with other protection systems in order to prevent adverse impact to the BPS, premature operation, and excessive isolation of facilities. TOs and TOPs should share any changes to the S Line RAS with TPs and PCs so that they can accurately reflect the S Line RAS when planning. TOs, TOPs



## - 121 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

BPS reliability.

Finding 20 – Lack of Coordination of the SONGS Separation Scheme : SCE did not coordinate the SONGS separation scheme with other protection systems, including protection and turbine control systems on the two SONGS generators. As a result, SCE did not realize that Units 2 and 3 at SONGS would trip after operation of the separation scheme. Recommendation 20 : SCE should ensure that the SONGS separation scheme is coordinated with other protection schemes, such as the generation protection and turbine control systems on the units at SONGS and UFLS schemes. SCE

Finding 21 – Effect of SONGS Separation Scheme on SONGS Units : The SONGS units tripped due to their turbine control systems detecting unacceptable acceleration following operation of the SONGS separation scheme. Recommendation 21 : GOs and GOPs should evaluate the sensitivity of the acceleration control functions in turbine control systems to verify that transient perturbations or fault conditions in the transmission system resulting in unit acceleration will not result in unit trip without allowing time for protective devices to clear the fault on the transmission system. GOs, GOPs

Finding 22 – Lack of Review and Studying Impact of SPSs : Although WECC equates SPSs with RASs, prior to October 1, 2011, WECC’s definition of RAS excluded many protection systems that would be included within NERC’s definition of SPS. As a result, WECC did not review and assess all NERC-defined SPSs in its region, and WECC’s TOPs did not perform the required reviewand assessment of all NERCdefined SPSs in their areas. Recommendation 22 : WECC RE, along with TOs, GOs, and Distribution Providers (DPs), should periodically review the purpose and impact of RASs, including Safety Nets and Local Area Protection Schemes, to ensure they are properly classified, are still necessary, serve their intended purposes, are coordinated properly with other protection systems, and do not have unintended consequences on reliability. WECC RE and the appropriate TOPs should promptly conduct these reviews for the SONGS separation scheme and the S Line RAS. WECC RE, TOs, GOs, DPs, TOPs

Finding 23 – Effect of Inadvertent Operation of SONGS Separation Scheme on BPS Reliability : The inquiry’s simulation of the event shows that the inadvertent operation of the SONGS separation scheme under normal system operations could lead to a voltage collapse and blackout in the SDG&E areas under certain high load conditions. Recommendation 23 : CAISO and SCE should promptly verify that the inadvertent operation of the SONGS separation scheme does not pose an unacceptable risk to BPS reliability. Until this verification can be completed, they should consider all actions to minimize this risk, up to and including, temporarily removing the SONGS separation scheme from

service. CAISO, SCE

Finding 24 – Not Recognizing Relay Settings When Establishing SOLs : An affected TO did not properly establish the SOL for two transformers, as the SOL did not Recommendation 24 : TOs should reevaluate their facility ratings methodologies and implementation of the methodologies to ensure that their ratings are equal to the most limiting piece of equipment, including relay TOs



## - 122 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

recognize that the most limiting elements (protective relays) were set to trip below the established emergency rating. As a result, the transformers tripped prior to the facilities being loaded to their emergency ratings during the restoration process, which delayed the restoration of power to the Yuma load pocket. settings. No relay settings should be set below a facility’s emergency rating. When the relay setting is determined to be the most limiting piece of equipment, consideration should be given to reviewing the setting to ensure that it does not unnecessarily restrict the transmission loadability.

Finding 25 – Margin Between Overload Relay Protection Settings and Emergency Rating : Some affected TOs set overload relay protection settings on transformers just above the transformers’ emergency rating, resulting in facilities being automatically removed from service before TOPs have sufficient time to take control actions to mitigate the resulting overloads. One TO in particular set its transformers’ overload protection schemes with such narrow margins between the emergency ratings and the relay trip settings that the protective relays tripped the transformers following an N-1 contingency. Recommendation 25 : TOs should review their transformers’ overload protection relay settings with their TOPs to ensure appropriate margins between relay settings and emergency ratings developed by TOPs. For example, TOs could consider using the settings of Reliability Standard PRC-023-1 R.1.11 even for those transformers not classified as BES. PRC-023- 1 R.1.11 requires relays to be set to allow the transformer to be operated at an overload level of at least 150% of the maximum applicable nameplate rating, or 115% of the highest operator established emergency transformer rating, whichever is greater. TOs, TOPs

Finding 26 –Relay Settings and Proximity to Emergency Ratings : Some TOs set relays to isolate facilities for loading conditions slightly above their thirty minute emergency ratings. As a result, several transmission lines and transformers tripped within seconds of exceeding their emergency ratings, leaving TOPs insufficient time to mitigate overloads. Recommendation 26 : TOs should evaluate load responsive relays on transmission lines and transformers to determine if the settings can be raised to provide more time for TOPs to take manual action to mitigate overloads that are within the short-time thermal capability of the equipment instead of allowing relays to prematurely isolate the transmission lines. If the settings cannot be raised to allow more time for TOPs to take manual action, TOPs must ensure that the settings are taken into account in developing facility ratings and that automatic isolation does not result in cascading outages. TOs, TOPs



## ANGULAR SEPARATION

FINDING RECOMMENDATION APPLICABLE ENTITIES Finding 27 – Phase Angle Difference Following Loss of Transmission Line : A TOP did not have tools in place to determine the phase angle difference between the two terminals of its 500 kV line after the line tripped. Yet, it informed the RC and another TOP that the line would be restored Recommendation 27 : TOPs should have: (1) the tools necessary to determine phase angle differences following the loss of lines; and (2) mitigation and operating plans for reclosing lines with large phase angle differences. TOPs should also train operators to effectively respond to phase angle differences. These plans should be developed based on the seasonal and next-day contingency analyses TOPs



## - 123 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

quickly, when, in fact, this could not have been accomplished. that address the angular differences across opened system elements.



## - 124 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Appendix C: Comparison of August 2003 and September 2011

Blackouts

On August 14, 2003, an estimated 50 mi llion people throughout the Midwest and Northeast United States and Ontario, Canada, experienced an electric power blackout. A day later, the joint U.S.-Canada Power System Outage Task Force began investigating the causes of the blackout and considering ways to prevent such outages in the future. The

task force detailed its findings and recommendations in an April 2004 report. 105 

### A comparison of the findings and recommenda tions in this April 2004 report and the

instant report on the September 8, 2011, blackout reveals commonalities between the

two events.

Although the August 2003 and Septem ber 2011 blackouts were triggered by different initiating events—tree touches in 20 03 compared to a switching error in 2011—

both blackouts had common underl ying causes. First, affected entities in both events did not conduct adequate long-term and operations planning studies necessary to understand vulnerabilities on their systems. Second, affected entities in both events had inadequate situational awareness leading up to and during the disturbances. In addition to these two underlying causes, both events were exacerbated by protection system relays that tripped facilities without allowing op erators sufficient time to take mitigating measures. These similarities are highlighted below, with excerpts from both reports to illustrate specific comparisons.

Inadequate Long-Term and Operations Planning The 2003 Blackout Report states that “F irstEnergy was not [operating its system securely] because the company had not conducted the long-term and operational planning studies needed to understand [cer tain] vulnerabilities and their operational

implications.” 106 

### Similarly, this inquiry’s report fo und that several entities’ operational and long-term studies did not ad equately ensure the reliable operation of their systems. Specifically, both reports described releva nt planning studies that: (1) did not adequately identify and study critical external facilities; (2) did not adequately analyze potential contingency scenario s; and (3) were based on inaccurate models and invalid system operating limits (SOLs).

105 Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations (U.S.

‐ Canada Power System Outage Task Force: April 2004) (2003 Blackout Report).

106 2003 Blackout Report at 23.



## - 125 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Inadequate Long Term and Operations Planning

Issue 2003 Blackout 2011 Blackout Insufficient Analysis in Seasonal Studies “[T]he studies FirstEnergy relied on .. . were not robust, thorough, or upto-date. This left FE’s planners and operators with a deficient understanding of their system’s capabilities and risks under a range of system conditions.” (P. 39).

“FE’s 2003 Summer Study focusedprimarily on single-contingency (N1) events, and did not consider significant multiple contingency losses and security. . . . Overall, the summer study posited less stressful system conditions than actually occurred August 14, 2003 (when load was well below historic peak demand).” (P 39). “TOPs do not always run their individual seasonal planning studies based on the multiple WECC base cases (heavy and light load summer, heavy and light load winter, and heavy spring), but, instead, may focus on only one load level.” (Finding 7)

“Seasonal planning studies do not adequately consider all facilities that may affect BPS reliability, including external facilities and lower-voltage facilities.”(Finding 6)

“In the seasonal planning process, at least one TOP did not share with neighboring TOPs overload relay trip settings on transformers and transmission lines that impacted external BPS systems.” (Finding 8)

Inadequate Identification and Study of Critical External Facilities “On August 14 four or five capacitor banks within the Cleveland-Akron area had been removed from service for routine inspection. . . . These static reactive power sources are important for voltage support. . . . The unavailability of the critical reactive resources was not known to those outside of FirstEnergy.” (PP. 26-27).

“NERC policy requires that critical facilities be identified and that neighboring control areas and reliability coordinators be made aware of the status of those facilities to identify the impact of those conditions on their own facilities. However, FE never identified these capacitor banks as critical and so did not pass on status information to others.” (P. 27). “Not all of the affected TOPs conduct next-day studies or share them with their neighbors and WECC RC. . . .TOPs were not alerted to contingencies on neighboring systems that could impact their internal system and the need to plan for such contingencies.” (Finding 1)

“In conducting next-day studies, some affected TOPs focus primarily on the TOPs’ internal SOLs and the need to stay within established Rated Path limits, without adequate consideration of some lower voltage facilities.” (Finding 3)

“[In conducting next-day studies,] . . . the RC does not study sub-100 kV facilities that impact BPS reliability unless it has specifically been alerted to issues with such facilities by individual TOPs...” (Finding 3)



## - 126 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Inaccurate Dynamic Models “The after-the-fact models developed to simulate August 14 conditions and events found that the dynamic modeling assumptions for generator and load power factors in regional planning and operating models were frequently inaccura te.” (P. 160). “. . . neither the tripping of the SONGS units nor the system collapse of SDG&E and CFE could be detected using WECC dynamic models because some of the elements of the event are not explicitly included in those models.” (Finding 10)



### To mitigate these concerns, the 2003 Bl ackout Report recommended that “NERC should work with the regional reliability councils to esta blish regional power system models that enable the sharing of consistent and validated data among entities in the

region,” 107 

### and “[c]larify criteria for identification of operationally critical facilities, and

improve dissemination of updated in formation on unplanned outages.” 108 

### This inquiry’s report likewise recommends that entities cooperate and coordinate more effectively across all planning horizons, especially by in creasing visibility in both external systems and lower voltage facilities that could impact BPS reliability.

Inadequate Situational Awareness The 2003 Blackout Report stated, “A prin cipal cause of the August 14 blackout

was a lack of situational awareness, which was in turn the result of inadequate reliability

tools and backup capabilities.” 109 

### Similarly, the instant inquiry determined that inadequate real-time situational awareness contributed to the cascading outages. In both events, for example, the affected entities’ real-time monitoring tools were not adequate to alert operators to system conditio ns and contingencies. Also, some of the

affected entities in both events did not use their real-time tools to monitor system

conditions. As a result of these situational awareness issues, affected entities in both events were not aware that they were no longer operating in a secure N-1 state and were

not alerted to the need to take corrective actions. 

## Inadequate Situational Awareness

Issue 2003 Blackout 2011 Blackout System Visibility “MISO [the Reliability Coordinator] had interpretive and operational tools and a large amount of system data, but had a limited view of FE’s system.” (P. 67). “Affected TOPs have limited realtime visibility outside their systems, typically monitoring only one external bus. As a result, they lack adequate situational awareness of external contingencies that could impact

107 2003 Blackout Report at 160.

108 2003 Blackout Report at 3.

109 2003 Blackout Report at 159.



## - 127 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Inadequate Situational Awareness

Issue 2003 Blackout 2011 Blackout their systems. They also may not fully understand how internal contingencies could affect SOLs in their neighbors’ systems.” (Finding 11)

Inadequate Real-Time Monitoring Tools “FE’s operational monitoring equipment was not adequate to alert FE’s operators regarding important deviations in operating conditions and the need for corrective action.” (P. 19).

“FE’s control room operators lost the alarm function that provided audible and visual indications when a significant piece of equipment changed from an acceptable to a problematic condition.” (P. 51).

MISO’s incomplete tool set and the failure to supply its state estimator with correct system data on August 14 contributed to the lack of situational awareness.” (P. 159). “Affected TOPs’ real-time tools are not adequate or, in one case, operational to provide the situational awareness necessary to identify contingencies and reliably operate their systems.” (Finding 12) “. . . a TOP lost the ability to conduct [Real Time Contingency Analysis] RTCA more than 30 minutes prior to and throughout the course of the event ...[and] did not notify WECC RC or any of its neighboring TOPs...”(Finding 15)

Operating in an Unsecure State “FE’s operators were not aware that the system was operating outside first contingency limits . . . because they did not conduct a contingency analysis.” (P. 64).

“MISO’s reliability coordinators were using non-real-time data to support real-time “flowgate” monitoring. This prevented MISO from detecting an N-1 security violation in FE’s system and from assisting FE in necessary relief actions.” (P. 19).

“Since FE’s operators were not aware and did not recognize events as they were occurring, they took no actions to return the system to a reliable state.” (P. 65). “The cascading nature of the event that led to uncontrolled separation of San Diego, IID, Yuma, and CFE indicates that an [interconnection reliability operating limit] IROL was violated . . . In addition, the established SOLs of 2,200 MWon Path 44 and 1,800 MW on HNG are invalid...”(Finding 18)

“One affected TOP operated in an unsecured N-1 state. . . . when it relied on post-contingency mitigation plans for its internal contingencies and subsequent overloads and trips, while assuming there would be sufficient time to mitigate the contingencies.” (Finding 13)



### To remedy these weaknesses in situational awareness, the 2003 Blackout Report recommended that entities [e]valuate and ad opt better real-time tools for operators and reliability coordinators.” 110 Similarly, this inquiry’s report recommends that operators

110 2003 Blackout Report at 159.



## - 128 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

develop and effectively utilize the real-time tools at their disposal and include all facilities that can im pact BPS reliability.

Protection Systems During both events, protection system settings exacerbated and accelerated the cascading nature of the outages. As stated in the 2003 Blackout Report, zone 3 relay settings “did not cause the blackout, [but] it is certain that they greatly expanded and

accelerated the spread of the cascade.” 111 

### Similarly, load responsive relay settings accelerated the September 8th cascade and e ffectively eliminated the window in which operators could have taken mitigating actions.



## Protection Systems

Issue 2003 Blackout 2011 Blackout Overly Conservative Relay Protection Settings “A few lines have zone 3 settings designed with overload margins close to the long-term emergency limit of the line. . . .Thus, it is possible for a zone 3 relay to operate on line load or overload in extreme contingency conditions even in the absence of a fault.” (P. 80) “Some affected TOs set overload relay protection settings on transformers just above the transformers’ emergency rating, resulting in facilities being automatically removed from service before TOPs have sufficient time to take control actions . . . following an N-1 contingency.” (Finding 25)

Cascading Relay Overload Trips “[B]ecause these zone 2 and 3 relays tripped after each line overloaded, these relays were the common mode of failure that accelerated the geographic spread of the cascade.” (P. 80) “Some TOs set relays to isolate facilities for loading conditions slightly above their thirty minute emergency ratings. As a result, several transmission lines and transformers tripped within seconds of exceeding their emergency ratings, leaving TOPs insufficient time to mitigate overloads.” (Finding 26)

Relay Protection Acting Too Quickly to Allow System Operators to Take Action “[T]he speed of the zone 2 and 3 operations across Ohio and Michigan eliminated any possibility . . . that either operator action or automatic intervention could have limited or mitigated the growing cascade.” (P. 80). “Some affected TOs set overload relay protection settings on transformers just above the transformers’ emergency rating, resulting in facilities being automatically removed from service before TOPs have sufficient time to take control actions...” (Finding 25)

“. . . several transmission lines

111 2003 Blackout Report at 82. Zone 3 relays “provide breaker failure and relay backup for remote distance

faults on a transmission line.” Id. at 80.



## - 129 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Protection Systems

Issue 2003 Blackout 2011 Blackout and transformers tripped within seconds of exceeding their emergency ratings, leaving TOPs insufficient time to mitigate overloads. (Finding 26)



### After seeing the consequences of conservative zone 3 settings, the 2003 Blackout Report recommended that “[i]ndu stry is to review zone 3 relays on lines of 230 kV and

higher.” 112 

### This inquiry’s report similarly recommends that Transmission Owners review their facilities’ overload relay prot ection settings to ensure the appropriate margin between relay settings and emergency ratings.

112 2003 Blackout Report at 158.



## - 130 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Appendix D: Benchmarking the Model

I. Introduction and Background

The inquiry’s Modeling and Simulation Te am replicated system conditions on September 8, 2011, and the events leading up to the blackout. The model reflects the state of the electric system before and during the event, with the real power output of generators dispatched to the values recorded in SCADA data. With any major event on the BPS, it is important to accurately mode l the system before and during the event in order to: (1) verify the Sequence of Events ; (2) support reconciliation of disparate measurement data; and (3) simulate and evaluate hypothetical scenarios, or “what-if”

scenarios.

In order to ensure the accuracy of these tasks, the Modeling and Simulation Team benchmarked the model to recorded SCADA and PMU measurements using the following guidelines. Key facilities and inte rfaces in the affected area were generally benchmarked to within 5% or 10 MVA accuracy to the measured data. Generator reactive outputs were also checked agains t recorded values to ensure that the representation of reactive power margin was reasonably accurate. The team also

monitored most other facilities in the affected area to ensure that the flows and voltage were reasonably close to measured data. Many of these other facilities also met the same guidelines used to benchmark the key facilities and interfaces.

The iterative process between benchmarking and case alteration has traditionally been time-consuming. The team pursued me thods that would ulti mately decrease the amount of time spent benchmarking so that results could quickly be used to identify problem areas in the case and make appropriate adjustments. Because the team received

SCADA and PMU measurement data from many sources and entities, the data was: (1) organized into a consistent format, useful for automated benchmarking; and (2) crosschecked and verified for accuracy. In organizing the data, the team also considered how each data point would map back to both po wer flow and dynamics results. The team ultimately achieved a single process to: (1) import power flow results; (2) import dynamics results; (3) compare the results to measured data from many sources at various quasi-steady state times during th e event; (4) export tables showing the percentage accuracy; and (5) export graphs sh owing the accuracy of the results relative to measured data throughout the event.



## - 131 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## II. Discussion

The locations and measurements that the team selected for benchmarking were naturally predicated on the available measurements. While the team compared each available data point to the model results, it did not benchmark the model to all available data points. Instead the team focused its benchmarking effort on a “study area” that included SDG&E, IID, the APS Yuma load po cket, and portions of CFE and SCE. The team gave preference to measurements that we re available in multiple data sources with some reasonable agreement between the different sources, and particular preference to

those locations where PMU measurements were available, because these measurements could also be benchmarked against a full dynamics simulation.

Following each set of simulations, the te am reviewed the benchmarking data both graphically and tabularly, and tuned the modeli ng case and simulation parameters in an attempt to bring the case closer to measured reality. The team would then re-run the simulation, and repeat this process.

Custom Interfaces Even though the team selected the best possible set of benchmarking data, and a substantial amount of work went into calibrating the study area of the modeling case to those measurements, inconsistencies between some data points persisted. These

inconsistencies arose due to the multitud e of subtle settings and parameters for equipment, such as a changed tap on a single transformer affecting reactive power flow. For this reason, the team developed “custom interfaces” to benchmark an aggregation of points. If an aggregated, modeled sub-system was very close to the actual measurements for that system, then the simulation could be trusted to accurately reflect the system. For example, if reactive power flow was misalloc ated to a pair of adjacent transformers sourcing a sub-system, the specific reactive flow on each transf ormer may not be of particular importance to the model. Howeve r, the reactive flow to the aggregate load being served by those transformers may havea significant impact on a neighboring subsystem, and be crucial to effective benchmarking.



## T he custom interfaces were also defined so as to indicate the amount of flow into or across a particular sub-system . For example, the calculated flows at the “IID North 92 kV System” interface give an idea of the amount and nature of the load in the northern IID 92 kV system. The custom interfaces selected include:

 IID North 92 kV System: All transmission sources for the nort hern IID 92 kV system, including the 230/92 kV transformers at Coachella Valley and Ramon, the 161/92 kV transformers at Coachella Valley and Avenue 58, and the 92 kV lines between the northern and southern IID systems.



## - 132 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

 IID South 92 kV System: All transmission sources for the so uthern IID 92 kV system, including the 230 kV transformer at El Centro, the 161/92 kV transformers at El Centro and Niland, and the 92 kV lines between the southe rn and northern IID systems.

 Yuma Pocket: Interfaces between the Yuma area 69 kV system (including portions of both APS and WALC service territories) and higher-voltage sy stems, including the 500/69 kV transformers at N. Gila, the 161/69 kV transformers at Gila, an d the 161 kV line from Pilot Knob to Yucca.

 Southwest California Desert Imports: All transmission sources into the IID/SDG&E/CFE/Yuma area other than Path 44.



### Figure 1: Key Facilities and Interfaces



## Key Facilities and Interfaces

The team chose key facilities and interfaces in the affected area as a way to quickly evaluate the model before fine-tuning it on a more granular level. These key

facilities and interfaces were benchmarked to within 5% or 10 MVA accuracy to the measured data throughout the entire event. The key facilities and interfaces are listed

below.

 WECC Path 44

 Southwest California Desert Imports

 IID Northern 92 kV System

 Niland-Blythe 161 kV Transmission Line

 IID Southern 92 kV System

 Imperial Valley-El Centro 230 kV Transmission Line (“S” Line)

 Miguel-Imperial Valley 500 kV Transmission Line

 Yuma Pocket

 El Centro-Pilot Knob 161 kV Transmission Line

 Pilot Knob-Knob 161 kV Transmission Line

 Pilot Knob-Yucca 161 kV Transmission Line

 Julian Hinds-Mirage 230 kV Transmission Line

 Julian Hinds-Eagle Mountain 230 kV Transmission Line



## - 133 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## III. Results

The following graphs demonstrate the benc hmarking results. Each plot gives both power flow (see “TSS” in graph legend) 113 

### and dynamic simulation (see “DYD” in

graph legend) 114 

### results at each selected time step, with the corresponding SCADA

and/or PMU measurement, as available. In some instances, known issues with

measured data are annotated on the charts, such as SCADA measurement errors for Coachella Valley during the interval following the initiating event.

The simulated MW values follow the measurements more closely than the simulated MVAR values. This is due to complexity involved in tuning voltage at each bus due to incomplete data, such as unknown tap values on large transformers. Overall, the

MVA values are within our be nchmarking guidelines.

The team also provided a table that compares : (1) the base case at 15:27:00 to the measured data; and (2) the case just prior to the loss of the Coachella Valley

transformers at 15:28:11 to the measured data. This table does not compare the dynamics values to the base case at 15:27:00 because the power flow base case was the foundation for the dynamics simulation , meaning the values would be equal.

113 Time Sequence Simulation.

114 Dynamics Data.



## - 134 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 135 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 136 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 137 - Differences due to

SCADA measurement

error at Coachella Valley and Ramon

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 138 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 139 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 140 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 141 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 142 - Difference due to

SCADA measurement captured during power swing

Difference due to

SCADA measurement captured during power swing

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 143 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 144 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 145 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 146 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## - 147 -

### FERC/NERC Staff Report on the September 8, 2011 Blackout

Key facilities and interfaces in the affe cted area were generally benchmarked to within 5% or 10 MVA accuracy to the measured data.

Base Case 15:28:11

Key Facility/ Interface Type Measured Power Flow Simulation Delta (Value) Delta (%) Measured Power Flow Simulation Delta (Value) Delta (%) Dynamics Simulation

MVA 1310.25 1323.61 -13.36 -1.02% 2453.57 2471.50 -17.93 -0.73% 2457.19

MW 1296.85 1292.09 4.77 0.37% 2434.48 2452.40 -17.92 -0.74% 2442.90 WECC Path 44

MVAR -186.93 -287.17 100.24 -53.63% -305.49 -306.68 1.19 -0.39% -264.64

MVA 1328.45 1336.38 -7.93 -0.60% 333.50 349.91 -16.41 -4.92% 347.90

MW 1301.35 1307.75 -6.40 -0.49% 310.76 316.68 -5.92 -1.90% 310.03 Southwest California Desert Imports MVAR 266.96 275.16 -8.19 -3.07% -121.02 -148.82 27.80 -22.97% -157.84

MVA 475.15 476.81 -1.66 -0.35% 416.84 480.17 - 63.32 115 - 15.19% 116 466.78

MW 471.60 473.23 -1.63 -0.35% 414.58 477.61 -63.04 -15.21% 464.39 IID North 92 kV System

MVAR -58.01 -58.31 0.30 -0.52% -43.43 -49.46 6.03 -13.88% -47.15

MVA 67.49 69.45 -1.97 -2.92% 108.41 118.34 -9.93 -9.16% 117.37

MW -65.13 -65.99 0.86 -1.32% -96.10 -100.93 4.83 -5.02% -101.43 Niland - Blythe 161 kV Line MVAR 17.68 21.67 -3.99 -22.56% 50.17 61.79 -11.62 -23.16% 59.07

MVA 105.60 110.97 -5.37 -5.08% 114.63 132.43 - 17.80 117 - 15.53% 118 124.85

MW 104.67 110.91 -6.24 -5.96% 106.14 114.84 -8.70 -8.20% 108.87 IID South 92 kV System

MVAR -14.03 -3.73 -10.30 73.42% -43.29 -65.93 22.65 -52.31% -61.12

MVA 96.08 105.20 -9.12 -9.49% 125.31 119.52 5.79 4.62% 302.90

MW 94.24 104.76 -10.53 -11.17% -109.22 -101.41 -7.81 7.15% 302.90 Imperial Valley - El Centro 230 kV Line (“S” Line) MVAR -18.75 -9.58 -9.17 48.89% 61.42 63.26 -1.83 -2.98% 0.05

MVA 1088.80 1095.22 -6.42 -0.59% 225.19 214.12 11.07 4.91% 77.98

MW -1087.30 -1093.10 5.80 -0.53% -188.50 -191.82 3.32 -1.76% 77.31 Miguel - Imperial Valley 500 kV Line MVAR 57.18 68.13 -10.94 -19.14% -123.20 -95.15 -28.05 22.77% -10.26

MVA 280.78 281.61 -0.83 -0.30% 279.55 283.18 -3.63 -1.30% 282.79

MW 280.11 281.41 -1.30 -0.46% 278.63 282.81 -4.18 -1.50% 281.63 Yuma Pocket

MVAR -19.46 -10.70 -8.76 45.00% -22.66 -14.46 -8.20 36.20% -25.68

115 Large differences due to SCADA measurement errors at Coachella Valley and Ramon

116 Id.

117 The team experienced difficulty in calibrating the MVAR flows in this area, but are generally confident in the benchmarking because the MW values are within 10 MW. The MVA differences in the model appear to increase during this event. The representation of the system in this area of the model appears to assum e that the IID

South 92 kV system is a load serving local network. However, the actual transmission system operates in parallel

with the rest of the BPS. It was difficult to calibrate the flows at the 92 kV to 161 kV interfaces because of the

differences between the representation of the system in the model versus th e parallel nature of the actual system.

118 Id.



## 148

### FERC/NERC Staff Report on the September 8, 2011 Blackout

MVA 9.15 9.83 -0.69 -7.51% 18.27 21.17 -2.90 -15.87% 18.51

MW -8.90 -8.60 -0.30 3.40% 15.73 17.04 -1.31 -8.33% 16.10 El Centro - Pilot Knob 161 kV Line MVAR 2.09 4.76 -2.68 - 127.99% 9.29 12.56 -3.27 -35.17% 9.14

MVA 74.27 68.50 5.77 7.76% 135.67 140.14 -4.46 -3.29% 141.54

MW -74.06 -66.62 -7.44 10.05% -120.25 -116.14 -4.11 3.42% -118.13 Pilot Knob - Knob 161 kV Line MVAR 5.54 15.94 -10.40 - 187.80% 62.83 78.42 -15.59 -24.81% 77.96

MVA 47.34 41.17 6.18 13.05% 132.82 128.54 4.28 3.22% 130.96

MW 46.92 40.94 5.98 12.74% 127.39 121.67 5.72 4.49% 122.63 Pilot Knob - Yucca 161 kV Line MVAR 6.33 4.30 2.03 32.04% -37.60 -41.46 3.87 -10.28% -45.95

MVA 291.92 287.50 4.42 1.51% 273.49 276.69 -3.21 -1.17% 276.32

MW 291.87 287.49 4.38 1.50% 273.46 276.67 -3.21 -1.18% 276.29 Julian Hinds - Mirage 230 kV Line MVAR -5.45 -2.12 -3.33 61.10% -3.98 -3.48 -0.49 12.43% 4.45

MVA 55.47 56.59 -1.12 -2.01% 71.70 68.07 3.63 5.06% 66.96

MW 53.29 55.35 -2.07 -3.88% 71.06 65.89 5.18 7.28% 66.26 Julian Hinds - Eagle Mountain 230 kV Line MVAR -15.43 -11.78 -3.64 23.62% 9.56 17.13 -7.57 -79.17% 9.64



## 149

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## Appendix E: Inquiry Team Members

FERC Staff

Office of Enforcement

Heather Polzin Jeremy Medovoy

Catherine Collins

Samuel Backfield

Thomas Lemon Cherise Ojo

Office of Electric Reliability Alan Phung

Alireza Ghassemian Boris Voynik

David Burnham Eddy Lim

Gilbert Lowe

Jacob Lucas John Spivak

Ken Githens

Kent Davis

Leonard Chamberlin

Louise Nutter Mahmood Mirheydar

Michelle Veloso

Monica Taba

Pablo Ovando Perry Servedio

Sasan Jalali Terrance Clingan

Terrence Simon

Thomas Reina Victor Barry

Office of Energy Policy & Innovation Mary Cain



## 150

### FERC/NERC Staff Report on the September 8, 2011 Blackout



## 151 NERC Staff

Ben McMillan Bob Cummings

Chris McManus

Dave Nevius Dmitry Kosterev (Technical Consultant, from BPA) Earl Shockley

Ed Ruck

Eric Allen Greg Henry

James Merlo

Jim Griffith

Jim Robinson

Jule Tate Kimberly Mielcarek

Mark Vastano

Phil Tatro Phil Winston (Technical Consultant, from Southern Company)

Roman Carter Terry Brinker

Department of Energy Liason

James McGlone

Nuclear Regulatory Commission Staff Singh Matharu