i
i

TABLE OF CONTENTS

I. EXECUTIVE SUMMARY .......................................................................................- 1 -

A. SYNOPSIS OF THE DISTURBANCE AND SYSTEM RECOVERY.................................- 1 -

B. MAP OF AFFECTED AREA AND KEY FACILITIES INVOLVED IN THE EVENT ............- 3 -

C. KEY FINDINGS, CAUSES, AND RECOMMENDATIONS..........................................- 5 -

II. INTRODUCTION ............................................................................................- 10 -

A. INQUIRY PROCESS .....................................................................................- 10 -

B. SYSTEM OVERVIEW.................................................................................... - 15 -

III. SEQUENCE OF EVENTS ..................................................................................- 23 -

A. PHASE 1: PRE-DISTURBANCE CONDITIONS...................................................- 24 -

B. PHASE 2: TRIP OF THE HASSAYAMPA-NORTH GILA 500 KV LINE ....................- 30 -

C. PHASE 3: TRIP OF THE COACHELLA VALLEY 230/92 KV TRANSFORMER AND

VOLTAGE DEPRESSION...............................................................................- 35 -

D. PHASE 4: TRIP OF RAMON 230/92 KV TRANSFORMER AND COLLAPSE OF IID’S 

NORTHERN 92 KV SYSTEM .........................................................................- 40 -

E. PHASE 5: YUMA LOAD POCKET SEPARATES FROM IID AND WALC ..................- 44 -

F. PHASE 6: HIGH-SPEED CASCADE, OPERATION OF THE SONGS SEPARATION 

SCHEME AND ISLANDING OF SAN DIEGO,IID, CFE, AND YUMA.......................- 48 -

G. PHASE 7: COLLAPSE OF THE SAN DIEGO/CFE/YUMA ISLAND .........................- 54 -

H. SYSTEM RESTORATION............................................................................... - 61 -

IV. CAUSES, FINDINGS, AND RECOMMENDATIONS ................................................- 64 -

A. PLANNING ................................................................................................- 64 -

1. NEXT-DAY PLANNING ............................................................................- 64 -

2. SEASONAL PLANNING .............................................................................- 73 -

3. NEAR- AND LONG-TERM PLANNING ........................................................ - 80 -

B. SITUATIONAL AWARENESS..........................................................................- 86 -

C. SYSTEM ANALYSIS .....................................................................................- 96 -

1. CONSIDERATION OF BES EQUIPMENT.......................................................- 97 -

2. IROL DERIVATIONS ...............................................................................- 98 -

3. IMPACT OF PROTECTION SYSTEMS ON EVENT............................................- 101 -

4. ANGULAR SEPARATION .......................................................................... - 111 -

APPENDICES 114
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 1 -

I. EXECUTIVE SUMMARY

A. Synopsis of the Disturbance and System Recovery

On the afternoon of September 8, 2011, an 11-minute system disturbance 

occurred in the Pacific Southwest, leading to cascading outages and leaving 

approximately 2.7 million customers without power.0F

1 The outages affected parts of 

Arizona, Southern California, and Baja California, Mexico. All of the San Diego area lost 

power, with nearly one-and-a-half million customers losing power, some for up to 12

hours. The disturbance occurred near rush hour, on a business day, snarling traffic for 

hours. Schools and businesses closed, some flights and public transportation were 

disrupted, water and sewage pumping stations lost power, and beaches were closed due 

to sewage spills. Millions went without air conditioning on a hot day. 

The loss of a single 500 kilovolt (kV)1F

2 transmission line initiated the event, but 

was not the sole cause of the widespread outages. The system is designed, and should be 

operated, to withstand the loss of a single line, even one as large as 500 kV. The affected

line—Arizona Public Service’s (APS) Hassayampa-N. Gila 500 kV line (H-NG)—is a 

segment of the Southwest Power Link (SWPL), a major transmission corridor that 

transports power in an east-west direction, from generators in Arizona, through the 

service territory of Imperial Irrigation District (IID), into the San Diego area. It had 

tripped on multiple occasions, as recently as July 7, 2011, without causing cascading 

outages.

With the SWPL’s major east-west corridor broken by the loss of H-NG, power 

flows instantaneously redistributed throughout the system, increasing flows through 

lower voltage systems to the north of the SWPL, as power continued to flow into San

Diego on a hot day during hours of peak demand. Combined with lower than peak

 1 “Customers” are not the same as “people” in utility parlance. The term customer generally refers to a single 

meter, whether at a residence, an apartment building, or a factory. Thus, a single customer could represent one 

or more persons, and a single person could be two customers, for example, if the same utility served both an 

individual’s residence and his small business. Estimates of “people” affected by blackouts generally are prepared 

by increasing the customer numbers by a multiplier, often two or three.

2 A list of acronyms used in this report is included in Appendix A.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 2 -

generation levels in San Diego and Mexico,2F

3 this instantaneous redistribution of power 

flows created sizeable voltage deviations and equipment overloads to the north of the 

SWPL. Significant overloading occurred on three of IID’s 230/92 kV transformers 

located at the Coachella Valley (CV) and Ramon substations, as well as on Western 

Electricity Coordinating Council (WECC) Path 44,3F

4 located south of the San Onofre 

Nuclear Generating Station (SONGS) in Southern California. 

The flow redistributions, voltage deviations, and resulting overloads had a ripple 

effect, as transformers, transmission lines, and generating units tripped offline, initiating

automatic load shedding throughout the region in a relatively short time span. Just 

seconds before the blackout, Path 44 carried all flows into the San Diego area as well as 

parts of Arizona and Mexico. Eventually, the excessive loading on Path 44 initiated an 

intertie separation scheme at SONGS, designed to separate SDG&E from SCE. The 

SONGS separation scheme separated SDG&E from Path 44, led to the loss of the SONGS

nuclear units, and eventually resulted in the complete blackout of San Diego and 

Comisión Federal de Electricidad’s (CFE) Baja California Control Area. During the 11 

minutes of the event, the WECC Reliability Coordinator (WECC RC) issued no directives 

and only limited mitigating actions were taken by the Transmission Operators (TOPs) of 

the affected areas. 

As a result of the cascading outages stemming from this event, customers in the 

SDG&E, IID, APS, Western Area Power Administration-Lower Colorado (WALC), and 

CFE territories lost power, some for multiple hours extending into the next day. 

Specifically,

• SDG&E lost 4,293 Megawatts (MW) of firm load, affecting approximately 1.4 million customers.

• CFE lost 2,150 MW of net firm load, affecting approximately 1.1 million customers.4F

5

• IID lost 929 MW of firm load, affecting approximately 146,000 customers.

 3 Total summer peak generation for San Diego Gas and Electric’s (SDG&E) territory and Comisión Federal de 

Electricidad’s (CFE) Baja California Control Area is 5,774 MW. On September 8, 2011, the total generation for 

SDG&E and CFE’s Baja California Control Area was 4,168, a difference of 1,606 MW.

4 Path 44 is one of 81 Rated Paths in the WECC region. A Rated Path is composed of “an individual transmission 

line or a combination of parallel transmission lines.” WECC 2011 Path Rating Catalog, January 2011, at item 1-i. Path 

44, also referred to as “South of SONGS,” is an aggregation of five 230 kV lines that delivers power in a northsouth direction from the Southern California Edison (SCE) footprint in the Los Angeles area into the SDG&E 

footprint.

5 CFE is Mexico’s state-owned utility. Only its Baja California Control Area was affected on September 8, 2011. 

The inquiry is particularly grateful to CFE for its willingness to share data and information to assist the inquiry in 

developing the most accurate conclusions and recommendations.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 3 -

• APS lost 389 MW of firm load, affecting approximately 70,000 customers.

• WALC lost 74 MW of firm load, 64 MW of which affected APS’s customers. The remaining 10 MW 

affected 5 WALC customers.

After the blackout, the affected entities promptly instituted their respective 

restoration processes.5F

6 All of the affected entities had access to power from their own or 

neighboring systems and, therefore, did not need to use “black start” plans.6F

7 Although 

there were some delays in the restoration process due to communication and 

coordination issues between entities, the process was generally effective. SDG&E took 12

hours to restore 100% of its load, and CFE took 10 hours to restore 100% of its load. IID, 

APS, and WALC restored power to 100% of their customers in approximately 6 hours. 

The affected entities also worked to restore generators and transmission lines that 

tripped during the event. IID and APS restored generation—333 MW for IID and 76 MW 

for APS—in 5 hours. Meanwhile, CFE restored 1,915 MW of tripped generation in 56

hours; SDG&E restored 2,229 MW of tripped generation in 39 hours; and SCE restored 

2,428 MW of tripped generation in 87 hours. IID restored its 230 kV transmission 

system in 12 hours and its 161 kV system in 9 hours; APS restored H-NG in 2 hours; 

SDG&E restored its 230 kV system in 12 hours; WALC restored its 161 kV system in 1.5

hours; and CFE restored its 230 kV system in 13 hours and its 115 kV system in 10 hours.

B. Map of Affected Area and Key Facilities Involved in the Event

The following map, showing the areas affected by the September 8th event and 

the key facilities involved during the event, can be used as a reference throughout the 

report:

 6 The term “affected entities” in this report refers to TOPs and Balancing Authorities (BAs) that were affected by 

the event. The affected entities include SDG&E, IID, APS, WALC, SCE, CFE, and the California Independent System 

Operator (CAISO).

7 Black start plans work to energize systems using internal generation to get from shutdown to operating 

condition without assistance from the Bulk Electric System (BES).
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 4 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 5 -

C. Key Findings, Causes, and Recommendations7F

8

The September 8, 2011, event showed that the system was not being operated in a 

secure N-1 state.8F

9 This failure stemmed primarily from weaknesses in two broad areas—

operations planning and real-time situational awareness—which, if done properly, would

have allowed system operators to proactively operate the system in a secure N-1 state 

during normal system conditions and to restore the system to a secure N-1 state as soon 

as possible, but no longer than 30 minutes. Without adequate planning and situational 

awareness, entities responsible for operating and overseeing the transmission system 

could not ensure reliable operations within System Operating Limits (SOLs) or prevent 

cascading outages in the event of a single contingency.9F

10 As demonstrated in Appendix 

C, inadequate situational awareness and planning were also identified as causes of the 

2003 blackout that affected an estimated 50 million people in the United States and 

Canada.

The inquiry also identified other underlying factors that contributed to the event, 

including: (1) not identifying and studying the impact on Bulk-Power System (BPS)10F

11

 8 While this section highlights the most significant causes, findings, and recommendations, the report details the 

complete list of findings, causes, and recommendations in section IV. In addition, for ease of reference all of the 

findings and recommendations are summarized in table format in Appendix B.

9 The North American Electric Reliability Corporation’s (NERC) mandatory Reliability Standards applicable to the 

BES require that the BES be operated so that it generally remains in a reliable condition, without instability, 

uncontrolled separation or cascading, even with the occurrence of any single contingency, such as the loss of a 

generator, transformer, or transmission line. This is commonly known as the “N-1 criterion.” N-1 contingency 

planning allows entities to identify potential N-1 contingencies before they occur and to adopt mitigating 

measures, as necessary, to prevent instability, uncontrolled separation, or cascading. As the Federal Energy 

Regulatory Commission (Commission) stated in Order No. 693 with regard to contingency planning, “a single 

contingency consists of a failure of a single element that faithfully duplicates what will happen in the actual 

system. Such an approach is necessary to ensure that planning will produce results that will enhance the 

reliability of that system. Thus, if the system is designed such that failure of a single element removes from 

service multiple elements in order to isolate the faulted element, then that is what should be simulated to assess 

system performance.” Mandatory Reliability Standards for the Bulk Power System, Order No. 693, FERC Stats. &

Regs. ¶ 31,242, at P 1716 (2007), order on reh’g, Mandatory Reliability Standards for the Bulk-Power System, 120 

FERC ¶ 61,053 (Order No. 693-A) (2007).

10 A contingency is the unexpected failure of an electrical system component.

11 The BPS is defined by Section 215(a) (1) of the Federal Power Act as “facilities and control systems necessary 

for operating an interconnected electric energy transmission network (or any portion thereof), and electric 

energy from generating facilities needed to maintain transmission system reliability.” The meaning of BPS and 

BES differ somewhat and, thus, this report uses each term in its proper context. With respect to reliability, the 

Commission has jurisdiction over all users, owners, and operators of the BPS. In Order No. 693 at P 75, the 

Commission adopted, at least for an initial period, the BES definition as the threshold for application of the NERC

Reliability Standards. Thus, this report uses BES when referring to entities’ specific facilities or elements that are 

subject to the Reliability Standards, but BPS when discussing the overall reliability impact. On January 25, 2012, 

NERC filed a petition with the Commission for approval of a revised definition of the BES. The proposed definition 

of BES would cover all elements operated at 100 kV or higher, with a list of specific inclusions and exclusions. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 6 -

reliability of sub-100 kV facilities in planning and operations;11F

12 (2) the failure to 

recognize Interconnection Reliability Operating Limits (IROLs) in the Western 

Interconnection;12F

13 (3) not studying and coordinating the effect of protection systems, 

including Remedial Action Schemes (RASs), during plausible contingency scenarios;13F

14

and (4) not providing effective tools and operating instructions for use when reclosing 

lines with large phase angle differences across the reclosing breakers.14F

15 

With regard to operations planning, some of the affected entities’ seasonal, nextday, and real-time studies do not adequately consider: (1) operations of facilities in 

external networks, including the status of transmission facilities, expected generation 

output, and load forecasts; (2) external contingencies that could impact their systems or 

internal contingencies that could impact their neighbors’ systems; and (3) the impact on 

BPS reliability of internal and external sub-100 kV facilities. As a result, these entities’ 

operations studies did not accurately predict the impact of the loss of APS’s H-NG or the 

loss of IID’s three 230/92 kV transformers. If the affected entities had more accurately 

predicted the impact of these losses prior to the event, these entities could have taken

appropriate pre-contingency measures, such as dispatching additional generation to 

mitigate overloads and prevent cascading outages. 

To improve operations planning in the WECC region, this report makes several 

recommendations designed to ensure that TOPs and BAs,15F

16 as appropriate: (1) obtain 

information on the operations of neighboring BAs and TOPs, including transmission 

outages, generation outages and schedules, load forecasts, and scheduled interchanges; 

(2) identify and plan for external contingencies that could impact their systems and 

 Transformers with the primary terminal and at least one secondary terminal operated at 100 kV or higher are on 

the list of specific inclusions. See North American Electric Reliability Corp., Docket No. RM12-6-000. This report 

takes no position on the petition.

12 This report does not attempt to define the limits of which sub-100 kV facilities impact BPS reliability. Certainly, 

many facilities below 100 kV do not impact BPS reliability. The sub-100 kV facilities in this event affected the BPS 

because they were in parallel to significant transmission corridors.

13 This report recommends that WECC RC should work with TOPs to consider whether any SOLs in the Western 

Interconnection constitute IROLs. As part of this effort, WECC RC should: (1) work with affected TOPs to 

consider whether Path 44 and H-NG should be recognized as IROLs; and (2) validate existing SOLs and ensure that 

they take into account all transmission and generation facilities and protection systems that impact BPS reliability.

14 This failure caused the derived SOLs on H-NG and Path 44 to be invalid on the day of the event.

15 As discussed in more detail in connection with Finding and Recommendation 27 below, when a line trips, the 

phase angle at one end of the line may be much larger than the phase angle at the other end. If the difference 

between the two angles is too great, reclosing the line could cause damage to generators or even system 

instability. 

16 See “Reliability Responsibilities” section at page 16 below.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 7 -

internal contingencies that could impact their neighbors’ systems; and (3) consider 

facilities operated at less than 100 kV that could impact BPS reliability. This effort 

should include a coordinated review of planning studies to ensure that operation of the 

affected Rated Paths will not result in the loss of non-consequential load, system 

instability, or cascading outages, with voltage and thermal limits within applicable 

ratings for N-1 contingencies originating from within or outside an entity’s footprint.

The September 8th event also exposed entities’ lack of adequate real-time 

situational awareness of conditions and contingencies throughout the Western 

Interconnection. For example, many entities’ real-time tools, such as State Estimator 

and Real-Time Contingency Analysis (RTCA), are restricted by models that do not 

accurately or fully reflect facilities and operations of external systems to ensure 

operation of the BPS in a secure N-1 state. Also, some entities’ real-time tools are not 

adequate or operational to alert operators to significant conditions or potential 

contingencies on their systems or neighboring systems. The lack of adequate situational 

awareness limits entities’ ability to identify and plan for the next most critical 

contingency to prevent instability, uncontrolled separation, or cascading outages. If 

some of the affected entities had been aware of real-time external conditions and run (or 

reviewed) studies on the conditions prior to the onset of the event, they would have been 

better prepared for the impacts when the event started and may have avoided the 

cascading that occurred. 

To improve situational awareness in the WECC region, this report makes several 

recommendations: (1) expand entities’ external visibility in their models through, for 

example, more complete data sharing; (2) improve the use of real-time tools to ensure 

the constant monitoring of potential internal or external contingencies that could affect 

reliable operations; and (3) improve communications among entities to help maintain 

situational awareness. In addition, TOPs should review their real-time monitoring tools, 

such as State Estimator and RTCA, to ensure that such tools represent critical facilities 

needed for the reliable operation of the BPS. These improvements will enable system 

operators to utilize real-time operating tools to proactively operate the system in a secure 

N-1 state. 

In addition to the planning and situational awareness issues, several other factors 

contributed to the September 8th event. For example, WECC RC and affected entities do 

not consistently recognize the adverse impact that sub-100 kV facilities can have on BPS 

reliability. The prevailing SOLs should have included the effects of facilities that had not 

been identified and classified as part of the BES, as well as the effects of critical facilities 

such as Special Protection Systems (SPSs) and the SONGS separation scheme. Relevant 

to the event, these entities did not consider IID’s 92 kV network and facilities, including 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 8 -

the CV and Ramon 230/92 kV transformers, as part of the BES, despite some previous 

studies indicating their impact on the BPS due to the fact they were electrically in 

parallel with higher-voltage facilities.16F

17 If these facilities had been designated as part of 

the BES, or otherwise incorporated into planning and operations studies and actively 

monitored and alarmed in RTCA systems, the cascading outages may have been avoided. 

Accordingly, the inquiry makes a recommendation to ensure that facilities that can 

impact BPS reliability, regardless of voltage level, are considered for classification as part 

of the BES and/or studied as part of entities’ planning in various time horizons. 

The inquiry also found some significant issues with protection system settings

and coordination. For example, IID used conservative overload relay trip settings on its

CV transformers. The relays were set to trip at 127% of the transformers’ normal rating, 

which is just above the transformers’ emergency rating (110% of normal rating). Such a 

narrow margin between the emergency rating and overload trip setting resulted in the 

facilities being automatically removed from service without providing operators enough 

time to mitigate the overloads. As a result of these settings, both CV transformers 

tripped within 40 seconds of H-NG tripping, initiating cascading outages. To avoid a 

similar problem in the future, the inquiry recommends that IID and other Transmission 

Owners (TOs) review their transformers’ overload protection relay settings. A good 

guideline for protective relay settings is Reliability Standard PRC-023-1 R1.11, which 

states that relays be “set to allow the transformer to be operated at an overload level of at 

least 150% of the maximum applicable nameplate rating, or 115% of the highest operator 

established emergency transformer rating, whichever is greater.” TOPs should also plan 

to take proper pre-contingency mitigation measures with due consideration for the 

applicable emergency ratings and overload protection settings (MW and time delay) 

before a facility loads to its relay trip point and is automatically removed from service.

The SONGS separation scheme’s operation provides another example of the lack 

of studies on, and coordination of, protection systems. This scheme, classified by SCE as 

a “Safety Net,”17F

18 had a significant impact on BPS reliability, separating SDG&E from 

SCE, resulting in the loss of both SONGS nuclear generators, and blacking out SDG&E 

 17 See, e.g., CFE’s Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19.

18 A Safety Net protection system protects the power system from unexpected, low-probability events that are 

outside the normal planning criteria, but which may lead to a complete system collapse. Safety Nets operate to 

minimize the severity of the event and attempt to prevent a system collapse or cascading outages. A Safety Net 

is typically intended to handle severe disturbances resulting from extreme, though perhaps not well-defined, 

events. A Safety Net is subject to review by the WECC Remedial Action Scheme Reliability Subcommittee if 

unintended operation would result in cascading or other performance standard violations. WECC Guideline: 

Remedial Action Scheme Classification, February 9, 2009.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 9 -

and CFE. Nevertheless, none of the affected entities, including SCE, as the owner and 

operator of the scheme, studied its impact on BPS reliability. The September 8th event 

shows that all protection systems and separation schemes, including Safety Nets, RASs, 

and SPSs, should be studied and coordinated periodically to understand their impact on 

BPS reliability to ensure their operation, inadvertent operation, or misoperation does not 

have unintended or undesirable effects. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 10 -

II. INTRODUCTION

A. Inquiry Process

On September 9, 2011, the Commission and NERC jointly announced an inquiry

to determine the causes of the outages and make recommendations for preventing such 

events in the future. The purpose of the inquiry was not to determine whether there may 

have been violations of applicable regulations, requirements, or standards subject to the 

Commission’s jurisdiction. Thus, while this report describes conduct which may warrant 

future investigations under Part 1b of the Commission’s regulations,18F

19 or actions by 

NERC under its Compliance Monitoring and Enforcement Program,19F

20 it draws no 

conclusions about whether violations occurred.

The inquiry was composed of smaller teams with particular subject-matter 

expertise, primarily from Commission and NERC professional staff, each of which

conducted rigorous analyses of a key issue or issues involved in the event. Those teams 

and their primary responsibilities were as follows:

• Sequence of Events – developed a precise and accurate sequence of events (SOE) to provide a 

foundation for root cause analysis, computer model simulations, and other analytical aspects of the 

inquiry.

• System Modeling and Simulation – developed an accurate system modeling case, 

benchmarked the case to actual conditions at critical times, replicated system conditions leading up 

to and during the outage, and simulated alternate “what if” scenarios. 

• Root Cause and Human Performance Analysis – performed in a systematic evaluation 

of the root causes and contributing factors and identified areas requiring further inquiry.

• Operations Tools, Supervisory Control and Data Acquisition

(SCADA)/Energy Management System (EMS), Communications, and 

Operations Planning – considered all aspects of the blackout related to operator and 

reliability coordinator knowledge of system conditions, actions or inactions, and communications, 

particularly the observability of the electric system and effectiveness of operational reliability 

assessment tools.

• Frequency/Area Control Error (ACE) Analysis – reviewed potential frequency 

anomalies related to the blackout, and analyzed underfrequency generator, load, and tie line 

tripping.

 19 18 C.F.R. Part 1b (2011).

20 NERC Compliance Monitoring and Enforcement Program, Appendix 4C to the NERC Rules of Procedure, 

January 31, 2012.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 11 -

• System Planning, Design, and Studies – analyzed factors used in setting SOLs and actual 

limits in effect on the day of the blackout, determined whether those limits were exceeded, and 

analyzed the extent to which actual system conditions varied from the assumptions used in setting 

the SOLs.

• Transmission and Generation Performance, Protection, Control, 

Maintenance, and Damage – analyzed the causes of automatic facility operations and 

generator trips, analyzed transmission and generation facility maintenance practices, and identified 

equipment damage.

• Restoration Review – reviewed the appropriateness and effectiveness of the restoration plans 

implemented, as well as the effectiveness of the coordination of these plans among the affected 

entities and WECC RC.

Each team not only examined its own subject area to determine what may have 

contributed to the event, but also considered lessons learned and potential 

recommendations for preventing such events in the future.

The inquiry devoted substantial time and resources to determine and study the 

causes of the event and develop meaningful recommendations with the goal of 

preventing similar events in the future. The team’s analyses were extensive, involving 

the review of high-quality data from various reliability entities in the WECC region and 

simulations of the event using sophisticated computer models. Described below in 

summary form are the primary steps the inquiry took to complete its analysis.

Data Gathering

The inquiry received and reviewed more than 20 gigabytes of data from 

approximately 500 data requests sent to entities in and around the affected areas. On 

September 19, 2011, the inquiry also began site visits with various entities involved in the 

outages, including entities with responsibility for balancing load and generation, 

transmission operation, and reliability coordination. During the site visits, the inquiry

toured control centers, conducted dozens of interviews and depositions, and viewed 

equipment involved in the event. These visits and depositions allowed the inquiry to 

learn about control room operations and practices, system status and conditions on the 

day of the event, operating procedures, planning, operations, and real-time tools, and 

restoration planning and procedures. The inquiry also conducted dozens of follow-up 

meetings and issued follow-up data requests. 

Of particular use to the inquiry were phasor measurement unit (PMU) records. 

PMUs are complex, multi-functional, high resolution recording devices installed widely 

throughout the Western Interconnection pursuant to a voluntary WECC-wide initiative. 

PMUs provide continuous, high-speed (30 scans per second) records of system 

conditions, including frequency, voltage, and phase angle relations. The continuous 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 12 -

nature of the data available through the PMUs, as well as their wide distribution 

throughout the power system, proved especially valuable to the inquiry in forming an 

accurate picture of the SOE and state of the system at particular points in time

throughout the disturbance. 

SOE Methodology

More than 100 notable events occurred in less than 11 minutes on September 8, 

2011. The inquiry’s SOE team established a precise and accurate sequence of outagerelated events to form a critical building block for the other parts of the inquiry. It 

provided, for example, a foundation for the root cause analysis, computer-based 

simulations, and other event analyses. Although entities time-stamp much of the data 

related to specific events, their time-stamping methodologies vary, and not all of the 

time-stamps were synchronized to the National Institute of Standards and Technology 

(NIST) standard clock in Boulder, Colorado. Validating the precise timing of specific 

events became a time-consuming, important, and sometimes difficult task. The 

availability of global positioning system (GPS)-time synchronized PMU data on 

frequency, voltage, and related power angles made this task much easier than in previous 

blackout inquiries and investigations. 

To develop the SOE, the SOE team started by resolving discrepancies between the 

multiple sources of data, sign convention inconsistencies, and incorrect data. The SOE 

team then developed an events database starting with all known events and times. 

Initial sources for the development of the database included preliminary reports filed by 

the affected entities as well as initial responses to data requests. The team then 

examined each record in the database to verify event times using available SCADA and 

PMU data. As the frequency, line flow, or voltage data suggested that additional events 

might have occurred on the system, the team added other possible events and verified 

them through additional data requests.

The SOE team developed multiple iterations of an SOE narrative document based 

on the database and the available SCADA and PMU data. Some iterations of the SOE 

narrative required that more data be requested of affected entities, and ultimately 

multiple data requests were sent to each entity. After the team completed the SOE 

narrative, the inquiry’s Modeling and Simulation team verified the SOE using power 

flow, voltage stability, and dynamic stability analyses. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 13 -

Power Flow and Dynamics Analysis

The inquiry’s Modeling and Simulation team, after validating the SOE,

considered several “what if” scenarios. The Modeling and Simulation team’s work is 

described in more detail in Appendix D. Power flow analyses study power systems under 

quasi-steady-state conditions by matching load and generation to obtain voltage 

magnitude and angle at each bus and the real and reactive power flowing through each 

transmission facility. Dynamic stability analyses study the impact of disturbances on 

frequency, voltage, and rotor angle stability, and determine whether transients in the 

power system are stable, thus allowing the power system to return to a quasi-steady-state 

operating condition following a disturbance.20F

21 

As the first step in performing power flow and dynamic stability analyses, the 

Modeling and Simulation team developed and benchmarked a modeling case of system 

conditions prior to the event. The team started with the WECC heavy summer base case 

and made adjustments based on State Estimator snapshots, EMS data, actual generation 

and schedules, PMU data, and a base case prepared by a separate team (led by CAISO)

that studied the event. The team further adjusted and benchmarked the base case using

SCADA and PMU data to match the system conditions for the entire event. The team 

devoted considerable time and effort to resolving discrepancies between the various 

sources of data to best calibrate the modeling case to actual measured data. As 

illustrated by Figure 1, on the next page, and described in more detail in Appendix D, 

the Modeling and Simulation team achieved a significant degree of accuracy. This figure 

compares Path 44 flows simulated by the Modeling and Simulation team to actual Path 

44 PMU data.

After developing and benchmarking a valid case, the Modeling and Simulation

team simulated the entire SOE using both power flow and dynamic simulations. This 

replication of the SOE established the validity of the model and enabled meaningful 

simulation of several alternative scenarios, developed to answer “what if” questions 

regarding the event. For example, the inquiry considered what would have happened if 

some of the affected entities had dispatched generation at certain locations during the 

event, if overload relays had been set at different levels, or if RASs, Safety Nets, or other 

SPSs had been designed or operated differently.

 21 Transient stability refers to the ability of synchronous generators to move to a new quasi-steady-state 

operating point while remaining synchronized after the system experiences a disturbance.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 14 -

Outreach Sessions

After developing a list of preliminary findings and recommendations, the inquiry

conducted outreach meetings with various industry associations and groups, including 

CAISO, WECC, the American Public Power Association (APPA), the North American 

Transmission Forum (NATF), the Edison Electric Institute (EEI), the National Rural 

Electric Cooperative Association (NRECA), and representatives from Regional Entities 

(REs), Regional Transmission Organizations, and Independent System Operators. Team 

members shared the inquiry’s preliminary findings and recommendations on a nonpublic basis with members of these organizations to obtain feedback and, with respect to 

the recommendations, input as to their practicality and feasibility. The inquiry 

considered the feedback and input provided by these organizations and incorporated 

much of it into the findings and recommendations included in this report.

Figure 1: Comparison of Actual and Simulated Path 44 Flows
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 15 -

B. System Overview

This subsection provides an overview of: (1) the Western Interconnection and its 

position in the North American electric grid; (2) the reliability entities responsible for 

operating the grid; (3) a description of the affected entities; and (4) a discussion of the 

interconnected nature of these entities.

The Western Interconnection and Its Position in the North American

Electric Grid

NERC shares its mission of ensuring the reliability of the BPS in North America 

with eight REs through a series of delegation of authority agreements. WECC is the 

designated RE responsible for coordinating and promoting BPS reliability in the Western 

Interconnection. In its capacity as the RE, WECC monitors and enforces compliance 

with Reliability Standards by the users, owners, and operators of the BPS. WECC also 

functions as an Interconnection-wide planning facilitator, aiding in transmission and 

resource integration planning at the request of its members, as well as a provider of data, 

analysis, and studies related to transmission planning and reliability issues.

The WECC region extends from Canada to Mexico. It includes the provinces of 

Alberta and British Columbia, the northern portion of Baja California, Mexico, the states 

of Washington, Oregon, California, Idaho, Nevada, Utah, Arizona, Colorado, Wyoming,

and portions of Montana, South Dakota, New Mexico, and Texas. See Figure 2, on the 

next page. The WECC region is nearly 1.8 million square miles in size, has over 126,000 

miles of transmission, and serves a population of 78 million. WECC contains 37 BAs and 

53 TOPs. Due to the diverse characteristics of this extensive region, WECC encounters 

unique challenges in day-to-day coordination of its interconnected system. WECC is tied 

to the Eastern Interconnection through a number of high-voltage direct current

transmission ties.

WECC also operates two RC offices that provide situational awareness and realtime monitoring of the entire Western Interconnection. WECC RC was an affected 

entity, and will be discussed with other affected entities below.

 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 16 -

Reliability Responsibilities

NERC categorizes the entities responsible for planning and operating the BPS in 

a reliable manner into multiple functional entity types. The NERC functional entity 

types most relevant to this event are BAs, TOs, TOPs, Generator Operators (GOPs),

Planning Coordinators (PCs), Transmission Planners (TPs), and RCs. These functions 

are described in more detail in NERC’s Reliability Functional Model.21F

22 Some of the 

affected entities conduct multiple reliability functions.

 Balancing Authority

The BA integrates resource plans ahead of time, maintains in real time the 

balance of electricity resources (generation and interchange) and electricity demand or 

load within its footprint, and supports the Interconnection frequency in real time. There 

 22 NERC Reliability Functional Model, Version 5, 

http://www.nerc.com/files/Functional_Model_V5_Final_2009Dec1.pdf.

Figure 2: Map of WECC Region
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 17 -

are 37 BAs in the WECC footprint. The following five BAs were affected by the event: 

APS, IID, WALC, CAISO, and CFE.

 Transmission Owner, Transmission Operator and Generator Operator

The TO owns and maintains transmission facilities. The TOP is responsible for 

the real-time operation of the transmission assets under its purview. The TOP has the 

authority to take corrective actions to ensure that its area operates reliably. The TOP

performs reliability analyses, including seasonal and next-day planning and RTCA, and 

coordinates its analyses and operations with neighboring BAs and TOPs to achieve 

reliable operations. It also develops contingency plans, operates within established 

SOLs, and monitors operations of the transmission facilities within its area. There are 53 

TOPs in the WECC region. The following seven TOPs were affected by the event: APS, 

IID, WALC, CAISO, CFE, SDG&E, and SCE. The GOP operates generating unit(s) and 

performs the functions of supplying energy and other services required to support 

reliable system operations, such as providing regulation and reserve capacity.

 Planning Coordinator

The PC is responsible for coordinating and integrating transmission facility and 

service plans, resource plans, and protection systems.22F

23

 Transmission Planner

The TP is responsible for developing a long-term (generally one year and beyond) 

plan for the reliability of the interconnected bulk transmission systems within its portion 

of the Planning Coordinator Area.

 Reliability Coordinator

The RC and TOP have similar roles, but different scopes. The TOP directly 

maintains reliability for its own defined area. The RC is the “highest level of authority” 

according to NERC, and maintains reliability for the Interconnection as a whole. Thus, 

the RC is expected to have a “wide-area” view of the entire Interconnection, beyond what

any single TOP could observe, to ensure operation within IROLs.

The RC oversees both transmission and balancing operations, and it has the 

authority to direct other functional entities to take certain actions to ensure reliable 

 23 PCs are the same as Planning Authorities (PAs) with respect to NERC registration and the Reliability Standards.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 18 -

operation. The RC, for example, may direct a TOP to take whatever action is necessary 

to ensure that IROLs are not exceeded.23F

24 The RC performs reliability analyses including 

next-day planning and RTCA for the Interconnection, but these studies are not intended 

to substitute for TOPs’ studies of their own areas. Other responsibilities of the RC

include responding to requests from TOPs to assist in mitigating equipment overloads. 

The RC also coordinates with TOPs on system restoration plans, contingency plans, and 

reliability-related services.

Descriptions of Affected Entities

The following entities were affected by the September 8th event:

 WECC RC

In its capacity as the RC, WECC is the highest level of authority responsible for 

the reliable operation of the BPS in the Western Interconnection. WECC RC oversees 

the operation of the Western Interconnection in real time, receiving data from entities 

throughout the entire Interconnection, and providing high-level situational awareness 

for the entire system. WECC RC can direct the entities it oversees to take certain actions

in order to preserve system reliability. Although WECC is both an RE and an RC, these 

two functions are organizationally separated. 

 Imperial Irrigation District 

IID, which encompasses the Imperial Valley, the eastern part of Coachella Valley 

in Riverside County, and a small portion of San Diego County, in California, owns and 

operates generation, transmission, and distribution facilities in its service area to provide 

comprehensive electric service to its customers. Thus, IID is a vertically integrated 

utility. IID’s generation consists of hydroelectric units on the All-American Canal as well 

as oil-, nuclear-, coal-, and gas-fired generation facilities, with a total net capability of 

514 MW. IID purchases power from other electric utilities to meet its peak demands in 

summer, which can exceed 990 MW. IID’s transmission system consists of 

approximately 1,400 miles of 500, 230, 161, and 92 kV lines, as well as 26 transmission 

substations. Among other NERC registrations, IID is a TOP, BA, and TP responsible for 

 24 For example, IRO-005-1 R.5 requires that “[e]ach [RC] shall identify the cause of any potential or actual SOL or 

IROL violations. The [RC] shall initiate the control action or emergency procedure to relieve the potential or 

actual IROL violation without delay, and no longer than 30 minutes. The [RC] shall be able to utilize all resources, 

including load shedding, to address an IROL violation.”
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 19 -

resource and transmission planning, load balancing, and frequency support for its 

footprint.

 Arizona Public Service 

APS is a vertically integrated utility that serves a 50,000 square mile territory 

spanning 11 of Arizona’s 15 counties. Among other NERC registrations, APS is the TOP 

and BA for its territory. APS engages in both marketing and grid operation functions, 

which are separated. APS owns and operates transmission facilities at the 500 

(including H-NG), 345, 230, 115, and 69 kV levels, and owns approximately 6,300 MW 

of installed generation capacity. APS’s 2011 peak load was 7,087 MW.

 Western Area Power Administration – Lower Colorado

WALC is one of the four entities constituting the Western Area Power 

Administration, a federal power marketer within the United States Department of 

Energy. WALC operates in Arizona, Southern California, Colorado, Utah, New Mexico, 

and Nevada, and is registered with NERC as a BA, TOP, and PC for its footprint. As a net 

exporter of energy, WALC’s territory has over 6,200 MW of generation, serving at most 

2,100 MW of peak load. A majority of WALC’s generation is federal hydroelectric 

facilities, with the balance consisting of thermal generation owned and operated by 

independent power producers. WALC also operates an extensive transmission network 

within its footprint, and is interconnected with APS, SCE, and nine other balancing 

areas.

 San Onofre Nuclear Generating Station 

SONGS is a two-unit nuclear generation facility capable of producing 

approximately 2,200 MW of power, and is located north of San Diego.24F

25 SONGS 

produces approximately 19% of the power used by SCE customers and 25% of the power 

used by SDG&E customers. SONGS is jointly owned by SCE (78.21%), SDG&E (20%), 

and the City of Riverside (1.79%). SCE, as TO and GO, is responsible for ensuring the 

safe and reliable operation of SONGS within the grid.

 California Independent System Operator 

 25 SONGS is currently in the midst of an extended outage. According to a March 2012 press release by CAISO, if 

both SONGS units remain offline for the summer, “San Diego and portions of the Los Angeles Basin may face local 

reliability challenges.” http://www.caiso.com/Documents/SummerGridOutlook ComplicatedPossibleExtendedOutage-NuclearPowerPlant.pdf.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 20 -

CAISO runs the primary market for wholesale electric power and open-access 

transmission in California, and manages the high-voltage transmission lines that make 

up approximately 80% of California’s power grid. CAISO operates its market through 

day-ahead and hour-ahead markets, as well as scheduling power in real time as 

necessary. Among other registrations, CAISO is PC and BA for most of California, 

including the city of San Diego. It also acts as TOP for several entities within its 

footprint, including SDG&E and SCE. CAISO likewise engages in modeling and planning 

functions in order to ensure long-term grid reliability, as well as identifying

infrastructure upgrades necessary for grid function.

 San Diego Gas and Electric 

SDG&E is a utility that serves both electricity and natural gas to its customers in 

San Diego County and a portion of southern Orange County, and is the primary utility 

for the city of San Diego. SDG&E owns relatively little generation—approximately 600 

MW—although generation owned by others in its footprint brings the total generation 

capacity of the area above 3,350 MW. Peak load for the area can exceed 4,500 MW in 

the summer. SDG&E also operates an extensive high-voltage transmission network at 

the 500, 230, and 138 kV levels. SDG&E, operating as a TOP within CAISO’s BA

footprint, has delegated part of its responsibilities as a TOP to CAISO. 

 Comisión Federal de Electricidad – Baja California Control Area

CFE is the only electric utility in Mexico, servicing up to 98% of the total 

population. CFE’s Baja California Control Area is not connected to the rest of Mexico’s 

electric grid but is connected to the Western Interconnection. CFE’s Baja California 

Control Area covers the northwest corner of Mexico, including the cities of Tijuana, 

Rosarito, Tecate, Ensenada, Mexicali, and San Luis Rio Colorado. CFE’s Baja California 

Control Area operates transmission systems at the 230, 161, 115, and 69 kV levels, and 

owns 2,039 MW of gross generating capacity and the rights to a 489 MW independent 

power producer within the Baja California Control Area. CFE’s Baja California Control 

Area had a net peak load of 2,184 MW for summer 2010. CFE’s Baja California Control 

Area is connected at the 230 kV level with SDG&E through two transmission lines on 

WECC Path 45. CFE functions as the TO, TOP, and BA for its Baja California Control 

Area under the oversight of WECC RC. For the remainder of this report, “CFE” refers

only to its Baja California Control Area.

 Southern California Edison 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 21 -

SCE is a large investor-owned utility which provides electricity in central, coastal, 

and southern California. SCE is a wholly-owned subsidiary of Edison International, 

which is also based in California. Among other NERC registrations, SCE operates as a 

TOP within CAISO’s BA footprint, and has delegated part of its responsibilities as a TOP 

to CAISO. SCE is also registered as TP, and is responsible for the reliability assessments 

of the SONGS separation scheme. SCE owns 5,490 circuit miles of transmission lines, 

including 500, 230, and 161 kV lines. SCE also operates a subtransmission system of 

7,079 circuit miles at the 115, 66, 55, and 33 kV levels. Of the affected entities, SCE is 

interconnected with APS, IID, and SDG&E at various transmission voltage levels. SCE 

owns over 5,600 MW of generation, including a majority share in SONGS, and its peak

load exceeds 22,000 MW. Along with SONGS staff, SCE is responsible for the safe and 

reliable operation of the nuclear facility.

Interconnected Operations

The September 8th event exemplifies the interconnected operations of three 

parallel transmission corridors through which power flows into the area where the 

blackout occurred. Typically, BAs, through dispatch, balance the flows on these 

corridors so that no one corridor experiences overloads in an N-1 situation, but this did 

not happen on September 8th. 

The first transmission corridor consists of the 500 kV H-NG, which is one of 

several transmission lines forming Path 49 (“East of River”). Along with two 500 kV 

lines, one from North Gila to Imperial Valley and another from Imperial Valley to 

Miguel, they form the SWPL. The majority of the SWPL is geographically parallel to the 

United States-Mexico border. The SWPL meets the SDG&E and IID systems at the 

Imperial Valley substation. This is shown as the “H-NG Corridor” on Figure 3, on the 

next page.

The second corridor is Path 44, also known as “South of SONGS,” operated by 

CAISO. This corridor includes the five 230 kV lines in the northernmost part of the 

SDG&E system that connect SDG&E with SCE at SONGS. 

The third transmission corridor, shown as the “S Corridor” on Figure 3, consists 

of lower voltage (230, 161 and 92 kV) facilities operated by IID and WALC in parallel 

with those of SCE, SDG&E, and APS. The only major interconnection between IID and 

SDG&E is through the 230 kV “S” Line, which connects the SDG&E/IID jointly-owned

Imperial Valley Substation (operated by SDG&E) to IID’s El Centro Switching Station. 

The S Line interconnects the southern IID system with SDG&E and APS at Imperial 

Valley, which is also a terminus for the SWPL segment from Miguel and the SWPL 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 22 -

segment from North Gila. WALC is connected to the SCE system and the rest of the 

Western Interconnection by 161 kV ties at Blythe, to IID by the 161 kV tie between 

WALC’s Knob and IID’s Pilot Knob substations, and to APS by a 69 kV tie via Gila at 

North Gila.

The eastern end of the SWPL, which terminates at APS’s Hassayampa hub, is 

connected to SCE via a 500 kV line that connects APS’s Palo Verde and SCE’s Devers 

substations. The northern IID system is connected to SCE’s Devers substation via a 230 

kV transmission line that connects from Devers to IID’s CV substation. These 

connections, along with SDG&E’s connection to SCE via Path 44’s terminus at SONGS,

make the SWPL, Path 44, and IID’s and WALC’s systems operate as electrically parallel 

transmission corridors.25F

26 The following simplified diagram illustrates the 

interconnected nature of these three parallel corridors. Red lines represent 500 kV, blue 

lines represent 230 kV, and green lines represent 161 kV.

 26 Power transfers from APS to SDG&E and CFE generally flow across the SWPL, but, due to parallel path flows, 

also known as loop flows, some of the power transfers flow through IID’s and WALC’s systems. Loop flow refers 

to power flow along any transmission paths that are in parallel with the most direct geographic or contract path.

Serrano Devers Palo Verde

SONGS

Miguel Imperial Valley

SDG&E

WAPA 

Lower 

Colorado

Hassayampa

WECC

CFE

Imperial 

Irrigation District

161 kV 92 kV

Path 

44

S-Corridor

H-NG 

Corridor

S-Line

North Gila

Figure 3: Three Parallel Corridors
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 23 -

III. SEQUENCE OF EVENTS26F

27

The 11 minutes of the disturbance are divided into seven phases, as highlighted in 

Figure 4, on the next page. This figure displays the progressive loading of the five 230 

kV tie lines from SCE north of San Diego that form Path 44. This section describes how 

the loss of various elements during an 11-minute period combined to exceed the 8,000 

amp setting on the SONGS separation scheme. After sustained loading on Path 44 above

8,000 amps, the SONGS separation scheme operated. Once the SONGS separation 

scheme operated, San Diego and IID, CFE, and Yuma, Arizona, blacked out in less than 

30 seconds. This section is divided into subsections for each phase, including the key 

events during the phase, their causes and effects, and, where relevant, what the affected 

entities knew and did not know as the events were unfolding. Each section begins with a 

brief summary. A final subsection describes restoration efforts after the blackout.

A set of graphics is included at the end of each phase to demonstrate the effect of 

the events during the phase. The first graphic in each set depicts the aggregate loading 

in amps on the five South of SONGS lines.27F

28 The bottom portion of the graphic shows 

all of the phases, while the majority of the graphic shows an expanded view of the phase 

being discussed. The second graphic in each set represents the loading on key facilities 

after each phase. The third graphic in each set shows how power flows redistributed

through Arizona, Southern California, and Mexico after each phase. Phases 6 and 7 have 

multiple power flow graphics. Phases 1 and 7 include only the second and third type of 

graphics. 

 27 All times are in Pacific Daylight Time (PDT) unless otherwise noted. Times are listed to millisecond (three 

decimal places) or tenth-of-second (decimal place) accuracy when possible. If milliseconds or tenth-of-seconds 

are not listed, the event is reconciled to the nearest second.

28 Path 44 flows (complex power in volt amperes, current in amps) were calculated from SONGS PMU data. 

Those readings differ somewhat from disturbance monitoring equipment that was unavailable until completion of 

the inquiry’s analysis. The differences are explained by variances in how some minor auxiliary loads are measured 

and in measurement equipment tolerances.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 24 -

The following figure shows all seven phases of the disturbance.

A. Phase 1: Pre-Disturbance Conditions

Phase 1 Summary:

• Timing: September 8, 2011, before H-NG trips at 15:27:39

• A hot, shoulder season day with some generation and transmission maintenance outages

• Relatively high loading on some key facilities: H-NG at 78% of its normal rating, CV transformers 

at 83%

• 44 minutes before loss of H-NG, IID’s RTCA results showed that the N-1 contingency loss of the 

first CV transformer would result in an overload of the second transformer above its trip point

• An APS technician skipped a critical step in isolating the series capacitor bank at the North Gila 

substation

September 8, 2011, was a relatively normal, hot day in Arizona, Southern 

California, and Baja California, Mexico, with heavy power imports into Southern 

California from Arizona. In fact, imports into Southern California were approximately 

2,750 MW, just below the import limit of 2,850 MW. September is generally considered 

a “shoulder” season, when demand is lower than peak seasons and generation and 

transmission maintenance outages are scheduled. By September 8th, entities 

throughout the WECC region, including some of the affected entities, had begun 

Trip of Hassayampa 

– North Gila 500 kV 

Line

Trip of Coachella 

Valley 230/92 kV 

Transformers

Trip of Pilot 

Knob – El 

Centro 161 kV 

Line

South of SONGS 

Separation

Trip of Pilot Knob 

161/92 kV & Yucca 

161/69 kV 

Transformers

Trip or Gila 161/69 kV 

Transformers & 

Yuma Cogen

Trip of Ramon 

230/92 kV 

Transformer

1 2 3 4 5 6 7

Disturbance Phases

Figure 4: Seven Phases of the Disturbance
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 25 -

generation and transmission outages for maintenance purposes. For example, on 

September 8th maintenance outages included over 600 MW of generation in Baja 

California28F

29 and two 230 kV transmission lines in SDG&E’s territory. However, there 

were no major forced outages or major planned transmission outages that would result 

in a reduction of the SOLs in the area. 

 Pre-Disturbance Conditions in IID

Despite September being considered a shoulder month, temperatures in IID’s 

service territory reached 115 degrees on September 8th.29F

30 IID’s load headed toward 

near-peak levels of more than 900 MW, which required it to dispatch local combustion 

turbine generation in accordance with established operating procedures. Prior to the 

event, loading on IID’s CV transformers reached approximately 125 megavolt amperes 

(MVA) per transformer, which is approximately 83% of the transformers’ normal limit. 

Loading on IID’s Ramon transformer was 153 MVA, which is approximately 68% of its 

normal limit. 

IID’s S Line ties IID to SDG&E, and through SDG&E, to generation in Mexico at 

La Rosita. It also ties CFE and IID, through SDG&E’s La Rosita international 

transmission line. Before the event, IID was importing power on the S Line, and thus 

power was flowing northward from the jointly owned Imperial Valley substation to IID’s 

El Centro substation. Flows on the S Line would reverse multiple times during the event. 

When power flowed on the S Line from south to north, the implication was that IID was

supplied radially through SDG&E. Throughout the event, as power flowed from north to 

south, the implication was that flows intended for SDG&E and/or CFE were moving 

through IID’s system. Eventually, in Phase 6, south to north flows on the S Line would 

activate a RAS that would ultimately trip more than 400 MW of generation at La Rosita

and the S Line, thereby worsening the loading on Path 44.

Forty-four minutes prior to the loss of H-NG on September 8, 2011, IID’s RTCA 

results showed that the N-1 contingency loss of the first CV transformer would result in 

an overload of the second transformer above its trip point. The IID operator was not

actively monitoring the RTCA results and, therefore, was not alerted to the need to take 

any corrective actions. At the time of the event, IID operators did not keep the RTCA 

 29 The generation was known as Termoelectrica de Mexicali, and will be hereafter referred to as “TDM.” It is also 

shown as “TDM” on the Map of Affected Entities.

30 According to IID, the temperature in El Centro, California reached 115 degrees on September 8, 2011.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 26 -

display visible, and RTCA alarms were not audible. By reducing loading on the CV 

transformers at this pre-event stage, the operator could have mitigated the severe effects 

on the transformers that resulted when H-NG tripped. Since the event, IID has required, 

and now requires, its operators to have RTCA results displayed at all times. The loading 

on IID’s CV transformers was pivotal to this event. Loading on the CV transformers is 

influenced by: (1) the pre-contingency flow on H-NG; (2) load and generation in IID’s 92 

kV network; (3) flow on the S Line; and (4) to a lesser extent, generation connected to 

the Imperial Valley substation. See Figure 5, below.

 Pre-Disturbance Conditions in CFE

At 15:07 CFE’s Presidente Juarez Unit 11 tripped, which required CFE to activate 

its Baja California Control Area contingency reserves to restore its ACE. At 15:15 PDT 

CFE returned its ACE to where it had been before the unit tripped. Although still 

complying with the spinning reserve requirements, CFE was short on non-spinning 

reserve, with all of its available resources in use or already deployed.

800

1000

1200

1400

1600

1800

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

150%

160%

170%

180%

190%

200%

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

900

950

1000

Pre-Contingency 

Hassaympa-North Gila 

Line Loading 

(MW)

Transformer Rating (%)

All 92kV Generation Level (MW)

Post-Contingency Coachella Valley Transformer Loading

All 92kV Generation and Normal Rating

Event Conditions

Below normal rating Below emergency rating Below trip zone Trip Zone

Figure 5: Post-Contingency CV Transformer 

Loading Based on All IID 92 kV Generation
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 27 -

 Pre-Disturbance Focus of WECC RC

Prior to the event, WECC RC operators were monitoring unscheduled flow on 

several paths in Northern California. WECC RC did not view any of the scheduled

transmission or generation outages as significant. As illustrated by the chart below, two 

minutes before the event (at 15:25), major paths in the blackout area were operating 

below their Path ratings:

Major Paths in the Blackout Area Established Path 

Ratings/Flow Limits

Path Loadings in 

MW and %

500 kV H-NG

(Part of Corridor 1 into blackout area) 1,800 MW30F

31 1,397 MW

78%

Path 44 

(Corridor 2 into blackout area) 2,200 MW31F

32 1,302 MW

59%

230 kV S Line 

(Part of Corridor 3 into blackout area) 239 MW 90 MW38%

SDG&E Import SOL 2,850 MW 2,539 MW

89%

SDG&E to CFE Path 45 800 MW S-N;

408 MW N-S

241 MW N-S

60%

 Pre-Disturbance Conditions in APS

APS manages H-NG, a segment of the SWPL. At 13:57:46, the series capacitors32F

33

at APS’s North Gila substation were automatically bypassed due to phase imbalance 

protection. APS sent a substation technician to perform switching to isolate the 

capacitor bank. The technician was experienced in switching capacitor banks, having 

performed switching approximately a dozen times. APS also had a written switching 

order for the specific H-NG series capacitor bank at North Gila. After the APS system 

operator and the technician verified that they were working from the same switching 

order, the operator read steps 6 through 16 of the switching order to the technician. The 

technician repeated each step after the operator read it, and the operator verified the 

 31 The limit of H-NG is a portion of the rating of Path 49. The inquiry determined that the limit is approximately 

1,800 MW.

32 With one segment of the SWPL out, the limit increases to 2,500 MW.

33 A series capacitor is a power system device that is connected in series with a transmission line. It increases the 

transfer capability of the line by reducing the voltage drop across the line and by increasing the reactive power 

injection into the line to compensate for the reactive power consumption. In simple terms, a 50% series 

compensated line means it has the equivalent of 50% of the electric distance (or impedance) of the otherwise 

uncompensated line. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 28 -

technician had correctly understood the step. The technician then put a hash mark 

beside each of steps 6 through 16 to indicate that he was to perform those steps. The 

technician did not begin to perform any of steps 6 through 16 until after all steps had 

been verified with the system operator. 

The technician successfully performed step 6, verifying that the capacitor breaker

was closed, placing it in “local” and tagging it out with “do not operate” tags. However, 

because he was preoccupied with obtaining assistance from a maintenance crew to hang 

grounds33F

34 for a later step, he accidentally wrote the time that he had completed step 6 

on the line for step 8. For several minutes, he had multiple conversations about 

obtaining assistance to hang the grounds. He then looked back at the switching order to 

see what step should be performed next. His mistake in writing the time for step 6 on 

the line for step 8 caused him to pick up with step 9, rather than step 7.34F

35 Thus, he 

skipped two steps, one of them the crucial step (step 8) of closing a line switch to place 

H-NG in parallel with the series capacitor bank. This step would bypass the capacitor 

bank, resulting in almost zero voltage across the bank and virtually zero current through 

the bank. Because he skipped step 8, when he began to crank open the hand-operated 

disconnect switch to isolate the capacitor bank, it began arcing under load.35F

36 He could 

not manage to toggle the gearing on the switch to enable its closure, so he stayed under 

the arcing 500 kV line, determined to crank open the switch far enough to break the arc, 

thereby preventing additional damage to the equipment. Figure 6, on the next page, is 

a schematic of the APS series capacitor bank, showing steps seven through nine.

 34 Grounds are temporary protective connections that are run from conductive parts of lines, structures, and 

equipment, to earth or some other grounding system that substitutes for earth. If the isolated equipment is 

accidentally energized, grounds are intended to: (1) limit the voltage rise at the worksite to a safe value; and (2) 

provide a pathway for fault current to flow, thereby allowing upstream protective devices to trip.

35 In human performance analysis, this is known as a “place keeping” error, by failing to physically mark steps as 

they are completed.

36 An electric arc is a luminous discharge of current that is formed when a strong current jumps a gap in a circuit.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 29 -

Phase 1 Graphics36F

37

 37 For the dial graphics shown here, green indicates available capacity on the facility, red indicates that the facility 

is fully loaded to its normal limit, blue indicates the amount by which the facility is overloaded, and gray indicates 

that the facility has tripped or load has been lost. For the power flow graphics, black borders indicate islanding, 

and gray areas bounded by black are those where load was lost.

Figure 6: APS Series Capacitor Bank

Pre-disturbance (000)

Steps 7, 8 

Step 9 (Close) 

(Open)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 30 -

B. Phase 2: Trip of the Hassayampa-North Gila 500 kV Line

Phase 2 Summary:

• Timing: 15:27:39 to 15:28:16, just before CV transformer No. 2 trips 

• H-NG trips due to fault; APS operators believe they will restore it quickly and tell WECC RC

• H-NG flow redistributed to Path 44 (84% increase in flow), IID, and WALC systems

• CV transformers immediately overloaded above their relay setting 

• At end of Phase 2, loading on Path 44 at 5,900 out of 8,000 amps needed to initiate SONGS 

separation scheme

At 15:27:39, the arc that had developed on each phase of the disconnect switch 

lengthened as the switch continued to open, to the point where two phases came into 

contact. This caused H-NG to trip to clear this phase-to-phase (A to C) fault. The highspeed protection system correctly detected the fault and tripped the line in 2.6 cycles (43 

milliseconds). After discussion with the technician, APS operators erroneously believed 

that they could return the line to service in approximately 15 minutes, even though they

had no situational awareness of a large phase angle difference caused by the outage. 

More time would have been needed to redispatch generation to reduce the phase angle 

difference to the allowed value. APS system operators informed CAISO, Salt River 

Project (SRP), and WECC RC that the line would be reclosed quickly, even though they 

were unaware that this was not possible because of the large phase angle difference that 

existed between Hassayampa and North Gila. The inquiry’s simulation indicates that the

15:27:00
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 31 -

post-contingency angular difference was beyond the allowed North Gila synch-check 

relay reclosing angle setting of 60 degrees, and there would not have been adequate 

generation for redispatch to reduce the phase angle difference to within the allowed 

value. APS operators were only able to see the angular difference on EMS displays after 

isolating the North Gila capacitor bank and re-energizing H-NG from the Hassayampa 

substation (before closing at North Gila). 

H-NG, which has a flow limit of 1,800 MW37F

38 with a 30 minute emergency rating 

of 2,431 MW, was carrying 1,391 MW flowing from east to west along the SWPL at the 

time of the trip. As a result of the line trip, flows redistributed across the remaining lines 

into the San Diego, Imperial Valley, and Yuma areas. The IID and WALC systems, 

located between the two parallel high voltage Paths, were forced to carry approximately 

23% of the flow that had initially been carried by H-NG. The majority of the flow 

diverted to Path 44, as discussed below.

Immediately after the loss of H-NG, the loading on both of IID’s CV transformers 

increased to 130% of their normal rating and 118.5% of their emergency rating. The time 

overcurrent relays on the CV transformers picked up because the current flow was above 

the overcurrent relay setting, and began timing according to their very inverse38F

39 time 

delay. The CV transformers would both trip within 40 seconds of the loss of H-NG. At 

the same time, loading on IID’s Ramon 230/92 kV transformer increased to 94% of its 

normal rating and 85% of its emergency rating. Three seconds after the loss of H-NG, 

SCADA metering for the CV transformer banks stopped recording accurate readings due 

to remote terminal unit (RTU) exceeding maximum scale. IID and WECC RC no longer 

had accurate information about or situational awareness of the loading on these 

important transformers.

IID also experienced increased loading on several of its 161 kV lines immediately 

after the loss of H-NG: Blythe-Niland and Knob-Pilot Knob loading increased by 49% 

and 55%, respectively. Flows on IID’s S Line reversed from south to north (SDG&E to 

IID) to north to south (IID to SDG&E) during this phase of the event, indicating that 

 38 See footnote 31, supra.

39 “Very inverse” describes the time/current characteristic of the relays’ time delay which is inversely 

proportional to the current magnitude sensed by the relay. That is, the greater the current, the less time before 

the relay will trip.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 32 -

flows intended for SDG&E were being routed through IID’s 161 and 92 kV systems. 

While IID was aware of the flow changes on the S Line, it was unable to see the loss of HNG in real time. 

Flows on WALC’s Gila 161/69 kV transformers increased from approximately 12 

MVA to 60 MVA, still well below their normal limits of 75 MVA each, but indicative of 

the sudden increase in flows on WALC’s system just after the loss of H-NG. WALC also

experienced significant voltage drops on its 161 kV system, particularly at Blythe (6.9% 

drop) and Kofa (6.7% drop) substations, due to the increased flows on that system.

The loss of H-NG interrupted the southern 500 kV path into San Diego. The 

majority of the flow diverted to the northern entry to SDG&E, Path 44. Flow on Path 44

increased by approximately 84%, from 1,293 MW to 2,362 MW. This flow equates to a 

tie current of 5,900 amps relative to the 8,000 amps required to initiate the SONGS 

separation scheme.

Because so much of the flow on H-NG was intended for San Diego, the inquiry

considered whether increasing internal generation in SDG&E’s area would have avoided 

the cascading outages.39F

40 Figure 7, on the next page, illustrates post-contingency 

loading on the CV transformers based on pre-contingency loading on H-NG and the 

generation level at IID’s and SDG&E’s jointly owned Imperial Valley substation. The red 

area on the graph indicates the large zone in which loading below H-NG’s 1,800 MW 

SOL would load the CV transformers above their trip point. This area demonstrates the 

non-secure N-1 operating point of the CV transformers. It shows that the operating 

conditions that would reduce the loading on the transformer are: increased generation 

at Imperial Valley, reduced flow on H-NG before it tripped, or both. For example, the 

graph indicates that for the same amount of transfer on H-NG, additional generators

connected at Imperial Valley would reduce the post-contingency loading on the CV 

transformers.

 40 The inquiry’s analysis is not intended to suggest specific generation adjustments that could have been made 

by specific entities on September 8, 2011, but rather to show the extent to which the affected entities are 

interdependent. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 33 -

In general, adding generation in San Diego, CFE, or Imperial Valley and backing 

down generation in APS’s system (east of Path 49) would reduce the loading on IID’s 92 

kV system for the loss of H-NG. For example, an additional 600 MW of generation at 

Imperial Valley and a reduction of generation in APS’s system by the same amount

would have reduced the pre-contingency loading on H-NG by 20% and improved the 

post-contingency voltage in WALC’s Blythe area by approximately 4%. Under this 

condition, the loading on the CV transformers for the loss of H-NG would be 

approximately 111% of their normal rating (166 MVA), well below their trip setting of 

127%. This is a further demonstration of the importance of including all facilities when 

deriving SOLs.

After seeing the alarm for the loss of H-NG, the WECC RC operator promptly 

called the line’s operator, APS. APS told WECC RC it could get H-NG restored within 

minutes. While WECC RC was monitoring Rated Paths, it took no action specific to Path 

44, believing it would take five or ten minutes for APS to restore H-NG. As the entire 

event took only 11 minutes, WECC RC did not issue any directives in connection with the 

loss of H-NG.

Shortly after H-NG tripped, at 15:27:49, one of the combustion turbines at CFE’s 

Central La Rosita substation tripped while producing 156 MW. This trip may have been

800

1000

1200

1400

1600

1800

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

150%

160%

170%

180%

190%

200%

Pre-Contingency 

Hassaympa-North Gila 

Line Loading 

(MW)

Transformer Rating (%)

Imperial Valley Generation Level (MW)

Post-Contingency Coachella Valley Transformer Loading

Imperial Valley Generation and Normal Rating

Event Conditions

Below normal rating Below emergency rating Below trip zone Trip Zone

Figure 7 : Post-Contingency CV Transformer 

Loading Based on Imperial Valley Generation
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 34 -

triggered by transients40F

41 caused by the initial fault at North Gila and subsequent trip of 

H-NG. Loss of this unit further increased the flow on Path 44, raising the current to 

6,200 amps out of the 8,000 needed to initiate the SONGS separation scheme. 

However, the La Rosita trip alone was not significant in causing the cascading that 

followed.41F

42 CFE was also unaware in real time that H-NG had tripped. After losing the 

Central La Rosita unit, CFE was unable to recover its ACE with its own resources, and at 

15:30, it requested 158 MW of emergency assistance from CAISO for the remainder of 

the hour.

Phase 2 Graphics

 41 See footnote 21. CFE stated that the trip was triggered by transients.

42 The Modeling and Simulation team conducted a “what if” simulation and determined that, even without the 

inadvertent tripping of 160 MW of generation at La Rosita, the overloads and ensuing blackout would still have 

occurred. 

Hassyampa –

N. Gila 500 

kV line trip

CCM Unit 1 

generator trip

South of SONGS – Calculated Phase Current
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 35 -

 

 

C. Phase 3: Trip of the Coachella Valley 230/92 kV Transformer and 

Voltage Depression

15:27:39 – The Hassayampa- North Gila 500 kV line 

tripped.

15:27:40
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 36 -

Phase 3 Summary:

• Timing: 15:28:16, when CV transformer bank No. 2 tripped, to just before 15:32:10, when Ramon 

transformer tripped

• Both CV transformers tripped within 40 seconds of H-NG tripping

• IID knew losing both CV transformers would overload Ramon transformer and S Line connecting it 

with SDG&E

• Severe low voltage in WALC’s 161 kV system

• At end of Phase 3, loading on Path 44 at 6,700 amps out of 8,000 needed to initiate SONGS 

separation scheme

At 15:28:16, less than a minute after H-NG tripped, IID’s CV transformer bank 

No. 2 tripped on the 230 kV side. The CV overload protection relays detected an 

overload immediately after H-NG was lost. The overloads were caused by through-flows

on IID’s 92 and 161 kV systems which parallel APS’s 500 kV system. The normal ratings 

for these transformers are 150 MVA, but immediately after H-NG tripped, each CV

transformer was carrying more than 191 MVA. The relays were set to trip at 

approximately 127%42F

43 of the transformers’ normal ratings, or 191.2 MVA at nominal 

voltage. The inverse time relays took 37.5 seconds to trip bank No. 2 and 38.2 seconds to 

trip bank No. 1. Thus, CV bank No. 1 tripped only 677 milliseconds after bank No. 2, 

again on the 230 kV side. Although the primary winding or high side voltages of the CV

transformers are 230 kV, the banks were not considered as elements of the BES because 

their secondary winding or low side voltages are below 100 kV. As discussed in detail in 

Section IV, because these transformers and the underlying 92 kV system were not 

classified as elements of the BES, IID, neighboring TOPs, and WECC RC did not assess 

the impact of critical external contingencies on overloading the CV banks, the effect of 

losing the CV banks and the subsequent impact on the Ramon bank, and, finally their 

overall adverse effect on BPS reliability. 

IID was aware of the potential for local cascading if the CV transformers tripped. 

IID’s next-day plan for September 8, 2011, which was not based on updated studies,

indicated that if both CV transformers tripped,43F

44 the Ramon 230/92 kV transformer 

would trip and the S Line tie with SDG&E would overload to 109% of its normal rating. 

The next-day plan also indicated that this overloading, in turn, would result in tripping 

generation because the S Line RAS trips generation supplied to Imperial Valley when the 

S Line loads to 108% of its normal rating. IID’s next-day mitigation plan for loss of the 

CV transformers required starting turbines at Coachella and Niland and asking CAISO to 

 43 IID’s transformer protection philosophy specifies trip settings at 120% of normal ratings. IID chose the closest 

available relay tap, which was approximately 127% of the normal rating.

44 This contingency scenario had nothing to do with H-NG tripping. IID’s studies did not show any effect on the 

CV banks resulting from the loss of H-NG. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 37 -

redispatch generation to relieve the S Line. This was a post-contingency mitigation plan. 

But after the event, IID’s operator admitted that if the CV transformers tripped on 

overload, he would have “very little time to mitigate the Ramon [transformer], if at all.” 

Even the quickest-starting turbines take about 10 minutes to start and ramp to full load, 

but IID effectively had only four minutes before the Ramon transformer would trip, after 

the loss of the CV transformers. 

The loss of the CV banks caused flows on the S Line between SDG&E and IID to 

again reverse direction. Because its load exceeded internal generation, IID began pulling 

power from SCE through SDG&E due to the loss of key facilities in IID’s northern 

system. The tripping of the second CV bank also open-ended the Coachella ValleyRamon 230 kV “KS” Line (at CV), which was carrying about 41 MVA. This further 

increased loading on the Mirage-Ramon 230 kV line and through-flow from IID’s 230 kV 

collector system through Devers, but had little effect on the overall disturbance. By 

15:31:35, IID’s operators switched in 92 kV capacitor banks at Avenue 42, Avenue 58, 

and Highline due to low voltage.

The loss of IID’s two CV transformers caused the aggregate current on Path 44 to 

increase from 6,200 amps to 6,600 of the 8,000 amps necessary to trigger the SONGS 

separation scheme. However, by the end of this Phase aggregate Path 44 current

reached 6,700 amps.

The loss of the CV banks caused a severe voltage depression on the WALC 161 kV 

system south of Blythe. During this period, loads in that area (largely irrigation pumps) 

were highly susceptible to motor stalling, which can create additional reactive demand 

and exacerbate transmission loading, both of which contribute to additional voltage 

decline. See Figure 8, on the next page. At 15:28:18, the Blythe 161 kV bus alarmed at 

142 kV (0.882 per unit). 44F

45 WALC continued to experience severe low voltage on its 161 

kV system until the S Line tripped at 15:38:02.4.

 45 Other alarms and low voltage readings followed throughout WALC’s system one to nine seconds later, 

including the Parker-Kofa 161 kV line, which alarmed for overload at 169 MVA (167 MVA rating); Kofa 161 kV bus 

voltage recorded at 143 kV (0.888 per unit); Knob 161 kV bus voltage recorded at 142 kV (0.882 per unit); Parker 

161 kV bus voltage recorded at 149 kV (0.925 per unit); Gila and Goldmine 161 kV bus voltages recorded at 144 kV 

(0.894 per unit); and Parker 230 kV bus voltage recorded at 222 (0.965 per unit). 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 38 -

On September 8, 2011, CAISO had partial visibility of IID’s system, but could not 

see that the CV banks had tripped. Prior to the event CAISO and IID had been working 

together to increase their mutual visibility and those efforts are continuing. Currently, 

CAISO receives loading data from the 230 kV side of the CV transformers.

Despite the fact that it did not consider the CV banks to be part of the BES, 

WECC RC does observe much of IID’s 92 kV system in real time, including the CV banks. 

The WECC RC operator did notice the CV transformers trip, but he was focused on when 

APS would return H-NG to service.

Trip of Coachella 

Valley 230/92 kV 

Transformers 

Trip of Hassayampa 

– North Gila 500 kV 

Line

Ramon 161/92 kV 

Transformer Trip

Trip of Over 400 

MW in Northern 

IID 92 kV Load 

Pocket

Yucca 161/69 kV 

Transformers 1 and 

2 Trip

El Centro – Pilot 

Knob 161 kV Line 

Trip 

Figure 8: Blythe 161kV Voltage

Ramon 230/92 kV
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 39 -

Phase 3 Graphics

Coachella Valley 

230/92 kV 

transformers

trip

South of SONGS – Calculated Phase Current

15:28:17 – Two Coachella Valley 230/92 kV transformers 

and the Coachella Valley Ramon 230 kV “KS” line tripped. 

(030)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 40 -

D. Phase 4: Trip of Ramon 230/92 kV Transformer and Collapse of IID’s 

Northern 92 kV System

Phase 4 Summary:

• Timing: 15:32:10 to just before 15:35:40

• IID’s Ramon 230/92 transformer tripped at 15:32:10, was set for 207% of its normal rating instead 

of its design setting of 120%, which allowed it to last approximately four minutes longer than CV 

transformers

• IID experienced undervoltage load shedding, generation and transmission line loss in its 92 kV 

system

• Path 44 loading increased from approximately 6,700 amps, to as high as 7,800 amps, and ended at 

around 7,200 amps (out of 8,000 needed to initiate the SONGS separation scheme)

At 15:32:10.621, less than five minutes after the trip of H-NG, IID’s Ramon 

230/92 kV transformer tripped on the 92 kV side. The normal rating for this

transformer was 225 MVA, and its relays were set to trip above 207% of its normal 

rating, or 466 MVA. Before it tripped, the SCADA metering for the Ramon bank had 

stopped recording accurate readings due to RTUs exceeding maximum scale, just as for 

the CV banks. Following the loss of the CV transformers, the inverse time relays took 

less than four minutes to trip the Ramon transformer. IID had intended to set the 

Ramon transformer to trip at 120% of its normal rating. Had it been set at this level, the 

Ramon transformer would have tripped almost immediately after the loss of the CV 

transformers, approximately four minutes earlier than the time of its actual trip. IID 

believed that the Ramon transformer would overload beyond the trip point upon the loss 

of both CV transformers. Its next-day plan noted, “the Ramon Bank #1 transformer will

15:28:18
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 41 -

overload and relay out of service because the overcurrent settings are set to trip at 

120%.” IID’s next-day plan relied on a post-contingency operating philosophy of starting 

the Coachella Gas Turbines to mitigate overloads following the loss of the CV 

transformers, but the plan was unrealistic as IID would not have had time to start any 

additional generation between the loss of the CV transformer banks and the loss of the 

Ramon transformer.

Within less than one second after the loss of the Ramon transformer, automatic 

distribution undervoltage protection in IID’s northern 92 kV system began tripping 

distribution feeders and shedding load. From 15:32:11 to 15:33:46, 444 MW of IID’s load 

tripped, with nearly half of the load being shed within 10 seconds of the Ramon 

transformer tripping. As illustrated in Figure 9, below, the severe voltage depression 

following the loss of the Ramon transformer appears to have prompted a local voltage 

collapse within IID’s northern 92 kV system, evidenced by both the steep drop-off in 

voltage as well as a sharp rise in reactive power flow due to motor stalling. 

Ramon 161/92 kV 

Transformer Trip

Trip of Over 400 

MW in Northern 

IID 92 kV Load 

Pocket

Over-Voltage Trip 

of 92 kV System 

Capacitors

Figure 9: 92kV Voltage (per unit) at Avenue 58

Ramon 230/92 kV 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 42 -

The loss of IID’s northern resources and subsequent system response caused IID 

to lose multiple generators connected to its 92 kV system, including IID’s Niland Gas 

Turbine 2 (generating 45 MW), IID’s CV Gas Turbine 4 (generating 20 MW), 

independent power producer Colmac’s unit (generating 46 MW), and IID’s Drop 4 Unit 2 

Hydro Generator (generating 10.3 MW).

IID also began losing transmission lines. The Blythe-Niland 161 kV “F” Line, 

which saw increased loading during Phase 2, tripped at 15:32:13 (approximately 3 

seconds after loss of the Ramon banks). Its normal rating was 165 MVA, and it was set to 

trip at 129% of the normal rating (212 MVA at nominal voltage) with a 3-second time 

delay.45F

46 The Niland-CV 161 kV “N” Line, carrying 83 MVA, tripped approximately 2 

seconds later at 15:32:15.29 due to Zone 3 distance protection.46F

47

In WALC’s territory, the Blythe-Goldmine-Knob and Parker-Kofa 161 kV lines 

overloaded approximately four seconds after the Ramon transformer tripped, at 

15:32:14, but did not trip. These lines each had a normal rating of 167 MVA, but were 

loaded to 177 MVA. Power flows redistributed through the Parker and Blythe areas after 

IID lost the Blythe-Niland line. WALC took some actions in an attempt to arrest the 

voltage depression it was experiencing, including a directive to start hydropower 

generation units Parker 3 and 4 for voltage support at 15:34:07. At the time, Parker area 

voltage was at 150 kV (0.932 per unit). WALC also switched in shunt capacitors on the 

69 kV system at Gila and Kofa. At the time, voltage at Gila was at 65.5 kV (0.906 per 

unit) while Kofa was at 59 kV (0.86 per unit). 

CAISO attempted to bring on generation through its exceptional dispatch47F

48

process to bring Path 44 back within its limit of 2,500 MW, anticipating that it had 30 

minutes to do so. At 15:35, it dispatched the Larkspur No. 2 peaking unit (rated 50 MW)

 46 Based on the last available SCADA scan before the line tripped, the voltage at Blythe was at 123.1 kV (0.765 per 

unit) and the line was loaded to 172 MVA. Based on these measurements, the line was carrying 807 amps at the 

last time recorded; the relay was set to trip with a 3-second time delay at 762 amps.

47 A distance relay is a relay that compares observed voltage and current on a line and operates when that ratio is 

below its preset value. Zone 3 relays are typically set to protect against faults that are more than one substation 

away from the observed line as backup protection. An appropriate time delay should be set in the relay to give 

the remote station relays the opportunity to operate and isolate the minimum amount of equipment necessary to 

clear the fault. A common issue with the application of Zone 3 relays is that they can restrict the loading on 

transmission lines (e.g. the N Line) during abnormal system conditions like those present on September 8th.

48 CAISO’s exceptional dispatch process involves calling on generators outside of the market automated 

dispatch process.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 43 -

within San Diego, which has a 20-minute start-up time. Also at this time, APS began 

taking steps to restore H-NG by completing the bypass of the series capacitor bank.

During Phase 4, aggregate loading on the South of SONGS 230 kV transmission 

lines increased from approximately 6,700 amps to as high as 7,800 amps. The loading 

settled around 7,200 amps and remained there for the rest of Phase 4.

Phase 4 Graphics

 

Ramon 230/92 kV 

transformer trip

Multiple line, 

generator and load 

trips

Voltage collapse in 

pocket, followed 

by load tripping

South of SONGS – Calculated Phase Current

Time: 15:32:10 – The Ramon 230/92 kV transformer 

tripped and IID shed 444 MW of load. (110)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 44 -

15:32:35

 

E. Phase 5: Yuma Load Pocket Separates from IID and WALC

Phase 5 Summary:

• Timing: 15:35:40 to just before 15:37:55

• The Gila and Yucca transformers tripped, isolating the Yuma load pocket to a single tie with SDG&E

• Path 44 loading increased from 7,200 to 7,400 amps after Gila transformer tripped, and ended at 

7,800 amps after loss of the Yucca transformers and YCA generator (very close to the 8,000 amps 

needed to initiate the SONGS separation scheme)

At 15:35:40, approximately eight minutes after H-NG tripped, WALC’s Gila 

161/69 kV transformers tripped due to time-overcurrent protection. The two 

transformers are each rated 75 MVA, but the 69 kV bus section that connects the 

transformers to the rest of the 69 kV substation is rated 1,200 amps (143 MVA at 

nominal voltage), and the overcurrent protection is set accordingly at 1,200 amps. The 

bus was carrying 1,312 amps at the time of the trip.

One minute later, at 15:36:40, the Yucca 161/69 kV transformers 1 and 2 tripped 

when their common 69 kV breaker tripped due to overload protection. Bank No. 1 is 

owned by IID and is rated 73 MVA, and bank No. 2 is owned by APS and is rated 75 

MVA. The IID Yucca generator and four out of the six APS combustion turbines

connected to APS’s 69 kV system were offline at the time of the event, as was the IID 

GT21 combustion turbine on the 161 KV side. These generators may have supported load 

Time: 15:32:35
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 45 -

in the area had they been in service. Almost immediately, the Pilot Knob breaker on the 

Pilot Knob-Yucca 161 kV “AX” transmission line, which is effectively the 161 kV breaker 

for the Yucca 161/69 kV transformers, received a direct transfer trip from the Yucca 

transformer overload protection, thereby tripping the AX Line. As a result of the loss of 

the Yucca and Gila transformers, the Yuma load pocket was isolated to only one tie to the 

SDG&E system, causing loading on each N. Gila 500/69 kV transformer bank to increase

from 57 MVA to 164 MVA. 

Less than one second after the Yucca transformers and AX Line tripped, at 

15:35:40, the Yuma Cogeneration Associates (YCA) combined cycle plant on the Yuma 

69 kV system tripped. The combustion turbine is rated at 35 MW and the heat recovery 

unit is rated at 17 MW, totaling 52 MW. It appears that both units were fully loaded at 

the time of the trip. The cause of the trip is unknown, but the loss of the YCA unit 

hastened the collapse of the Yuma load pocket.

Approximately one minute later, at 15:37:41, a common 161 kV breaker tripped 

IID’s Pilot Knob 161/92 kV transformers Nos. 2 and 5 for No. 2 overload protection. The 

overload protection was set to trip the banks at 121% of the normal rating (37.5 MVA at 

nominal voltage).

At WALC’s request, between 15:36:48 and 15:36:52, SCE directed Metropolitan 

Water District operators to drop 80 MW of pumping load attached to the Gene 

substation (near Parker) to improve 230 kV voltage support at Parker in an attempt to 

arrest declining voltages.

As it had done during Phase 4, CAISO ordered exceptional dispatch to bring Path 

44 below its 2,500 MW limit. At 15:36:00, CAISO called SCE and ordered an exceptional 

dispatch of Larkspur Peaking Unit No. 1 (rated 50 MW), and Kearny GT2 and GT3 (each 

rated 59 MW) to go to full load. The Larkspur unit takes 20 minutes to start, and the 

Kearny units are 10-minute “quick start” peaking generators. All of these units were 

offline at the time, and they were unable to come online before the system collapsed.48F

49 

The tripping of the Gila 161/69 kV transformers caused the aggregate loading on 

Path 44 to increase from approximately 7,200 amps to approximately 7,400 amps, out of 

the 8,000 amps necessary to initiate the SONGS separation scheme. After the loss of the 

 49 Larkspur generation is connected to the SDG&E 69 kV system south of Otay Mesa, and Kearny generation is 

connected to the SDG&E 69 kV system in northern San Diego.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 46 -

Yucca 161/69 kV transformers, the YCA plant, and the Pilot Knob 161/92 kV 

transformers, the loading further increased to approximately 7,800 amps.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 47 -

Phase 5 Graphics

 

Gila 161/69 kV 

transformers trip

YCA 

generating 

units trip

Yucca 161/69 kV 

transformers trip

Pilot Knob 

161/92 kV 

transformers 

trip

 

15:37:42

 

South of SONGS – Calculated Phase Current

Time: 15:35:31 – Yuma Cogen (YCA) tripped. (180)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 48 -

F. Phase 6: High-Speed Cascade, Operation of the SONGS Separation 

Scheme and Islanding of San Diego, IID, CFE, and Yuma

Phase 6 summary:

• Timing: 15:37:55 to 15:38:21.2

• IID’s El Centro-Pilot Knob line tripped, forcing all of IID’s southern 92 kV system to draw from 

SDG&E via the S Line

• S Line RAS operates, tripping generation at Imperial Valley and worsening the loading on Path 44

• S Line RAS trips S Line, isolating IID from SDG&E

• Path 44 exceeds trip point of 8,000 amps, to as high as 9,500 amps

• SONGS separation scheme operates and creates SDG&E/CFE/Yuma island

When the El Centro-Pilot Knob 161 kV line tripped at 15:37:55 (10 minutes after 

loss of H-NG), it isolated the southern IID 92 kV system onto a single transmission line 

from SDG&E: the S Line. Forcing all of the remaining load in IID to draw through the 

SDG&E system pushed the aggregate current on Path 44 to 8,400 amps, well above the 

trip point of 8,000 amps. If the aggregate current on Path 44 remained above 8,000 

Time: 15:37:42
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 49 -

amps, the definite minimum time relay49F

50 would initiate the SONGS separation scheme 

to separate SDG&E from SCE at SONGS. 

IID’s El Centro-Pilot Knob 161 kV line open-ended at El Centro when a 161 kV 

breaker at El Centro tripped on Zone 3 relay protection50F

51 with a one second delay. The 

apparent impedance detected on the Zone 3 relay at El Centro was hovering near its trip 

zone immediately following the Pilot Knob 161/92 kV transformer trips (12 seconds 

earlier), but did not cross into the Zone 3 tripping region until this time.

By this time in the event, the South of SONGS lines were San Diego’s only source 

of critical imported generation, and were also keeping IID and CFE’s Baja California 

Control Area from going dark. If the aggregate current was brought below 8,000 amps, 

the blackout could have been avoided, but at this point no operator action could have 

occurred quickly enough to save the South of SONGS Path. Had there been formal 

operating procedures that recognized the need to promptly shed load as the aggregate 

current approached 8,000, and had operators been trained on the 8,000 amp set point, 

it is possible that operation of the SONGS separation scheme could have been averted by 

earlier control actions.

Milliseconds after the loss of IID’s El Centro-Pilot Knob 161 kV line, at

15:37:55.890, NextEra’s Buck Boulevard combustion turbine generator tripped due to 

operation of SCE’s Blythe Energy RAS, dropping 128 MW of generation.51F

52 This was 

caused by a reduction of counter-flows on the Julian Hinds-Mirage 230 kV line that had 

been created by heavy flows from the Julian Hinds-Eagle Mountain 230 kV line feeding 

toward the WALC 161 kV system to support the heavy north to south 161 kV flows toward 

Pilot Knob. When the El Centro-Pilot Knob 161 kV line tripped, those counter-flows 

disappeared, initiating the RAS operation. The Buck Boulevard heat recovery unit 

ramped down by 82 MW over the next few minutes. The Buck Boulevard combined cycle 

plant was generating 409 MW (535 MW rating) at the time the combustion turbine

tripped. Tripping the Buck Boulevard generator did not increase loading on Path 44, 

because it is not located south of Path 44.

 50 A definite minimum time relay can operate in one of two ways. When current reaches a certain value, the 

relay will operate with a definite time delay that reflects the relay’s fastest operating time. Before the relay 

reaches that value, the time for the relay to operate is inversely proportional to its observed current magnitude. 

During the event, the relay operated while following the latter characteristic.

51 See footnote 47, supra.

52 The Blythe Energy RAS, among other functions, trips generation owned by NextEra to protect the Julian HindsMirage 230 kV line from overloading with east to west flows for a potential loss of the Julian Hinds-Eagle 

Mountain 230 kV line. Buck Boulevard is connected to SCE’s 230 kV system in the Blythe area. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 50 -

Just three seconds after the loss of IID’s El Centro-Pilot Knob 161 kV line, at 

15:37.58.2, the S Line RAS at Imperial Valley Substation initiated the tripping of two 

combined cycle generators at Central La Rosita in Mexico. The S Line RAS currently

protects El Centro’s 161/92 kV transformer No. 2 by initially tripping a combination of 

CLR II generators when the flow on the S Line exceeds 269 MW flowing northward from 

SDG&E into IID. Two combustion turbines were loaded to 152 MW (193.5 MW rating), 

and 153 MW (193.5 MW rating), respectively, and the associated steam heat recovery 

unit (which also tripped following loss of the turbines) was loaded to 127 MW (159.3 

rating), totaling 432 MW of generation. 

Loss of the CLR II generation drove the South of SONGS flows from about 8,400 

amps to about 9,500 amps, which remained above the 8,000 amp setting of the SONGS 

separation scheme. The inquiry’s simulation showed that had the S Line tripped without 

the S Line RAS tripping the CLR II generation, the flow on Path 44 would have fallen 

below 8,000 amps to settle at an estimated 7,730 amps, and the SONGS separation 

scheme might not have operated.52F

53

Approximately four seconds after the S Line RAS tripped the CLR II generators, 

at 15:38:02.4 the S Line RAS tripped the S Line itself due to flow above 289 MW toward 

IID from SDG&E. Tripping of this line created an IID island. IID reported that from 

15:37:59 to 15:40:24, 507.85 MW of load tripped on its system, mostly in the southern 92 

kV system. 

The tripping of the S Line meant that IID was no longer pulling power from 

SDG&E and CFE through Path 44, so the aggregate Path 44 flows decreased from 

approximately 9,500 amps to approximately 8,700 amps, but were still above the 8,000 

amps required to trigger the SONGS separation scheme.

At 15:38:21.2, not quite 11 minutes after H-NG tripped, the SONGS separation 

scheme operated, reconfiguring the SONGS 230 kV switchyard and isolating the SONGS 

generators onto the SCE system to the north. This reconfiguration effectively separated

 53 The inquiry’s simulation showed that if the S Line RAS tripped only the S Line, IID’s system would still have 

collapsed, but San Diego and the Yuma load pocket would likely have survived. Voltages would have remained 

acceptable, and the 230 kV system around SONGS may have experienced minor overloads. While this would have 

resulted in a large phase angle difference on H-NG, the fact that the SONGS separation scheme would not have 

operated would have allowed time for system operators to make the load and generation changes necessary to 

reduce the phase angle difference.

Had the S Line RAS not operated at all, or only operated to trip the CLR II generators, Path 44 flows would have 

settled above the 8,000 amp threshold and thus the SONGS separation scheme would still have operated. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 51 -

all five South of SONGS 230 kV transmission lines from the SONGS units and the SCE 

system, and separated SDG&E from the rest of the Western Interconnection. Operation 

of the SONGS separation scheme created an island consisting of the SDG&E system, the 

remaining Yuma-area load connected through the 500 kV system from Miguel to North 

Gila, and CFE’s Baja California Control Area.

September 8, 2011, was the first time that the SONGS separation scheme had 

ever activated, and its effects on neighboring systems had not been studied. Although 

this sequence of events has focused on how the loss of elements combined over the 11

minutes to exceed the 8,000 amp SONGS separation scheme trigger, in real time, no 

entity was monitoring that limit or recognized the potential consequences of its 

operation.

WECC RC, responsible for the reliable operation of the BPS, and with having a 

wide area view of the BPS, did not have any alarm that would alert operators before 

operation of the separation scheme. Although WECC RC operators were monitoring the 

Path limit on Path 44, they were not watching the aggregate flows with respect to the 

SONGS separation scheme trigger. WECC RC operators noticed the five South of 

SONGS breakers open after the scheme had already operated.

CAISO, the TOP for SDG&E and SCE, did not have any alarms specifically tied to 

the operation of the SONGS separation scheme either. CAISO only has alarms for when 

Path 44 exceeds its Path rating, but had no ability to monitor the SONGS separation 

scheme, set at 3,100 MW (8,000 amps). After the loss of H-NG, which caused Path 44 to 

exceed its Path rating, CAISO operators were primarily concerned with returning flows 

on Path 44 to below the Path rating of 2,500 MW, but believed they had 30 minutes to 

do so. Unlike Path ratings, the separation scheme would not allow CAISO operators 30 

minutes to reduce flows on Path 44. CAISO did attempt to dispatch additional 

generation within SDG&E to reduce flows on Path 44. The other method to reduce flows 

would have been to manually shed load in SDG&E in time to prevent operation of the 

SONGS separation scheme. SDG&E estimates that it could have shed approximately 240 

MW in between two and two-and-a-half minutes. However, SDG&E was never 

instructed to shed load and was unaware of the need to shed load.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 52 -

Phase 6 Graphics

El Centro – Pilot 

Knob 161 kV line 

trip

Imperial Valley –

El Centro 230 kV 

CLR “S” line trip

generating 

units trip SONGS 

separation

15:38:02.4

15.38.21.2

South of SONGS – Calculated Phase Current

Time: 15:37:55 – The El Centro-Pilot Knob 161 kV line 

tripped. (230)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 53 -

Time: 15:37:55.110

Time: 15:38:02.4

Time: 15:38:21.2
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 54 -

G. Phase 7: Collapse of the San Diego/CFE/Yuma Island

Phase 7 Summary:

• Timing: Just after 15:38:21.2 to 15:38:38

• Underfrequency Load Shedding (UFLS) was not able to prevent the SDG&E/CFE/Yuma island 

from collapsing 

• SONGS nuclear units shut down even though they remained connected to the SCE side of the 

SONGS separation scheme

During phase 7 of the event the SDG&E/CFE/Yuma island broke into three 

separate islands, all of which collapsed due to an imbalance between generation and 

demand, resulting in severe underfrequency which tripped both loads and generation.

The SDG&E/CFE/Yuma island created by operation of the SONGS separation 

scheme had a significant imbalance between generation and load from the beginning. As 

a result, the frequency in the island rapidly declined. By less than a second after the 

SONGS separation scheme activated (15:38:22), the UFLS programs of SDG&E, APS,

and CFE had all began activating within the island. Figures 10 and 11, below show the 

frequency within the island as it collapses. All steps of the UFLS systems activated and 

system frequency in the island briefly stalled at approximately 57.2 hertz (Hz). CFE’s 

UFLS analysis showed 512 MW of load shed by 15:38:21.901. 

However, the same analysis showed that three CFE generators, totaling 459 MW,

tripped offline beginning at 15:38:21.905, partially negating CFE’s UFLS actions. In 

addition, a number of smaller generators, totaling about 130 MW, tripped only 0.5 

seconds later while CFE was still connected to SDG&E and while SDG&E’s UFLS

program was still working to shed load.53F

54 See Figure 11, below. The net effect of CFE’s 

UFLS actions and generator trips—512 MW shed by UFLS and 590 MW of tripped 

generation—was that CFE’s imports from SDG&E increased from approximately 440 

MW to approximately 520 MW. This worsened CFE’s system conditions and increased 

the stress on SDG&E before SDG&E’s underfrequency separation protection systems 

opened the ties between CFE and SDG&E. SDG&E also had three generators with 

underfrequency protection that operated at 57.3 Hz, above the frequency at which the 

system leveled out. Due to these early generation losses, the frequency continued to 

decline below 57 Hz, which was the underfrequency setting for the majority of generators

in the island. Thus, the island blacked out, shortly after separating into three subislands.

 54 The fact that several generators tripped during load shedding suggests that CFE may benefit from analyzing 

whether its UFLS program and generator underfrequency protection systems are coordinated. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 55 -

SDG&E, CFE, and APS UFLSs Operation

56.50

57.00

57.50

58.00

58.50

59.00

59.50

60.00

60.50

21.0 21.2 21.4 21.6 21.8 22.0 22.2 22.4 22.6 22.8 23.0 23.2

Time 15:38 (s)

1

4

3

2

5

6

7

8 9

SDG&E's Block 9 Load shed (3080 MW) 

complete. Frequency is at ~57.25 Hz

SDG&E 

UFLS 

Blocks' 

pick-up 

points

Separation

6 Blocks of CFE Load shed (512 MW) 

complete. Frequency is at ~57.4 Hz

5 Blocks of APS-Yuma Load shed (209 

MW) complete. Frequency is at ~57.4 Hz

CFE Generator 

(150 MW) Trips

Two CFE Generators 

(309 MW) Trip

Smaller 

CFE 

Generators 

(130 MW) 

Trip

Figure 10: Frequency, Voltage in the SDG&E/Yuma/CFE Island

Figure 11: Frequency Performance in the SDG&E/Yuma/CFE Island
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 56 -

The CFE island separated from SDG&E after their only two remaining ties 

tripped in rapid succession. At 15:38:22.2, the Otay Mesa-Tijuana 230 kV transmission 

line open-ended at Tijuana in CFE’s territory due to underfrequency protection.54F

55 Less 

than a second later, at 15:38:23.13, the Imperial Valley-La Rosita 230 kV transmission 

line open-ended at Imperial Valley in SDG&E’s territory by underfrequency 

protection.55F

56 According to CFE, its UFLS program was not designed for the operation of 

a SDG&E/CFE/Yuma island, but for the operation of a “southern WECC island.” 

The Yuma island separated from SDG&E at 15:38:23.12, when the Imperial 

Valley-North Gila 500 kV transmission line tripped by underfrequency protection. APS’s

UFLS operated on 26 out of the 28 feeders in the Yuma area prior to the loss of the local 

Yucca steam generators that were on line. However, there was insufficient local 

generation to stabilize the load pocket in Yuma. At 15:38:38, the Yuma island internal 

units tripped on underfrequency protection. 

At about the same time that it separated from CFE and APS’s Yuma pocket, 

SDG&E lost four generating units, totaling 570 MW, due to the generators’ 

underfrequency protection.56F

57 

Although the SONGS generators remained connected to the SCE side of the 

switchyard at SONGS, at about 15:38:27.5, or approximately six seconds after the 

SONGS separation scheme initiated, the SONGS turbines both experienced a brief 

acceleration in speed and tripped due to turbine control logic. At the same time, local 

system frequency at SONGS was observed to spike from 59.974 Hz to 61.203 Hz. After 

the initial impulse caused by the system separation, the frequency in the main body of 

the Western Interconnection peaked near 60.170 Hz. This can be seen on Figures 12 and

13, on the next page. The turbine trip initiated a reactor shutdown, and the units began 

coasting down. A little more than a second later, at 15:38:28.963, SONGS Unit 3 

electrically disconnected from the system, and less than three seconds after the reactors 

shut down, at 15:38:30.209, SONGS Unit 2 electrically disconnected from the system. 

 55 The Tijuana end opened instantaneously. Subsequently, the Otay Mesa end of the line in SDG&E’s territory

opened at 15:38:23.044 by underfrequency protection (with 1-second delay).

56 The line’s underfrequency setting was 57.9 Hz, with 1-second delay. The instantaneous underfrequency 

protection scheme at La Rosita in CFE’s territory failed to operate due to a bad fuse connection.

57 At 15:38:23.000, the Palomar Energy Center combustion turbines CT1 and CT2 tripped on underfrequency, 

followed by the heat recovery unit ST at 15:38:23.07 (all set to trip at 57.3 Hz with a 750 millisecond time delay). 

CT1 was loaded to 160 MW, CT2 was loaded to 165 MW, and ST was loaded to 195 MW at the time of the trips. It is 

believed that additional unit Goal Line LP, generating 50 MW, tripped around the same time due to a 58 Hz 

frequency with a 1-second time delay.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 57 -

Loss of the 2,300 MW of SONGS’ generation effectively reduced the loss of load for the 

main body of the Western Interconnection from a 3,400 MW loss to a net 1,100 MW load 

loss. This made the recovery from the resulting overfrequency event much easier. The 

SONGS generators did not lose offsite power because the SONGS switchyard was still 

connected to the SCE system.

 

Figure 12 : Frequency Excursion in WECC 

Interconnection Immediately after the SONGS 

Figure 13: SONGS Generation Trips and 

Auxiliary Loads Transfer to 230 kV Bus
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 58 -

By 15:38:38, the SDG&E, CFE and Yuma islands had all collapsed, leaving 

approximately 2.7 million customers without power.

Phase 7 Graphics

 

Time: 15:38:30 – The South of SONGS Separation Scheme 

operates and both SONGS units tripped. (300)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 59 -

 

 

Yuma Separates (Time: 15.38.23.12)

CFE Separates (Time: 15.38.23.13)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 60 -

 

SDG&E, CFE, and Yuma Blackout ( by 15.38.30)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 61 -

H. System Restoration

None of the affected entities needed to implement black start plans because they 

all were able to access sources of power from their own or a neighbor’s system that was 

still energized. The restoration process generally proceeded as expected, and some 

entities restored load more quickly than they had expected. The following charts 

indicate how long it took the affected entities to fully restore their lost load, generation, 

and transmission. 

LOAD RESTORATION EFFORTS

Entities

Demand 

Interrupted 

(MW)

Time Until

Demand Fully

Restored

Date 

Restored

100 % 

Demand 

Restored 

(hrs)

Number of

Customers

Affected

SDG&E 4,293 03:23 9/9/11 12 1.4 Million

CFE 2,205 01:37 9/9/11 10 1.1 Million

IID 929 21:40 9/8/11 6 146,000

APS 389 21:12 9/8/11 6 70,000

WALC 74 22:23 9/8/11 6.5 557F

58

 58 The majority of WALC’s lost load (64 MW) affected APS customers. SCE lost 117 fringe load customers who 

were served by the SDG&E system.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 62 -

GENERATION RESTORATION EFFORTS

Entities Generation 

Lost (MW)

Time

Generation 

Restored 

Date Restored Generation 

Restored (hrs)

SCE 2,428 06:33 9/12/11 87

SDG&E58F

59 2,229 06:20 9/10/11 39

CFE 1,915 23:43 9/10/11 56

IID 333 20:42 9/8/11 5

APS 76 20:37 9/8/11 5

TRANSMISSION RESTORATION EFFORTS

Entities

Final 

Transmission 

Restored (kV)

Time

Transmission 

Restored 

Date 

Restored

Transmission 

Restored (hrs)

IID 230

161

03:37

00:31

9/9/11

9/9/11

12

9

SDG&E 500

230

17:36

03:47

9/8/11

9/9/11

2

12

APS 500 16:51 9/8/11 1.5

WALC 161 17:09 9/8/11 1.559F

60

CFE 230

115

04:03

01:58

9/9/11

9/9/11

12.5

10

 

 59 According to SDG&E, after restoring the SDG&E transmission systems, CAISO took over restoring SDG&E’s 

generation.

60 This represents the time it took WALC to restore its 161/69 kV Gila transformers, however, none of WALC’s 

transmission lines were lost in the outage.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 63 -

WECC RC could have taken a more active role in coordinating the restoration

efforts. WECC RC has the largest area of visibility and more advanced real-time study 

tools than the TOPs. During a multi-system restoration, issues are likely to arise 

between neighboring BAs and TOPs that may require either a neutral decision maker, or 

rapid technical analysis of unplanned system conditions. WECC RC is uniquely situated 

to provide such assistance. WECC RC should clarify its role, and the real-time 

information it can provide, in emergency situations like a multi-system restoration. 

WECC RC should also specifically address the issue of coordination among other 

functional entities (like BAs and TOPs) in its operating area, outlining the areas of 

responsibility during system restoration and other emergencies. 

The inquiry reviewed recordings and other data about restoration which 

disclosed the following incidents that could have benefitted from better WECC RC 

coordination and assistance in real time:

• A 30-minute debate occurred between SCE, which was attempting to provide cranking power to 

SDG&E to restore SDG&E’s system, and the SONGS operators, about the conditions necessary for 

resetting the SONGS separation scheme lockout relay.

• Recordings showed a lack of clarity among WECC RC, CAISO, and SDG&E about responsibilities 

for restoration efforts. Among other things, this resulted in a SONGS operator making a unilateral 

decision to open a circuit breaker on the line responsible for restoring power to SDG&E’s system, 

leaving the line in a less reliable configuration (connected to a single bus).
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 64 -

IV. CAUSES, FINDINGS, AND RECOMMENDATIONS

A. Planning

Next-Day Planning

 Background

TOPs are required to perform next-day studies to identify and plan for potential 

limitations on their system in the day-ahead timeframe, and to coordinate these studies 

with their neighboring TOPs.60F

61 These studies provide a proactive mechanism to ensure 

that the system can be operated reliably and allow time to develop effective operating 

solutions.61F

62 These solutions include, among other things, effective control actions 

needed to return the system to a secure state in anticipated normal and contingency 

system conditions. The development of these plans in the day-ahead timeframe is 

critical because it would be nearly impossible, due to the complexity of the BPS, for 

control room operators to return the system to a secure operating state under stressed 

conditions without effective action plans developed in advance. The adequacy of nextday studies depends on how extensively and accurately facilities and next-day system 

conditions are incorporated into the models used for the studies. This includes 

consideration of a reasonably accurate, current, and complete list of external 

contingencies that could impact a TOP’s system as well as internal contingencies that 

could impact external SOLs. Consistency of study inputs among all TOPs and BAs is also 

critical for reliable operation.

The inquiry found that the affected TOPs’ and BAs’ procedures for conducting 

next-day studies and models used in these studies vary considerably. As explained more 

fully below, APS does not conduct next-day studies, relying, instead, on two sets of 

studies, conducted on a seasonal and annual basis, that consider a list of possible, 

predetermined contingency scenarios and provide plans to mitigate the contingencies if 

violated. Meanwhile, IID has a policy of conducting next-day studies each day, but 

between April and October of 2011, it failed to perform the required studies on a daily 

basis. All other affected TOPs conduct next-day studies, but they use models that do not 

adequately reflect next-day operations of facilities in networks external to them. These 

 61 See NERC Reliability Standard TOP-002-2b R11.

62 See, e.g., NERC Reliability Standard TOP-002-2b (“Current operations plans and procedures are essential to be 

prepared for reliable operations, including response for unplanned events.”).
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 65 -

TOPs’ next-day studies also do not consider a full list of internal and external 

contingencies that could impose limitations on their daily operations or external

operations. Moreover, most of these TOPs’ next-day studies do not consider the impact 

of sub-100 kV facilities on BPS reliability, such as the impact of IID’s CV transformers. 

WECC RC is the highest level of authority responsible for reliable operation of the 

BPS in the Western Interconnection, with the authority to prevent or mitigate emergency 

operating conditions in the next-day and real-time timeframes. As such, WECC RC also 

conducts next-day studies for the entire Western Interconnection and builds its model 

from the previous day’s peak State Estimator case, which includes all facilities operated 

at 100 kV and above and some sub-100 kV facilities. WECC RC then incorporates 

forecast information, which typically includes transmission outages as provided by 

TOPs, generation outages or derates of 50 MW or greater as provided by TOPs, as well as 

load forecasts, expected net interchange, and unit commitment forecast data from BAs. 

While WECC RC has a more extensive representation of facilities throughout the WECC 

footprint in its model than any individual TOP, it does not necessarily monitor or alarm 

for certain lower voltage facilities and facilities deemed non-BES that can impact BPS 

reliability. Moreover, because some of the forecasted information can change between 

the time the TOPs and BAs provide it to WECC RC and the time WECC RC runs its nextday studies, WECC RC’s next-day studies might not accurately reflect next-day 

operations. 

The September 8th event exposed four weaknesses with the foregoing procedures 

for conducting next-day studies in WECC’s region. These weaknesses are detailed in the 

following four findings. A common theme prevails in all four findings: the affected

entities do not accurately account for external next-day operating conditions or potential 

external contingencies that could impact their systems.

Finding 1 Failure to Conduct and Share Next-Day Studies:

• Not all of the affected TOPs conduct next-day studies or share them with 

their neighbors and WECC RC. As a result of failing to exchange studies, on 

September 8, 2011 TOPs were not alerted to contingencies on neighboring 

systems that could impact their internal system and the need to plan for such 

contingencies.

Recommendation 1: 

• All TOPs should conduct next-day studies and share the results with 

neighboring TOPs and the RC (before the next day) to ensure that all 

contingencies that could impact the BPS are studied.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 66 -

Failure to Conduct Next-Day Studies

APS does not conduct next-day studies. Instead, it relies on two sets of studies, 

conducted on a seasonal and annual basis, for its daily operations. First, APS uses its 

summer and winter seasonal studies for the non-WECC Rated Paths within its 

transmission system. APS performs these studies on a model that it builds from the 

WECC heavy summer base case. In a coordinated effort with other entities in Arizona, it 

updates this WECC base case with anticipated loads and resources from the state. APS

then adds a detailed representation of the entire state’s network, including its own 

subtransmission system down to the 12 kV distribution system, to finalize the summer 

model. To create its winter model, APS modifies the summer model with winter peak 

conditions throughout Arizona.

Once these summer and winter models are complete, APS studies a set of 

predetermined contingencies, and relies on the results to determine the response of its 

transmission system to single and multiple contingencies during peak load conditions 

with planned outages modeled. The studies’ list of contingencies is based on past 

studies, operating experience, and engineering judgment. The studies also establish 

mitigating measures for contingencies that do not meet loading or voltage guidelines.

Second, APS relies on a single manual, developed annually, as a guide for its daily 

operations on four Rated Paths within its system. This manual is the result of studies of 

possible, predetermined contingencies on Rated Paths. The results and operational 

instructions in this manual are based on seasonal models that APS develops in 

coordination with four WECC regional study groups, led by CAISO. CAISO first sends a 

base case to each study group to update with topology changes for the upcoming 

season. Individual members of each study group also update the model with details from 

their systems. CAISO then incorporates all of the updates and stresses key Paths in 

California before sending the model back to the study groups. APS uses this model as a 

starting point to study the four Rated Paths in its system. APS analyzes the resulting 

peak-load model using a predetermined set of single and double contingency events that 

are focused primarily on high-voltage transmission outages to determine required 

actions to secure the system for the next most critical N-1 event.62F

63 The manual directs 

APS to rerate relevant Path(s) and identifies necessary mitigating measures as long as 

the contingency (or multiple contingency) scenario is included in the manual. The 

manual, however, may not include a particular contingency (or multiple contingency) 

 63 APS’s manual covers only 500 kV and 345 kV facilities, and nothing lower.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 67 -

scenario, or may not accurately reflect the internal and external system topology for the 

day in question, resulting in the potential for unforeseen circumstances. 

Thus, APS uses seasonal studies for non-Rated Paths and the manual for Rated 

Paths as tools in the day-ahead timeframe, without any additional analysis to validate 

that the tools remain valid for the next day’s specific configuration and operation, such 

as transmission or generation outages external to APS’s footprint that were not 

anticipated at the time the base seasonal study was performed. APS maintains that these 

tools are sufficient for day-ahead purposes because they include the most severe 

contingencies identified in its system. This viewpoint overlooks the purpose of next-day 

studies—to plan for next-day operations in light of conditions that change daily. By 

relying on tools based on studies conducted on a seasonal and annual basis, APS cannot 

account for all plausible daily scenarios. For typical days that fall within the boundaries 

of the underlying studies and analysis, APS’s tools may be viable. For atypical days 

where conditions fall outside the studied boundaries, however, this approach may not be 

adequate. For example, September 8, 2011, was an atypical day not contemplated by 

APS’s manual, as the manual did not account for various generation outages in effect for 

maintenance.

Between April and October 2011, IID also did not consistently perform adequate 

next-day analyses for each day. Although IID had a policy of conducting separate nextday analyses for each new day, it failed to consistently perform the required 

analyses. Specifically, IID produced a document each new day showing various changes 

in weather, load and generation forecasts, planned facility outages, potential contingency 

violations, or mitigation measures for identified contingencies, but did not always 

perform the underlying power flow studies for each day between April and October

2011. On average, between April 2011 and October 2011 IID actually performed a study 

no more than two times per week. For the other days, IID simply referenced past 

studies. For example, it appears that IID did not perform a separate, updated study for

September 8, 2011, because the powerflow study case provided for this day does not 

match the contingency results included in the daily operations guide for the day. In 

other words, it appears that for September 8, 2011, IID simply changed the forecasted 

data without actually performing the next-day study. Instead, IID referenced a previous 

study. The referenced study, however, was not valid because it did not match the load 

and generation dispatch data for the day, and there were differences in projected 

overloads reported as potential contingencies. IID’s next-day studies were purportedly 

reviewed by IID for accuracy, but these discrepancies were not identified. IID discovered 

this issue during the course of the inquiry and is in the process of implementing 

corrective actions to ensure accurate next-day analyses are completed in the future.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 68 -

Finally, the inquiry heard on more than one occasion from TOPs, including APS, 

that WECC RC was responsible for conducting next-day studies or that WECC RC should 

conduct next-day studies that TOPs are currently responsible for conducting. WECC 

RC’s next-day studies for the entire Interconnection, however, are not intended to 

substitute for the TOPs’ next-day studies of their own systems. 

Failure to Effectively Share and Coordinate Next-Day Studies

In addition to finding that not all entities conduct next-day studies, the inquiry 

found problems with sharing and coordination among the affected TOPs that do conduct 

such studies. The affected TOPs do not consistently share their studies with neighboring 

TOPs, BAs, and the RC. TOPs generally provide studies to WECC RC only if the RC 

identifies an issue in its study and specifically asks to review a TOP’s study. In addition, 

WECC RC’s method of sharing its next-day studies with other entities is not effective. 

Specifically, WECC RC’s practice is to share the results of its next-day studies when 

conditions warrant, or when it receives a request for a study result.63F

64 WECC RC posts 

on a secure Internet portal a list of limitations or SOLs identified by its next-day studies 

for individual TOPs and BAs to view, but it is up to TOPs and BAs to access this list. 

Also, this list contains only issues that WECC RC deems significant and does not include 

basic, next-day operating conditions, such as scheduled outages.

One example of the adverse consequences of these sharing and coordination 

issues relates to the 600-plus MW of TDM generation that was offline for maintenance 

on September 8th. The TDM generation outage was included in WECC RC’s and 

CAISO’s next-day studies, and posted on CAISO’s website, but not incorporated into 

other entities’ next-day models and studies.64F

65 WECC RC receives outage information 

from TOPs and BAs through its Coordinated Outage System (COS). While TOPs and BAs 

submit their own information into COS, they cannot access information submitted by 

others. IID could have benefitted from knowledge of the TDM outages. The TDM units 

radially connect to the Imperial Valley substation, jointly owned by IID and SDG&E. If 

the TDM units had been online, they could have mitigated northern IID overloads on the 

 64 See WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at

http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/WECC RC Operations Planning.pdf. 

65 CAISO knew about the outages because the TDM units participate in the CAISO market. CAISO posts daily 

outage unit status reports on its public website that provide the best available data at the date and time of the 

report, for generation units that participate in CAISO’s market. These outages are posted at 

http://www.caiso.com/market/Pages/OutageManagement/UnitStatus.aspx. In CAISO’s archives, the TDM units 

are shown on outage on September 7 and 8, at a minimum. Dispatch details, however, are not included. WECC 

RC receives CAISO’s outage unit status reports daily by email and was aware of the outages. However, IID and 

APS did not know about the TDM outages.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 69 -

CV and Ramon transformers that resulted when H-NG tripped. If IID had learned about 

these outages from WECC RC or CAISO, it could have incorporated the outages in the 

day-ahead timeframe and dispatched additional generation, or taken other control 

actions, to compensate for the overloads on its system caused by having these generators 

offline and the H-NG tripping. 

The September 8th event illustrates that conducting next-day studies and 

sharing the results of such studies are critical to allow TOPs to identify and plan 

for potential contingencies. 

Finding 2 Lack of Updated External Networks in Next-Day Study Models:

 • When conducting next-day studies, some affected TOPs use models for 

external networks that are not updated to reflect next-day operating 

conditions external to their systems, such as generation schedules and 

transmission outages. As a result, these TOPs’ next-day studies do not 

adequately predict the impact of external contingencies on their systems or 

internal contingencies on external systems.

Recommendation 2:

• TOPs and BAs should ensure that their next-day studies are updated to 

reflect next-day operating conditions external to their systems, such as 

generation and transmission outages and scheduled interchanges, which can 

significantly impact the operation of their systems. TOPs and BAs should 

take the necessary steps, such as executing nondisclosure agreements, to 

allow the free exchange of next-day operations data between operating 

entities. Also, RCs should review the procedures in the region for 

coordinating next-day studies, ensure adequate data exchange among BAs 

and TOPs, and facilitate the next-day studies of BAs and TOPs.

As a starting point for their next-day studies, the affected TOPs use models from 

either a TOP’s seasonal base case or the previous day’s EMS model, if available. The 

seasonal base case represents next-day operating conditions internal to the TOPs’ 

systems, but leaves external networks exactly as they were represented in the WECC 

seasonal base case. The affected TOPs’ EMS models sometimes include only one or two 

buses outside each TOP’s internal footprint. Thus, neither type of day-ahead model

contains actual day-ahead forecasts of system conditions external to each TOP’s system. 

For example, leading into September 8th, the affected TOPs had limited knowledge of 

the current status of transmission facilities, expected generation output, and load 

predictions outside their footprints. Consequently, their next-day studies could not 

adequately predict the impact of external contingencies on their systems or of internal 

contingencies on external systems.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 70 -

IID’s next-day study for September 8th illustrates the adverse effects of not 

accounting for external next-day planned operations. IID used the WECC heavy summer 

seasonal base case to model external conditions for its next-day study for September 8th. 

This base case reflects that most external generation is online to meet summer peak 

loads. A heavy summer base case does not accurately represent a shoulder season day 

like September 8th. By September, both generation and transmission maintenance had 

started.

For example, on September 8th TDM generator units in Mexico, totaling more 

than 600 MW, were offline for maintenance. These units are external to IID and radially 

connect to IID’s jointly owned Imperial Valley substation. When online, this generation 

can help to mitigate overloads on the CV and Ramon transformers in IID’s system. 

Because IID relied on a heavy summer seasonal model for external networks and did not 

incorporate any updates about the TDM generation, its next-day study did not reflect the 

maintenance outage of these units. With the TDM generation incorrectly represented as 

being online, IID’s next-day study did not correctly identify how much the loss of H-NG 

would overload IID’s transformers in its 92 kV system. In fact, IID’s next-day study for 

September 8, 2011, did not show that the loss of H-NG would overload the CV

transformers to their trip point.65F

66 If IID had learned about the TDM outages (whether 

from CAISO’s website or BY some other method) and incorporated the information into 

its model, it could have dispatched additional generation, adjusted load, or taken other 

control actions before the loss of H-NG to mitigate such overloading. 

As mentioned above, WECC RC receives next-day data from the entities through 

interfaces such as the COS. WECC RC is well-situated to facilitate data-sharing among 

the 37 BAs and 53 TOPs in the WECC footprint. Given the large number of BAs and 

TOPs in the WECC region, some of which are relatively small in size and resources, 

central coordination and facilitation may be necessary to ensure that all BAs and TOPs 

accurately reflect next-day operating conditions external to their system.66F

67 WECC RC 

has been working to facilitate data sharing by drafting and circulating a universal 

 66 The heavy summer base case has more than 1,000 MW more generation in the affected area than was 

available on September 8, 2011. In addition to not representing the offline generation, IID’s study, by relying on 

the heavy summer base case, did not accurately reflect the flow on H-NG. The heavy summer base case shows 

flow on H-NG as 1,118 MW, while actual flow on H-NG at the time of the trip was 1,391 MW.

67 Under current WECC RC procedures, the RC only shares the results of its operational planning analyses if the 

results indicate the need for specific operational actions to prevent or mitigate an instance of exceeding an 

operating limit. WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at

http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/WECC RC Operations Planning.pdf.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 71 -

nondisclosure agreement. As this report was being finalized, less than 30 of the 

approximately 100 discrete entities within WECC had signed the agreement.67F

68

Finding 3 Sub-100 kV Facilities Not Adequately Considered in

Next-Day Studies: 

• In conducting next-day studies, some affected TOPs focus primarily on the 

TOPs’ internal SOLs and the need to stay within established Rated Path 

limits, without adequate consideration of some lower voltage facilities. As a 

result, these TOPs risk overlooking facilities that may become overloaded

and impact the reliability of the BPS. Similarly, the RC does not study sub100 kV facilities that impact BPS reliability unless it has specifically been 

alerted to issues with such facilities by individual TOPs or the RC has 

otherwise identified a particular sub-100 kV facility as affecting the BPS.

Recommendation 3: 

• TOPs and RCs should ensure that their next-day studies include all internal 

and external facilities (including those below 100 kV) that can impact BPS 

reliability.

The September 8th event showed that some sub-100 kV facilities can have 

significant impacts on BPS reliability, such as causing instability or cascading outages. 

Yet, it appears that these facilities are not adequately considered in the day-ahead 

timeframe. For example, IID’s 92 kV network runs parallel to two major transmission

Paths: (1) Path 44, which connects to the SWPL via the Palo Verde-Devers 500 kV line 

(part of Path 49) and runs to the north of IID; and (2) the SWPL, which runs to the south 

of IID. Given the parallel nature of its system, IID’s 92 kV system is forced to carry a 

significant portion of any east-west power flows whenever segments of Path 44 or the 

SWPL are out of service. 

Because none of the affected TOPs, besides IID, considered IID’s 92 kV network

in their next-day studies, they were not aware how their internal contingencies could 

affect IID’s 92 kV network, or how an overload on IID’s 92 kV network could affect their 

systems. For example, APS does not routinely study IID’s lower voltage facilities, 

including the CV and Ramon transformers, in the day-ahead timeframe. APS uses 

seasonal studies and its operations manual as its tools in the day-ahead timeframe. 

While the model used for the seasonal studies physically has IID’s 92 kV network 

represented, neither the model nor the operations manual are used to consider the next 

day’s specific configuration and operation, such as transmission or generation outages 

external to APS’s footprint that were not anticipated at the time the seasonal study and 

 68 The agreement does address market concerns by requiring entities who participate in data-sharing to respect 

the separation of market and operations functions.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 72 -

manual were updated. As a result, APS was not able to predict what occurred on IID’s 

system—increased flows and overloading on its 92 and 161 kV transformers and 

transmission lines—when H-NG tripped offline. Similarly, affected TOPs other than IID 

do not consider in their day-ahead planning how the loss of the CV and Ramon 

transformers, leading to the S Line RAS operation, could adversely affect their internal 

systems. Accordingly, TOPs should revise their next-day study practices to account for 

all facilities, including those operated below 100 kV, that impact BPS reliability.

WECC RC also did not adequately consider sub-100 kV facilities not identified as 

BES that can have significant impacts on BPS reliability. While WECC RC does model 

IID’s CV transformers in its next-day studies, prior to September 8, 2011, it did not “flag” 

them in its studies for active monitoring.68F

69 This means that WECC RC had data showing 

that the transformers would overload under certain conditions, but the overloads were 

not identified by alarms to be seen by RC operators. WECC RC did not actively monitor 

the CV transformers in its next-day studies because they are below 100 kV and IID had 

not alerted WECC RC to any issues that would warrant monitoring of the transformers. 

Given the CV transformers’ impact on BPS reliability, WECC RC should actively monitor 

these transformers.69F

70

Finding 4 Flawed Process for Estimating Scheduled Interchanges: 

• WECC RC’s process for estimating scheduled interchanges is not adequate to ensure that 

such values are accurately reflected in its next-day studies. As a result, its next-day 

studies may not accurately predict actual power flows and contingency overloads.

Recommendation 4:

 

• WECC RC should improve its process for predicting interchanges in the dayahead timeframe. 

Interchanges are energy transfers that cross BA Areas. Interchanges can affect 

flows across transmission systems, so forecasting accurate interchanges is important in 

the day-ahead timeframe to plan for potential overloading. WECC RC’s process for 

estimating scheduled interchanges is not adequate to ensure that the scheduled 

interchanges incorporated into its next-day studies are accurate. Under this process, by 

10:00 AM each day BAs provide WECC RC with all interchanges they have approved for 

 69 To aid in effectively and efficiently processing and analyzing reliability data for the entire Western 

Interconnection, WECC RC has the option of flagging a subset of facilities for active monitoring in its studies. It 

has since updated this feature to flag the CV transformers for monitoring.

70 WECC RC has implemented new procedures since September 8, 2011, to monitor RTCA results for the CV 

transformers.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 73 -

the next day. The BAs typically submit this information once per day without any 

subsequent updates. WECC RC then validates these scheduled interchanges by 

comparing the values with what the BAs provided the prior day and with what WECC 

RC’s state estimator observed in the prior days and weeks.

The accuracy of interchange data in WECC RC’s next-day studies could be 

improved by allowing for updates closer to real time. BAs’ interchange data are likely to 

change after their 10:00 AM submittal to WECC RC. Some BAs have automated 

systems, which send updates of interchange data to WECC RC. Most BAs submit the 

data manually, only once at 10:00 AM. Inclusion of a process or requirement for BAs to 

update their scheduled interchanges after their 10:00 AM submission would increase the 

likelihood of accurate interchange data.

The accuracy of interchange data affected WECC RC’s next-day study for 

September 8, 2011. Specifically, the scheduled interchanges reflected in WECC RC’s 

next-day study for September 8, 2011, were not sufficiently accurate to predict that IID’s 

CV 230/92 kV transformers would overload to their trip point upon the loss of H-NG. 

After the event, WECC RC ran its next-day study using actual interchanges, and found 

that the CV transformers would overload beyond their tripping threshold upon the loss 

of H-NG. If WECC RC had used more accurate net interchange data and flagged the CV

transformers for monitoring, it could have learned of the issues with these transformers 

and alerted IID or issued directives for control actions to mitigate the situation, such as 

increasing generation or shedding load.

Seasonal Planning

 Background

Following a set of disturbances in the Western Interconnection during the 

summer of 1996, WECC established a new seasonal planning structure designed to avert 

system-wide disturbances while maximizing the commercial availability of transmission 

capacity. This new structure involved the creation of the Operating Transfer Capability

Policy Committee (OTCPC). The purpose of the OTCPC was to provide coordinated 

standard development and determination of seasonal Operating Transfer Capabilities

(OTCs), or Operating Transfer Limits,70F

71 within the Western Interconnection.71F72 

 71 OTCs are now known as SOLs.

72 The OTCPC itself was abolished and replaced with a new structure in June 2011; however, planning for the 

seasonal period in which the blackout occurred was performed under the OTCPC structure, so the inquiry’s 

analysis focused on the OTCPC structure. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 74 -

Among other things, the OTCPC was designed to be responsible for determining 

which transmission Paths should be studied, facilitating OTC dispute resolution, 

ensuring that seasonal studies maintain consistent standards and methodologies, and 

approving seasonal studies of OTC limits. To that end, the OTCPC was charged with 

reviewing and approving study plans and technical simulation results; developing 

policies and procedures addressing seasonal OTCs; establishing working groups such as 

subregional study groups and the Operating Procedures Review Group; addressing OTC 

seams issues between subregions; and providing technical guidance. 

The seasonal study plans that are reviewed and approved by the OTCPC were

created by a set of four subregional study groups (sometimes referred to as SRSGs or 

simply subregions). There were four groups: (1) the California/Mexico Operations 

Study Subcommittee (OSS); (2) the Northwest Operational Planning Study Group 

(NOPSG); (3) the Rocky Mountain Subregional Study Group (RMSG); and (4) the 

Southwest Area Study Group (SASG). The affected entities were members of two of 

these groups: the OSS (CAISO, SDG&E, SCE, CFE, and IID) and the SASG (APS, 

WALC).

On an annual basis, each subregional study group reviewed the Paths in its 

subregion to determine which Paths should be studied and the system conditions under 

which they should be studied. Then, seasonally, the four subregional study group chairs 

submitted their recommendations of which Paths to study to the OTCPC for review and 

approval. Following OTCPC’s approval, the studies were performed in accordance with 

the OTC study process. This process began with establishment of an initial “base case” 

by WECC staff, with input from representatives of each subregional group. The “base 

case” is a computer model of projected or starting power system conditions for a specific 

point in time. For the 2010-2011 planning year, five base cases were used.72F

73 Once the 

comments from the four subregional representatives were incorporated, the final cases 

were made available via WECC’s web site for adjustment and modification by 

subregional members in order to forecast expected seasonal conditions on the system. 

The subregional members performed their own seasonal studies, and then met to discuss 

the results. A subregional seasonal planning case was produced on this basis, but no 

further studies were performed. Subregional seasonal cases were shared among the four 

subregions via liaisons from the other subregions. No comprehensive WECC-wide Path 

rating study was prepared on the basis of the four subregional studies.

 73 These included low summer load, high summer load, low winter load, high winter load, and high spring load.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 75 -

In addition to, and apart from, the seasonal planning studies just described, 

TOPs also conduct their own seasonal studies focusing on their own internal networks. 

These internal studies follow a different process from the seasonal Path rating studies,

though both begin with the WECC base case. Internal seasonal studies, however, are not 

aggregated or reviewed at the subregional level. Instead, TOPs generally replace the 

information from the WECC base case with more accurate and granular detail for their 

own areas only. Once updated, the TOPs perform contingency analyses for their own 

internal purposes. They then share with their neighbors the results of these operational 

studies, which typically contain only the default data from the WECC base case for 

everything outside of their own areas. 

The inquiry identified a number of issues relating to both types of seasonal 

planning by the affected entities. These issues impaired the accuracy and effectiveness of 

the seasonal studies by excluding, in various ways, pertinent issues and information that 

should have been taken into consideration.

Finding 5 Lack of Coordination in Seasonal Planning Process:

 • The seasonal planning process in the WECC region lacks effective 

coordination. Specifically, the four WECC subregions do not adequately 

integrate and coordinate studies across the subregions, and no single entity 

is responsible for ensuring a thorough seasonal planning process. Instead of 

conducting a full contingency analysis based on all of the subregions’ studies, 

the subregions rely on experience and engineering judgment in choosing 

which contingencies to discuss. As a result, individual TOPs may not identify 

contingencies in one subregion that may affect TOPs in the same or another 

subregion.

Recommendation 5:

• WECC RE should ensure better integration and coordination of the various 

subregions’ seasonal studies for the entire WECC system. To ensure a 

thorough seasonal planning process, at a minimum, WECC RE should 

require a full contingency analysis of the entire WECC system, using one 

integrated seasonal study, and should identify and eliminate gaps between 

subregional studies. Individual TOPs should also conduct a full contingency 

analysis to identify contingencies outside their own systems that can impact 

the reliability of the BPS within their system and should share their seasonal 

studies with TOPs shown to affect or be affected by their contingencies.

No comprehensive WECC-wide seasonal studies are performed. With respect to 

seasonal Path rating studies, a representative or leader from each subregion adapts the 

WECC base case on the basis of input from subregional members, and then makes these 

revised cases available to the other subregional members for review, comment, and 

approval. The subregional leader then conducts the seasonal studies concentrating only 

on the rated Paths in the subregion. The results of the seasonal Path rating studies are 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 76 -

shared and discussed first among the subregion’s members, and then with the other 

subregions, but neither WECC RE nor the OTCPC performs or mandates any further 

seasonal studies, and no new WECC-wide seasonal study is performed to reflect the 

input of all of the subregions. Instead, representatives of the subregional groups gather 

informally to discuss the results of their seasonal studies and rely on experience and 

engineering judgment to identify and resolve any issues. 

The events of September 8, 2011, illustrate that this process is not adequate: the 

tripping of one line in a rated Path—H-NG, which is part of Path 49—ultimately led to 

the tripping of other lines in other rated Paths, including Paths 44 and 45. Focusing 

exclusively on Path ratings—and solely on a subregional basis—ignores network facilities 

that can impact rated Paths (and vice-versa) and does not account for the 

interrelationships of Paths and other facilities across WECC’s subregions. 

With respect to the internal seasonal studies, there is even less coordination. 

TOPs generally perform internal seasonal studies using models that include detailed data 

for their own system, but default to WECC base case data, which may not be sufficiently 

detailed or updated, for everything else. TOPs perform contingency analysis for their 

own internal areas using this model. No study is done to identify the impact of external 

contingencies on the TOP’s system, or the impact of the TOP’s internal contingencies on 

the SOLs of other TOPs. TOPs provide the results of their internal seasonal studies to 

neighboring TOPs for informational purposes, after which those TOPs may or may not 

provide comments. 

In all, this situation indicates that the TOPs’ internal seasonal planning studies 

are too heavily reliant upon the assumptions underlying and reflected in a single WECC

base case, and do not consider and study impacts of variations from that base case. 

The September 8th event demonstrated one example where better integration of 

seasonal studies across two subregions is needed. When H-NG (part of Path 49) tripped, 

approximately 12% of the flow from that line, which is located in the SASG subregion, 

was transferred across IID’s 230/92kV transformers, via the IID 92kV local network to 

the southern IID 161 kV network, which are all in the OSS subregion. This additional 

flow on IID’s CV transformers ultimately resulted in cascading outages and impacted 

Paths 44 and 45. The affected entities were unaware of this potential inter-Path impact, 

because the SASG and OSS studies had not been jointly considered. Moreover, since the 

subregional studies concentrate only on Path ratings, this flow transfer was not 

apparent. If the seasonal studies of SASG and OSS had been better coordinated and 

more rigorously analyzed, the potential for the loss of H-NG to overload IID’s 92 kV 

network could have been identified and mitigation plans developed.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 77 -

Finding 6 External and Lower-Voltage Facilities Not Adequately Considered 

in Seasonal Planning Process: 

• Seasonal planning studies do not adequately consider all facilities that may 

affect BPS reliability, including external facilities and lower-voltage 

facilities. 

Recommendation 6: 

• TOPs should expand the focus of their seasonal planning to include external 

facilities and internal and external sub-100 kV facilities that impact BPS 

reliability.

As noted above, TOPs performing subregional Path rating studies do not 

sufficiently account for the impact of facilities external to their subregion, or facilities 

within their subregion that are not part of a rated Path. Moreover, no WECC-wide Path 

rating study is performed to harmonize and analyze the impact of one subregion on the 

rest of the subregions. 

The problem with this approach is illustrated in the example cited above: The 

tripping of a part of one rated Path, H-NG, which is part of Path 49, led to the tripping of 

portions of other rated Paths. The mechanism whereby these other trips were triggered 

was the transfer of flow across low-voltage (below 100 kV) facilities that were located in a 

different subregion. Under the approach to Path rating studies in place at the time, it 

would have been impossible for WECC RE or TOPs to anticipate and study this 

possibility, because it occurred across subregions, indirectly, via lower-voltage facilities. 

Even if seasonal Path rating studies had been performed across subregions, these studies 

would not have anticipated this possibility, unless they also took into account lowervoltage facilities, which they presently do not.

The internal seasonal planning studies of the various TOPs are subject to similar 

omissions, although these studies encompass more than just the rated Paths and contain 

more detail than the Path rating studies. The practices of individual TOPs differ, but 

none contains sufficient detail and accuracy with respect to facilities outside their own 

footprints, as well as lower-voltage facilities. IID, for example, has explained that it 

“does not identify or study components outside of the IID territory below 100 kV for 

impacts on the BPS reliability in its territory,” nor does it “identify or study components 

inside of the IID territory below 100 kV for impacts on the BPS reliability outside of its 

territory.” 

Similarly, while CAISO studies in its seasonal planning process “all of the 

transmission components that it operates, some of which are below 100 kV,” it has also 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 78 -

acknowledged that it “does not have the necessary information to accurately study 

transmission components below 100 kV outside of its territory to determine if they have 

an impact on the BPS reliability in [CAISO’s] service territory.” 

The events of September 8, 2011, demonstrate that sub-100 kV facilities in 

parallel with BPS systems can have a significant effect on BPS reliability. The loss of HNG caused the overloading and tripping of both 230/92 kV transformers at CV, which in 

turn caused another sub-100 kV transformer to trip at Ramon, which led to the 

cascading outages discussed in detail above. This possibility was not studied as part of 

the seasonal studies by any of the TOPs, other than IID, because the CV transformers’

secondary windings are below 100 kV. The seasonal studies conducted by affected TOPs, 

other than IID, did not study the impact of the CV transformers. If the CV transformer 

contingency overloads had been identified as limiting elements in the seasonal plans, the 

cascading outages might have been avoided or lessened by having pre-contingency 

mitigation in place, such as increasing generation on IID’s 92 kV system.

Finding 7 Failure to Study Multiple Load Levels: 

• TOPs do not always run their individual seasonal planning studies based on 

the multiple WECC base cases (heavy and light load summer, heavy and light 

load winter, and heavy spring), but, instead, may focus on only one load 

level. As a result, contingencies that occur during the shoulder seasons (or 

other load levels not studied) might be missed.

Recommendation 7:

• TOPs should expand the cases on which they run their individual planning 

studies to include multiple base cases, as well as generation maintenance

outages and dispatch scenarios during high load shoulder periods. 

WECC created five base cases for the 2010-2011 season— heavy and light load 

summer, heavy and light load winter, and heavy spring—intended to capture the 

spectrum of possible loading configurations at different times of the year. The inquiry 

found that some of the affected TOPs deemed it unnecessary to run individual planning 

studies based on the multiple WECC base cases. Instead, these TOPs identified some 

subset of these base cases that they concluded were most relevant to their concerns and 

ran studies based on only that subset of base cases. Some TOPs employed only one base 

case—the heavy load summer base case—for planning the season during which the 

September 8, 2011 blackout occurred. By limiting the run of planning studies to a small 

subset of base cases, TOPs restrict their ability to anticipate and respond to 

contingencies arising in the context of load levels that vary significantly from those in the 

subset of base cases upon which their studies were predicated. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 79 -

As noted above, September 8, 2011 was a very hot day in the region, and 

scheduled flows in the IID footprint were near record peaks. The high demand on 

September 8th was indeed similar to what would have been modeled in a heavy load 

summer seasonal study. The generation picture, however, was very different. By 

September 8, 2011 generation maintenance—which is not typically scheduled for 

summer peak days—had begun. The “heavy peak” summer study base cases that were 

actually used for September 8th therefore had built into them the incorrect assumption 

that there would be minimal maintenance—i.e., that most generation would be on line—

and thus did not account for the normal resumption of facility maintenance in the 

shoulder season. 

If IID’s seasonal studies had assumed even a modest decrease in the available 

generation, they might have enabled IID to anticipate and prevent the events that 

occurred on its system. IID was unaware of the TDM maintenance outages, but if it had 

conducted a shoulder season study, it might have been operating in a mode that more 

accurately reflected actual operating conditions on that day and could have potentially 

avoided the overloading of CV transformers to the tripping point. This lack of awareness

illustrates the risks of not separately modeling the shoulder months such as September, 

when facility maintenance has begun but demand could remain or become very high. 

During these times, generation to serve load may come from other areas, changing flow 

patterns from those that typically occur on a normal summer peak day in which most 

generation is on line. 

Finding 8 Not Sharing Overload Relay Trip Settings: 

• In the seasonal planning process, at least one TOP did not share with 

neighboring TOPs overload relay trip settings on transformers and 

transmission lines that impacted external BPS systems.

Recommendation 8: 

• TOPs should include in the information they share during the seasonal 

planning process the overload relay trip settings on transformers and 

transmission lines that impact the BPS, and separately identify those that 

have overload trip settings below 150% of their normal rating, or below 115% 

of the highest emergency rating, whichever of these two values is greater.

As discussed in greater detail below, the relay trip settings of IID’s CV 230/92 kV

transformers were set very low, just above the facilities’ emergency rating. These 

settings effectively meant that IID’s system operators had very little time to respond to 

the overload resulting from the loss of H-NG beyond emergency ratings and could not 

rely on post-contingency mitigation. If IID’s neighbors had been aware of the relay trip 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 80 -

settings on these transformers when preparing their seasonal studies, they would have 

been able to plan for the possibility of the CV transformers tripping at a lower trip point. 

As a general matter, TOPs should be aware of the relay trip settings of facilities in 

neighboring areas that have the potential to impact portions of the BPS within their own 

areas, regardless of whether or not those facilities have been defined as, or deemed to be, 

BES facilities. This concern is particularly acute where the overload trip points of the 

facility in question are set below 150% of their normal rating, or below 115% of their 

emergency rating, because, as discussed below, such settings sharply limit the amount of 

time available for operators to implement post-contingency mitigation measures. These 

settings require that all entities that could be affected are aware and able to implement 

pre-contingency mitigation.

Near-and Long-Term Planning

 Background

TPs and PCs conduct near- and long-term studies to ensure their systems are 

planned for reliable operation under normal operating conditions. In addition, the 

system facilities must remain stable in the event of single and multiple contingency 

scenarios. Near-term studies consider potential contingencies one to five years past the 

study date, and long-term studies consider potential contingencies six to ten years past 

the study date. The near- and long-term planning process in the WECC region involves a 

coordinated effort among individual TPs and PCs at the local level, Subregional Planning 

Groups (SPGs)73F

74 at the regional level, and WECC RE at the Interconnection-wide level. 

It is a multi-step process, performed annually. 

First, TPs and PCs submit data about their internal networks to their respective 

SPG for each horizon year studied (i.e., years one through ten). These data include 

forecasted load levels and facilities projected to be in or out of service. Also, these data 

assume peak load conditions and, thus, reflects that most internal generation is online. 

Second, SPGs add information to these data based on their broad knowledge of planning 

projects and reliability issues within their respective regions. For example, an SPG 

 74 There are five SPGs in the WECC region, each representing a specific area and composed of various members 

and stakeholders, including individual owners and operators of transmission networks, representatives of local 

government agencies, and independent developers. SPGs allow for the joint consideration of issues among 

individual members. APS, IID, and WALC are members of WestConnect, which performs the SPG function in the 

Southwest region. SDG&E and SCE are members of CAISO, which performs the SPG function in parts of California. 

The SPGs are involved in near- and long-term planning only and are unrelated to the SRSGs, discussed above, 

which deal with seasonal planning.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 81 -

might add data for a particular horizon year based on its knowledge of a merchant 

generator’s desire to connect to the grid. SPGs also consider future projects needed for 

reliability and the effect of environmental regulations on the future operation of 

generator units. Third, SPGs merge all of their members’ cases to create a regional case. 

Fourth, WECC RE merges the various regional cases from all the SPGs to create the base 

case for each horizon year. WECC RE makes these cases available on its website for TPs, 

PCs, and SPGs to access. Finally, TPs and PCs add their own subtransmission facilities 

to the base cases to run their near- and long-term studies. TPs and PCs typically choose 

a list of contingencies to study based on past experience and engineering judgment.

As discussed below, this multi-step process has several shortcomings, which left 

the affected entities unprepared for the September 8th event.

Finding 9 Gaps in Near- and Long-Term Planning Process: 

• Gaps exist in WECC RE’s, TPs’ and PCs’ processes for conducting near- and 

long-term planning studies, resulting in a lack of consideration for: (1) 

critical system conditions; (2) the impact of elements operated at less than 

100 kV on BPS reliability; and (3) the interaction of protection systems, 

including RASs. As a consequence, the affected entities did not identify 

during the planning process that the loss of a single 500 kV transmission line 

could potentially cause cascading outages. Planning studies conducted 

between 2006 and 2011 should have identified the critical conditions that 

existed on September 8th and proposed appropriate mitigation strategies.

Recommendation 9:

• WECC RE should take actions to mitigate these and any other identified gaps 

in the procedures for conducting near- and long-term planning studies. The 

September 8th event and other major events should be used to identify 

shortcomings when developing valid cases over the planning horizon and to 

identify flaws in the existing planning structure. WECC RE should then 

propose changes to improve the performance of planning studies on a 

subregional- and Interconnection-wide basis and ensure a coordinated 

review of TPs’ and PCs’ studies. TOPs, TPs and PCs should develop study 

cases that cover critical system conditions over the planning horizon; 

consider the benefits and potential adverse effects of all protection systems, 

including RASs, Safety Nets (such as the SONGS separation scheme), and 

overload protection schemes; study the interaction of RASs and Safety Nets; 

and consider the impact of elements operated at less than 100 kV on BPS 

reliability.

The affected entities’ near- and long-term planning studies for horizon year 2011 

(i.e., the studies conducted in 2001 through 2010) did not identify that the loss of a 

single 500 kV line in APS’s territory would cause cascading outages across the territories 

of SDG&E, CFE, IID, and WALC. Several gaps in the near- and long-term planning 

process contributed to these omissions. First, TPs and PCs submit peak load data to 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 82 -

WECC for incorporation into the base case and, thus, the data assume that most internal 

generation is online to meet peak conditions. As a result, the models for 2011 did not 

contain accurate, realistic representations of online generation. Running studies under 

the assumption that most generation is online provided an unrealistic portrayal of 

system transfers on the day of the event. 

Indeed, system transfers following the loss of H-NG were higher than the 

transfers seen in the base case used for near- and long-term studies. Significant flows 

from H-NG transferred across IID’s and WALC’s systems and onto Path 44. Flow on 

Path 44 increased by approximately 84% following the loss of the line. These large 

system transfers went undetected in near- and long-term studies, and the affected

entities were not alerted to the need to plan for these critical system conditions. To avoid 

this problem in the future, TPs and PCs should study more generation dispatch scenarios 

to provide a more realistic projection of system transfers following contingencies.

Second, TPs and PCs do not run a full list of external contingencies during the 

near- and long-term planning process. Instead, they rely on experience and engineering 

judgment, focusing on previously identified contingencies. This can be particularly 

problematic in today’s operating environment in which the nature and limitations of the 

system are rapidly changing. For example, as part of its near- and long-term planning 

IID studied potential contingencies on four WECC Rated Paths, but did not study the 

loss of H-NG. As a result, IID was not prepared for the effect on its system when that 

line tripped. Also, while IID’s CV 230/92 kV transformers are included in the base case, 

some of the affected TPs and PCs did not study the potential loss of these facilities. By 

not considering a complete list of external contingencies that could impact their systems, 

TPs’ and PCs’ studies for horizon year 2011 were not sufficient to identify and plan for 

the impact of external contingencies on their internal systems or internal contingencies 

on neighboring systems.

Third, TPs and PCs do not study external subtransmission facilities in the nearand long-term planning process. Individual TPs and PCs add their own subtransmission 

facilities after the base case has been created by WECC RE, but do not add external 

subtransmission equipment. If external subtransmission systems were included in the 

base case, entities could identify the parallel flow on such lower-voltage systems that can 

result from transmission contingency outages. This consideration is particularly 

important for lower voltage systems that parallel external high voltage systems. For 

example, when APS’s H-NG tripped, approximately 12% of its flow transferred to IID’s 

92 kV system. This increased flow and overloading on IID’s system had a ripple effect, 

causing cascading outages throughout neighboring territories. Because the affected

entities did not study external subtransmission systems in their near- and long-term 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 83 -

studies, they did not identify the potential for overloading on IID’s 92 kV system or the 

impact on their systems from this overloading.

Fourth, TPs and PCs do not sufficiently study the interaction of protection 

systems in external networks in their near- and long-term planning studies. For 

example, some of the affected TPs and PCs did not study the interaction between the 

overload protection on IID’s three 230/92 kV transformers, or between the protection on 

these transformers and the S Line RAS. Based on the pre-event conditions, the loss of 

one CV transformer would automatically result in the loss of the second, followed 

automatically by the loss of the Ramon transformer, which in turn, would result in either 

voltage collapse and load shedding, or overloading on the S Line. The S Line RAS is 

designed to mitigate overloads by tripping generation in Mexico that supplies power to 

IID. However, operation in this manner only served to further overload IID and WALC 

facilities and exacerbate system conditions on the day of the event. The affected entities 

should have studied the interaction of these schemes to prepare for the impacts on their 

systems.

Finding 10 Benchmarking WECC Dynamic Models: 

• The inquiry obtained a very good correlation between the simulations and 

the actual event until the SONGS separation scheme activated. After 

activation of the scheme, however, neither the tripping of the SONGS units 

nor the system collapse of SDG&E and CFE could be detected using WECC 

dynamic models because some of the elements of the event are not explicitly 

included in those models. Sample simulations of the islanded region showed 

that by adding known details from the actual event, including UFLS

programs and automatic capacitor switching, the simulation and event 

become more closely aligned following activation of the SONGS separation 

scheme.

Recommendation 10:

 

• WECC dynamic models should be benchmarked by TPs against actual data 

from the September 8th event to improve their conformity to actual system 

performance. In particular, improvements to model performance from 

validation would be helpful in analysis of under and/or over frequency 

events in the Western Interconnection and the stability of islanding 

scenarios in the SDG&E and CFE areas. 

The inquiry simulated the dynamic system response of the September 8th event 

from prior to the loss of H-NG through the separation of Path 44 and the unsuccessful 

islanding of SDG&E and CFE. The team obtained very good correlation between the 

simulation model and the actual event until the SONGS separation scheme activated. 

However, neither the tripping of the SONGS units nor the system collapse of SDG&E and 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 84 -

CFE could be predicted using existing WECC dynamic models entities use to perform 

near- and long-term planning. 

This inability to use the existing system models to reproduce the actual event is 

also evident in the post-event analysis that was prepared by SDG&E on the effectiveness 

of UFLS programs following the September 8th event.74F

75 The SDG&E post-event

analysis shows that the UFLS performance should have prevented the SDG&E system 

from frequency collapse, similar to the “as is” results shown in Figure 14, below. 

However, the SDG&E analysis does not explain why the simulation results are so 

different than the actual system responses—i.e., successful islanding operation versus

system collapse.

The inquiry’s Modeling and Simulation team was able to obtain a simulation 

more closely aligned with actual measured performance by performing several sensitivity 

 75 Preliminary Analysis of SDG&E Off-Nominal UFLS Program Effectiveness Following September 8, 2011 Pacific 

Southwest Event, Performed by SDG&E, December 7, 2011.

Figure 14: Actual and Simulated Frequency at Miguel 500 kV Bus
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 85 -

studies and adding details from the actual event, including UFLS performance, PMU 

data, and generation tripped in CFE’s and SDG&E’s territories. For example, one 

sensitivity study (referred to here as “Test 3”) simulated approximately:

a) 3,080 MW of UFLS in SDG&E 1.3 seconds after Path 44 tripped (compared to 2,760 MW in 

“as-is” case)

b) 520 MW of UFLS in CFE after Path 44 tripped, but prior to SDG&E separation from CFE/APS 

(compared to 900 MW modeled in “as-is” case)

c) 589 MW of generation tripped in CFE after Path 44 tripped, but prior to SDG&E separation 

from CFE/APS (compared to zero in “as-is” case)

d) 1,000 MW of generation tripped in SDG&E immediately after SDG&E separated from 

CFE/APS (compared to zero in “as-is” case)

Figure 15, below, shows results of “Test 3.” As can be seen, this simulation 

more closely follows the actual event than the “as-is” model used in Figure 14.

 

The simulation studies explain the ineffectiveness of the UFLS program, despite 

up to 75% of SDG&E load that was shed within 1.3 seconds of the SONGS separation 

scheme operating. The simulation analysis confirmed findings in the inquiry’s SOE that 

the frequency collapse was caused by generation trips and UFLS misoperations within 

Figure 15: Miguel Frequency Actual and Simulated for “Test 3”
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 86 -

CFE shortly after Path 44’s separation, followed by additional generation trips within 

SDG&E around the time it separated from CFE/APS. 

B. Situational Awareness

Background

TOPs, BAs, and RCs have system operators who constantly monitor their 

networks to maintain situational awareness of system conditions, identify potential 

system disturbances, and institute mitigating measures, as necessary. The affected

entities utilize a range of tools to perform these functions. All of the entities use SCADA 

systems as their main monitoring tool. SCADA systems typically consist of a central

computer that receives information from various RTUs and intelligent electronic devices 

(IEDs), located throughout the system. SCADA systems provide control center operators 

with real-time measurements of system conditions and can send alarms to signal a 

problem.

Most of the affected entities also use several other tools to study and analyze the 

information received from their SCADA systems. Two of the most important tools are 

State Estimator and RTCA. State Estimator gathers the available measurements from 

the SCADA system and calculates estimated real-time values for the whole system. 

RTCA then takes the information from State Estimator and studies “what if” scenarios. 

For example, RTCA determines the potential effects of losing a specific facility, such as a 

generator, transmission line, or transformer, on the rest of the system. In addition to 

studying the effects of various contingencies, RTCA can prioritize contingencies. It can 

also provide mitigating actions and send alarms (visual and/or audible) to operators to 

alert them to potential contingencies. 

While most of the affected entities have and use these tools, the inquiry identified

several concerns with entities’ ability to adequately monitor, identify, and plan for the 

next most critical contingency in real time. Several areas for improvement are described 

in the findings below. 

 

PMUs did not play a role in observing the September 8th event in real time, but 

may prove increasingly important in situational awareness. Of the affected entities, 

CAISO, SCE, and APS are equipped with PMUs. PMUs are widely distributed 

throughout WECC as the result of a WECC-wide initiative known as the Western 

Interconnection Synchrophasor Program (WISP). Their high sampling speed (up to 30 

samples per second) and excellent GPS-based time synchronization offer new granularity 

in information about voltage phase angles and other grid conditions. PMUs are expected 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 87 -

to be used to identify and monitor for grid stress, grid robustness, dangerous 

oscillations, frequency instability, voltage instability, and reliability margins. While not 

yet sufficiently integrated to have been used by the affected entities in their control 

rooms on September 8th, as discussed earlier, PMU data proved valuable in constructing 

the sequence of events and other post-event analysis. 

Finding 11 Lack of Real-Time External Visibility: 

• Affected TOPs have limited real-time visibility outside their systems, 

typically monitoring only one external bus. As a result, they lack adequate 

situational awareness of external contingencies that could impact their 

systems. They also may not fully understand how internal contingencies 

could affect SOLs in their neighbors’ systems.

Recommendation 11: 

• TOPs should engage in more real-time data sharing to increase their

visibility and situational awareness of external contingencies that could 

impact the reliability of their systems. They should obtain sufficient data to 

monitor significant external facilities in real time, especially those that are 

known to have a direct bearing on the reliability of their system, and 

properly assess the impact of internal contingencies on the SOLs of other 

TOPs. In addition, TOPs should review their real-time monitoring tools, 

such as State Estimator and RTCA, to ensure that such tools represent 

critical facilities needed for the reliable operation of the BPS.

Although all of the affected TOPs use SCADA to monitor their own systems, some 

TOPs’ situational awareness is hindered by their limited visibility into neighboring 

systems. Some of the affected TOPs’ real-time external visibility is limited to one or two 

buses outside their systems. The September 8, 2011, event demonstrated that more 

expansive visibility into neighboring systems is necessary for these TOPs to maintain 

situational awareness of external conditions and contingencies that could impact their 

systems and internal conditions and contingencies that could impact their neighbors’ 

systems. During the 11-minute time span of the September 8th event, entities observed 

changes in flows into their systems, but were unable to understand the cause or 

significance of these changes and lacked sufficient time to take corrective actions. If 

affected entities had seen and run studies based on real-time external conditions prior to 

the event, they could have been better prepared to redispatch generation or take other 

control actions and deal with the impacts when the event started.

IID, for example, is adjacent to APS, and the changes in flows on APS’s system, 

especially on its 500 kV lines, can affect the flows on IID’s system and vice versa. Yet, 

IID’s visibility into APS’s system is limited to information about the tie line between 

them. In fact, IID’s visibility into all of its neighbors is limited to one or two buses 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 88 -

outside its system.75F

76 As a result, IID did not learn in real-time that H-NG tripped. IID

also did not understand prior to the event how changes in flows or the loss of H-NG 

would affect its system. Immediately after H-NG tripped, IID observed loading on its CV 

transformers escalate rapidly, but it had not been prepared for this escalation. 

If IID had greater visibility into APS’s system and IID had an equivalent on its 

RTCA that modeled the external network using APS’s real-time data instead of pseudogenerators modeled at the end of each tie line, IID’s RTCA could have more accurately 

studied the results of a post-contingency loss of H-NG on its system before it occurred. 

After seeing the more accurate RTCA results, IID could have initiated appropriate 

control actions before H-NG tripped. Also, having real-time status of the H-NG would 

have better prepared IID to deal with the effects of its loss in real time. 

In addition to IID not having adequate situational awareness of APS’s system, the 

affected TOPs and BAs external to IID were not aware in real time of the effect of the 

post-contingency loss of IID’s three 230/92 kV transformers on their systems. Losses of 

the CV and Ramon transformers can cause SOL violations on neighboring systems. 

Indeed, on September 8th, these transformer outages had a significant ripple effect and 

led to the cascading nature of the event. Yet, entities outside IID’s footprint were not 

prepared for these outages and, except for WECC RC, were unaware of the outages in 

real time because of a lack of adequate visibility into IID’s system. For example, at the 

time of the event, CAISO’s visibility into IID’s system stopped at the tie line into IID’s El 

Centro station. 

The September 8th event exposed the negative consequences of TOPs having 

limited external visibility into neighboring systems. Providing TOPs with the ability to 

observe and model external system conditions and events on a continuous real-time 

basis will allow them to study and plan for the impact of external conditions and 

contingencies before it is too late to react, as was the case on September 8th. 

 76 IID has made efforts, even before the September 8th event, to receive more data points from adjacent utilities 

and is currently continuing this effort with all of its neighbors.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 89 -

Finding 12 Inadequate Real-Time Tools: 

• Affected TOPs’ real-time tools are not adequate or, in one case, operational 

to provide the situational awareness necessary to identify contingencies and 

reliably operate their systems.

Recommendation 12:

• TOPs should take measures to ensure that their real-time tools are adequate, operational,

and run frequently enough to provide their operators the situational awareness necessary 

to identify and plan for contingencies and reliably operate their systems. 

Although many of the affected TOPs have and use real-time tools such as State 

Estimator and RTCA, some of the tools are not adequate or operational to provide the 

situational awareness necessary to effectively monitor and operate their systems. Also, 

some TOPs run or view these tools infrequently, while others run RTCA, for example, 

every five minutes. 

The alarming function on IID’s RTCA provides an example of a real-time tool that 

does not adequately maximize situational awareness capabilities. IID’s RTCA does not 

provide operators with any audible alarms or pop-up visual alerts when an overload is 

predicted to occur. Instead, IID’s RTCA uses color codes on a display that the operator 

must call up manually to learn of significant potential contingencies. For example, IID’s 

RTCA might show that on the next contingency, a specific element will become 

overloaded. However, as currently designed, the operator must go to the specific page 

related to this element to view this result. The result will be color coded on this page, but 

this code does not function as an alarm.

This design feature of IID’s RTCA had negative consequences on the day of the 

event. Forty-four minutes prior to the loss of H-NG, IID’s RTCA results showed that the 

N-1 contingency loss of the first CV transformer would result in overloading of the 

second CV transformer to its tripping point. If IID had taken action at this precontingency stage, it could have avoided the loss of both transformers. The IID operator, 

however, did not view the appropriate RTCA display and, therefore, was not alerted to 

the need to take action. If the operator had reviewed the RTCA results and taken 

necessary corrective actions, he could have relieved loading on the transformers at this 

pre-event stage, and thus mitigated the severe effects on the CV transformers that 

resulted when H-NG tripped.76F

77 

 77 Since the event, IID has initiated changes to its RTCA program. First, it is working with a vendor to install an 

audible alarm feature. Second, IID has instructed its operators to constantly leave the RTCA result display screen 

on, rather than periodically calling it up.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 90 -

One affected entity, APS, has State Estimator and RTCA capability, but neither 

tool is operational. As a result, APS has limited capability to monitor and operate its 

system to withstand potential real-time contingencies. Instead of using RTCA, APS 

relies on a set of previously studied contingencies and pre-determined plans to mitigate 

them. These studies are included in a manual that is created annually and usually

updated several times a year.77F

78 By relying on pre-determined studies, APS cannot 

account and prepare for all potential contingency scenarios in real time. RTCA would 

provide APS with a more realistic analysis of its next potential contingency because the

RTCA analysis is based on real-time conditions, as measured by State Estimator. 

Without RTCA, APS operators are not fully prepared to identify and plan for the next 

most critical contingency on its system. 

RTCA would have allowed APS operators to study the impact of the loss of its HNG. Although APS could have studied this contingency in its manual and seasonal 

studies, it could not have studied it based on real-time operating conditions that only 

State Estimator can provide. For example, APS’s manual and seasonal studies did not 

study the loss of H-NG together with the multiple generator outages that existed on the 

day of the event.78F

79 As a result, APS was unprepared for the actual consequences of 

losing H-NG on September 8, 2011, including overloads on IID’s 92 kV system and

potential difficulty reclosing H-NG due to large phase angle differences.79F

80

 78 APS can also ask WECC RC or an APS engineer for a current-day study, but it usually relies on its manual for 

operations. APS also relies on WECC RC to notify it of any major post-contingency issues detected by WECC RC’s 

RTCA results, but WECC RC might not consistently and promptly notify individual TOPs of all major issues.

79 APS has indicated that it has had difficulty obtaining generator outage information from other BAs due to 

market and/or tariff concerns.

80 Prior to the event, APS had been working with a vendor to build its RTCA capability and, since the event; it has 

accelerated its efforts to make RTCA operational.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 91 -

Finding 13 Reliance on Post-Contingency Mitigation Plans: 

• One affected TOP operated in an unsecured N-1 state on September 8, 2011, 

when it relied on post-contingency mitigation plans for its internal 

contingencies and subsequent overload and tripping, while assuming there 

would be sufficient time to mitigate the contingencies. Post-contingency 

mitigation plans are not viable under all circumstances, such as when 

equipment trips on overload relay protection that prevents operators from 

taking timely control actions. If this TOP had used pre-contingency 

measures on September 8th, such as dispatching additional generation, to 

mitigate first contingency emergency overloads for its internal 

contingencies, the cascading outages that were triggered by the loss of H-NG

might have been avoided with the prevailing system conditions on 

September 8, 2011.

Recommendation 13: 

• TOPs should review existing operating processes and procedures to ensure that postcontingency mitigation plans reflect the time necessary to take mitigating actions, 

including control actions, to return the system to a secure N-1 state as soon as possible 

but no longer than 30 minutes following a single contingency. As part of this review, 

TOPs should consider the effect of relays that automatically isolate facilities without 

providing operators sufficient time to take mitigating measures.

Before September 8, 2011, IID consistently relied on post-contingency mitigation 

plans, rather than proactively responding on a pre-contingency basis, for RTCA results 

showing that the N-1 loss of one CV transformer would result in overloading on the 

second CV transformer. Post-contingency plans can work to prevent a second 

contingency as long as operators have sufficient time to take mitigating actions. Postcontingency mitigation is not an appropriate choice for the CV transformers, which are 

set to trip by overload protection relays without allowing operators enough time to take 

mitigating actions. Specifically, the transformers’ overload protection scheme is set with 

a thin margin between the emergency rating and the relay trip point. The normal rating 

of the transformers is 150 MVA, the emergency rating is 165 MVA, and the relay trip 

point is set at 190.5 MVA, or 127% of the normal rating. Thus, when the transformers 

reach their emergency rating, operators may have the mistaken belief that they have 

sufficient time to take mitigating actions, when, in fact, the operators will have very little 

time before the transformers will trip offline, because they will soon reach the relay trip 

setting. As shown below, pre-contingency mitigation measures are necessary when 

operators are faced with settings that leave such little margin between the emergency 

rating and overload trip point. 

On multiple days during the summer of 2011, IID’s RTCA results showed that an 

N-1 contingency tripping of one of the CV transformers would result in overloading on 

the second transformer. IID continued to operate in this state on multiple days without 

taking any pre-contingency mitigating actions. For example, IID did not dispatch 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 92 -

additional generation on a pre-contingency basis to control the loading on one CV

transformer to prevent overloading on the second CV transformer. There were 

potentially severe consequences of not taking pre-contingency actions. Specifically, IID’s 

next-day study for September 8th detailed that the loss of both CV transformers would 

overload: (1) IID’s Ramon transformer to its trip point; and (2) the S Line, which, in 

turn, would cause the S Line RAS to trip generation in Mexico that supplies power to the

Imperial Valley substation. In short, on multiple days in summer 2011, IID’s RTCA 

results showed that the loss of one CV transformer would overload the second 

transformer, and IID’s next-day study revealed the cascading outages that would stem 

from the loss of both transformers. Yet, IID did not institute pre-contingency mitigating 

measures, such as dispatching additional generation. 

Instead, IID relied on post-contingency plans. On most days in summer 2011, the 

level of overloading on the CV transformers gave IID just enough time to successfully use 

a post-contingency mitigation plan to start generation after the loss of the first 

transformer to avoid the loss of the second transformer. However, on at least two days 

observed by the inquiry, a post-contingency plan would not allow the operator enough 

time to implement necessary procedures to mitigate the problem. On those two days, the 

loading on both CV transformers was high enough that only pre-contingency mitigation 

measures could have prevented the loss of the second transformer upon the loss of the 

first. On the first of those two days, IID was simply fortunate that the N-1 contingency 

loss of the first transformer never occurred. The second of the two days was September 

8, 2011.

Forty-four minutes prior to the loss of H-NG, IID’s RTCA results showed that the 

N-1 contingency loss of the first CV transformer would result in overloading of the 

second transformer to approximately 139% of its normal rating—leading to the loss of 

the transformer by relay action. If IID had taken action at this pre-contingency stage, 

IID might have been able to avoid the loss of both transformers.80F

81 After H-NG tripped, 

the relays took less than 40 seconds to trip both CV transformers. Operators had no

time to mitigate the overloads before the transformers were removed from service.

 81 The inquiry understands that the IID operator did not see these RTCA results and, thus, would not have known 

of the need for pre-contingency mitigating measures. There is no indication, however, that IID would have used 

pre-contingency measures regardless of the results. IID consistently relied on post-contingency measures.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 93 -

Finding 14 WECC RC Staffing Concerns: 

• WECC RC staffs a total of four operators at any one time to meet the 

functional requirements of an RC, including continuous monitoring, 

conducting studies, and giving directives. The September 8th event raises 

concerns that WECC RC’s staffing is not adequate to respond to emergency 

conditions.

Recommendation 14: 

• WECC RC should evaluate the effectiveness of its staffing level, training and 

tools. Based on the results of this evaluation, it should determine what 

actions are necessary to perform its functions appropriately as the RC and 

address any identified deficiencies.

WECC RC performs its reliability coordination functions through two offices. 

Although each office is capable of monitoring the entire Interconnection, during normal 

operations the offices have primary responsibility for monitoring different parts of the 

Western Interconnection. WECC RC’s Vancouver, Washington, office is primarily 

responsible for monitoring the Pacific Northwest (excluding PacifiCorp East), California, 

and CFE’s territory in Mexico. WECC RC’s Loveland, Colorado, office is primarily 

responsible for monitoring the Desert Southwest area, Rocky Mountain area, 

PacifiCorp’s East area, Sierra Pacific Power Company’s area, IID’s area, and the Los 

Angeles intermountain area. Each office staffs two on-shift operators at all times. Each 

center dedicates an operator to the real-time desk (real-time operator) and the other 

operator to the study desk (study desk operator). 

The real-time operator’s primary responsibilities include monitoring limits and 

operating parameters, identifying exceedances, evaluating mitigation plans, and 

directing corrective actions. The study desk operator’s primary responsibilities include 

monitoring expected post-contingency conditions to identify potential exceedances, 

evaluating actions being taken, and directing corrective action as necessary. The study 

desk operator also reviews WECC RC’s next-day study for accuracy, conducts real-time 

studies to evaluate system conditions, and monitors EMS applications, such as RTCA, to 

identify any performance issues and request corrective actions, as necessary. The realtime operator and study desk operator also have some joint responsibilities, including 

reporting events that impact the BPS, identifying events or system conditions that 

require notification to adjacent RCs, and monitoring and testing primary and backup 

internal communication systems. Through these responsibilities, WECC RC is 

responsible for the reliable operation of the BPS in the WECC footprint, and it has the 

ultimate authority to prevent or mitigate emergency operating situations in both nextday and real-time timeframes.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 94 -

In addition, WECC RC is responsible for providing information to the entities in 

its footprint, including the 53 TOPs and 37 BAs. Some of this information is provided 

over the telephone. During the event, in addition to performing the many RC functions 

they are responsible for performing, the RC operators had to answer phone calls 

providing or seeking information on the disturbance.

Given WECC RC’s responsibility and authority, four total operators—two in each 

regional office—might not be sufficient to effectively perform its function, particularly 

during emergency conditions. Several examples from the September 8th event highlight 

this concern. 

First, after the loss of H-NG, many alarms began sounding in WECC RC’s control 

rooms, as voltage dropped and facilities overloaded. With so many alarms sounding in 

an emergency situation, the real-time operator had a difficult time prioritizing which 

alarms to monitor. WECC RC has eight unique categories, or “buckets,” of alarms within 

its EMS applications, grouped according to importance. Buckets 1 and 2 contain the 

highest priority alarms. Bucket 1 includes all 500 and 345 kV circuit breaker status 

changes, frequency and Path violations, status of generators greater than 50 MW and 

associated circuit breakers, and critical bus voltages. Bucket 2 includes all 220/230 kV 

circuit breaker status changes and automatic voltage regulator status.81F

82 Buckets 3 

through 8 include lesser priority items, such as RAS status changes, non-critical bus 

voltages, and circuit breaker status changes below 220 kV. Operators receive audible 

alarms for buckets 1 and 2 and typically leave bucket 1’s display on the screen constantly 

and use one other screen to display all other buckets. It is a constant process to 

continually monitor the alarms, even during normal operating conditions, and it might 

not be possible for one real-time operator to keep track of and prioritize multiple alarms 

sounding at once. Also, both operators had numerous phone calls to field from entities 

throughout the affected areas, reporting and requesting information. Overburdening the 

real-time operator in this way could undermine his or her ability to perform the critical 

functions of monitoring system conditions and directing necessary corrective actions. 

Accordingly, WECC RC should consider whether additional operators are necessary to 

adequately perform these functions. 

A second indication that the current RC staffing levels might not be sufficient 

came during the September 8th event when the study desk operator had to abandon his 

duties in order to provide support to the real-time operator by fielding phone calls and 

monitoring conditions. On this day, the RC operators were able to call for an engineer to 

 82 The CV 230/92 kV transformers are included in bucket 2.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 95 -

conduct some studies. Because the September 8th event occurred during the afternoon, 

an engineer was available. Finding an engineer to substitute for the study desk operator

may not always be so easy. Late at night and early in the morning, no engineers are on 

duty. That the study desk operator needed to leave his responsibilities to support the 

real-time operator may indicate that one real-time operator and one study desk operator

per office might not be sufficient to fulfill WECC’s reliability coordination functions. 

Alternatively, additional training and enhanced tools may enable an entity to 

accomplish more with the same number of personnel. While the inquiry observed a 

sampling of WECC RC’s tools to be adequate during its site visit, WECC RC is in the best 

position to identify the combination of additional staff, enhanced tools, or training that 

best addresses the concerns identified by this report.

Finding 15 Failure to Notify WECC RC and Neighboring TOPs Upon Losing 

RTCA: 

• On September 8, 2011, at least one affected TOP lost the ability to conduct 

RTCA more than 30 minutes prior to and throughout the course of the event 

due to the failure of its State Estimator to converge. The entity did not notify 

WECC RC or any of its neighboring TOPs, preventing this entity from 

regaining situational awareness.

Recommendation 15: 

• TOPs should ensure procedures and training are in place to notify WECC RC 

and neighboring TOPs and BAs promptly after losing RTCA capabilities.

When entities temporarily lose their RTCA capability due to technical issues, they 

become blind to the next most severe contingency on their system, and they do not know 

what pre-contingency measures might be necessary. Thus, when they lose RTCA, they 

must take immediate action to try to regain their situational awareness. For example, 

after losing RTCA an entity should contact WECC RC, so the RC can monitor the entity’s

system and inform it of any significant issues. In such instances, the RC should also 

notify neighboring entities of any major contingencies that could impact their systems. 

Between 13:59 and the start of the event on September 8, 2011, WALC lost its 

RTCA when its State Estimator stopped solving. 82F

83 As a result, WALC lost its ability to 

identify and study post-contingency violations and to take pre-contingency mitigating 

measures, as necessary. When it lost its RTCA, WALC should have contacted WECC RC 

and asked it to monitor WALC’s area. WECC RC could have then notified WALC 

regarding any significant problems and could have also contacted WALC’s neighbors if it 

 83 By not solving, or converging, the State Estimator stopped providing estimated values for the system.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 96 -

learned of any SOLs in WALC that were impacting the neighbors’ systems.83F

84 Prior to 

the event on September 8, 2011, WALC experienced several post-contingency SOL 

violations, but, without its RTCA capability, remained unaware of them. WECC RC’s 

RTCA results showed these violations. WALC, however, did not notify WECC RC when it 

lost RTCA and, thus, WECC RC was unaware that it should notify WALC of the 

violations. An entity should never be operating in an unknown state, as WALC was when 

it lacked functional RTCA and State Estimator, and did not ask any other entity to assist 

it with situational awareness.

Finding 16 Discrepancies Between RTCA and Planning Models: 

• WECC’s model used by TOPs to conduct RTCA studies is not consistent with

WECC’s planning model and produces conflicting solutions.

Recommendation 16: 

• WECC should ensure consistencies in model parameters between its

planning model and its RTCA model and should review all model parameters 

on a consistent basis to make sure discrepancies do not occur.

The usefulness of RTCA study results and other real-time studies depend on the 

models used in the studies. Inaccurate models jeopardize the accuracy of studies, as well 

as entities’ ability to respond appropriately to potential contingencies identified by the 

studies. The inquiry’s simulation of the September 8th event discovered that a 

discrepancy exists between WECC RC’s model used to conduct RTCA studies and the 

model used for WECC’s planning studies. Specifically, the impedance of IID’s CV 

transformers differed by a factor of two between the WECC models. WECC’s planning 

model has an impedance of 0.1 per unit, while WECC RC’s RTCA model has an 

impedance of 0.05 per unit. This difference resulted in an error of approximately 16% in 

the RTCA model compared to the planning model with respect to loading on the CV 

transformers.

Although the inquiry did not perform a comprehensive comparison of all 

parameters in WECC’s various models, this discrepancy between the RTCA and planning 

models on such important facilities calls into question the validity of other parameters in 

WECC’s models.

I. System Analysis

 84 While not at issue in this event, the RC should also notify TOPs if it loses its RTCA, so that TOPs know that the 

RC is not able to observe their systems.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 97 -

Consideration of BES Equipment

 Background

The BES is generally defined as all facilities operating at voltages above 100 kV, 

although certain sub-100 kV facilities with a significant impact on the BPS may be 

considered a part of the BES. Each RE currently determines its specific procedure for 

determining what is or is not BES. If a facility is not considered BES, relevant TOPs, 

BAs, and RCs may not study and model the impact of that facility.

Finding 17 Impact of Sub-100 kV Facilities on BPS Reliability: 

• WECC RC and affected TOPs and BAs do not consistently recognize the 

adverse impact sub-100 kV facilities can have on BPS reliability. As a result, 

sub-100 kV facilities might not be designated as part of the BES, which can 

leave entities unable to address the reliability impact they can have in the 

planning and operations time horizons. If, prior to September 8, 2011, 

certain sub-100 kV facilities had been designated as part of the BES and, as a 

result, were incorporated into the TOPs’ and RC’s planning and operations 

studies, or otherwise had been incorporated into these studies, cascading 

outages may have been avoided on the day of the event.

Recommendation 17:

• WECC, as the RE, should lead other entities, including TOPs and BAs, to

ensure that all facilities that can adversely impact BPS reliability are either 

designated as part of the BES or otherwise incorporated into planning and 

operations studies and actively monitored and alarmed in RTCA systems.

WECC RC, as well as TOPs and BAs impacted by the event, did not consider IID’s 

92 kV network and facilities (including the CV and Ramon transformers) as BES

elements. IID did not reconsider whether the CV and Ramon transformers should be 

studied like BES facilities even after a draft study sponsored by CFE (and shared with 

IID) suggested the existence of a through-flow issue between the 500 kV substations at 

Devers and Imperial Valley, adversely impacting IID’s 92 kV network (including the CV 

and Ramon transformers) during contingencies on BPS systems, including H-NG.84F

85 

Because the Reliability Standards apply to BES facilities, if the CV transformers had been 

considered BES facilities, IID would have been required to study the impact they could 

have on BPS reliability.85F

86 Also, WECC RC and the affected TOPs would likely have 

included the facilities in their studies and been aware of the impact the loss of H-NG

 85 See CFE’s Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19.

86 See, e.g., NERC Reliability Standard TOP-002-2b R11 (TOPs “shall perform seasonal, next-day, and current-day 

Bulk Electric System studies to determine SOLs”).
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 98 -

would have on IID’s 92 kV system, as well as the impact various trips within IID’s 92 kV 

system would have on the rest of the BPS. The inquiry determined that, during the 

event, approximately 12% (168 MW) of the original flow on H-NG was transferred 

through IID’s 92 kV system, making the 92 kV system part of a bulk power path as well 

as a significant looped transmission facility. The cascading outages that resulted from 

the loss of H-NG demonstrated the significant potential for IID’s 92 kV system, including 

the CV transformers, to impact BPS reliability.

IROL Derivations

 Background

In order to ensure the reliable operation of the BPS, entities are required to 

identify and plan for IROLs, which are SOLs that, if violated, can cause instability, 

uncontrolled separation, and cascading outages. Once an IROL is identified, system 

operators are then required to create plans to mitigate the impact of exceeding such a 

limit to maintain system reliability.

Finding 18 Failure to Establish Valid SOLs and Identify IROLs: 

• The cascading nature of the event that led to uncontrolled separation of San 

Diego, IID, Yuma, and CFE indicates that an IROL was violated on 

September 8, 2011, even though WECC RC did not recognize any IROLs in 

existence on that day. In addition, the established SOL of 2,200 MW on Path 

44 and 1,800 MW on H-NG are invalid for the present infrastructure, as 

demonstrated by the event.

Recommendation 18.1: 

• WECC RC should recognize that IROLs do exist on its system and, thus, 

should study IROLs in the day-ahead timeframe and monitor potential IROL 

exceedances in real-time. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 99 -

Recommendation 18.2: 

• WECC RC should work with TOPs to consider whether any SOLs in the 

Western Interconnection constitute IROLs. As part of this effort, WECC RC 

should: (1) work with affected TOPs to consider whether Path 44 and H-NG 

should be recognized as IROLs; and (2) validate existing SOLs, and ensure 

that they take into account all transmission and generation facilities and 

protection systems that impact BPS reliability. 

The NERC Glossary defines an IROL as an SOL that, if violated, could expose a 

widespread area of the BPS to instability, uncontrolled separation, or cascading outages 

that adversely impact the reliability of the BPS. Each IROL is associated with a 

maximum time limit (Tv) that the IROL can be exceeded before the risk to the 

Interconnection or another RC area becomes greater than acceptable. The time limit can 

vary, but any IROL’s Tv must be less than or equal to 30 minutes.86F

87

For this event, the loss of H-NG should have been associated with an IROL with a 

Tv for this N-1 contingency of essentially no minutes, because the cascading from the 

loss of H-NG began within seconds. However, neither WECC RC nor any of the affected 

entities have previously identified this IROL. The WECC region historically has 

maintained an operating philosophy of not recognizing IROLs.87F

88 Instead, entities in the 

WECC region believe that as long as they operate within the conditions they have 

studied, they will not face the risk of IROLs and will not need to calculate IROLs. The 

September 8th event undermines this philosophy. 

Prior to the event, the WECC system was supplying loads in the various balancing 

authority areas in the range of 85-95% of their recorded peak loads. The power flows on 

all the Paths in the WECC region were below their maximum ratings and voltages were 

within acceptable levels. In particular, the two major transmission corridors into the 

blackout area, namely Path 44 and H-NG, were loaded respectively to 1,302 MW and 

 87 As defined by the NERC Glossary of Terms, an IROL’s Tv is “[t]he maximum time that an [IROL] can be violated 

before the risk to the interconnection or other Reliability Coordinator Area(s) becomes greater than acceptable. 

Each [IROL’s] Tv shall be less than or equal to 30 minutes.” NERC Glossary of Terms, February 8, 2012, at 26.

88 As described by WECC in a February 16, 2012 Webinar on its SOL Methodology revision, “The WECC operating 

philosophy is to operate only in conditions that have been studied. Therefore, under these normal operating 

conditions, there are never IROL conditions (only SOLs). An IROL condition may be created by the occurrence of 

one or more unanticipated contingencies. When this occurs, under WECC Reliability Standards, bulk electric 

system operators are required to resolve the IROL condition within 20 minutes (stability) or 30 minutes 

(thermal).” http://www.wecc.biz/awareness/Reliability/Documents /SOL Methodology Presentation 

02.16.2012.pdf (emphasis in original).
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 100 -

1,372 MW. Compared to their maximum SOL ratings of 2,200 MW and 1,800 MW,

these loadings represent 59% and 78% of their maximum ratings—well within current 

limits. Path 44 and H-NG ratings of 2,200 MW and 1,800 MW may be invalid for the 

present infrastructure because cascading outages due to a single contingency occurred at 

loadings well below the SOL ratings.

During the 11-minute disturbance, the single contingency of the sudden loss of HNG resulted in a series of cascading outages, with multiple elements exceeding their 

applicable ratings and leading to a widespread blackout of the area.

Accordingly, WECC RC should lead all relevant TOPs in the blackout area to 

study and report on the appropriateness of identifying Path 44 and H-NG as IROL paths. 

WECC RC should similarly assess transfer Paths outside this blackout area to ensure that 

there are no other similar reliability issues in the Western Interconnection. Existing 

operating processes and procedures should be reviewed to ensure corrective control 

capabilities are provided to system operators to enable them to return the system to a 

secure N-1 state as soon as possible, but no longer than 30 minutes following a single 

contingency.

WECC RC has a proposed new SOL Methodology document (current effective 

date of June 4, 2012), which acknowledges the need to establish IROLs, and the RC’s 

responsibility to monitor IROLs.88F

89 It recognizes that “Stability SOLs may qualify as 

IROLs depending on the potential consequences of exceeding the limit and the impact on 

BES reliability. WECC RC makes this determination by collaborating with TOPs to 

understand the nature of the stability SOL, understanding the conditions that result in 

the establishment of the stability SOL, and determining the BES impacts of exceeding 

the stability SOL.”89F

90 WECC RC also has a proposed multi-step process for determining 

whether thermal or voltage SOLs are IROLs. In general, WECC RC will look at whether 

potential IROLs cause “Widespread Adverse System Impacts,” or “potential cascading.” 

“Widespread Adverse System Impacts” is defined as “loading of three or more additional 

BES Facilities beyond 125% of their applicable emergency thermal Facility Rating, or 

[t]hree or more additional BES Facilities with bus voltages experiencing voltages less 

than 90%.”90F

91 “Potential cascading” is defined as “when studies indicate that a 

 89 See WECC System Operating Limits Methodology for the Operations Horizon, Version 6.1, available at

http://www.wecc.biz/awareness/Reliability/ WECC RC Operating Procedures/WECC FAC 011-EFFECTIVE DATE 6-4-

2012 SOL Methodology for the Operations Horizon.pdf.

90 Id. at 5.

91 Id. at 6.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 101 -

contingency results in severe loading on a Facility, triggering a chain reaction of Facility 

disconnection by relay action, equipment failure, or forced immediate manual 

disconnection of the Facility (for example, public safety concerns, or no time for the 

operator to implement mitigation actions).”91F

92 

Impact of Protection Systems on Event

 Protection System Coordination

When an abnormal system condition is detected on the BPS, relay protection 

systems operate to isolate the problem while causing minimum disturbance to the power 

system. This requires the relay to be selective in determining which elements to

interrupt. The only method of obtaining this selectivity is to perform coordination 

studies. The inquiry discovered that two TOs did not properly coordinate a protection 

system and a third TO implemented a protection scheme without performing any 

coordination studies at all. This lack of coordination of protection systems resulted in 

circuits unnecessarily being interrupted, which had an undesirable effect on BPS 

reliability during the September 8th event.

Finding 19 Lack of Coordination of the S Line RAS: 

• Several TOs and TOPs did not properly coordinate a RAS by: (1) not 

performing coordination studies with the overload protection schemes on 

the facilities that the S Line RAS is designed to protect; and (2) not assessing 

the impact of setting relays to trip generation sources and a 230 kV 

transmission tie line prior to the operation of a single 161/92 kV 

transformer’s overload protection. As a result, BES facilities were isolated in 

excess of those needed to maintain reliability, with adverse impact on BPS 

reliability.

Recommendation 19:

• The TOs and TOPs responsible for design and coordination of the S Line RAS should 

revisit its design basis and protection settings to ensure coordination with other 

protection systems in order to prevent adverse impact to the BPS, premature operation,

and excessive isolation of facilities. TOs and TOPs should share any changes to the S Line 

RAS with TPs and PCs so that they can accurately reflect the S Line RAS when planning.

Operation of the S Line RAS isolates facilities beyond what is necessary to ensure 

reliability. The S Line RAS is a directional overload scheme, located at the Imperial 

Valley substation, which is jointly owned by SDG&E and IID. The S Line RAS was 

originally implemented to protect the sole 230/161 kV transformer at El Centro from 

 92 Id.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 102 -

overloads due to increased flow on the S Line.92F

93 At the time, this was the only transfer 

point from the 230 kV line to the 161 kV system, and subsequently the 92 kV system, in 

IID’s southern area. After implementing this RAS, IID has since installed a 230/92 kV 

transformer at El Centro, providing another path from the 230 kV system to the lower 

voltage networks.

IID’s current intention for the S Line RAS is to reduce loading on the S Line by 

tripping generation and, if insufficient to reduce flow, tripping the S Line at Imperial 

Valley Substation before transformer overload protection operates to trip the 161/92 kV 

transformer at El Centro. Tripping the S Line before allowing the El Centro 161/92 kV 

transformer’s overload protection to take action effectively results in the removal of the 

230 kV source at the El Centro substation, which normally feeds a 230/92 kV 

transformer and a 230/161 kV transformer. Thus, the design of the S Line RAS 

intentionally isolates networked BES facilities to mitigate an overload on a non-BES 

facility (El Centro 161/92 kV transformer) to support reliability of the local system. 

While this action alone does not constitute miscoordination, proper coordination of a 

RAS should take into account, through system studies, the potential impact on BPS 

reliability, including potential interaction with other RASs and protection systems. 

During the September 8th event, the S Line RAS operated as designed, in that it 

tripped when it reached the settings that IID had prescribed. However, if one considers 

the purpose of the S Line RAS, which was to protect the El Centro transformer from 

overloads, the S Line RAS operated long before it was needed. At the time that the S 

Line RAS operated, the El Centro 161/92 transformer was only loaded to 38% of its 

normal rating, and its overload trip point is 178% of its normal rating. Thus, the El 

Centro 161/92 transformer could have carried at least four times as much load before the 

transformer’s overload protection system would have operated. Even though the El 

Centro transformer that the S Line RAS was designed to protect was nowhere near 

overloading, the S Line RAS tripped important generation and a 230 kV line. This calls 

into question the coordination of the S Line RAS with the transformer overload 

protection systems at El Centro. 

IID provided SDG&E with the S Line RAS settings to implement. IID did not 

perform any studies to coordinate the S Line RAS with IID’s protection systems. SDG&E 

did some studies to verify that the RAS coordinated with SDG&E’s protection systems. 

There is no indication that the S Line RAS was coordinated with IID’s transformer 

 93 The S Line RAS also serves as secondary protection for other IID facilities if a RAS on the Imperial Valley to 

Miguel 500 kV line fails to operate.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 103 -

overload protection at the El Centro station at which the S Line terminates. At a 

minimum, IID, SDG&E and CAISO (as the TOP for SDG&E) should work together to 

ensure the proper coordination of the S Line RAS.

To make matters worse, during the September 8th event, San Diego was relying 

on generation at Imperial Valley from the south when the S Line RAS tripped that 

generation. Loss of the Imperial Valley generation caused San Diego to pull even more 

power from the north, increasing the loading on Path 44 and causing the SONGS 

separation scheme to further exceed its trip point. If not tripped by the S Line RAS, 

generation at Imperial Valley could have helped SDG&E survive after the operation of 

the SONGS separation scheme. The inquiry’s simulation showed that, had the S Line 

RAS tripped only the S Line without tripping the generation, the SONGS separation 

scheme would not have operated, and only IID would have lost power.93F

94

Finding 20 Lack of Coordination of the SONGS Separation Scheme: 

• SCE did not coordinate the SONGS separation scheme with other protection 

systems, including protection and turbine control systems on the two SONGS 

generators. As a result, SCE did not realize that Units 2 and 3 at SONGS 

would trip after operation of the separation scheme.

Recommendation 20:

• SCE should ensure that the SONGS separation scheme is coordinated with 

other protection schemes, such as the generation protection and turbine 

control systems on the units at SONGS and UFLS schemes.

SCE, the TO and TOP of the SONGS separation scheme, did not perform any 

protection system coordination studies for the separation scheme it implemented at 

SONGS. The scheme is intended to isolate five 230 kV lines simultaneously if its preset 

value is exceeded for a sustained period. If SCE had coordinated the separation scheme 

with other protection and generation control systems at SONGS, it may have recognized 

the potential for the operation of the SONGS separation scheme to cause the SONGS 

generators to trip. Coordination in this context requires system studies to assess the 

impact of operation of the RAS on the power system, including potential interaction with 

other RASs and protection systems, such as UFLS schemes.

In addition to the consequences at SONGS itself, the lack of coordination of the 

systems means that, when the scheme operates, the system enters an unknown state. 

 94 See footnote 53.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 104 -

During the event, the operation of the protection scheme significantly contributed to the 

blackout of SDG&E, CFE, and Yuma—an effect neither coordinated nor adequately 

studied prior to the event. The inquiry’s simulation indicates that SDG&E, CFE and, 

Yuma would not have been blacked out if the SONGS separation scheme had not 

operated, with limited impact to the rest of the Western Interconnection.

Finding 21 Effect of SONGS Separation Scheme on SONGS Units: 

• The SONGS units tripped due to their turbine control systems detecting 

unacceptable acceleration following operation of the SONGS separation 

scheme. 

Recommendation 21: 

• GOs and GOPs should evaluate the sensitivity of the acceleration control 

functions in turbine control systems to verify that transient perturbations or 

fault conditions in the transmission system resulting in unit acceleration will 

not result in unit trip without allowing time for protective devices to clear 

the fault on the transmission system.

When the SONGS separation scheme operated, turbines at SONGS began to 

accelerate in excess of their control system setting causing both units to trip offline. The 

tripping of the SONGS units in this manner raises questions about the sensitivity of the 

turbine control system’s settings. The units are expected to withstand severe faults on 

the transmission system and allow the transmission protection systems to operate 

without the generators tripping offline. The coordination required for this protection is 

not a traditional relay-to-relay coordination; rather, the setting for the acceleration 

function should be coordinated with capabilities of the turbine and with the system 

response anticipated following operation of transmission protection systems for faults 

under various system conditions. The setting should also be coordinated with the system 

response following operation of the SONGS separation scheme. Had the turbine control 

system acceleration function been coordinated in this manner, the trip of the units may 

have been avoided.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 105 -

Protection System Studies

Finding 22 Lack of Review and Studying Impact of SPSs: 

• Although WECC equates SPSs with RASs, prior to October 1, 2011, WECC’s 

definition of RAS excluded many protection systems that would be included 

within NERC’s definition of SPS. As a result, WECC did not review and 

assess all NERC-defined SPSs in its region, and WECC’s TOPs did not 

perform the required review and assessment of all NERC-defined SPSs in 

their areas. 

Recommendation 22: 

• WECC RE, along with TOs, GOs, and Distribution Providers (DPs), should 

periodically review the purpose and impact of RASs, including Safety Nets 

and Local Area Protection Schemes, to ensure they are properly classified, 

are still necessary, serve their intended purposes, are coordinated properly

with other protection systems, and do not have unintended consequences on 

reliability. WECC RE and the appropriate TOPs should promptly conduct 

these reviews for the SONGS separation scheme and the S Line RAS.

The NERC definition of an SPS concludes with “Also called Remedial Action 

Scheme.”94F

95 This implies that all SPSs are RASs and vice versa, but prior to October 1, 

2011, the WECC region did not equate SPSs with RASs.95F

96 WECC created four 

classifications of protection systems that fall under the NERC definition of SPS, and,

instead of including all of these classifications in the RAS definition, WECC only 

identified a subset of those protection systems as RASs. Safety Nets, Wide Area 

Protection Systems (WAPS), and Local Area Protection Systems (LAPS) were excluded 

from the WECC definition of a RAS even though they are SPSs as defined by NERC.

For example, SCE did not study the impact of the SONGS separation scheme on 

BPS reliability because it believed, by classifying this scheme as a Safety Net, that it was 

not required to be studied. SCE also did not submit the separation scheme to WECC for 

review by the Remedial Action Scheme Reliability Subcommittee (RASRS). The inquiry 

determined that the SONGS separation scheme is indeed an SPS/RAS as defined by 

NERC, because it altered the BPS configuration by separating Path 44 and redistributing

generation in the absence of any faulted equipment. WECC, SDG&E, and SCE did not

study the impact that the SONGS separation scheme could have on BPS reliability and,

 95 NERC Glossary of Terms, February 8, 2012, at 46.

96 On October 1, 2011, WECC revised its definition of RAS to include Safety Nets and Local Area Protection 

Schemes.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 106 -

thus, were unaware of its severe impact on the BPS when the scheme operated: blacking 

out SDG&E and CFE and leading to the loss of the SONGS generators. 

Another protection system that did not get the necessary scrutiny due to WECC’s 

narrow definition of RAS was the S Line RAS. The S Line is a 230 kV transmission line 

that serves as a major tie between SDG&E & IID. It runs from IID’s and SDG&E’s jointly 

owned Imperial Valley station on one end to IID’s El Centro station on the other. The S

Line RAS, as IID and SDG&E called it, was classified as a LAPS by WECC, which called it 

the “S Line Scheme.” Thus, the RAS received no periodic assessments. Like the SONGS 

scheme, the S Line RAS appears to be a SPS/RAS as defined by NERC, because it is an 

automatic protection system that took action other than isolating a faulted facility by 

tripping generation in Mexico for loading on a tie line between SDG&E and IID. 

The S Line RAS was implemented for two reasons: (1) to protect IID’s system 

from overload during an N-2 event at SDG&E’s Miguel substation; and (2) to protect 

IID’s lone 230/161kV transformer at El Centro from overloads due to generation 

additions at Imperial Valley substation. The inquiry questions whether the scheme is

still necessary, as both of the concerns that originally triggered installation of the S Line 

RAS have been mitigated. IID added a new transformer bank at El Centro, mitigating 

the concern for overloads on the 230/161kV transformer. Also, reconfigurations at 

Miguel along with the modifications to a RAS at Miguel have mitigated the concern of 

adverse effects on IID’s system as a result of an N-2 event at Miguel. Since LAPSs are 

not periodically reviewed, the arguably outdated S Line RAS was still active during the 

September 8th event, and its operation contributed to IID’s uncontrolled separation and 

the operation of the SONGS separation scheme by tripping over 400 MW of generation 

before the S Line itself tripped. At a minimum, SDG&E, IID and CAISO should 

participate in the review of the S Line RAS.

The SPSs that operated during the event suggest that WECC’s previous exclusion 

of certain NERC-defined SPSs from WECC’s RAS definition had an adverse impact on 

BPS reliability. 

Finding 23 Effect of Inadvertent Operation of SONGS Separation Scheme 

on BPS Reliability:

• The inquiry’s simulation of the event shows that the inadvertent operation of 

the SONGS separation scheme under normal system operations could lead to 

a voltage collapse and blackout in the SDG&E areas under certain high load 

conditions.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 107 -

Recommendation 23:

• CAISO and SCE should promptly verify that the inadvertent operation of the 

SONGS separation scheme does not pose an unacceptable risk to BPS 

reliability. Until this verification can be completed, they should consider all 

actions to minimize this risk, up to and including temporarily removing the 

SONGS separation scheme from service.

The inquiry conducted a simulation to evaluate what would happen if the SONGS 

separation scheme inadvertently operated during normal system operations (e.g., in the 

absence of any outages, overloads, or SOL violations). Based on this simulation, the 

inquiry determined that under certain high load conditions, the operation of the scheme 

could result in voltage collapse and a blackout in SDG&E’s and CFE’s territories. The 

inquiry conducted a voltage stability study using a Power-Voltage (P-V) curve to estimate 

the amount of SDG&E load that could reliably be supplied after an inadvertent operation 

of the SONGS separation scheme. The P-V curve below in Figure 16 demonstrates that 

such operation would lead to a voltage collapse and a blackout in the SDG&E and CFE 

territories under certain high load conditions.

400

420

440

460

480

500

520

540

560

580

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000 7500 8000

North Gila Voltage (kV)

San Diego Load

PV Curve for North Gila 500-kV Bus

North Gila 500-kV Base Case North Gila 500-kV Path 44 Open

Figure 16: PV Curve for North Gila 500 kV Bus
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 108 -

Specifically, the system is most likely to collapse when the SDG&E load exceeds 

3,500 MW. In 2010, SDG&E’s load exceeded this amount for 851 hours,96F

97 meaning that 

the system was exposed to a potential blackout for approximately 10% of the year. This 

shows the potential risk to BPS reliability during normal system operations as a result of 

the inadvertent operation of the SONGS separation scheme. Accordingly, given the lack 

of studies done on the scheme, the inquiry recommends that the inadvertent operation of 

the SONGS separation scheme be reviewed promptly to ensure it does not pose an 

unacceptable risk to BPS reliability. Until this verification can be completed, CAISO and 

SCE should consider all actions needed to minimize this risk, up to and including 

temporarily removing the scheme from service.

Moreover, if SCE and CAISO were to decide to temporarily remove the scheme 

from the service, the inquiry does not believe that BPS reliability would be jeopardized. 

Indeed, inquiry simulations conducted for the day of the event show that if the scheme 

had not operated, the system, with the exception of collapses in the IID and Yuma areas, 

would have stabilized with minor overloads in the area around SONGS, acceptable 

voltages in the SDG&E area, and sufficient reactive margins in the critical portion of 

SCE’s system.

Finding 24 Not Recognizing Relay Settings When Establishing SOLs: 

• An affected TO did not properly establish the SOL for two transformers, as 

the SOL did not recognize that the most limiting elements (protective relays) 

were set to trip below the established emergency rating. As a result, the 

transformers tripped prior to the facilities being loaded to their emergency 

ratings during the restoration process, which delayed the restoration of 

power to the Yuma load pocket.

Recommendation 24: 

• TOs should reevaluate their facility ratings methodologies and 

implementation of the methodologies to ensure that their ratings are equal 

to the most limiting piece of equipment, including relay settings. No relay 

settings should be set below a facility’s emergency rating. When the relay 

setting is determined to be the most limiting piece of equipment, 

consideration should be given to reviewing the setting to ensure that it does 

not unnecessarily restrict the transmission loadability.

TOs are required to designate and share their facilities’ SOLs. An SOL is the 

value that satisfies the most limiting element of a facility beyond which the system 

 97 SDG&E Annual Electric Balancing Authority Area and Planning, FERC Form No. 714 (2010). 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 109 -

cannot operate reliably. The inquiry’s relay loadability calculations show that APS failed 

to properly establish the SOL for two of its 500/69 kV transformers in North Gila, 

because the transformers’ relay loadability or load limit was actually set below their 

emergency ratings. A facility cannot operate above its relay load limit, as operation in 

excess of a load limit results in the facility being removed from service. Thus, these 

settings prevented the TOP from taking advantage of the short term emergency ratings 

identified by the transformers’ SOLs. These settings resulted in difficulties restoring 

power to the Yuma load pocket, as operators believed they could load the transformers 

up to their emergency rating. Instead, the transformers tripped below the emergency 

rating, delaying the restoration of power to Yuma.

If the SOL derivation had considered the transformer relay load limit, the TO 

could have (1) provided an SOL that accurately reflected the relay load limit so the 

system operator could have limited the transformer loading appropriately, or (2) 

reviewed the relay load limit to determine whether it unnecessarily limited the 

transformer loadability, and if so, raised the transformer relay setting threshold above 

the transformer emergency rating while coordinating the setting with the transformer 

short-time thermal capability. 

Load-Responsive Phase Protection Systems Set Too Close to Normal or 

Emergency Ratings

BES facilities at a minimum are required to have normal and emergency ratings.

The normal rating is a continuous rating or a rating that a facility can be operated to on a 

daily basis that specifies the amount of electrical loading a facility can support. The 

emergency rating specifies the level of electrical loading a facility can support for a finite 

period of time. Operating a facility beyond its normal and/or emergency rating for an 

extended period of time will expose certain equipment in that facility to the risk of 

thermal damage. In order to prevent thermal damage to facilities, some TOs implement 

overload protection systems that are designed to automatically isolate the facilities if

operated beyond their emergency rating. 

A problem arises when overload protection systems are set in close proximity to a 

facility’s normal or emergency ratings. Setting the overload protection close to the 

normal or emergency ratings restricts facility loading and prevents operators from 

having sufficient time to take remedial action to mitigate an overload before the facility is 

automatically isolated by the overload protection system.97F

98 As the Commission stated

 98 NERC Reliability Standard PRC-023-1 R1.11 provides the following guidance on setting of overload protection 

systems on transformers: “Set the relays to allow the transformer to be operated at an overload level of at least 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 110 -

in Order No. 733, “manual mitigation of thermal overloads is best left to system 

operators, who can take appropriate actions to support Reliable Operation of the BulkPower System.”98F

99 Protective relay settings limited transmission loadability with 

extremely conservative overload protection settings, resulting in cascading outages

during the September 8th event. These settings resulted in facilities being automatically 

removed from service by relays before operators had an opportunity to take remedial 

action.

Finding 25 Margin Between Overload Relay Protection Settings and 

Emergency Rating:

 • Some affected TOs set overload relay protection settings on transformers 

just above the transformers’ emergency rating, resulting in facilities being 

automatically removed from service before TOPs have sufficient time to take 

control actions to mitigate the resulting overloads. One TO in particular set 

its transformers’ overload protection schemes with such narrow margins

between the emergency ratings and the relay trip settings that the protective 

relays tripped the transformers following an N-1 contingency.

Recommendation 25: 

• TOs should review their transformers’ overload protection relay settings 

with their TOPs to ensure appropriate margins between relay settings and 

emergency ratings developed by TOPs. For example, TOs could consider 

using the settings of Reliability Standard PRC-023-1 R.1.11 even for those 

transformers not classified as BES. PRC-023-1 R.1.11 requires relays to be 

set to allow the transformer to be operated at an overload level of at least 

150% of the maximum applicable nameplate rating, or 115% of the highest 

operator established emergency transformer rating, whichever is greater.

Relay loadability calculations indicate that the relay settings on a number of 

transmission facilities limited transmission loadability to slightly above the emergency 

rating. For example, the relays on IID’s CV transformers were set to trip at 127% of their 

normal rating. The parallel CV transformers were loaded to 130%, which was above their 

127% overload relay trip point, immediately after the loss of H-NG. Both transformers 

tripped less than 40 seconds later. If the transformers’ overload trip point had been in 

accordance with PRC-023-1 R.1.11, the trip point would not have been exceeded 

immediately after the loss of the H-NG, and IID operators might have had time to take 

actions to prevent cascading.99F

100 

 150% of the maximum applicable nameplate rating, or 115% of the highest operator established emergency 

transformer rating, whichever is greater.”

99 Transmission Relay Loadability Reliability Standard, 130 FERC ¶ 61,221, at P 212 (2010).

100 IID originally used conservative settings because the CV transformers are rare, expensive, load-serving 

transformers. IID has indicated, however, that it will increase the overload relay settings on the CV transformers 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 111 -

During the September 8th event, IID was unaware that the overload relay setting 

for the Ramon 230/92 kV transformer had been mistakenly set at 207% of its normal 

rating. IID intended the Ramon transformer to have been set to trip at 120% of its 

normal rating. After the event, IID reduced the Ramon transformer’s trip setting from 

207% to 120%, making it more likely to trip during high-loading conditions or conditions 

similar to those that precipitated the blackout, decreasing the opportunity for its 

operators to take mitigating actions during such conditions. This setting actually 

increased the risk of future cascading outages like the one which occurred on September 

8, 2011.

Finding 26 Relay Settings and Proximity to Emergency Ratings: 

• Some TOs set relays to isolate facilities for loading conditions slightly above 

their thirty minute emergency ratings. As a result, several transmission 

lines and transformers tripped within seconds of exceeding their emergency 

ratings, leaving TOPs insufficient time to mitigate overloads.

Recommendation 26: 

• TOs should evaluate load responsive relays on transmission lines and 

transformers to determine if the settings can be raised to provide more time 

for TOPs to take manual action to mitigate overloads that are within the 

short-time thermal capability of the equipment instead of allowing relays to 

prematurely isolate the transmission lines. If the settings cannot be raised 

to allow more time for TOPs to take manual action, TOPs must ensure that 

the settings are taken into account in developing facility ratings and that 

automatic isolation does not result in cascading outages.

In addition to the problematic protection settings of the CV transformers, which 

precipitated the cascade, the inquiry discovered that several other facilities, including a 

number of IID’s 161 kV transmission lines and two of WALC’s 161/69 kV transformers, 

had relay protection settings which were only slightly above those facilities’ emergency 

ratings. These conservative settings severely limited TOPs’ response time before the 

facilities were isolated, preventing the operators from taking effective mitigating action 

against the cascade. While the inquiry did not determine whether less conservative relay 

settings on these other facilities would have mitigated the cascade, the applied settings 

nevertheless do not leave operators sufficient time to take mitigating steps to prevent or 

ameliorate the consequences of future events.

Angular Separation

 to 150% of their normal rating, and will relocate an additional 230/92 kV transformer from another substation to 

CV.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 112 -

When a transmission line trips or goes out of service, the phase angle will 

generally increase between its two terminal points. When angle differences become 

large, facilities connected to the system can lose synchronization, causing the system to 

become unstable. Also, if the phase angle is too large, closing the line breaker back into 

service with a large angle difference may result in damage to nearby generator turbine 

shafts, and the resulting power swings and oscillations could lead to system instability or 

collapse. To enable successful reclosing, studies should be run to determine the 

maximum phase angle difference allowable for a line to be closed back in and safeguards 

be put into place to prevent reclosure with excessive phase angle difference. Should the 

phase angle difference exceed the established limit, generation or load must be adjusted 

to reduce it to the level that allows the line to be closed.

Finding 27 Phase Angle Difference Following Loss of Transmission Line:

• A TOP did not have tools in place to determine the phase angle difference 

between the two terminals of its 500 kV line after the line tripped. Yet, it

informed the RC and another TOP that the line would be restored quickly, 

when, in fact, this could not have been accomplished.

Recommendation 27:

• TOPs should have: (1) the tools necessary to determine phase angle 

differences following the loss of lines; and (2) mitigation and operating plans 

for reclosing lines with large phase angle differences. TOPs should also train 

operators to effectively respond to phase angle differences. These plans 

should be developed based on the seasonal and next-day contingency 

analyses that address the angular differences across opened system 

elements. 

The inquiry’s simulation shows that after H-NG tripped, the voltage phase angle 

between the two terminals increased from 20 degrees to approximately 72 degrees. On 

the day of the event, APS’s synchro-check relay was set at 60 degrees,100F

101 meaning APS 

would not have been able to reclose H-NG until it reduced the phase angle difference 

from 72 to 60 degrees, or changed the relay setting to allow the breaker to close. 

Specifically, the 60 degree setting would not have allowed APS to reclose H-NG until 

appropriate generation on both sides of North Gila was dispatched or load reductions in 

the areas west of North Gila were implemented to reduce the difference of the voltage 

phase angle to 60 degrees. 

 101 Based on additional studies, APS has since determined the maximum settings on its synchro-check relay at 

North Gila to allow a maximum phase angle difference of 75 degrees to reclose a line. To add margin, APS has 

implemented the relay setting at 70 degrees.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 113 -

Although APS operators are trained to effectively respond to phase angle 

differences,101F

102 APS currently lacks the tools necessary to determine phase angle 

differences following the loss of a transmission line until the line is reenergized.102F

103 The 

training, therefore, does little good if the operators cannot determine whether a phase

angle difference exists in the first place. Generally, APS operators can monitor phase 

angles through SCADA, but in order to receive and review this data, the transmission 

line must be energized. After H-NG tripped, and prior to reenergizing the line, for 

example, APS had no way to know if the line could be reclosed within the permissive 60 

degree setting of its synchro-check relay. It lacked situational awareness of the phase 

angle difference. Yet, APS informed WECC RC and CAISO that it believed the line could 

be reclosed quickly, when, in fact, this could not have been done due to the phase angle 

difference.103F

104 

To avoid a similar situation in the future, TOPs should ensure that they have 

adequate tools to determine phase angles after the loss of transmission lines. For 

example, they can install PMUs throughout their system, as APS plans to do, to increase 

their situational awareness of phase angles. Moreover, TOPs should ensure that their 

operators are trained to respond to phase angle differences by, for example, 

redispatching generation. In addition, TOPs should not underestimate the time required 

to reclose a line, particularly without first knowing the phase angle difference. Here, for 

example, APS likely could not have reclosed the line quickly, even had it known the 

phase angle difference, given system conditions on the day of the event.

Indeed, the inquiry conducted a series of power flow simulations and found that 

significant amounts of generation redispatch were needed to close the phase angle 

difference. Figure 17, on the next page, shows the relationship between the voltage 

phase angle of H-NG as generation is redispatched between California and Arizona. The 

dispatched approach adjusts the available generation nearest the Hassayampa and North 

Gila buses. As generation is dispatched to its maximum output in the vicinity of the two 

 102 APS provides its certified operators with two training classes, Power System Dynamics and Dynamics of 

Disturbances, both of which address power angles and their ramifications. In addition, APS provides its new 

operator trainees with training on power angles.

103 APS plans to expand its use of PMUs to enable it to determine phase angle differences even without a line 

being energized. Through the PMU data, APS would be able to determine voltage and angle measurements on 

live buses in its substations, through which it could calculate phase angle differences.

104 APS did not intentionally mislead WECC RC and CAISO with this statement. Rather, it did not expect that 

there would have been such a large phase angle difference, as it had not previously experienced such a 

difference. Moreover, APS determined that the line was not damaged and, thus, it did not believe there would be 

any issues closing the line.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 114 -

stations, other generators farther out are adjusted to effect the change in voltage phase 

angles.

The blue line in Figure 17 illustrates that with the particular conditions of the 

September 8th event, approximately 1,800 MW needed to be redispatched on both ends 

of H-NG (and close to the terminals, in Southern California and Arizona) in order to 

close the voltage phase angle from 72 degrees to 60 degrees (i.e., to within the permissive 

60 degree setting of the synchro-check relay). The green line shows that more 

generation—more than twice as much—must be redispatched if units are chosen in 

Northern California to close the angle between Hassayampa and North Gila. 

While system operators could redispatch generation from available spinning 

reserves or commit units in the Southern and/or Northern California area, it is 

questionable how quickly 1,800 MW could be dispatched. 

Figure 17: Phase Angle of H-NG vs. Generation Shift
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 115 -

Appendix A: List of Acronyms Used in Report

ACE Area Control Error

APS Arizona Public Service

BA Balancing Authority

BES Bulk Electric System

BPS Bulk-Power System

CAISO California Independent System Operator, Inc.

CFE Comisión Federal de Electricidad

CV Coachella Valley

EMS Energy Management System

GO Generator Owner

GOP Generator Operator

H-NG APS’s Hassayampa-North Gila 500 kV transmission line

IEEE Institute of Electrical and Electronics Engineers

IID Imperial Irrigation District

IROL Interconnection Reliability Operating Limit

kV Kilovolt

LAPS Local Area Protection System

MVA Megavolt-ampere

MW Megawatt

NERC North American Electric Reliability Corporation

OSS California/Mexico Operations Study Subcommittee

OTC Operating Transfer Capabilities

OTCPC Operating Transfer Capability Policy Committee

PC Planning Coordinator

PMU Phasor Measurement Unit

RAS Remedial Action Scheme

RC Reliability Coordinator

RE Regional Entity

RTCA Real-Time Contingency Analysis

SASG Southwest Area Study Group

SCADA Supervisory Control and Data Acquisition

SCE Southern California Edison

SDG&E San Diego Gas & Electric

SE State Estimator

SOE Sequence of Events

SOL System Operating Limit

SONGS San Onofre Nuclear Generating Station

SPS Special Protection System
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 116 -

SRP Salt River Power

SWPL Southwest Power Link

TO Transmission Owner

TOP Transmission Operator

TP Transmission Planner

UFLS Underfrequency Load Shedding

VAR Volt-Ampere Reactive

WALC Western Area Power Administration – Lower Colorado

WAPS Wide Area Protection System

WECC Western Electricity Coordinating Council

YCA Yuma Cogeneration Associates
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 117 -

Appendix B: Table of Findings and Recommendations

The following table provides a complete list of findings and corresponding 

recommendations, each of which are discussed in detail at Section IV of the 

report.

NEXT-DAY PLANNING

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 1 – Failure to Conduct 

and Share Next-Day Studies: 

Not all of the affected TOPs 

conduct next-day studies or share 

them with their neighbors and 

WECC RC. As a result of failing 

to exchange studies, on 

September 8, 2011 TOPs were not 

alerted to contingencies on 

neighboring systems that could 

impact their internal system and 

the need to plan for such 

contingencies.

Recommendation 1: All TOPs should 

conduct next-day studies and share the 

results with neighboring TOPs and the RC 

(before the next day) to ensure that all 

contingencies that could impact the BPS are 

studied.

TOPs

Finding 2 – Lack of Updated 

External Networks in Next-Day 

Study Models: When conducting 

next-day studies, some affected 

TOPs use models for external 

networks that are not updated to 

reflect next-day operating 

conditions external to their 

systems, such as generation 

schedules and transmission 

outages. As a result, these TOPs’ 

next-day studies do not 

adequately predict the impact of 

external contingencies on their 

systems or internal contingencies 

on external systems.

Recommendation 2: TOPs and BAs should 

ensure that their next-day studies are 

updated to reflect next-day operating 

conditions external to their systems, such as 

generation and transmission outages and 

scheduled interchanges, which can 

significantly impact the operation of their 

systems. TOPs and BAs should take the 

necessary steps, such as executing 

nondisclosure agreements, to allow the free 

exchange of next-day operations data 

between operating entities. Also, RCs should 

review the procedures in the region for 

coordinating next-day studies, ensure 

adequate data exchange among BAs and 

TOPs, and facilitate the next-day studies of 

BAs and TOPs.

TOPs, BAs, RCs

Finding 3 –Sub-100 kV Facilities 

Not Adequately Considered in 

Next-Day Studies: In conducting 

next-day studies, some affected 

TOPs focus primarily on the 

TOPs’ internal SOLs and the need 

to stay within established Rated 

Path limits, without adequate 

consideration of some lower 

voltage facilities. As a result, 

these TOPs risk overlooking 

facilities that may become 

overloaded and impact the 

reliability of the BPS. Similarly, 

Recommendation 3: TOPs and RCs should 

ensure that their next-day studies include all 

internal and external facilities (including 

those below 100 kV) that can impact BPS 

reliability.

TOPs, RCs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 118 -

the RC does not study sub-100 kV 

facilities that impact BPS 

reliability unless it has specifically 

been alerted to issues with such 

facilities by individual TOPs or 

the RC has otherwise identified a 

particular sub-100 kV facility as 

affecting the BPS.

Finding 4 – Flawed Process for 

Estimating Scheduled 

Interchanges: WECC RC’s 

process for estimating scheduled 

interchanges is not adequate to 

ensure that such values are 

accurately reflected in its nextday studies. As a result, its nextday studies may not accurately 

predict actual power flows and 

contingency overloads.

Recommendation 4: WECC RC should 

improve its process for predicting 

interchanges in the day-ahead timeframe.

WECC RC

SEASONAL PLANNING

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 5 – Lack of Coordination 

in Seasonal Planning Process: 

The seasonal planning process in 

the WECC region lacks effective 

coordination. Specifically, the 

four WECC subregions do not 

adequately integrate and 

coordinate studies across the 

subregions, and no single entity is 

responsible for ensuring a 

thorough seasonal planning 

process. Instead of conducting a 

full contingency analysis based on 

all of the subregions’ studies, the 

subregions rely on experience and 

engineering judgment in choosing 

which contingencies to discuss. 

As a result, individual TOPs may 

not identify contingencies in one 

subregion that may affect TOPs in 

the same or another subregion.

Recommendation 5: WECC RE should 

ensure better integration and coordination of 

the various subregions’ seasonal studies for 

the entire WECC system. To ensure a 

thorough seasonal planning process, at a 

minimum, WECC RE should require a full 

contingency analysis of the entire WECC 

system, using one integrated seasonal study, 

and should identify and eliminate gaps 

between subregional studies. Individual 

TOPs should also conduct a full contingency 

analysis to identify contingencies outside 

their own systems that can impact the 

reliability of the BPS within their system and 

should share their seasonal studies with 

TOPs shown to affect or be affected by their 

contingencies.

WECC RE, TOPs

Finding 6 –External and LowerVoltage Facilities Not Adequately 

Considered in Seasonal Planning 

Process: Seasonal planning 

studies do not adequately 

consider all facilities that may 

affect BPS reliability, including 

external facilities and lowervoltage facilities.

Recommendation 6: TOPs should expand 

the focus of their seasonal planning to 

include external facilities and internal and 

external sub-100 kV facilities that impact 

BPS reliability.

TOPs

Finding 7 – Failure to Study 

Multiple Load Levels: TOPs do 

Recommendation 7: TOPs should expand 

the cases on which they run their individual 

TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 119 -

not always run their individual 

seasonal planning studies based 

on the multiple WECC base cases 

(heavy and light load summer, 

heavy and light load winter, and 

heavy spring), but, instead, may 

focus on only one load level. As a 

result, contingencies that occur 

during the shoulder seasons (or 

other load levels not studied) 

might be missed.

planning studies to include multiple base 

cases, as well as generation maintenance 

outages and dispatch scenarios during high 

load shoulder periods.

Finding 8 – Not Sharing 

Overload Relay Trip Settings: In 

the seasonal planning process, at 

least one TOP did not share with 

neighboring TOPs overload relay 

trip settings on transformers and 

transmission lines that impacted 

external BPS systems.

Recommendation 8: TOPs should include in 

the information they share during the 

seasonal planning process the overload relay 

trip settings on transformers and 

transmission lines that impact the BPS, and 

separately identify those that have overload 

trip settings below 150% of their normal 

rating, or below 115% of the highest 

emergency rating, whichever of these two 

values is greater.

TOPs

NEAR- AND LONG-TERM PLANNING

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 9 – Gaps in Near- and 

Long-Term Planning Process: 

Gaps exist in WECC RE’s, TPs’ 

and PCs’ processes for conducting 

near- and long-term planning 

studies, resulting in a lack of 

consideration for: (1) critical 

system conditions; (2) the impact 

of elements operated at less than 

100 kV on BPS reliability; and (3) 

the interaction of protection 

systems. As a consequence, the 

affected entities did not identify 

during the planning process that 

the loss of a single 500 kV 

transmission line could 

potentially cause cascading 

outages. Planning studies 

conducted between 2006 and 

2011 should have identified the 

critical conditions that existed on 

September 8th and proposed 

appropriate mitigation strategies.

Recommendation 9: WECC RE should take 

actions to mitigate these and any other 

identified gaps in the procedures for 

conducting near- and long-term planning 

studies. The September 8th event and other 

major events should be used to identify 

shortcomings when developing valid cases 

over the planning horizon and to identify 

flaws in the existing planning structure. 

WECC RE should then propose changes to 

improve the performance of planning studies 

on a subregional- and Interconnection-wide 

basis and ensure a coordinated review of 

TPs’ and PCs’ studies. TOPs, TPs and PCs 

should develop study cases that cover critical 

system conditions over the planning horizon; 

consider the benefits and potential adverse 

effects of all protection systems, including 

RASs, Safety Nets (such as the SONGS 

separation scheme), and overload protection 

schemes; study the interaction of RASs and 

Safety Nets; and consider the impact of 

elements operated at less than 100 kV on 

BPS reliability.

WECC RE, TOPs, 

TPs, PCs

Finding 10 – Benchmarking

WECC Dynamic Models: The 

inquiry obtained a very good 

correlation between the 

simulations and the actual event 

until the SONGS separation 

scheme activated. After 

Recommendation 10: WECC dynamic 

models should be benchmarked by TPs 

against actual data from the September 8th 

event to improve their conformity to actual 

system performance. In particular, 

improvements to model performance from 

validation would be helpful in analysis of 

TPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 120 -

activation of the scheme, 

however, neither the tripping of 

the SONGS units nor the system 

collapse of SDG&E and CFE could 

be detected using WECC dynamic 

models because some of the 

elements of the event are not 

explicitly included in those 

models. Sample simulations of 

the islanded region showed that 

by adding known details from the 

actual event, including UFLS 

programs and automatic 

capacitor switching, the 

simulation and event become 

more closely aligned following 

activation of the SONGS 

separation scheme.

under and/or over frequency events in the 

Western Interconnection and the stability of 

islanding scenarios in the SDG&E and CFE 

areas.

SITUATIONAL AWARENESS

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 11 – Lack of Real-Time 

External Visibility: Affected 

TOPs have limited real-time 

visibility outside their systems, 

typically monitoring only one 

external bus. As a result, they 

lack adequate situational 

awareness of external 

contingencies that could impact 

their systems. They also may not 

fully understand how internal 

contingencies could affect SOLs 

in their neighbors’ systems.

Recommendation 11: TOPs should engage in 

more real-time data sharing to increase their 

visibility and situational awareness of 

external contingencies that could impact the 

reliability of their systems. They should 

obtain sufficient data to monitor significant 

external facilities in real time, especially 

those that are known to have a direct bearing 

on the reliability of their system, and 

properly assess the impact of internal 

contingencies on the SOLs of other TOPs. In 

addition, TOPs should review their real-time 

monitoring tools, such as State Estimator 

and RTCA, to ensure that such tools 

represent critical facilities needed for the 

reliable operation of the BPS.

TOPs

Finding 12 – Inadequate RealTime Tools: Affected TOPs’ realtime tools are not adequate or, in 

one case, operational to provide 

the situational awareness 

necessary to identify 

contingencies and reliably 

operate their systems.

Recommendation 12: TOPs should take 

measures to ensure that their real-time tools 

are adequate, operational, and run 

frequently enough to provide their operators 

the situational awareness necessary to 

identify and plan for contingencies and 

reliably operate their systems.

TOPs

Finding 13 – Reliance on PostContingency Mitigation Plans: 

One affected TOP operated in an 

unsecured N-1 state on 

September 8, 2011, when it relied 

on post-contingency mitigation 

plans for its internal 

contingencies and subsequent 

overload and tripping, while 

assuming there would be 

Recommendation 13: TOPs should review 

existing operating processes and procedures 

to ensure that post-contingency mitigation 

plans reflect the time necessary to take 

mitigating actions, including control actions, 

to return the system to a secure N-1 state as 

soon as possible but no longer than 30 

minutes following a single contingency. As 

part of this review, TOPs should consider the 

effect of relays that automatically isolate 

TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 121 -

sufficient time to mitigate the 

contingencies. Post-contingency 

mitigation plans are not viable 

under all circumstances, such as 

when equipment trips on 

overload relay protection that 

prevents operators from taking 

timely control actions. If this 

TOP had used pre-contingency 

measures on September 8th, such 

as dispatching additional 

generation, to mitigate first 

contingency emergency overloads 

for its internal contingencies, the 

cascading outages that were 

triggered by the loss of H-NG 

might have been avoided with the 

prevailing system conditions on 

September 8, 2011.

facilities without providing operators 

sufficient time to take mitigating measures.

Finding 14 – WECC RC Staffing 

Concerns: WECC RC staffs a 

total of four operators at any one 

time to meet the functional 

requirements of an RC, including 

continuous monitoring, 

conducting studies, and giving 

directives. The September 8th 

event raises concerns that WECC 

RC’s staffing is not adequate to 

respond to emergency conditions.

Recommendation 14: WECC RC should 

evaluate the effectiveness of its staffing level, 

training and tools. Based on the results of 

this evaluation, it should determine what 

actions are necessary to perform its 

functions appropriately as the RC and 

address any identified deficiencies.

WECC RC

Finding 15 – Failure to Notify 

WECC RC and Neighboring TOPs 

Upon Losing RTCA: On 

September 8, 2011, at least one 

affected TOP lost the ability to 

conduct RTCA more than 30 

minutes prior to and throughout 

the course of the event due to the 

failure of its State Estimator to 

converge. The entity did not 

notify WECC RC or any of its 

neighboring TOPs, preventing 

this entity from regaining 

situational awareness.

Recommendation 15: TOPs should ensure 

procedures and training are in place to notify 

WECC RC and neighboring TOPs and BAs 

promptly after losing RTCA capabilities.

TOPs

Finding 16 – Discrepancies

Between RTCA and Planning 

Models: WECC’s model used by 

TOPs to conduct RTCA studies is 

not consistent with WECC’s 

planning model and produces 

conflicting solutions.

Recommendation 16: WECC should ensure 

consistencies in model parameters between 

its planning model and its RTCA model and 

should review all model parameters on a 

consistent basis to make sure discrepancies 

do not occur.

WECC

CONSIDERATION OF BES EQUIPMENT
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 122 -

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 17 – Impact of Sub-100 

kV Facilities on BPS Reliability: 

WECC RC and affected TOPs and 

BAs do not consistently recognize 

the adverse impact sub-100 kV 

facilities can have on BPS 

reliability. As a result, sub-100 

kV facilities might not be 

designated as part of the BES, 

which can leave entities unable to 

address the reliability impact they 

can have in the planning and 

operations time horizons. If, 

prior to September 8, 2011, 

certain sub-100 kV facilities had 

been designated as part of the 

BES and, as a result, were 

incorporated into the TOPs’ and 

RC’s planning and operations 

studies, or otherwise had been 

incorporated into these studies, 

cascading outages may have been 

avoided on the day of the event.

Recommendation 17: WECC, as the RE 

should lead other entities, including TOPs 

and BAs, to ensure that all facilities that can 

adversely impact BPS reliability are either 

designated as part of the BES or otherwise 

incorporated into planning and operations 

studies and actively monitored and alarmed 

in RTCA systems.

WECC RE, TOPs, 

BAs

IROL DERIVATIONS

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 18 – Failure to Establish 

Valid SOLs and Identify IROLs: 

The cascading nature of the event 

that led to uncontrolled 

separation of San Diego, IID, 

Yuma, and CFE indicates that an 

IROL was violated on September 

8, 2011, even though WECC RC 

did not recognize any IROLs in 

existence on that day. In 

addition, the established SOL of 

2,200 MW on Path 44 and 1,800 

MW on H-NG are invalid for the 

present infrastructure, as 

demonstrated by the event.

Recommendation 18.1: WECC RC should 

recognize that IROLs do exist on its system 

and, thus, should study IROLs in the dayahead timeframe and monitor potential 

IROL exceedances in real-time. 

Recommendation 18.2: WECC RC should 

work with TOPs to consider whether any 

SOLs in the Western Interconnection 

constitute IROLs. As part of this effort, 

WECC RC should: (1) work with affected 

TOPs to consider whether Path 44 and H-NG 

should be recognized as IROLs; and (2) 

validate existing SOLs, and ensure that they 

take into account all transmission and 

generation facilities and protection systems 

that impact BPS reliability.

WECC RC, TOPs

PROTECTION SYSTEMS

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 19 – Lack of 

Coordination of the S Line RAS: 

Several TOs and TOPs did not 

properly coordinate a RAS by: (1) 

Recommendation 19: The TOs and TOPs 

responsible for design and coordination of 

the S Line RAS should revisit its design basis 

and protection settings to ensure 

TOs, TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 123 -

not performing coordination 

studies with the overload 

protection schemes on the 

facilities that the S Line RAS is 

designed to protect; and (2) not 

assessing the impact of setting 

relays to trip generation sources 

and a 230 kV transmission tie 

line prior to the operation of a 

single 161/92 kV transformer’s 

overload protection. As a result, 

BES facilities were isolated in 

excess of those needed to 

maintain reliability, with adverse 

impact on BPS reliability.

coordination with other protection systems 

in order to prevent adverse impact to the 

BPS, premature operation, and excessive 

isolation of facilities. TOs and TOPs should 

share any changes to the S Line RAS with 

TPs and PCs so that they can accurately 

reflect the S Line RAS when planning.

Finding 20 – Lack of 

Coordination of the SONGS 

Separation Scheme: SCE did not 

coordinate the SONGS separation 

scheme with other protection 

systems, including protection and 

turbine control systems on the 

two SONGS generators. As a 

result, SCE did not realize that 

Units 2 and 3 at SONGS would 

trip after operation of the 

separation scheme.

Recommendation 20: SCE should ensure 

that the SONGS separation scheme is 

coordinated with other protection schemes, 

such as the generation protection and 

turbine control systems on the units at 

SONGS and UFLS schemes.

SCE

Finding 21 – Effect of SONGS 

Separation Scheme on SONGS 

Units: The SONGS units tripped 

due to their turbine control 

systems detecting unacceptable 

acceleration following operation 

of the SONGS separation scheme.

Recommendation 21: GOs and GOPs should 

evaluate the sensitivity of the acceleration 

control functions in turbine control systems 

to verify that transient perturbations or fault 

conditions in the transmission system 

resulting in unit acceleration will not result 

in unit trip without allowing time for 

protective devices to clear the fault on the 

transmission system.

GOs, GOPs

Finding 22 – Lack of Review and 

Studying Impact of SPSs: 

Although WECC equates SPSs 

with RASs, prior to October 1, 

2011, WECC’s definition of RAS 

excluded many protection 

systems that would be included 

within NERC’s definition of SPS. 

As a result, WECC did not review 

and assess all NERC-defined 

SPSs in its region, and WECC’s 

TOPs did not perform the 

required review and assessment 

of all NERC-defined SPSs in their 

areas.

Recommendation 22: WECC RE, along with 

TOs, GOs, and Distribution Providers (DPs), 

should periodically review the purpose and 

impact of RASs, including Safety Nets and 

Local Area Protection Schemes, to ensure 

they are properly classified, are still 

necessary, serve their intended purposes, are 

coordinated properly with other protection 

systems, and do not have unintended 

consequences on reliability. WECC RE and 

the appropriate TOPs should promptly 

conduct these reviews for the SONGS 

separation scheme and the S Line RAS.

WECC RE, TOs, 

GOs, DPs, TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 124 -

Finding 23 – Effect of 

Inadvertent Operation of SONGS 

Separation Scheme on BPS 

Reliability: The inquiry’s 

simulation of the event shows 

that the inadvertent operation of 

the SONGS separation scheme 

under normal system operations 

could lead to a voltage collapse 

and blackout in the SDG&E areas 

under certain high load 

conditions.

Recommendation 23: CAISO and SCE 

should promptly verify that the inadvertent 

operation of the SONGS separation scheme 

does not pose an unacceptable risk to BPS 

reliability. Until this verification can be 

completed, they should consider all actions 

to minimize this risk, up to and including, 

temporarily removing the SONGS separation 

scheme from service.

CAISO, SCE

Finding 24 – Not Recognizing 

Relay Settings When 

Establishing SOLs: An affected 

TO did not properly establish the 

SOL for two transformers, as the 

SOL did not recognize that the 

most limiting elements 

(protective relays) were set to trip 

below the established emergency 

rating. As a result, the 

transformers tripped prior to the 

facilities being loaded to their 

emergency ratings during the 

restoration process, which 

delayed the restoration of power 

to the Yuma load pocket.

Recommendation 24: TOs should reevaluate 

their facility ratings methodologies and 

implementation of the methodologies to 

ensure that their ratings are equal to the 

most limiting piece of equipment, including 

relay settings. No relay settings should be 

set below a facility’s emergency rating. 

When the relay setting is determined to be 

the most limiting piece of equipment, 

consideration should be given to reviewing 

the setting to ensure that it does not 

unnecessarily restrict the transmission 

loadability.

TOs

Finding 25 – Margin Between

Overload Relay Protection 

Settings and Emergency Rating: 

Some affected TOs set overload 

relay protection settings on 

transformers just above the 

transformers’ emergency rating, 

resulting in facilities being 

automatically removed from 

service before TOPs have 

sufficient time to take control 

actions to mitigate the resulting 

overloads. One TO in particular 

set its transformers’ overload 

protection schemes with such 

narrow margins between the 

emergency ratings and the relay 

trip settings that the protective

relays tripped the transformers 

following an N-1 contingency.

Recommendation 25: TOs should review 

their transformers’ overload protection relay 

settings with their TOPs to ensure 

appropriate margins between relay settings 

and emergency ratings developed by TOPs. 

For example, TOs could consider using the 

settings of Reliability Standard PRC-023-1 

R.1.11 even for those transformers not 

classified as BES. PRC-023-1 R.1.11 requires 

relays to be set to allow the transformer to be 

operated at an overload level of at least 150% 

of the maximum applicable nameplate 

rating, or 115% of the highest operator 

established emergency transformer rating, 

whichever is greater.

TOs, TOPs

Finding 26 –Relay Settings and 

Proximity to Emergency 

Ratings: Some TOs set relays to 

isolate facilities for loading 

conditions slightly above their 

thirty minute emergency ratings. 

As a result, several transmission 

lines and transformers tripped 

Recommendation 26: TOs should evaluate 

load responsive relays on transmission lines 

and transformers to determine if the settings 

can be raised to provide more time for TOPs 

to take manual action to mitigate overloads 

that are within the short-time thermal 

capability of the equipment instead of 

allowing relays to prematurely isolate the 

TOs, TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 125 -

within seconds of exceeding their 

emergency ratings, leaving TOPs 

insufficient time to mitigate 

overloads.

transmission lines. If the settings cannot be 

raised to allow more time for TOPs to take 

manual action, TOPs must ensure that the 

settings are taken into account in developing 

facility ratings and that automatic isolation 

does not result in cascading outages.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 126 -

ANGULAR SEPARATION

FINDING RECOMMENDATION APPLICABLE

ENTITIES

Finding 27 – Phase Angle 

Difference Following Loss of 

Transmission Line: A TOP did 

not have tools in place to 

determine the phase angle 

difference between the two 

terminals of its 500 kV line after 

the line tripped. Yet, it informed 

the RC and another TOP that the 

line would be restored quickly, 

when, in fact, this could not have 

been accomplished.

Recommendation 27: TOPs should have: (1) 

the tools necessary to determine phase angle 

differences following the loss of lines; and (2) 

mitigation and operating plans for reclosing 

lines with large phase angle differences. 

TOPs should also train operators to 

effectively respond to phase angle 

differences. These plans should be 

developed based on the seasonal and nextday contingency analyses that address the 

angular differences across opened system 

elements.

TOPs
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 127 -

Appendix C: Comparison of August 2003 and September 2011 

Blackouts

On August 14, 2003, an estimated 50 million people throughout the Midwest and 

Northeast United States and Ontario, Canada, experienced an electric power blackout. A 

day later, the joint U.S.-Canada Power System Outage Task Force began investigating the 

causes of the blackout and considering ways to prevent such outages in the future. The 

task force detailed its findings and recommendations in an April 2004 report.104F

105 A 

comparison of the findings and recommendations in this April 2004 report and the 

instant report on the September 8, 2011, blackout reveals commonalities between the 

two events.

Although the August 2003 and September 2011 blackouts were triggered by 

different initiating events—tree touches in 2003 compared to a switching error in 2011—

both blackouts had common underlying causes. First, affected entities in both events did 

not conduct adequate long-term and operations planning studies necessary to 

understand vulnerabilities on their systems. Second, affected entities in both events had 

inadequate situational awareness leading up to and during the disturbances. In addition 

to these two underlying causes, both events were exacerbated by protection system relays 

that tripped facilities without allowing operators sufficient time to take mitigating 

measures. These similarities are highlighted below, with excerpts from both reports to 

illustrate specific comparisons.

Inadequate Long-Term and Operations Planning 

The 2003 Blackout Report states that “FirstEnergy was not [operating its system 

securely] because the company had not conducted the long-term and operational 

planning studies needed to understand [certain] vulnerabilities and their operational 

implications.”105F

106 Similarly, this inquiry’s report found that several entities’ operational 

and long-term studies did not adequately ensure the reliable operation of their systems. 

Specifically, both reports described relevant planning studies that: (1) did not 

adequately identify and study critical external facilities; (2) did not adequately analyze

potential contingency scenarios; and (3) were based on inaccurate models and invalid

system operating limits (SOLs). 

 105 Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations 

(U.S.-Canada Power System Outage Task Force: April 2004) (2003 Blackout Report).

106 2003 Blackout Report at 23.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 128 -

Inadequate Long Term and Operations Planning

Issue 2003 Blackout 2011 Blackout

Insufficient Analysis in 

Seasonal Studies

“[T]he studies FirstEnergy relied on . 

. . were not robust, thorough, or upto-date. This left FE’s planners and 

operators with a deficient 

understanding of their system’s 

capabilities and risks under a range 

of system conditions.” (P. 39).

“FE’s 2003 Summer Study focused 

primarily on single-contingency (N1) events, and did not consider 

significant multiple contingency 

losses and security. . . . Overall, the 

summer study posited less stressful 

system conditions than actually 

occurred August 14, 2003 (when 

load was well below historic peak 

demand).” (P 39).

“TOPs do not always run their 

individual seasonal planning studies 

based on the multiple WECC base cases 

(heavy and light load summer, heavy 

and light load winter, and heavy 

spring), but, instead, may focus on only 

one load level.” (Finding 7) 

“Seasonal planning studies do not 

adequately consider all facilities that 

may affect BPS reliability, including 

external facilities and lower-voltage 

facilities.”(Finding 6)

“In the seasonal planning process, at 

least one TOP did not share with 

neighboring TOPs overload relay trip 

settings on transformers and 

transmission lines that impacted 

external BPS systems.” (Finding 8)

Inadequate 

Identification and 

Study of Critical 

External Facilities

“On August 14 four or five capacitor 

banks within the Cleveland-Akron 

area had been removed from service 

for routine inspection. . . . These 

static reactive power sources are 

important for voltage support. . . . 

The unavailability of the critical 

reactive resources was not known to 

those outside of FirstEnergy.” (PP. 

26-27).

“NERC policy requires that critical 

facilities be identified and that 

neighboring control areas and 

reliability coordinators be made 

aware of the status of those facilities 

to identify the impact of those 

conditions on their own facilities. 

However, FE never identified these 

capacitor banks as critical and so did 

not pass on status information to 

others.” (P. 27).

“Not all of the affected TOPs conduct 

next-day studies or share them with 

their neighbors and WECC RC. . . .TOPs 

were not alerted to contingencies on 

neighboring systems that could impact 

their internal system and the need to 

plan for such contingencies.” (Finding 

1)

“In conducting next-day studies, some 

affected TOPs focus primarily on the 

TOPs’ internal SOLs and the need to 

stay within established Rated Path 

limits, without adequate consideration 

of some lower voltage facilities.” 

(Finding 3)

“[In conducting next-day studies,] . . .

the RC does not study sub-100 kV 

facilities that impact BPS reliability 

unless it has specifically been alerted to 

issues with such facilities by individual 

TOPs...” (Finding 3)
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 129 -

Inaccurate Dynamic 

Models

“The after-the-fact models developed 

to simulate August 14 conditions and 

events found that the dynamic 

modeling assumptions for generator 

and load power factors in regional 

planning and operating models were 

frequently inaccurate.” (P. 160).

“. . . neither the tripping of the SONGS 

units nor the system collapse of SDG&E 

and CFE could be detected using WECC 

dynamic models because some of the 

elements of the event are not explicitly 

included in those models.” (Finding 

10) 

To mitigate these concerns, the 2003 Blackout Report recommended that “NERC 

should work with the regional reliability councils to establish regional power system 

models that enable the sharing of consistent and validated data among entities in the 

region,”106F

107 and “[c]larify criteria for identification of operationally critical facilities, and 

improve dissemination of updated information on unplanned outages.”107F

108 This inquiry’s 

report likewise recommends that entities cooperate and coordinate more effectively 

across all planning horizons, especially by increasing visibility in both external systems 

and lower voltage facilities that could impact BPS reliability.

Inadequate Situational Awareness 

The 2003 Blackout Report stated, “A principal cause of the August 14 blackout 

was a lack of situational awareness, which was in turn the result of inadequate reliability 

tools and backup capabilities.”108F

109 Similarly, the instant inquiry determined that 

inadequate real-time situational awareness contributed to the cascading outages. In 

both events, for example, the affected entities’ real-time monitoring tools were not 

adequate to alert operators to system conditions and contingencies. Also, some of the 

affected entities in both events did not use their real-time tools to monitor system 

conditions. As a result of these situational awareness issues, affected entities in both 

events were not aware that they were no longer operating in a secure N-1 state and were 

not alerted to the need to take corrective actions.

Inadequate Situational Awareness

Issue 2003 Blackout 2011 Blackout

System Visibility “MISO [the Reliability 

Coordinator] had interpretive and 

operational tools and a large 

amount of system data, but had a 

limited view of FE’s system.” (P. 

67).

“Affected TOPs have limited 

real-time visibility outside their 

systems, typically monitoring 

only one external bus. As a 

result, they lack adequate 

situational awareness of external 

contingencies that could impact 

 107 2003 Blackout Report at 160.

108 2003 Blackout Report at 3.

109 2003 Blackout Report at 159. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 130 -

Inadequate Situational Awareness

Issue 2003 Blackout 2011 Blackout

their systems. They also may not 

fully understand how internal 

contingencies could affect SOLs 

in their neighbors’ systems.”

(Finding 11) 

Inadequate RealTime Monitoring 

Tools

“FE’s operational monitoring 

equipment was not adequate to 

alert FE’s operators regarding 

important deviations in operating 

conditions and the need for 

corrective action.” (P. 19). 

“FE’s control room operators lost 

the alarm function that provided 

audible and visual indications 

when a significant piece of 

equipment changed from an 

acceptable to a problematic 

condition.” (P. 51).

MISO’s incomplete tool set and the 

failure to supply its state estimator 

with correct system data on August 

14 contributed to the lack of 

situational awareness.” (P. 159).

“Affected TOPs’ real-time tools 

are not adequate or, in one case, 

operational to provide the 

situational awareness necessary 

to identify contingencies and 

reliably operate their systems.”

(Finding 12)

“. . . a TOP lost the ability to 

conduct [Real Time Contingency 

Analysis] RTCA more than 30 

minutes prior to and throughout 

the course of the event ...[and] 

did not notify WECC RC or any 

of its neighboring 

TOPs...”(Finding 15)

Operating in an 

Unsecure State

“FE’s operators were not aware 

that the system was operating 

outside first contingency limits . . . 

because they did not conduct a 

contingency analysis.” (P. 64).

“MISO’s reliability coordinators 

were using non-real-time data to 

support real-time “flowgate” 

monitoring. This prevented MISO 

from detecting an N-1 security 

violation in FE’s system and from 

assisting FE in necessary relief 

actions.” (P. 19).

“Since FE’s operators were not 

aware and did not recognize events 

as they were occurring, they took 

no actions to return the system to a 

reliable state.” (P. 65).

“The cascading nature of the 

event that led to uncontrolled 

separation of San Diego, IID, 

Yuma, and CFE indicates that an 

[interconnection reliability 

operating limit] IROL was 

violated . . . In addition, the 

established SOLs of 2,200 MW 

on Path 44 and 1,800 MW on HNG are invalid...”(Finding 18)

“One affected TOP operated in 

an unsecured N-1 state. . . . when 

it relied on post-contingency 

mitigation plans for its internal 

contingencies and subsequent 

overloads and trips, while 

assuming there would be 

sufficient time to mitigate the 

contingencies.” (Finding 13)

To remedy these weaknesses in situational awareness, the 2003 Blackout Report 

recommended that entities [e]valuate and adopt better real-time tools for operators and 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 131 -

reliability coordinators.”109F110 Similarly, this inquiry’s report recommends that operators 

develop and effectively utilize the real-time tools at their disposal and include all 

facilities that can impact BPS reliability. 

Protection Systems 

During both events, protection system settings exacerbated and accelerated the 

cascading nature of the outages. As stated in the 2003 Blackout Report, zone 3 relay 

settings “did not cause the blackout, [but] it is certain that they greatly expanded and 

accelerated the spread of the cascade.”110F

111 Similarly, load responsive relay settings 

accelerated the September 8th cascade and effectively eliminated the window in which 

operators could have taken mitigating actions. 

Protection Systems

Issue 2003 Blackout 2011 Blackout

Overly Conservative 

Relay Protection 

Settings

“A few lines have zone 3 settings 

designed with overload margins 

close to the long-term emergency 

limit of the line. . . .Thus, it is 

possible for a zone 3 relay to 

operate on line load or overload in 

extreme contingency conditions 

even in the absence of a fault.” (P. 

80)

“Some affected TOs set overload 

relay protection settings on 

transformers just above the 

transformers’ emergency rating, 

resulting in facilities being 

automatically removed from 

service before TOPs have 

sufficient time to take control 

actions . . . following an N-1 

contingency.” (Finding 25)

Cascading Relay 

Overload Trips

“[B]ecause these zone 2 and 3 

relays tripped after each line 

overloaded, these relays were the 

common mode of failure that 

accelerated the geographic spread 

of the cascade.” (P. 80)

“Some TOs set relays to isolate 

facilities for loading conditions 

slightly above their thirty 

minute emergency ratings. As a 

result, several transmission 

lines and transformers tripped 

within seconds of exceeding 

their emergency ratings, leaving 

TOPs insufficient time to 

mitigate overloads.” (Finding 

26) 

Relay Protection 

Acting Too Quickly to 

Allow System 

Operators to Take

Action

“[T]he speed of the zone 2 and 3 

operations across Ohio and 

Michigan eliminated any 

possibility . . . that either operator 

action or automatic intervention 

could have limited or mitigated the 

growing cascade.” (P. 80).

“Some affected TOs set overload 

relay protection settings on 

transformers just above the 

transformers’ emergency rating, 

resulting in facilities being 

automatically removed from 

service before TOPs have 

 110 2003 Blackout Report at 159.

111 2003 Blackout Report at 82. Zone 3 relays “provide breaker failure and relay backup for remote distance 

faults on a transmission line.” Id. at 80.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 132 -

Protection Systems

Issue 2003 Blackout 2011 Blackout

sufficient time to take control 

actions...” (Finding 25)

“. . . several transmission lines 

and transformers tripped within 

seconds of exceeding their 

emergency ratings, leaving 

TOPs insufficient time to 

mitigate overloads. (Finding 26)

After seeing the consequences of conservative zone 3 settings, the 2003 Blackout 

Report recommended that “[i]ndustry is to review zone 3 relays on lines of 230 kV and 

higher.”111F

112 This inquiry’s report similarly recommends that Transmission Owners 

review their facilities’ overload relay protection settings to ensure the appropriate 

margin between relay settings and emergency ratings.

 112 2003 Blackout Report at 158. 
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 133 -

Appendix D: Benchmarking the Model

I. Introduction and Background

The inquiry’s Modeling and Simulation Team replicated system conditions on 

September 8, 2011, and the events leading up to the blackout. The model reflects the 

state of the electric system before and during the event, with the real power output of 

generators dispatched to the values recorded in SCADA data. With any major event on 

the BPS, it is important to accurately model the system before and during the event in 

order to: (1) verify the Sequence of Events; (2) support reconciliation of disparate 

measurement data; and (3) simulate and evaluate hypothetical scenarios, or “what-if” 

scenarios.

In order to ensure the accuracy of these tasks, the Modeling and Simulation Team 

benchmarked the model to recorded SCADA and PMU measurements using the 

following guidelines. Key facilities and interfaces in the affected area were generally 

benchmarked to within 5% or 10 MVA accuracy to the measured data. Generator 

reactive outputs were also checked against recorded values to ensure that the 

representation of reactive power margin was reasonably accurate. The team also 

monitored most other facilities in the affected area to ensure that the flows and voltage 

were reasonably close to measured data. Many of these other facilities also met the same 

guidelines used to benchmark the key facilities and interfaces.

The iterative process between benchmarking and case alteration has traditionally 

been time-consuming. The team pursued methods that would ultimately decrease the 

amount of time spent benchmarking so that results could quickly be used to identify 

problem areas in the case and make appropriate adjustments. Because the team received 

SCADA and PMU measurement data from many sources and entities, the data was: (1) 

organized into a consistent format, useful for automated benchmarking; and (2) crosschecked and verified for accuracy. In organizing the data, the team also considered how 

each data point would map back to both power flow and dynamics results. The team 

ultimately achieved a single process to: (1) import power flow results; (2) import 

dynamics results; (3) compare the results to measured data from many sources at 

various quasi-steady state times during the event; (4) export tables showing the 

percentage accuracy; and (5) export graphs showing the accuracy of the results relative 

to measured data throughout the event.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 134 -

II. Discussion

The locations and measurements that the team selected for benchmarking were 

naturally predicated on the available measurements. While the team compared each 

available data point to the model results, it did not benchmark the model to all available 

data points. Instead the team focused its benchmarking effort on a “study area” that 

included SDG&E, IID, the APS Yuma load pocket, and portions of CFE and SCE. The 

team gave preference to measurements that were available in multiple data sources with 

some reasonable agreement between the different sources, and particular preference to 

those locations where PMU measurements were available, because these measurements 

could also be benchmarked against a full dynamics simulation.

Following each set of simulations, the team reviewed the benchmarking data both 

graphically and tabularly, and tuned the modeling case and simulation parameters in an 

attempt to bring the case closer to measured reality. The team would then re-run the 

simulation, and repeat this process.

Custom Interfaces

Even though the team selected the best possible set of benchmarking data, and a 

substantial amount of work went into calibrating the study area of the modeling case to 

those measurements, inconsistencies between some data points persisted. These 

inconsistencies arose due to the multitude of subtle settings and parameters for 

equipment, such as a changed tap on a single transformer affecting reactive power flow. 

For this reason, the team developed “custom interfaces” to benchmark an aggregation of 

points. If an aggregated, modeled sub-system was very close to the actual measurements 

for that system, then the simulation could be trusted to accurately reflect the system. For 

example, if reactive power flow was misallocated to a pair of adjacent transformers 

sourcing a sub-system, the specific reactive flow on each transformer may not be of 

particular importance to the model. However, the reactive flow to the aggregate load 

being served by those transformers may have a significant impact on a neighboring subsystem, and be crucial to effective benchmarking.

The custom interfaces were also defined so as to indicate the amount of flow into 

or across a particular sub-system. For example, the calculated flows at the “IID North 92 

kV System” interface give an idea of the amount and nature of the load in the northern 

IID 92 kV system. The custom interfaces selected include:

• IID North 92 kV System: All transmission sources for the northern IID 92 kV system, including 

the 230/92 kV transformers at Coachella Valley and Ramon, the 161/92 kV transformers at 

Coachella Valley and Avenue 58, and the 92 kV lines between the northern and southern IID 

systems.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 135 -

• IID South 92 kV System: All transmission sources for the southern IID 92 kV system, including 

the 230 kV transformer at El Centro, the 161/92 kV transformers at El Centro and Niland, and the 

92 kV lines between the southern and northern IID systems.

• Yuma Pocket: Interfaces between the Yuma area 69 kV system (including portions of both APS 

and WALC service territories) and higher-voltage systems, including the 500/69 kV transformers at 

N. Gila, the 161/69 kV transformers at Gila, and the 161 kV line from Pilot Knob to Yucca.

• Southwest California Desert Imports: All transmission sources into the 

IID/SDG&E/CFE/Yuma area other than Path 44.

Figure 1: Key Facilities and Interfaces

Key Facilities and Interfaces

The team chose key facilities and interfaces in the affected area as a way to 

quickly evaluate the model before fine-tuning it on a more granular level. These key 

facilities and interfaces were benchmarked to within 5% or 10 MVA accuracy to the 

measured data throughout the entire event. The key facilities and interfaces are listed 

below.

• WECC Path 44

• Southwest California Desert Imports

• IID Northern 92 kV System

• Niland-Blythe 161 kV Transmission Line

• IID Southern 92 kV System

• Imperial Valley-El Centro 230 kV Transmission Line (“S” Line)

• Miguel-Imperial Valley 500 kV Transmission Line

• Yuma Pocket

• El Centro-Pilot Knob 161 kV Transmission Line

• Pilot Knob-Knob 161 kV Transmission Line

• Pilot Knob-Yucca 161 kV Transmission Line

• Julian Hinds-Mirage 230 kV Transmission Line

• Julian Hinds-Eagle Mountain 230 kV Transmission Line
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 136 -

III. Results

The following graphs demonstrate the benchmarking results. Each plot gives 

both power flow (see “TSS” in graph legend)112F

113 and dynamic simulation (see “DYD” in 

graph legend)113F

114 results at each selected time step, with the corresponding SCADA 

and/or PMU measurement, as available. In some instances, known issues with 

measured data are annotated on the charts, such as SCADA measurement errors for

Coachella Valley during the interval following the initiating event.

The simulated MW values follow the measurements more closely than the 

simulated MVAR values. This is due to complexity involved in tuning voltage at each bus 

due to incomplete data, such as unknown tap values on large transformers. Overall, the 

MVA values are within our benchmarking guidelines. 

The team also provided a table that compares: (1) the base case at 15:27:00 to the 

measured data; and (2) the case just prior to the loss of the Coachella Valley 

transformers at 15:28:11 to the measured data. This table does not compare the 

dynamics values to the base case at 15:27:00 because the power flow base case was the 

foundation for the dynamics simulation, meaning the values would be equal.

 113 Time Sequence Simulation.

114 Dynamics Data.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 137 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 138 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 139 -

Differences due to 

SCADA measurement 

error at Coachella 

Valley and Ramon
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 140 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 141 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 142 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 143 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 144 -

Difference due to 

SCADA measurement 

captured during power 

swing

Difference due to 

SCADA measurement 

captured during power 

swing
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 145 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 146 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 147 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 148 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

- 149 -
FERC/NERC Staff Report on the September 8, 2011 Blackout 

150

Key facilities and interfaces in the affected area were generally benchmarked to 

within 5% or 10 MVA accuracy to the measured data.

Base Case 15:28:11

Key 

Facility/

Interface

Type Measure

d

Power 

Flow 

Simulat

ion

Delta 

(Value

)

Delta (%) Measure

d

Power 

Flow 

Simulatio

n

Delta 

(Value) Delta (%) Dynamics Simulation

WECC 

Path 44

MVA 1310.25

1323.6

1 -13.36 -1.02% 2453.57 2471.50 -17.93 -0.73% 2457.19

MW 1296.85

1292.0

9 4.77 0.37% 2434.48 2452.40 -17.92 -0.74% 2442.90

MVA

R -186.93 -287.17

100.2

4 -53.63% -305.49 -306.68 1.19 -0.39% -264.64

Southwes

t 

California 

Desert 

Imports

MVA 1328.45

1336.3

8 -7.93 -0.60% 333.50 349.91 -16.41 -4.92% 347.90

MW 1301.35

1307.7

5 -6.40 -0.49% 310.76 316.68 -5.92 -1.90% 310.03

MVA

R 266.96 275.16 -8.19 -3.07% -121.02 -148.82 27.80 -22.97% -157.84

IID North 

92 kV 

System

MVA 475.15 476.81 -1.66 -0.35% 416.84 480.17 -63.32114F

115 -15.19%115F116 466.78

MW 471.60 473.23 -1.63 -0.35% 414.58 477.61 -63.04 -15.21% 464.39

MVA

R -58.01 -58.31 0.30 -0.52% -43.43 -49.46 6.03 -13.88% -47.15

Niland -

Blythe 

161 kV 

Line

MVA 67.49 69.45 -1.97 -2.92% 108.41 118.34 -9.93 -9.16% 117.37

MW -65.13 -65.99 0.86 -1.32% -96.10 -100.93 4.83 -5.02% -101.43

MVA

R 17.68 21.67 -3.99 -22.56% 50.17 61.79 -11.62 -23.16% 59.07

IID South 

92 kV 

System

MVA 105.60 110.97 -5.37 -5.08% 114.63 132.43 -17.80116F

117 -15.53%117F118 124.85

MW 104.67 110.91 -6.24 -5.96% 106.14 114.84 -8.70 -8.20% 108.87

MVA

R -14.03 -3.73 -10.30 73.42% -43.29 -65.93 22.65 -52.31% -61.12

Imperial 

Valley - El 

Centro

230 kV 

Line (“S” 

Line)

MVA 96.08 105.20 -9.12 -9.49% 125.31 119.52 5.79 4.62% 302.90

MW 94.24 104.76 -10.53 -11.17% -109.22 -101.41 -7.81 7.15% 302.90

MVA

R -18.75 -9.58 -9.17 48.89% 61.42 63.26 -1.83 -2.98% 0.05

Miguel -

Imperial 

Valley

500 kV 

Line

MVA 1088.80

1095.2

2 -6.42 -0.59% 225.19 214.12 11.07 4.91% 77.98

MW -1087.30

-

1093.1

0 5.80 -0.53% -188.50 -191.82 3.32 -1.76% 77.31

 115 Large differences due to SCADA measurement errors at Coachella Valley and Ramon

116 Id.

117 The team experienced difficulty in calibrating the MVAR flows in this area, but are generally confident in the 

benchmarking because the MW values are within 10 MW. The MVA differences in the model appear to increase 

during this event. The representation of the system in this area of the model appears to assume that the IID 

South 92 kV system is a load serving local network. However, the actual transmission system operates in parallel 

with the rest of the BPS. It was difficult to calibrate the flows at the 92 kV to 161 kV interfaces because of the 

differences between the representation of the system in the model versus the parallel nature of the actual 

system.

118 Id.
FERC/NERC Staff Report on the September 8, 2011 Blackout 

151

MVA

R 57.18 68.13 -10.94 -19.14% -123.20 -95.15 -28.05 22.77% -10.26

Yuma 

Pocket

MVA 280.78 281.61 -0.83 -0.30% 279.55 283.18 -3.63 -1.30% 282.79

MW 280.11 281.41 -1.30 -0.46% 278.63 282.81 -4.18 -1.50% 281.63

MVA

R -19.46 -10.70 -8.76 45.00% -22.66 -14.46 -8.20 36.20% -25.68

El Centro 

- Pilot 

Knob 161 

kV Line

MVA 9.15 9.83 -0.69 -7.51% 18.27 21.17 -2.90 -15.87% 18.51

MW -8.90 -8.60 -0.30 3.40% 15.73 17.04 -1.31 -8.33% 16.10

MVA

R 2.09 4.76 -2.68 -127.99% 9.29 12.56 -3.27 -35.17% 9.14

Pilot 

Knob -

Knob 161 

kV Line

MVA 74.27 68.50 5.77 7.76% 135.67 140.14 -4.46 -3.29% 141.54

MW -74.06 -66.62 -7.44 10.05% -120.25 -116.14 -4.11 3.42% -118.13

MVA

R 5.54 15.94 -10.40 -187.80% 62.83 78.42 -15.59 -24.81% 77.96

Pilot 

Knob -

Yucca 

161 kV 

Line

MVA 47.34 41.17 6.18 13.05% 132.82 128.54 4.28 3.22% 130.96

MW 46.92 40.94 5.98 12.74% 127.39 121.67 5.72 4.49% 122.63

MVA

R 6.33 4.30 2.03 32.04% -37.60 -41.46 3.87 -10.28% -45.95

Julian 

Hinds -

Mirage 

230 kV 

Line

MVA 291.92 287.50 4.42 1.51% 273.49 276.69 -3.21 -1.17% 276.32

MW 291.87 287.49 4.38 1.50% 273.46 276.67 -3.21 -1.18% 276.29

MVA

R -5.45 -2.12 -3.33 61.10% -3.98 -3.48 -0.49 12.43% 4.45

Julian 

Hinds -

Eagle 

Mountain 

230 kV 

Line

MVA 55.47 56.59 -1.12 -2.01% 71.70 68.07 3.63 5.06% 66.96

MW 53.29 55.35 -2.07 -3.88% 71.06 65.89 5.18 7.28% 66.26

MVA

R -15.43 -11.78 -3.64 23.62% 9.56 17.13 -7.57 -79.17% 9.64
FERC/NERC Staff Report on the September 8, 2011 Blackout 

152

Appendix E: Inquiry Team Members

FERC Staff

Office of Enforcement

Heather Polzin

Jeremy Medovoy

Catherine Collins

Samuel Backfield

Thomas Lemon

Cherise Ojo

Office of Electric Reliability

Alan Phung

Alireza Ghassemian

Boris Voynik

David Burnham

Eddy Lim

Gilbert Lowe

Jacob Lucas

John Spivak

Ken Githens

Kent Davis

Leonard Chamberlin

Louise Nutter

Mahmood Mirheydar

Michelle Veloso

Monica Taba

Pablo Ovando

Perry Servedio

Sasan Jalali

Terrance Clingan

Terrence Simon

Thomas Reina

Victor Barry

Office of Energy Policy & Innovation

Mary Cain
FERC/NERC Staff Report on the September 8, 2011 Blackout 

153

NERC Staff

Ben McMillan

Bob Cummings

Chris McManus

Dave Nevius

Dmitry Kosterev (Technical Consultant, from BPA)

Earl Shockley

Ed Ruck

Eric Allen

Greg Henry

James Merlo

Jim Griffith

Jim Robinson

Jule Tate

Kimberly Mielcarek

Mark Vastano

Phil Tatro

Phil Winston (Technical Consultant, from Southern Company)

Roman Carter

Terry Brinker

Department of Energy Liason

James McGlone

Nuclear Regulatory Commission Staff

Singh Matharu