# Presentation: 453363

## Joint U.S.-Canada Power System Outage Investigation

- Interim Report
- Causes of the
- August 14th Blackout in the
- United States and Canada

## Overview

- The report
- What caused the blackout?
- Reliability management
- What didn’t cause the blackout?
- How do we know this?
- Key events in the blackout
- Why did the cascade spread?
- Why did the cascade stop where it did?
- Next steps

## U.S.-Canada Interim Report

- Released November 19, 2003
- Result of an exhaustive bi-national investigation
- Working groups on electric system, nuclear plant performance and security
- Hundreds of professionals on investigation teams performed extensive analysis
- Interim report produced by the teams and accepted by the bi-national Task Force

## Conclusions of the Interim Report

- What caused the blackout
- Inadequate situational awareness by FirstEnergy
- Inadequate tree-trimming by FirstEnergy
- Inadequate diagnostic support by reliability coordinators serving the Midwest
- Explanation of the cascade and major events
- Nuclear plants performed well
- No malicious cyber attack caused blackout

## Slide 5

## What caused the blackout (1)

- FirstEnergy lost its system condition alarm system around 2:14pm, so its operators couldn’t tell later on that system conditions were degrading.
- FE lost many capabilities of its Energy Management System from the problems that caused its alarm failure – but operators didn’t realize it had failed
- After 3:05pm, FE lost three 345 kV lines due to contacts with overgrown trees, but didn’t know the lines had gone out of service.

## What caused the blackout (2)

- As each FE line failed, it increased the loading on other lines and drove them closer to failing.  FE lost 16 138kV lines between 3:39 and 4:06pm, but remained unaware of any problem until 3:42pm.
- FE took no emergency action to stabilize the transmission system or to inform its neighbors of its problems.
- The loss of FE’s Sammis-Star 345 kV line at 4:05:57pm was the start of the cascade beyond Ohio.

## What caused the blackout (3)

- MISO (FE’s reliability coordinator) had an unrelated software problem and for much of the afternoon was unable to tell that FE’s lines were becoming overloaded and insecure.
- AEP saw signs of FE’s problems and tried to alert FE, but was repeatedly rebuffed.
- PJM saw the growing problem, but did not have joint procedures in place with MISO to deal with the problem quickly and effectively.

## What caused the blackout (4)

- 1) FirstEnergy didn’t properly understand the condition of its system, which degraded as the afternoon progressed.
- FE didn’t ensure the security of its transmission system because it didn’t use an effective contingency analysis tool routinely.
- FE lost its system monitoring alarms and lacked procedures to identify that failure.
- After efforts to fix that loss, FE didn’t check to see if the repairs had worked.
- FE didn’t have additional monitoring tools to help operators understand system conditions after their main monitoring and alarm tools failed.

## What caused the blackout (5)

- 2) FE failed to adequately trim trees in its transmission rights-of-way.
- Overgrown trees under FE transmission lines caused the first three FE 345 kV line failures.
- These tree/line contacts were not accidents or coincidences
- Trees found in FE rights-of-way are not a new problem
- One tree over 42’ tall; one 14 years old; another 14” in diameter
- Extensive evidence of long-standing tree-line contacts

## What caused the blackout (6)

- 3) Reliability Coordinators did not provide adequate diagnostic support to compensate for FE’s failures.
- MISO’s state estimator failed due to a data error.
- MISO’s flowgate monitoring tool didn’t have real-time line information to detect growing overloads.
- MISO operators couldn’t easily link breaker status to line status to understand changing conditions.
- PJM and MISO lacked joint procedures to coordinate problems affecting their common boundaries.

## Reliability management (1)

- Fundamental rule of grid operations – deal with the grid in front of you and keep it secure.  HOW?
- 1) Balance supply and demand
- 2) Balance reactive power supply and demand to maintain voltages
- 3) Monitor flows to prevent overloads and line overheating
- 4) Keep the system stable

## Reliability management (2)

- 5) Keep the system reliable, even if or after it loses a key facility
- 6) Plan, design and maintain the system to operate reliably
- 7) Prepare for emergencies
- Training
- Procedures and plans
- Back-up facilities and tools
- Communications
- 8) The control area is responsible for its system

## What didn’t cause the blackout (1)

- 1) High power flow patterns across Ohio
- Flows were high but normal
- FE could limit imports if they became excessive
- 2) System frequency variations
- Frequency was acceptable
- 3) Low voltages on 8/14 and earlier
- FE voltages were above 98% through 8/13
- FE voltages held above 95% before 15:05 on 8/14

## What didn’t cause the blackout (2) 

- 4) Independent power producers and reactive power
- IPPs produced reactive power as required in their contracts
- Control area operators and reliability coordinators can order higher reactive power production from IPPs but didn’t on 8/14
- Reactive power must be locally generated and there are few IPPs that are electrically significant to the FE area in Ohio

## What didn’t cause the blackout (3)

- 5) Unanticipated availability or absence of new or out of service generation and transmission
- All of the plants and lines known to be in and out of service on 8/14 were in the MISO day-ahead and morning-of schedule analyses, which indicated the system could be securely operated
- 6) Peak temperatures or loads in the Midwest and Canada
- Conditions were normal for August
- 7) Master Blaster computer virus or malicious cyber attack

## How do we know this?

- The Task Force investigation team has over two hundred experts from the US and Canada government agencies, national laboratories, academics, industry, and consultants
- Extensive interviews, data collection, field visits, computer modeling, and fact-checking of all leads and issues
- Logical, systematic analysis of all possibilities and hypotheses to verify root causes and eliminate false explanations

## What happened on August 14

- At 1:31 pm, FirstEnergy lost the  Eastlake 5 power plant, an important source of reactive power for the Cleveland-Akron area
- Starting at 3:05 pm EDT, three 345 kV lines in FE’s system failed – within normal operating load limits -- due to contacts with overgrown trees

## What happened (2) -- Ohio

- Why did so many trees contact power lines?
- The trees were overgrown because rights-of-way hadn’t been properly maintained
- Lines sag lower in summer with heat and low winds, and sag more with higher current

## What happened (3) -- Ohio

- After the 345 kV lines were lost, at 3:39 pm FE’s 138 kV lines around Akron began to overload and fail; 16 overloaded and tripped out of service

## What happened (4) -- Ohio

- At 4:05 pm, after FirstEnergy’s Sammis-Star 345 kV line failed due to severe overload.

## What happened (5) -- cascade

- Before the loss of Sammis-Star, the blackout was only a local problem in Ohio
- The local problem became a regional problem because FE did not act to contain it nor to inform its neighbors and MISO about the problem
- After Sammis-Star fell at 4:05:57, northern Ohio’s load was shut off from its usual supply sources to the south and east, and the resulting overloads on the broader grid began an unstoppable cascade that flashed a surge of power across the northeast, with many lines overloading and tripping out of service.

## What happened (6) -- cascade

- 1) 4:06

- 2) 4:08:57

- 3) 4:10:37

- 4) 4:10:38.6

## What happened (7) -- cascade

- 5) 4:10:39

- 6) 4:10:44

- 7) 4:10:45

- 8) 4:13

## Power plants affected

- The blackout shut down 263 power plants (531 units) in the US and Canada, most from the cascade after 4:10:44 pm – but none suffered significant damage

## Affected areas

- When the cascade was over at 4:13pm, over 50 million people in the northeast US and the province of Ontario were out of power.

## Why the cascade spread

- Sequential tripping of transmission lines and generators in a widening geographic area, driven by power swings and voltage fluctuations.
- The result of automatic equipment operations (primarily relays and circuit breakers) and system design

## Why the cascade stopped

- Early line trips separated and protected areas from the cascade (southern Ohio).
- Higher voltage lines are better able to absorb voltage and current swings, so helped to buffer against the cascade (AEP, Pennsylvania).
- Areas with high voltage profiles and good reactive power margins weren’t swamped by the sudden voltage and power drain (PJM and New England).
- Areas with good internal balances of generation to load could reach internal equilibrium and island without collapsing (upstate New York and parts of Ontario's Niagara and Cornwall areas).

## Next steps

- Phase 1 investigation continues – more data analysis and modeling of the cascade
- Phase 2 – develop recommendations
- Public consultations in Cleveland, New York, Toronto to receive feedback on Interim Report and recommendations on how to prevent the next blackout
- Letters and comments welcome to US DOE and Natural Resources Canada websites
- Final report released in early 2004.