U.S.-Canada Power System Outage Task Force

Final Report on the

August 14, 2003 Blackout

in the

United States and Canada:

Causes and

Recommendations

April 2004

U.S.-Canada Power System Outage Task Force

Final Report on the

August 14, 2003 Blackout

in the

United States and Canada:

Causes and

Recommendations

April 2004
ii  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

U.S.-Canada Power System Outage Task Force 

 

March 31, 2004 

Dear Mr. President and Prime Minister: 

We are pleased to submit the Final Report of the U.S.-Canada Power System Outage 

Task Force. As directed by you, the Task Force has completed a thorough investigation 

of the causes of the August 14, 2003 blackout and has recommended actions to minimize 

the likelihood and scope of similar events in the future. 

The report makes clear that this blackout could have been prevented and that immediate 

actions must be taken in both the United States and Canada to ensure that our electric 

system is more reliable. First and foremost, compliance with reliability rules must be 

made mandatory with substantial penalties for non-compliance. 

We expect continued collaboration between our two countries to implement this report’s 

recommendations. Failure to implement the recommendations would threaten the 

reliability of the electricity supply that is critical to the economic, energy and national 

security of our countries. 

The work of the Task Force has been an outstanding example of close and effective 

cooperation between the U.S. and Canadian governments. Such work will continue as we 

strive to implement the Final Report’s recommendations. We resolve to work in 

cooperation with Congress, Parliament, states, provinces and stakeholders to ensure that 

North America’s electric grid is robust and reliable. 

We would like to specifically thank the members of the Task Force and its Working 

Groups for their efforts and support as we investigated the blackout and moved toward 

completion of the Final Report. All involved have made valuable contributions. We 

submit this report with optimism that its recommendations will result in better electric 

service for the people of both our nations. 

Sincerely, 

U.S. Secretary of Energy Minister of Natural Resources Canada 
Contents

Page

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  iii

1. Introduction ..................................................................... 1

2. Overview of the North American Electric Power System and Its Reliability Organizations ....... 5

The North American Power Grid Is One Large, Interconnected Machine .................... 5

Planning and Reliable Operation of the Power Grid Are Technically Demanding.............. 6

Reliability Organizations Oversee Grid Reliability in North America . . . . . . . . . . . . . . . . . . . . . . . 10

Key Parties in the Pre-Cascade Phase of the August 14 Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3. Causes of the Blackout and Violations of NERC Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

The Causes of the Blackout in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Linking Causes to Specific Weaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Institutional Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4. Context and Preconditions for the Blackout: The Northeastern Power Grid

Before the Blackout Began . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Electric Demands on August 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Generation Facilities Unavailable on August 14. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Unanticipated Outages of Transmission and Generation on August 14. . . . . . . . . . . . . . . . . . . . . . 27

Key Parameters for the Cleveland-Akron Area at 15:05 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Power Flow Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Voltages and Voltage Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Past System Events and Adequacy of System Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Model-Based Analysis of the State of the Regional Power System at 15:05 EDT,

Before the Loss of FE’s Harding-Chamberlin 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

System Frequency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5. How and Why the Blackout Began in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Chapter Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Phase 1: A Normal Afternoon Degrades: 12:15 EDT to 14:14 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Phase 2: FE’s Computer Failures: 14:14 EDT to 15:59 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Phase 3: Three FE 345-kV Transmission Line Failures and Many Phone Calls:

15:05 EDT to 15:57 EDT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Phase 4: 138-kV Transmission System Collapse in Northern Ohio: 15:39 to 16:08 EDT . . . . . . . . . 68

6. The Cascade Stage of the Blackout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Phase 5: 345-kV Transmission System Cascade in Northern Ohio and South-Central Michigan . . 77

Phase 6: The Full Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Phase 7: Several Electrical Islands Formed in Northeast U.S. and Canada: 16:10:46 EDT

to 16:12 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Why the Blackout Stopped Where It Did . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Under-Frequency and Under-Voltage Load-Shedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Why the Generators Tripped Off. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

7. The August 14 Blackout Compared With Previous Major North American Outages . . . . . . . . . . . . . 103

Incidence and Characteristics of Power System Outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Outage Descriptions and Major Causal Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Common or Similar Factors Among Major Outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Comparisons With the August 14 Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Contents (Continued) Page

Appendixes

Tables

Figures

iv  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

A. Members of the U.S.-Canada Power System Outage Task Force and Its Three Working Groups . . . 175

B. Description of Outage Investigation and Process for Development of Recommendations . . . . . . . . 179

C. List of Commenters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

D. NERC Actions to Prevent and Mitigate the Impacts of Future Cascading Blackouts . . . . . . . . . . . . . 193

E. List of Electricity Acronyms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

F. Electricity Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

G. Transmittal Letters from the Three Working Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

8. Performance of Nuclear Power Plants Affected by the Blackout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Findings of the U.S. Nuclear Working Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Findings of the Canadian Nuclear Working Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Perspective of Nuclear Regulatory Agencies on Potential Changes to the Grid . . . . . . . . . . . . . . . . 129

9. Physical and Cyber Security Aspects of the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Summary and Primary Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

SWG Mandate and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Cyber Security in the Electricity Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Information Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Cyber Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

10. Recommendations to Prevent or Minimize the Scope of Future Blackouts . . . . . . . . . . . . . . . . . . . 139

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

4.1. Generators Not Available on August 14. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2. Benchmarking Model Results to Actual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3. Comparison of Voltage Criteria (Percent) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.1. Changing Conditions That Affect System Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

8.1. U.S. Nuclear Plant Trip Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

8.2. Summary of Events for U. S. Nuclear Power Plants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

8.3. Summary of Shutdown Events for Canadian Nuclear Power Plants . . . . . . . . . . . . . . . . . . . . . . . . 128

2.1. Basic Structure of the Electric System ............................................... 5

2.2. North American Interconnections.................................................. 6

2.3. PJM Load Curve, August 18-24, 2003 ............................................... 7

2.4. Normal and Abnormal Frequency Ranges............................................ 7

2.5. NERC Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.6. NERC Regions and Control Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.7. NERC Reliability Coordinators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.8. Reliability Coordinators and Control Areas in Ohio and Surrounding States. . . . . . . . . . . . . . . . . 13

4.1. August 2003 Temperatures in the U.S. Northeast and Eastern Canada. . . . . . . . . . . . . . . . . . . . . . 25

4.2. Load Forecasts Below Actuals, August 11 through 14. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.3. MW and MVAr Output from Eastlake Unit 5 on August 14. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.4. Generation, Demand, and Interregional Power Flows on August 14, 2003, at 15:05 EDT. . . . . . . . 29
Figures (Continued) Page

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  v

4.5. Scheduled Imports and Exports for the Northeast Central Region, June 1 through

August 13, 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.6. Impacts of Transactions Flows on Critical Line Loadings, August 14, 2003 . . . . . . . . . . . . . . . . . . . 30

4.7. Actual Voltages Across the Ohio Area Before and On August 14, 2003. . . . . . . . . . . . . . . . . . . . . . . 34

4.8. Reactive Reserves Around Ohio on August 14, 2003, for Representative Generators in the Area. . . 37

4.9. Loss of the Perry Unit Hurts Critical Voltages and Reactive Reserves: V-Q Analyses . . . . . . . . . . . . 42

4.10. Impact of Perry Unit Outage on Cleveland-Akron Area Voltage Stability . . . . . . . . . . . . . . . . . . . . . 43

4.11. Frequency on August 14, 2003, up to 16:09 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.12. Hourly Deviations in Eastern Interconnection Frequency for the Month of August 2003 . . . . . . . . 44

5.1. Timeline: Start of the Blackout in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.2. Timeline Phase 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.3. Eastlake Unit 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.4. Timeline Phase 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.5. FirstEnergy 345-kV Line Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.6. Voltages on FirstEnergy’s 345-kV Lines: Impacts of Line Trips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.7. Timeline Phase 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.8. Harding-Chamberlin 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5.9. Hanna-Juniper 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.10. Cause of the Hanna-Juniper Line Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5.11. Star-South Canton 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.12. Cumulative Effects of Sequential Outages on Remaining 345-kV Lines . . . . . . . . . . . . . . . . . . . . . . 64

5.13. Timeline Phase 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.14. Voltages on FirstEnergy’s 138-kV Lines: Impact of Line Trips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.15. Simulated Effect of Prior Outages on 138-kV Line Loadings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6.1. Rate of Line and Generator Trips During the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.2. Area Affected by the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.3. Sammis-Star 345-kV Line Trip, 16:05:57 EDT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

6.4. Sammis-Star 345-kV Line Trip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.5. Line Flows Into Michigan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.6. Ohio 345-kV Lines Trip, 16:08:59 to 16:09:07 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.7. New York-Ontario Line Flows at Niagara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.8. First Power Swing Has Varying Impacts Across the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

6.9. Map of Zone 3 (and Zone 2s Operating Like Zone 3s) Relay Operations on August 14, 2003 . . . . . . 81

6.10. Michigan and Ohio Power Plants Trip. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

6.11. Transmission and Generation Trips in Michigan, 16:10:36 to 16:10:37 EDT . . . . . . . . . . . . . . . . . . 83

6.12. Flows on Keith-Waterman 230-kV Ontario-Michigan Tie Line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

6.13. Simulated 345-kV Line Loadings from 16:05:57 through 16:10:38.4 EDT . . . . . . . . . . . . . . . . . . . . 84

6.14. Simulated Regional Interface Loadings from 16:05:57 through 16:10:38.4 EDT. . . . . . . . . . . . . . . . 84

6.15. Michigan Lines Trip and Ohio Separates from Pennsylvania, 16:10:36 to 16:10:38.6 EDT . . . . . . . 85

6.16. Active and Reactive Power and Voltage from Ontario into Detroit . . . . . . . . . . . . . . . . . . . . . . . . . . 86

6.17. Measured Power Flows and Frequency Across Regional Interfaces, 16:10:30 to 16:11:00 EDT,

with Key Events in the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

6.18. Cleveland and Toledo Islanded, 16:10:39 to 16:10:46 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

6.19. Generators Under Stress in Detroit, as Seen from Keith PSDR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

6.20. Western Pennsylvania Separates from New York, 16:10:39 EDT to 16:10:44 EDT . . . . . . . . . . . . . . 89

6.21. Northeast Separates from Eastern Interconnection, 16:10:45 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . 89

6.22. PJM to New York Interties Disconnect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

6.23. New York and New England Separate, Multiple Islands Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

6.24. Generator Trips by Time and Cause. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

6.25. Events at One Large Generator During the Cascade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Figures (Continued) Page

vi  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

6.26. Measured Power Flows and Frequency Across Regional Interfaces, 16:10:45 to 16:11:30 EDT,

with Key Events in the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

6.27. Frequency Separation Between Ontario and Western New York . . . . . . . . . . . . . . . . . . . . . . . . . . 99

6.28. Electric Islands Reflected in Frequency Plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

6.29. Area Affected by the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

6.30. Cascade Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

7.1. North American Power System Outages, 1984-1997. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
1. Introduction

On August 14, 2003, large portions of the Midwest

and Northeast United States and Ontario, Canada,

experienced an electric power blackout. The outage affected an area with an estimated 50 million

people and 61,800 megawatts (MW) of electric

load in the states of Ohio, Michigan, Pennsylvania, New York, Vermont, Massachusetts, Connecticut, New Jersey and the Canadian province of

Ontario. The blackout began a few minutes after

4:00 pm Eastern Daylight Time (16:00 EDT), and

power was not restored for 4 days in some parts of

the United States. Parts of Ontario suffered rolling

blackouts for more than a week before full power

was restored. Estimates of total costs in the United

States range between $4 billion and $10 billion

(U.S. dollars).1 In Canada, gross domestic product

was down 0.7% in August, there was a net loss of

18.9 million work hours, and manufacturing shipments in Ontario were down $2.3 billion (Canadian dollars).2

On August 15, President George W. Bush and

then-Prime Minister Jean Chrétien directed that a

joint U.S.-Canada Power System Outage Task

Force be established to investigate the causes of

the blackout and ways to reduce the possibility of

future outages. They named U.S. Secretary of

Energy Spencer Abraham and Herb Dhaliwal,

Minister of Natural Resources, Canada, to chair

the joint Task Force. (Mr. Dhaliwal was later succeeded by Mr. John Efford as Minister of Natural

Resources and as co-chair of the Task Force.)

Three other U.S. representatives and three other

Canadian representatives were named to the

Task Force. The U.S. members were Tom Ridge,

Secretary of Homeland Security; Pat Wood III,

Chairman of the Federal Energy Regulatory Commission; and Nils Diaz, Chairman of the Nuclear

Regulatory Commission. The Canadian members

were Deputy Prime Minister John Manley, later

succeeded by Deputy Prime Minister Anne

McLellan; Kenneth Vollman, Chairman of the

National Energy Board; and Linda J. Keen, President and CEO of the Canadian Nuclear Safety

Commission.

The Task Force divided its work into two phases:

 Phase I: Investigate the outage to determine its

causes and why it was not contained.

 Phase II: Develop recommendations to reduce

the possibility of future outages and reduce the

scope of any that occur.

The Task Force created three Working Groups to

assist in both phases of its work—an Electric System Working Group (ESWG), a Nuclear Working

Group (NWG), and a Security Working Group

(SWG). The Working Groups were made up of

state and provincial representatives, federal

employees, and contractors working for the U.S.

and Canadian government agencies represented

on the Task Force.

The Task Force published an Interim Report on

November 19, 2003, summarizing the facts that

the bi-national investigation found regarding the

causes of the blackout on August 14, 2003. After

November 19, the Task Force’s technical investigation teams pursued certain analyses that were

not complete in time for publication in the Interim

Report. The Working Groups focused on the drafting of recommendations for the consideration of

the Task Force to prevent future blackouts and

reduce the scope of any that nonetheless occur. In

drafting these recommendations, the Working

Groups drew substantially on information and

insights from the investigation teams’ additional

analyses, and on inputs received at three public

meetings (in Cleveland, New York City, and

Toronto) and two technical conferences (in Philadelphia and Toronto). They also drew on comments filed electronically by interested parties on

websites established for this purpose by the

U.S. Department of Energy and Natural Resources

Canada.

Although this Final Report presents some new

information about the events and circumstances

before the start of the blackout and additional

detail concerning the cascade stage of the blackout, none of the comments received or additional

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  1
analyses performed by the Task Force’s investigators have changed the validity of the conclusions

published in the Interim Report. This report,

however, presents findings concerning additional

violations of reliability requirements and institutional and performance deficiencies beyond those

identified in the Interim Report.

The organization of this Final Report is similar to

that of the Interim Report, and it is intended to

update and supersede the Interim Report. It is

divided into ten chapters, including this introductory chapter:

 Chapter 2 provides an overview of the institutional framework for maintaining and ensuring

the reliability of the bulk power system in North

America, with particular attention to the roles

and responsibilities of several types of reliability-related organizations.

 Chapter 3 identifies the causes of the blackout

and identifies failures to perform effectively

relative to the reliability policies, guidelines,

and standards of the North American Electric

Reliability Council (NERC) and, in some cases,

deficiencies in the standards themselves.

 Chapter 4 discusses conditions on the regional

power system on and before August 14 and

identifies conditions and failures that did and

did not contribute to the blackout.

 Chapter 5 describes the afternoon of August 14,

starting from normal operating conditions, then

going into a period of abnormal but still potentially manageable conditions, and finally into

an uncontrollable blackout in northern Ohio.

 Chapter 6 provides details on the cascade phase

of the blackout as it spread in Ohio and then

across the Northeast, and explains why the system performed as it did.

 Chapter 7 compares the August 14, 2003, blackout with previous major North American power

outages.

 Chapter 8 examines the performance of the

nuclear power plants affected by the August 14

outage.

 Chapter 9 addresses issues related to physical

and cyber security associated with the outage.

 Chapter 10 presents the Task Force’s recommendations for preventing future blackouts and

reducing the scope of any that occur.

Chapter 10 includes a total of 46 recommendations, but the single most important of them is that

the U.S. Congress should enact the reliability provisions in H.R. 6 and S. 2095 to make compliance

with reliability standards mandatory and enforceable. If that could be done, many of the other recommended actions could be accomplished readily

in the course of implementing the legislation. An

overview of the recommendations (by titles only)

is provided on pages 3 and 4.

Chapter 2 is very little changed from the version

published in the Interim Report. Chapter 3 is new

to this Final Report. Chapters 4, 5, and 6 have been

revised and expanded from the corresponding

chapters (3, 4, and 5) of the Interim Report. Chapters 7, 8, and 9 are only slightly changed from

Chapters 6, 7, and 8 of the Interim Report. The

Interim Report had no counterpart to Chapter 10.

This report also includes seven appendixes:

 Appendix A lists the members of the Task Force

and the three working groups.

 Appendix B describes the Task Force’s investigative process for developing the Task Force’s

recommendations.

 Appendix C lists the parties who either commented on the Interim Report, provided suggestions for recommendations, or both.

 Appendix D reproduces a document released on

February 10, 2004 by NERC, describing its

actions to prevent and mitigate the impacts of

future cascading blackouts.

 Appendix E is a list of electricity acronyms.

 Appendix F provides a glossary of electricity

terms.

 Appendix G contains transmittal letters pertinent to this report from the three Working

Groups.

2  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 
 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  3

Overview of Task Force Recommendations: Titles Only

Group I. Institutional Issues Related to Reliability

1. Make reliability standards mandatory and enforceable, with penalties for noncompliance.

2. Develop a regulator-approved funding mechanism for NERC and the regional reliability councils,

to ensure their independence from the parties they oversee.

3. Strengthen the institutional framework for reliability management in North America.

4. Clarify that prudent expenditures and investments for bulk system reliability (including investments in new technologies) will be recoverable through transmission rates.

5. Track implementation of recommended actions to improve reliability.

6. FERC should not approve the operation of new RTOs or ISOs until they have met minimum

functional requirements.

7. Require any entity operating as part of the bulk power system to be a member of a regional reliability council if it operates within the council’s footprint.

8. Shield operators who initiate load shedding pursuant to approved guidelines from liability or

retaliation.

9. Integrate a “reliability impact” consideration into the regulatory decision-making process.

10. Establish an independent source of reliability performance information.

11. Establish requirements for collection and reporting of data needed for post-blackout analyses.

12. Commission an independent study of the relationships among industry restructuring, competition, and reliability.

13. DOE should expand its research programs on reliability-related tools and technologies.

14. Establish a standing framework for the conduct of future blackout and disturbance

investigations.

Group II. Support and Strengthen NERC’s Actions of February 10, 2004

15. Correct the direct causes of the August 14, 2003 blackout.

16. Establish enforceable standards for maintenance of electrical clearances in right-of-way areas.

17. Strengthen the NERC Compliance Enforcement Program.

18. Support and strengthen NERC’s Reliability Readiness Audit Program.

19. Improve near-term and long-term training and certification requirements for operators, reliability

coordinators, and operator support staff.

20. Establish clear definitions for normal, alert and emergency operational system conditions. Clarify

roles, responsibilities, and authorities of reliability coordinators and control areas under each

condition.

21. Make more effective and wider use of system protection measures.

22. Evaluate and adopt better real-time tools for operators and reliability coordinators.

23. Strengthen reactive power and voltage control practices in all NERC regions.

24. Improve quality of system modeling data and data exchange practices.

25. NERC should reevaluate its existing reliability standards development process and accelerate the

adoption of enforceable standards.

26. Tighten communications protocols, especially for communications during alerts and emergencies. Upgrade communication system hardware where appropriate.

27. Develop enforceable standards for transmission line ratings.

28. Require use of time-synchronized data recorders.

29. Evaluate and disseminate lessons learned during system restoration.

30. Clarify criteria for identification of operationally critical facilities, and improve dissemination of

updated information on unplanned outages.

31. Clarify that the transmission loading relief (TLR) process should not be used in situations involving an actual violation of an Operating Security Limit. Streamline the TLR process.

(continued on page 142)
Endnotes

4  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

1 See “The Economic Impacts of the August 2003 Blackout,”

Electric Consumer Research Council (ELCON), February 2,

2004.

2 Statistics Canada, Gross Domestic Product by Industry,

August 2003, Catalogue No. 15-001; September 2003 Labour

Force Survey; Monthly Survey of Manufacturing, August 2003,

Catalogue No. 31-001.

Overview of Task Force Recommendations: Titles Only (Continued)

Group III. Physical and Cyber Security of North American Bulk Power Systems

32. Implement NERC IT standards.

33. Develop and deploy IT management procedures.

34. Develop corporate-level IT security governance and strategies.

35. Implement controls to manage system health, network monitoring, and incident management.

36. Initiate U.S.-Canada risk management study.

37. Improve IT forensic and diagnostic capabilities.

38. Assess IT risk and vulnerability at scheduled intervals.

39. Develop capability to detect wireless and remote wireline intrusion and surveillance.

40. Control access to operationally sensitive equipment.

41. NERC should provide guidance on employee background checks.

42. Confirm NERC ES-ISAC as the central point for sharing security information and analysis.

43. Establish clear authority for physical and cyber security.

44. Develop procedures to prevent or mitigate inappropriate disclosure of information.

Group IV. Canadian Nuclear Power Sector

45. The Task Force recommends that the Canadian Nuclear Safety Commission request Ontario

Power Generation and Bruce Power to review operating procedures and operator training associated with the use of adjuster rods.

46. The Task Force recommends that the Canadian Nuclear Safety Commission purchase and install

backup generation equipment.
2. Overview of the North American Electric Power

System and Its Reliability Organizations

The North American Power Grid

Is One Large, Interconnected

Machine

The North American electricity system is one of

the great engineering achievements of the past 100

years. This electricity infrastructure represents

more than $1 trillion (U.S.) in asset value, more

than 200,000 miles—or 320,000 kilometers (km)

of transmission lines operating at 230,000 volts

and greater, 950,000 megawatts of generating

capability, and nearly 3,500 utility organizations

serving well over 100 million customers and 283

million people.

Modern society has come to depend on reliable

electricity as an essential resource for national

security; health and welfare; communications;

finance; transportation; food and water supply;

heating, cooling, and lighting; computers and

electronics; commercial enterprise; and even

entertainment and leisure—in short, nearly all

aspects of modern life. Customers have grown to

expect that electricity will almost always be available when needed at the flick of a switch. Most

customers have also experienced local outages

caused by a car hitting a power pole, a construction crew accidentally damaging a cable, or a

lightning storm. What is not expected is the occurrence of a massive outage on a calm, warm day.

Widespread electrical outages, such as the one

that occurred on August 14, 2003, are rare, but

they can happen if multiple reliability safeguards

break down.

Providing reliable electricity is an enormously

complex technical challenge, even on the most

routine of days. It involves real-time assessment,

control and coordination of electricity production

at thousands of generators, moving electricity

across an interconnected network of transmission

lines, and ultimately delivering the electricity to

millions of customers by means of a distribution

network.

As shown in Figure 2.1, electricity is produced at

lower voltages (10,000 to 25,000 volts) at generators from various fuel sources, such as nuclear,

coal, oil, natural gas, hydro power, geothermal,

photovoltaic, etc. Some generators are owned by

the same electric utilities that serve the end-use

customer; some are owned by independent power

producers (IPPs); and others are owned by customers themselves—particularly large industrial

customers.

Electricity from generators is “stepped up” to

higher voltages for transportation in bulk over

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  5

Figure 2.1. Basic Structure of the Electric System
transmission lines. Operating the transmission

lines at high voltage (i.e., 230,000 to 765,000 volts)

reduces the losses of electricity from conductor

heating and allows power to be shipped economically over long distances. Transmission lines are

interconnected at switching stations and substations to form a network of lines and stations called

a power “grid.” Electricity flows through the interconnected network of transmission lines from the

generators to the loads in accordance with the

laws of physics—along “paths of least resistance,”

in much the same way that water flows through a

network of canals. When the power arrives near a

load center, it is “stepped down” to lower voltages

for distribution to customers. The bulk power system is predominantly an alternating current (AC)

system, as opposed to a direct current (DC) system, because of the ease and low cost with which

voltages in AC systems can be converted from one

level to another. Some larger industrial and commercial customers take service at intermediate

voltage levels (12,000 to 115,000 volts), but most

residential customers take their electrical service

at 120 and 240 volts.

While the power system in North America is commonly referred to as “the grid,” there are actually

three distinct power grids or “interconnections”

(Figure 2.2). The Eastern Interconnection includes

the eastern two-thirds of the continental United

States and Canada from Saskatchewan east to the

Maritime Provinces. The Western Interconnection

includes the western third of the continental

United States (excluding Alaska), the Canadian

provinces of Alberta and British Columbia, and a

portion of Baja California Norte, Mexico. The third

interconnection comprises most of the state of

Texas. The three interconnections are electrically

independent from each other except for a few

small direct current (DC) ties that link them.

Within each interconnection, electricity is produced the instant it is used, and flows over virtually all transmission lines from generators to

loads.

The northeastern portion of the Eastern Interconnection (about 10 percent of the interconnection’s

total load) was affected by the August 14 blackout.

The other two interconnections were not

affected.1

Planning and Reliable Operation

of the Power Grid Are Technically

Demanding

Reliable operation of the power grid is complex

and demanding for two fundamental reasons:

 First, electricity flows at close to the speed of

light (186,000 miles per second or 297,600

km/sec) and is not economically storable in

large quantities. Therefore electricity must be

produced the instant it is used.

 Second, without the use of control devices too

expensive for general use, the flow of alternating current (AC) electricity cannot be controlled

like a liquid or gas by opening or closing a valve

in a pipe, or switched like calls over a longdistance telephone network.2 Electricity flows

freely along all available paths from the generators to the loads in accordance with the laws of

physics—dividing among all connected flow

paths in the network, in inverse proportion to

the impedance (resistance plus reactance) on

each path.

Maintaining reliability is a complex enterprise

that requires trained and skilled operators, sophisticated computers and communications, and careful planning and design. The North American

Electric Reliability Council (NERC) and its ten

Regional Reliability Councils have developed system operating and planning standards for ensuring the reliability of a transmission grid that are

based on seven key concepts:

 Balance power generation and demand

continuously.

 Balance reactive power supply and demand to

maintain scheduled voltages.

 Monitor flows over transmission lines and other

facilities to ensure that thermal (heating) limits

are not exceeded.

6  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Figure 2.2. North American Interconnections
 Keep the system in a stable condition.

 Operate the system so that it remains in a reliable condition even if a contingency occurs,

such as the loss of a key generator or transmission facility (the “N-1 criterion”).

 Plan, design, and maintain the system to operate reliably.

 Prepare for emergencies.

These seven concepts are explained in more detail

below.

1. Balance power generation and demand continuously. To enable customers to use as much

electricity as they wish at any moment, production by the generators must be scheduled or

“dispatched” to meet constantly changing

demands, typically on an hourly basis, and then

fine-tuned throughout the hour, sometimes

through the use of automatic generation controls to continuously match generation to actual

demand. Demand is somewhat predictable,

appearing as a daily demand curve—in the

summer, highest during the afternoon and evening and lowest in the middle of the night, and

higher on weekdays when most businesses are

open (Figure 2.3).

Failure to match generation to demand causes

the frequency of an AC power system (nominally 60 cycles per second or 60 Hertz) to

increase (when generation exceeds demand) or

decrease (when generation is less than demand)

(Figure 2.4). Random, small variations in frequency are normal, as loads come on and off

and generators modify their output to follow the

demand changes. However, large deviations in

frequency can cause the rotational speed of generators to fluctuate, leading to vibrations that

can damage generator turbine blades and other

equipment. Extreme low frequencies can trigger

automatic under-frequency “load shedding,”

which takes blocks of customers off-line in

order to prevent a total collapse of the electric

system. As will be seen later in this report, such

an imbalance of generation and demand can

also occur when the system responds to major

disturbances by breaking into separate

“islands”; any such island may have an excess

or a shortage of generation, compared to

demand within the island.

2. Balance reactive power supply and demand to

maintain scheduled voltages. Reactive power

sources, such as capacitor banks and generators, must be adjusted during the day to maintain voltages within a secure range pertaining to

all system electrical equipment (stations, transmission lines, and customer equipment). Most

generators have automatic voltage regulators

that cause the reactive power output of generators to increase or decrease to control voltages to

scheduled levels. Low voltage can cause electric

system instability or collapse and, at distribution voltages, can cause damage to motors and

the failure of electronic equipment. High voltages can exceed the insulation capabilities of

equipment and cause dangerous electric arcs

(“flashovers”).

3. Monitor flows over transmission lines and

other facilities to ensure that thermal (heating)

limits are not exceeded. The dynamic interactions between generators and loads, combined

with the fact that electricity flows freely across

all interconnected circuits, mean that power

flow is ever-changing on transmission and distribution lines. All lines, transformers, and

other equipment carrying electricity are heated

by the flow of electricity through them. The

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  7

Figure 2.3. PJM Load Curve, August 18-24, 2003

Figure 2.4. Normal and Abnormal Frequency

Ranges
flow must be limited to avoid overheating and

damaging the equipment. In the case of overhead power lines, heating also causes the metal

conductor to stretch or expand and sag closer to

ground level. Conductor heating is also affected

by ambient temperature, wind, and other factors. Flow on overhead lines must be limited to

ensure that the line does not sag into obstructions below such as trees or telephone lines, or

violate the minimum safety clearances between

the energized lines and other objects. (A short

circuit or “flashover”—which can start fires or

damage equipment—can occur if an energized

line gets too close to another object). Most transmission lines, transformers and other currentcarrying devices are monitored continuously to

ensure that they do not become overloaded or

violate other operating constraints. Multiple

ratings are typically used, one for normal conditions and a higher rating for emergencies. The

primary means of limiting the flow of power on

transmission lines is to adjust selectively the

output of generators.

4. Keep the system in a stable condition. Because

the electric system is interconnected and

dynamic, electrical stability limits must be

observed. Stability problems can develop very

quickly—in just a few cycles (a cycle is 1/60th of

a second)—or more slowly, over seconds or

minutes. The main concern is to ensure that

generation dispatch and the resulting power

flows and voltages are such that the system is

stable at all times. (As will be described later in

this report, part of the Eastern Interconnection

became unstable on August 14, resulting in a

cascading outage over a wide area.) Stability

limits, like thermal limits, are expressed as a

maximum amount of electricity that can be

safely transferred over transmission lines.

There are two types of stability limits: (1) Voltage stability limits are set to ensure that the

unplanned loss of a line or generator (which

may have been providing locally critical reactive power support, as described previously)

will not cause voltages to fall to dangerously

low levels. If voltage falls too low, it begins to

collapse uncontrollably, at which point automatic relays either shed load or trip generators

to avoid damage. (2) Power (angle) stability limits are set to ensure that a short circuit or an

unplanned loss of a line, transformer, or generator will not cause the remaining generators and

loads being served to lose synchronism with

one another. (Recall that all generators and

loads within an interconnection must operate at

or very near a common 60 Hz frequency.) Loss

of synchronism with the common frequency

means generators are operating out-of-step with

one another. Even modest losses of synchronism can result in damage to generation equipment. Under extreme losses of synchronism,

the grid may break apart into separate electrical

islands; each island would begin to maintain its

own frequency, determined by the load/generation balance within the island.

5. Operate the system so that it remains in a reliable condition even if a contingency occurs,

such as the loss of a key generator or transmission facility (the “N minus 1 criterion”). The

central organizing principle of electricity reliability management is to plan for the unexpected. The unique characteristics of electricity

8  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Local Supplies of Reactive Power Are Essential to Maintaining Voltage Stability

A generator typically produces some mixture of

“real” and “reactive” power, and the balance

between them can be adjusted at short notice to

meet changing conditions. Real power, measured

in watts, is the form of electricity that powers

equipment. Reactive power, a characteristic of

AC systems, is measured in volt-amperes reactive (VAr), and is the energy supplied to create or

be stored in electric or magnetic fields in and

around electrical equipment. Reactive power is

particularly important for equipment that relies

on magnetic fields for the production of induced

electric currents (e.g., motors, transformers,

pumps, and air conditioning.) Transmission

lines both consume and produce reactive power.

At light loads they are net producers, and at

heavy loads, they are heavy consumers. Reactive

power consumption by these facilities or devices

tends to depress transmission voltage, while its

production (by generators) or injection (from

storage devices such as capacitors) tends to support voltage. Reactive power can be transmitted

only over relatively short distances during heavy

load conditions. If reactive power cannot be supplied promptly and in sufficient quantity, voltages decay, and in extreme cases a “voltage

collapse” may result.
mean that problems, when they arise, can

spread and escalate very quickly if proper safeguards are not in place. Accordingly, through

years of experience, the industry has developed

a network of defensive strategies for maintaining reliability based on the assumption that

equipment can and will fail unexpectedly upon

occasion.

This principle is expressed by the requirement

that the system must be operated at all times to

ensure that it will remain in a secure condition

(generally within emergency ratings for current

and voltage and within established stability

limits) following the loss of the most important

generator or transmission facility (a “worst single contingency”). This is called the “N-1 criterion.” In other words, because a generator or

line trip can occur at any time from random failure, the power system must be operated in a

preventive mode so that the loss of the most

important generator or transmission facility

does not jeopardize the remaining facilities in

the system by causing them to exceed their

emergency ratings or stability limits, which

could lead to a cascading outage.

Further, when a contingency does occur, the

operators are required to identify and assess

immediately the new worst contingencies,

given the changed conditions, and promptly

make any adjustments needed to ensure that if

one of them were to occur, the system would

still remain operational and safe. NERC operating policy requires that the system be restored

as soon as practical but within no more than 30

minutes to compliance with normal limits, and

to a condition where it can once again withstand the next-worst single contingency without violating thermal, voltage, or stability

limits. A few areas of the grid are operated to

withstand the concurrent loss of two or more

facilities (i.e., “N-2”). This may be done, for

example, as an added safety measure to protect

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  9

Why Don’t More Blackouts Happen?

Given the complexity of the bulk power system

and the day-to-day challenges of operating it,

there are a lot of things that could go wrong—

which makes it reasonable to wonder why so few

large outages occur.

Large outages or blackouts are infrequent

because responsible system owners and operators practice “defense in depth,” meaning that

they protect the bulk power system through layers of safety-related practices and equipment.

These include:

1. A range of rigorous planning and operating

studies, including long-term assessments,

year-ahead, season-ahead, week-ahead, dayahead, hour-ahead, and real-time operational

contingency analyses. Planners and operators

use these to evaluate the condition of the system, anticipate problems ranging from likely

to low probability but high consequence, and

develop a good understanding of the limits and

rules for safe, secure operation under such

contingencies. If multiple contingencies occur

in a single area, they are likely to be interdependent rather than random, and should have

been anticipated in planning studies.

2. Preparation for the worst case. The operating

rule is to always prepare the system to be safe

in the face of the worst single contingency that

could occur relative to current conditions,

which means that the system is also prepared

for less adverse contingencies.

3. Quick response capability. Most potential

problems first emerge as a small, local situation. When a small, local problem is handled

quickly and responsibly using NERC operating

practices—particularly to return the system to

N-1 readiness within 30 minutes or less—

the problem can usually be resolved and

contained before it grows beyond local

proportions.

4. Maintain a surplus of generation and transmission. This provides a cushion in day-today operations, and helps ensure that small

problems don’t become big problems.

5. Have backup capabilities for all critical functions. Most owners and operators maintain

backup capabilities—such as redundant

equipment already on-line (from generation in

spinning reserve and transmission operating

margin and limits to computers and other

operational control systems)—and keep an

inventory of spare parts to be able to handle an

equipment failure.
a densely populated metropolitan area or when

lines share a common structure and could be

affected by a common failure mode, e.g., a single lightning strike.

6. Plan, design, and maintain the system to operate reliably. Reliable power system operation

requires far more than monitoring and controlling the system in real-time. Thorough planning, design, maintenance, and analysis are

required to ensure that the system can be operated reliably and within safe limits. Short-term

planning addresses day-ahead and week-ahead

operations planning; long-term planning

focuses on providing adequate generation

resources and transmission capacity to ensure

that in the future the system will be able to

withstand severe contingencies without experiencing widespread, uncontrolled cascading

outages.

A utility that serves retail customers must estimate future loads and, in some cases, arrange

for adequate sources of supplies and plan adequate transmission and distribution infrastructure. NERC planning standards identify a range

of possible contingencies and set corresponding

expectations for system performance under several categories of possible events, ranging from

everyday “probable” events to “extreme” events

that may involve substantial loss of customer

load and generation in a widespread area. NERC

planning standards also address requirements

for voltage support and reactive power, disturbance monitoring, facility ratings, system modeling and data requirements, system protection

and control, and system restoration.

7. Prepare for emergencies. System operators are

required to take the steps described above to

plan and operate a reliable power system, but

emergencies can still occur because of external

factors such as severe weather, operator error,

or equipment failures that exceed planning,

design, or operating criteria. For these rare

events, the operating entity is required to have

emergency procedures covering a credible

range of emergency scenarios. Operators must

be trained to recognize and take effective action

in response to these emergencies. To deal with a

system emergency that results in a blackout,

such as the one that occurred on August 14,

2003, there must be procedures and capabilities

to use “black start” generators (capable of

restarting with no external power source) and to

coordinate operations in order to restore the

system as quickly as possible to a normal and

reliable condition.

Reliability Organizations Oversee

Grid Reliability in North America

NERC is a non-governmental entity whose mission is to ensure that the bulk electric system in

North America is reliable, adequate and secure.

The organization was established in 1968, as a

result of the Northeast blackout in 1965. Since its

inception, NERC has operated as a voluntary organization, relying on reciprocity, peer pressure and

the mutual self-interest of all those involved to

ensure compliance with reliability requirements.

An independent board governs NERC.

To fulfill its mission, NERC:

 Sets standards for the reliable operation and

planning of the bulk electric system.

 Monitors and assesses compliance with standards for bulk electric system reliability.

 Provides education and training resources to

promote bulk electric system reliability.

 Assesses, analyzes and reports on bulk electric

system adequacy and performance.

 Coordinates with regional reliability councils

and other organizations.

 Coordinates the provision of applications

(tools), data and services necessary to support

the reliable operation and planning of the bulk

electric system.

 Certifies reliability service organizations and

personnel.

 Coordinates critical infrastructure protection of

the bulk electric system.

 Enables the reliable operation of the interconnected bulk electric system by facilitating information exchange and coordination among

reliability service organizations.

Recent changes in the electricity industry have

altered many of the traditional mechanisms,

incentives and responsibilities of the entities

involved in ensuring reliability, to the point that

the voluntary system of compliance with reliability standards is generally recognized as not adequate to current needs.3 NERC and many other

electricity organizations support the development

of a new mandatory system of reliability standards

10  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 
and compliance, backstopped in the United States

by the Federal Energy Regulatory Commission.

This will require federal legislation in the United

States to provide for the creation of a new electric

reliability organization with the statutory authority to enforce compliance with reliability standards among all market participants. Appropriate

government entities in Canada and Mexico are

prepared to take similar action, and some have

already done so. In the meantime, NERC encourages compliance with its reliability standards

through an agreement with its members.

NERC’s members are ten regional reliability

councils. (See Figure 2.5 for a map showing the

locations and boundaries of the regional councils.)

In turn, the regional councils have broadened

their membership to include all segments of the

electric industry: investor-owned utilities; federal

power agencies; rural electric cooperatives; state,

municipal and provincial utilities; independent

power producers; power marketers; and end-use

customers. Collectively, the members of the NERC

regions account for virtually all the electricity supplied in the United States, Canada, and a portion

of Baja California Norte, Mexico. The ten regional

councils jointly fund NERC and adapt NERC

standards to meet the needs of their regions. The

August 14 blackout affected three NERC regional

reliability councils—East Central Area Reliability

Coordination Agreement (ECAR), Mid-Atlantic

Area Council (MAAC), and Northeast Power Coordinating Council (NPCC).

“Control areas” are the primary operational entities that are subject to NERC and regional council

standards for reliability. A control area is a geographic area within which a single entity, Independent System Operator (ISO), or Regional

Transmission Organization (RTO) balances generation and loads in real time to maintain reliable

operation. Control areas are linked with each

other through transmission interconnection tie

lines. Control area operators control generation

directly to maintain their electricity interchange

schedules with other control areas. They also

operate collectively to support the reliability of

their interconnection. As shown in Figure 2.6,

there are approximately 140 control areas in North

America. The control area dispatch centers have

sophisticated monitoring and control systems and

are staffed 24 hours per day, 365 days per year.

Traditionally, control areas were defined by utility

service area boundaries and operations were

largely managed by vertically integrated utilities

that owned and operated generation, transmission, and distribution. While that is still true in

some areas, there has been significant restructuring of operating functions and some consolidation

of control areas into regional operating entities.

Utility industry restructuring has led to an

unbundling of generation, transmission and distribution activities such that the ownership and

operation of these assets have been separated

either functionally or through the formation of

independent entities called Independent System

Operators (ISOs) and Regional Transmission

Organizations (RTOs).

 ISOs and RTOs in the United States have been

authorized by FERC to implement aspects of the

Energy Policy Act of 1992 and subsequent FERC

policy directives.

 The primary functions of ISOs and RTOs are to

manage in real time and on a day-ahead basis

the reliability of the bulk power system and the

operation of wholesale electricity markets

within their footprint.

 ISOs and RTOs do not own transmission assets;

they operate or direct the operation of assets

owned by their members.

 ISOs and RTOs may be control areas themselves, or they may encompass more than one

control area.

 ISOs and RTOs may also be NERC Reliability

Coordinators, as described below.

Five RTOs/ISOs are within the area directly

affected by the August 14 blackout. They are:

 Midwest Independent System Operator (MISO)

 PJM Interconnection (PJM)

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  11

Figure 2.5. NERC Regions
 New York Independent System Operator

(NYISO)

 New England Independent System Operator

(ISO-NE)

 Ontario Independent Market Operator (IMO)

Reliability coordinators provide reliability oversight over a wide region. They prepare reliability

assessments, provide a wide-area view of reliability, and coordinate emergency operations in real

time for one or more control areas. They may operate, but do not participate in, wholesale or retail

market functions. There are currently 18 reliability coordinators in North America. Figure 2.7

shows the locations and boundaries of their

respective areas.

Key Parties in the Pre-Cascade

Phase of the August 14 Blackout

The initiating events of the blackout involved two

control areas—FirstEnergy (FE) and American

Electric Power (AEP)—and their respective reliability coordinators, MISO and PJM (see Figures

2.7 and 2.8). These organizations and their reliability responsibilities are described briefly in this

final subsection.

1. FirstEnergy operates a control area in northern Ohio. FirstEnergy (FE) consists of seven

electric utility operating companies. Four of

these companies, Ohio Edison, Toledo Edison,

The Illuminating Company, and Penn Power,

operate in the NERC ECAR region, with MISO

serving as their reliability coordinator. These

four companies now operate as one integrated

control area managed by FE.4

2. American Electric Power (AEP) operates a control area in Ohio just south of FE. AEP is both a

transmission operator and a control area

operator.

3. Midwest Independent System Operator

(MISO) is the reliability coordinator for

FirstEnergy. The Midwest Independent System

12  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Figure 2.6. NERC Regions and Control Areas
Operator (MISO) is the reliability coordinator

for a region of more than 1 million square miles

(2.6 million square kilometers), stretching from

Manitoba, Canada in the north to Kentucky in

the south, from Montana in the west to western

Pennsylvania in the east. Reliability coordination is provided by two offices, one in Minnesota, and the other at the MISO headquarters in

Indiana. Overall, MISO provides reliability

coordination for 37 control areas, most of which

are members of MISO.

4. PJM is AEP’s reliability coordinator. PJM is one

of the original ISOs formed after FERC orders

888 and 889, but was established as a regional

power pool in 1935. PJM recently expanded its

footprint to include control areas and transmission operators within MAIN and ECAR (PJMWest). It performs its duties as a reliability coordinator in different ways, depending on the

control areas involved. For PJM-East, it is

both the control area and reliability coordinator

for ten utilities, whose transmission systems

span the Mid-Atlantic region of New Jersey,

most of Pennsylvania, Delaware, Maryland,

West Virginia, Ohio, Virginia, and the District of

Columbia. The PJM-West facility has the reliability coordinator desk for five control areas

(AEP, Commonwealth Edison, Duquesne Light,

Dayton Power and Light, and Ohio Valley Electric Cooperative) and three generation-only

control areas (Duke Energy’s Washington

County (Ohio) facility, Duke’s Lawrence

County/Hanging Rock (Ohio) facility, and Allegheny Energy’s Buchanan (West Virginia)

facility.

Reliability Responsibilities of Control

Area Operators and Reliability

Coordinators

1. Control area operators have primary responsibility for reliability. Their most important

responsibilities, in the context of this report,

are:

N-1 criterion. NERC Operating Policy 2.A—

Transmission Operations:

“All CONTROL AREAS shall operate so that

instability, uncontrolled separation, or cascading outages will not occur as a result of

the most severe single contingency.”

Emergency preparedness and emergency

response. NERC Operating Policy 5—Emergency Operations, General Criteria:

“Each system and CONTROL AREA shall

promptly take appropriate action to relieve

any abnormal conditions, which jeopardize

reliable Interconnection operation.”

“Each system, CONTROL AREA, and Region

shall establish a program of manual and automatic load shedding which is designed to

arrest frequency or voltage decays that could

result in an uncontrolled failure of components of the interconnection.”

NERC Operating Policy 5.A—Coordination

with Other Systems:

“A system, CONTROL AREA, or pool that is

experiencing or anticipating an operating

emergency shall communicate its current

and future status to neighboring systems,

CONTROL AREAS, or pools and throughout the

interconnection....A system shall inform

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  13

Figure 2.7. NERC Reliability Coordinators

Figure 2.8. Reliability Coordinators and Control

Areas in Ohio and Surrounding States
other systems... whenever . . . the system’s

condition is burdening other systems or

reducing the reliability of the Interconnection . . . [or whenever] the system’s line loadings and voltage/reactive levels are such that

a single contingency could threaten the reliability of the Interconnection.”

NERC Operating Policy 5.C—Transmission

System Relief:

“Action to correct an OPERATING SECURITY

LIMIT violation shall not impose unacceptable stress on internal generation or transmission equipment, reduce system reliability

beyond acceptable limits, or unduly impose

voltage or reactive burdens on neighboring

systems. If all other means fail, corrective

action may require load reduction.”

Operating personnel and training: NERC Operating Policy 8.B—Training:

“Each OPERATING AUTHORITY should periodically practice simulated emergencies. The

scenarios included in practice situations

should represent a variety of operating conditions and emergencies.”

2. Reliability Coordinators such as MISO and

PJM are expected to comply with all aspects of

NERC Operating Policies, especially Policy 9,

Reliability Coordinator Procedures, and its

appendices. Key requirements include:

NERC Operating Policy 9, Criteria for Reliability Coordinators, 5.2:

Have “detailed monitoring capability of the

RELIABILITY AREA and sufficient monitoring

14  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Institutional Complexities and Reliability in the Midwest

The institutional arrangements for reliability in

the Midwest are much more complex than they

are in the Northeast—i.e., the areas covered by

the Northeast Power Coordinating Council

(NPCC) and the Mid-Atlantic Area Council

(MAAC). There are two principal reasons for this

complexity. One is that in NPCC and MAAC, the

independent system operator (ISO) also serves as

the single control area operator for the individual

member systems. In comparison, MISO provides

reliability coordination for 35 control areas in the

ECAR, MAIN, and MAPP regions and 2 others in

the SPP region, and PJM provides reliability coordination for 8 control areas in the ECAR and

MAIN regions (plus one in MAAC). (See table

below.) This results in 18 control-area-tocontrol-area interfaces across the PJM/MISO reliability coordinator boundary.

The other is that MISO has less reliability-related

authority over its control area members than PJM

has over its members. Arguably, this lack of

authority makes day-to-day reliability operations

more challenging. Note, however, that (1) FERC’s

authority to require that MISO have greater

authority over its members is limited; and (2)

before approving MISO, FERC asked NERC for a

formal assessment of whether reliability could be

maintained under the arrangements proposed by

MISO and PJM. After reviewing proposed plans

for reliability coordination within and between

PJM and MISO, NERC replied affirmatively but

provisionally. FERC approved the new MISOPJM configuration based on NERC’s assessment.

NERC conducted audits in November and

December 2002 of the MISO and PJM reliability

plans, and some of the recommendations of the

audit teams are still being addressed. The adequacy of the plans and whether the plans were

being implemented as written are factors in

NERC’s ongoing investigation.

Reliability Coordinator (RC)

Control

Areas in

RC Area

Regional Reliability

Councils Affected and

Number of Control Areas Control Areas of Interest in RC Area

MISO 37 ECAR (12), MAIN (9),

MAPP (14), SPP (2)

FE, Cinergy,

Michigan Electric Coordinated System

PJM 9 MAAC (1), ECAR (7),

MAIN (1)

PJM, AEP,

Dayton Power & Light

ISO New England 2 NPCC (2) ISONE, Maritime Provinces

New York ISO 1 NPCC (1) NYISO

Ontario Independent Market Operator 1 NPCC (1) IMO

Trans-Energie 1 NPCC (1) Hydro Québec
capability of the surrounding RELIABILITY

AREAS to ensure potential security violations

are identified.”

NERC Operating Policy 9, Functions of Reliability Coordinators, 1.7:

“Monitor the parameters that may have significant impacts within the RELIABILITY AREA

and with neighboring RELIABILITY AREAS

with respect to... sharing with other

RELIABILITY COORDINATORS any information

regarding potential, expected, or actual critical operating conditions that could negatively impact other RELIABILITY AREAS. The

RELIABILITY COORDINATOR will coordinate

with other RELIABILITY COORDINATORS and

CONTROL AREAS as needed to develop appropriate plans to mitigate negative impacts of

potential, expected, or actual critical operating conditions . . . .”

NERC Operating Policy 9, Functions of Reliability Coordinators, 6:

“Conduct security assessment and monitoring programs to assess contingency situations. Assessments shall be made in real time

and for the operations planning horizon at

the CONTROL AREA level with any identified

problems reported to the RELIABILITY COORDINATOR. The RELIABILITY COORDINATOR

is to ensure that CONTROL AREA, RELIABILITY

AREA, and regional boundaries are sufficiently modeled to capture any problems

crossing such boundaries.”

Endnotes

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  15

What Constitutes an Operating Emergency?

An operating emergency is an unsustainable

condition that cannot be resolved using the

resources normally available. The NERC Operating Manual defines a “capacity emergency” as

when a system’s or pool’s operating generation

capacity, plus firm purchases from other systems, to the extent available or limited by transfer capability, is inadequate to meet its demand

plus its regulating requirements. It defines an

“energy emergency” as when a load-serving

entity has exhausted all other options and can

no longer provide its customers’ expected

energy requirements. A transmission emergency exists when “the system’s line loadings

and voltage/ reactive levels are such that a single

contingency could threaten the reliability of the

Interconnection.” Control room operators and

dispatchers are given substantial latitude to

determine when to declare an emergency. (See

pages 66-67 in Chapter 5 for more detail.)

1 The province of Québec, although considered a part of the

Eastern Interconnection, is connected to the rest of the Eastern Interconnection only by DC ties. In this instance, the DC

ties acted as buffers between portions of the Eastern Interconnection; transient disturbances propagate through them less

readily. Therefore, the electricity system in Québec was not

affected by the outage, except for a small portion of the province’s load that is directly connected to Ontario by AC transmission lines. (Although DC ties can act as a buffer between

systems, the tradeoff is that they do not allow instantaneous

generation support following the unanticipated loss of a generating unit.)

2 In some locations, bulk power flows are controlled through

specialized devices or systems, such as phase angle regulators, “flexible AC transmission systems” (FACTS), and highvoltage DC converters (and reconverters) spliced into the AC

system. These devices are still too expensive for general

application.

3 See, for example, Maintaining Reliability in a Competitive

Electric Industry (1998), a report to the U.S. Secretary of

Energy by the Task Force on Electric Systems Reliability;

National Energy Policy (2001), a report to the President of the

United States by the National Energy Policy Development

Group, p. 7-6; and National Transmission Grid Study (2002),

U.S. Dept. of Energy, pp. 46-48.

4 The remaining three FE companies, Penelec, Met-Ed, and

Jersey Central Power & Light, are in the NERC MAAC region

and have PJM as their reliability coordinator. The focus of this

report is on the portion of FE in the ECAR reliability region

and within the MISO reliability coordinator footprint.

3. Causes of the Blackout

and Violations of NERC Standards

Summary

This chapter explains in summary form the causes

of the initiation of the blackout in Ohio, based on

the analyses by the bi-national investigation team.

It also lists NERC’s findings to date concerning

seven specific violations of its reliability policies,

guidelines, and standards. Last, it explains how

some NERC standards and processes were inadequate because they did not give sufficiently clear

direction to industry members concerning some

preventive measures needed to maintain reliability, and that NERC does not have the authority to

enforce compliance with the standards. Clear

standards with mandatory compliance, as contemplated under legislation pending in the U.S.

Congress, might have averted the start of this

blackout.

Chapters 4 and 5 provide the details that support

the conclusions summarized here, by describing

conditions and events during the days before and

the day of the blackout, and explain how those

events and conditions did or did not cause or contribute to the initiation of the blackout. Chapter 6

addresses the cascade as the blackout spread

beyond Ohio and reviews the causes and events of

the cascade as distinct from the earlier events in

Ohio.

The Causes of the Blackout in Ohio

A dictionary definition of “cause” is “something

that produces an effect, result, or consequence.”1

In searching for the causes of the blackout, the

investigation team looked back through the progression of sequential events, actions and inactions to identify the cause(s) of each event. The

idea of “cause” is here linked not just to what happened or why it happened, but more specifically

to the entities whose duties and responsibilities

were to anticipate and prepare to deal with the

things that could go wrong. Four major causes, or

groups of causes, are identified (see box on page

18).

Although the causes discussed below produced

the failures and events of August 14, they did not

leap into being that day. Instead, as the following

chapters explain, they reflect long-standing institutional failures and weaknesses that need to be

understood and corrected in order to maintain

reliability.

Linking Causes

to Specific Weaknesses

Seven violations of NERC standards, as identified

by NERC,2 and other conclusions reached by

NERC and the bi-national investigation team are

aligned below with the specific causes of the

blackout. There is an additional category of conclusions beyond the four principal causes—the

failure to act, when it was the result of preceding

conditions. For instance, FE did not respond to the

loss of its transmission lines because it did not

have sufficient information or insight to reveal the

need for action. Note: NERC’s list of violations has

been revised and extended since publication of

the Interim Report. Two violations (numbers 4

and 6, as cited in the Interim Report) were

dropped, and three new violations have been

identified in this report (5, 6, and 7, as numbered

here). NERC continues to study the record and

may identify additional violations.3

Group 1: FirstEnergy and ECAR failed to assess

and understand the inadequacies of FE’s

system, particularly with respect to voltage

instability and the vulnerability of the

Cleveland-Akron area, and FE did not operate

its system with appropriate voltage criteria

and remedial measures.

 FE did not monitor and manage reactive

reserves for various contingency conditions as

required by NERC Policy 2, Section B, Requirement 2.

 NERC Policy 2, Section A, requires a 30-minute

period of time to re-adjust the system to prepare

to withstand the next contingency.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  17
18  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Causes of the Blackout’s Initiation

The Ohio phase of the August 14, 2003, blackout

was caused by deficiencies in specific practices,

equipment, and human decisions by various

organizations that affected conditions and outcomes that afternoon—for example, insufficient

reactive power was an issue in the blackout, but

it was not a cause in itself. Rather, deficiencies in

corporate policies, lack of adherence to industry

policies, and inadequate management of reactive

power and voltage caused the blackout, rather

than the lack of reactive power. There are four

groups of causes for the blackout:

Group 1: FirstEnergy and ECAR failed to

assess and understand the inadequacies of

FE’s system, particularly with respect to

voltage instability and the vulnerability of

the Cleveland-Akron area, and FE did not

operate its system with appropriate voltage

criteria. (Note: This cause was not identified in

the Task Force’s Interim Report. It is based on

analysis completed by the investigative team

after the publication of the Interim Report.)

As detailed in Chapter 4:

A) FE failed to conduct rigorous long-term planning studies of its system, and neglected to

conduct appropriate multiple contingency or

extreme condition assessments. (See pages

37-39 and 41-43.)

B) FE did not conduct sufficient voltage analyses

for its Ohio control area and used operational

voltage criteria that did not reflect actual voltage stability conditions and needs. (See pages

31-37.)

C) ECAR (FE’s reliability council) did not conduct an independent review or analysis of

FE’s voltage criteria and operating needs,

thereby allowing FE to use inadequate practices without correction. (See page 39.)

D)Some of NERC’s planning and operational

requirements and standards were sufficiently

ambiguous that FE could interpret them to

include practices that were inadequate for reliable system operation. (See pages 31-33.)

Group 2: Inadequate situational awareness

at FirstEnergy. FE did not recognize or

understand the deteriorating condition of

its system.

As discussed in Chapter 5:

A) FE failed to ensure the security of its transmission system after significant unforeseen contingencies because it did not use an effective

contingency analysis capability on a routine

basis. (See pages 49-50 and 64.)

B) FE lacked procedures to ensure that its operators were continually aware of the functional

state of their critical monitoring tools. (See

pages 51-53, 56.)

C) FE control center computer support staff and

operations staff did not have effective internal

communications procedures. (See pages 54,

56, and 65-67.)

D) FE lacked procedures to test effectively the

functional state of its monitoring tools after

repairs were made. (See page 54.)

E) FE did not have additional or back-up monitoring tools to understand or visualize the status of their transmission system to facilitate

its operators’ understanding of transmission

system conditions after the failure of their primary monitoring/alarming systems. (See

pages 53, 56, and 65.)

Group 3: FE failed to manage adequately tree

growth in its transmission rights-of-way.

This failure was the common cause of the outage

of three FE 345-kV transmission lines and one

138-kV line. (See pages 57-64.)

Group 4: Failure of the interconnected grid’s

reliability organizations to provide effective

real-time diagnostic support.

As discussed in Chapter 5:

A) MISO did not have real-time data from

Dayton Power and Light’s Stuart-Atlanta

345-kV line incorporated into its state estimator (a system monitoring tool). This precluded

(continued on page 19)
 NERC is lacking a well-defined control area

(CA) audit process that addresses all CA responsibilities. Control area audits have generally not

been conducted with sufficient regularity and

have not included a comprehensive audit of the

control area’s compliance with all NERC and

Regional Council requirements. Compliance

with audit results is not mandatory.

 ECAR did not conduct adequate review or analyses of FE’s voltage criteria, reactive power

management practices, and operating needs.

 FE does not have an adequate automatic undervoltage load-shedding program in the Cleveland-Akron area.

Group 2: Inadequate situational awareness

at FirstEnergy. FE did not recognize or

understand the deteriorating condition of

its system.

Violations (Identified by NERC):

 Violation 7: FE’s operational monitoring equipment was not adequate to alert FE’s operators

regarding important deviations in operating

conditions and the need for corrective action as

required by NERC Policy 4, Section A, Requirement 5.

 Violation 3: FE’s state estimation and contingency analysis tools were not used to assess

system conditions, violating NERC Operating

Policy 5, Section C, Requirement 3, and Policy

4, Section A, Requirement 5.

Other Problems:

 FE personnel did not ensure that their

Real-Time Contingency Analysis (RTCA) was a

functional and effective EMS application as

required by NERC Policy 2, Section A, Requirement 1.

 FE’s operational monitoring equipment was not

adequate to provide a means for its operators to

evaluate the effects of the loss of significant

transmission or generation facilities as required

by NERC Policy 4, Section A, Requirement 4.

 FE’s operations personnel were not provided

sufficient operations information and analysis

tools as required by NERC Policy 5, Section C,

Requirement 3.

 FE’s operations personnel were not adequately

trained to maintain reliable operation under

emergency conditions as required by NERC Policy 8, Section 1.

 NERC Policy 4 has no detailed requirements for:

(a) monitoring and functional testing of critical

EMS and supervisory control and data acquisition (SCADA) systems, and (b) contingency

analysis.

 NERC Policy 6 includes a requirement to plan

for loss of the primary control center, but lacks

specific provisions concerning what must be

addressed in the plan.

 NERC system operator certification tests for

basic operational and policy knowledge.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  19

Causes of the Blackout’s Initiation (Continued)

MISO from becoming aware of FE’s system

problems earlier and providing diagnostic

assistance or direction to FE. (See pages

49-50.)

B) MISO’s reliability coordinators were using

non-real-time data to support real-time

“flowgate” monitoring. This prevented MISO

from detecting an N-1 security violation in

FE’s system and from assisting FE in necessary relief actions. (See pages 48 and 63.)

C) MISO lacked an effective way to identify the

location and significance of transmission line

breaker operations reported by their Energy

Management System (EMS). Such information would have enabled MISO operators to

become aware earlier of important line outages. (See page 48.)

D) PJM and MISO lacked joint procedures or

guidelines on when and how to coordinate a

security limit violation observed by one of

them in the other’s area due to a contingency

near their common boundary. (See pages

62-63 and 65-66.)

In the chapters that follow, sections that relate to

particular causes are denoted with the following

symbols: Cause 2

Inadequate

Situational

Awareness

Cause 3

Inadequate

Tree

Trimming

Cause 4

Inadequate

RC Diagnostic

Support

Cause 1

Inadequate

System

Understanding
Significant additional training is needed to

qualify an individual to perform system operation and management functions.

Group 3: FE failed to manage adequately tree

growth in its transmission rights-of-way. This

failure was the common cause of the outage of

three FE 345-kV transmission lines and

affected several 138-kV lines.

 FE failed to maintain equipment ratings

through a vegetation management program. A

vegetation management program is necessary to

fulfill NERC Policy 2, Section A, Requirement 1

(Control areas shall develop, maintain, and

implement formal policies and procedures to

provide for transmission security... including

equipment ratings.)

 Vegetation management requirements are not

defined in NERC Standards and Policies.

Group 4: Failure of the interconnected grid’s

reliability organizations to provide effective

diagnostic support.

Violations (Identified by NERC):

 Violation 4: MISO did not notify other reliability coordinators of potential system problems as

required by NERC Policy 9, Section C, Requirement 2.

 Violation 5: MISO was using non-real-time data

to support real-time operations, in violation of

NERC Policy 9, Appendix D, Section A, Criteria

5.2.

 Violation 6: PJM and MISO as reliability coordinators lacked procedures or guidelines between

their respective organizations regarding the

coordination of actions to address an operating

security limit violation observed by one of them

in the other’s area due to a contingency near

their common boundary, as required by Policy

9, Appendix C. Note: Policy 9 lacks specifics on

what constitutes coordinated procedures and

training.

Other Problems:

 MISO did not have adequate monitoring capability to fulfill its reliability coordinator responsibilities as required by NERC Policy 9,

Appendix D, Section A.

 Although MISO is the reliability coordinator for

FE, on August 14 FE was not a signatory to the

MISO Transmission Owners Agreement and

was not under the MISO tariff, so MISO did not

have the necessary authority as FE’s Reliability

Coordinator as required by NERC Policy 9, Section B, Requirement 2.

 Although lacking authority under a signed

agreement, MISO as reliability coordinator nevertheless should have issued directives to FE to

return system operation to a safe and reliable

level as required by NERC Policy 9, Section B,

Requirement 2, before the cascading outages

occurred.

 American Electric Power (AEP) and PJM

attempted to use the transmission loading relief

(TLR) process to address transmission power

flows without recognizing that a TLR would not

solve the problem.

 NERC Policy 9 does not contain a requirement

for reliability coordinators equivalent to the

NERC Policy 2 statement that monitoring

equipment is to be used in a manner that would

bring to the reliability coordinator’s attention

any important deviations in operating

conditions.

 NERC Policy 9 lacks criteria for determining the

critical facilities lists in each reliability coordinator area.

 NERC Policy 9 lacks specifics on coordinated

procedures and training for reliability coordinators regarding “operating to the most conservative limit” in situations when operating

conditions are not fully understood.

Failures to act by FirstEnergy or others to solve

the growing problem, due to the other causes.

Violations (Identified by NERC):

 Violation 1: Following the outage of the Chamberlin-Harding 345-kV line, FE operating personnel did not take the necessary action to

return the system to a safe operating state as

required by NERC Policy 2, Section A, Standard

1.

 Violation 2: FE operations personnel did not

adequately communicate its emergency operating conditions to neighboring systems as

required by NERC Policy 5, Section A.

Other Problems:

 FE operations personnel did not promptly take

action as required by NERC Policy 5, General

20  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 
Criteria, to relieve the abnormal conditions

resulting from the outage of the HardingChamberlin 345-kV line.

 FE operations personnel did not implement

measures to return system operation to within

security limits in the prescribed time frame

of NERC Policy 2, Section A, Standard 2, following the outage of the Harding-Chamberlin

345-kV line.

 FE operations personnel did not exercise the

authority to alleviate the operating security

limit violation as required by NERC Policy 5,

Section C, Requirement 2.

 FE did not exercise a load reduction program to

relieve the critical system operating conditions

as required by NERC Policy 2, Section A,

Requirement 1.2.

 FE did not demonstrate the application of

effective emergency operating procedures as

required by NERC Policy 6, Section B, Emergency Operations Criteria.

 FE operations personnel did not demonstrate

that FE has an effective manual load shedding

program designed to address voltage decays

that result in uncontrolled failure of components of the interconnection as required by

NERC Policy 5, General Criteria.

 NERC Policy 5 lacks specifics for Control Areas

on procedures for coordinating with other systems and training regarding “operating to the

most conservative limit” in situations when

operating conditions are not fully understood.

Institutional Issues

As indicated above, the investigation team identified a number of institutional issues with respect

to NERC’s reliability standards. Many of the institutional problems arise not because NERC is an

inadequate or ineffective organization, but rather

because it has no structural independence from

the industry it represents and has no authority to

develop strong reliability standards and to enforce

compliance with those standards. While many in

the industry and at NERC support such measures,

legislative action by the U.S. Congress is needed to

make this happen.

These institutional issues can be summed up

generally:

1. Although NERC’s provisions address many of

the factors and practices which contributed to

the blackout, some of the policies or guidelines

are inexact, non-specific, or lacking in detail,

allowing divergent interpretations among reliability councils, control areas, and reliability

coordinators. NERC standards are minimum

requirements that may be made more stringent

if appropriate by regional or subregional bodies,

but the regions have varied in their willingness

to implement exacting reliability standards.

2. NERC and the industry’s reliability community

were aware of the lack of specificity and detail

in some standards, including definitions of

Operating Security Limits, definition of

planned outages, and delegation of Reliability

Coordinator functions to control areas, but they

moved slowly to address these problems

effectively.

3. Some standards relating to the blackout’s

causes lack specificity and measurable compliance criteria, including those pertaining to

operator training, back-up control facilities,

procedures to operate when part or all of the

EMS fails, emergency procedure training,

system restoration plans, reactive reserve

requirements, line ratings, and vegetation

management.

4. The NERC compliance program and regionbased auditing process has not been comprehensive or aggressive enough to assess the capability of all control areas to direct the operation

of their portions of the bulk power system. The

effectiveness and thoroughness of regional

councils’ efforts to audit for compliance with

reliability requirements have varied significantly from region to region. Equally important,

absent mandatory compliance and penalty

authority, there is no requirement that an entity

found to be deficient in an audit must remedy

the deficiency.

5. NERC standards are frequently administrative

and technical rather than results-oriented.

6. A recently-adopted NERC process for development of standards is lengthy and not yet fully

understood or applied by many industry participants. Whether this process can be adapted to

support an expedited development of clear and

auditable standards for key topics remains to be

seen.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  21
7. NERC has not had an effective process to ensure

that recommendations made in various reports

and disturbance analyses are tracked for

accountability. On their own initiative, some

regional councils have developed effective

tracking procedures for their geographic areas.

Control areas and reliability coordinators operate

the grid every day under guidelines, policies, and

requirements established by the industry’s reliability community under NERC’s coordination. If

those policies are strong, clear, and unambiguous,

then everyone will plan and operate the system at

a high level of performance and reliability will be

high. But if those policies are ambiguous and do

not make entities’ roles and responsibilities clear

and certain, they allow companies to perform at

varying levels and system reliability is likely to be

compromised.

Given that NERC has been a voluntary organization that makes decisions based on member votes,

if NERC’s standards have been unclear, nonspecific, lacking in scope, or insufficiently strict,

that reflects at least as much on the industry community that drafts and votes on the standards as it

does on NERC. Similarly, NERC’s ability to obtain

compliance with its requirements through its

audit process has been limited by the extent to

which the industry has been willing to support the

audit program.

Endnotes

22  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

1 Webster’s II New Riverside University Dictionary, Riverside

Publishing Co., 1984.

2 A NERC team looked at whether and how violations of

NERC’s reliability requirements may have occurred in the

events leading up to the blackout. They also looked at

whether deficiencies in the requirements, practices and procedures of NERC and the regional reliability organizations

may have contributed to the blackout. They found seven specific violations of NERC operating policies (although some are

qualified by a lack of specificity in the NERC requirements).

The Standards, Procedures and Compliance Investigation

Team reviewed the NERC Policies for violations, building on

work and going beyond work done by the Root Cause Analysis Team. Based on that review the Standards team identified

a number of violations related to policies 2, 4, 5, and 9.

Violation 1: Following the outage of the ChamberlinHarding 345-kV line, FE did not take the necessary actions to

return the system to a safe operating state within 30 minutes.

(While Policy 5 on Emergency Operations does not address

the issue of “operating to the most conservative limit” when

coordinating with other systems and operating conditions are

not understood, other NERC policies do address this matter:

Policy 2, Section A, Standard 1, on basic reliability for single

contingencies; Policy 2, Section A, Standard 2, to return a system to within operating security limits within 30 minutes;

Policy 2, Section A, Requirement 1, for formal policies and

procedures to provide for transmission security; Policy 5,

General Criteria, to relieve any abnormal conditions that jeopardize reliable operation; Policy 5, Section C, Requirement 1,

to relieve security limit violations; and Policy 5, Section 2,

Requirement 2, which gives system operators responsibility

and authority to alleviate operating security limit violations

using timely and appropriate actions.)

Violation 2: FE did not notify other systems of an impending system emergency. (Policy 5, Section A, Requirement 1,

directs a system to inform other systems if it is burdening others, reducing system reliability, or if its lack of single contingency coverage could threaten Interconnection reliability.

Policy 5, Section A, Criteria, has similar provisions.)

Violation 3: FE’s state estimation/contingency analysis

tools were not used to assess the system conditions. (This is

addressed in Operating Policy 5, Section C, Requirement 3,

concerning assessment of Operating Security Limit violations, and Policy 4, Section A, Requirement 5, which

addresses using monitoring equipment to inform the system

operator of important conditions and the potential need for

corrective action.)

Violation 4: MISO did not notify other reliability coordinators of potential problems. (Policy 9, Section C, Requirement

2, directing the reliability coordinator to alert all control areas

and reliability coordinators of a potential transmission problem.)

Violation 5: MISO was using non-real-time data to support

real-time operations. (Policy 9, Appendix D, Section A, Criteria For Reliability Coordinators 5.2, regarding adequate facilities to perform their responsibilities, including detailed

monitoring capability to identify potential security violations.)

Violation 6: PJM and MISO as Reliability Coordinators

lacked procedures or guidelines between themselves on when

and how to coordinate an operating security limit violation

observed by one of them in the other’s area due to a contingency near their common boundary (Policy 9, Appendix 9C,

Emergency Procedures). Note: Since Policy 9 lacks specifics

on coordinated procedures and training, it was not possible

for the bi-national team to identify the exact violation that

occurred.

Violation 7: The monitoring equipment provided to FE

operators was not sufficient to bring the operators’ attention

to the deviation on the system. (Policy 4, Section A, System

Monitoring Requirements regarding resource availability and

the use of monitoring equipment to alert operators to the need

for corrective action.)

3 NERC has not yet completed its review of planning standards and violations.