# U.S.-Canada Power System Outage Task Force



# Final Report on the August 14, 2003 Blackout in the United States and Canada:

Causes and Recommendations

April 2004

# U.S.-Canada Power System Outage Task Force



# Final Report on the August 14, 2003 Blackout in the United States and Canada:

Causes and Recommendations

April 2004ii  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  

## U.S.-Canada Power System Outage Task Force

March 31, 2004

Dear Mr. President and Prime Minister:

We are pleased to submit the Final Report of the U.S.-Canada Power System Outage Task Force. As directed by you, the Task Force has completed a thorough investigation of the causes of the August 14, 2003 blackout and has recommended actions to minimize the likelihood and scope of similar events in the future. The report makes clear that this blackout could have been prevented and that immediate actions must be taken in both the United States and Canada to ensure that our electric system is more reliable. First and foremost, compliance with reliability rules must be made mandatory with substantial penalties for non-compliance. We expect continued collaboration between our two countries to implement this report’s recommendations. Failure to implement the recommendations would threaten the reliability of the electricity supply that is critical to the economic, energy and national security of our countries. The work of the Task Force has been an outstanding example of close and effective cooperation between the U.S. and Canadian governments. Such work will continue as we strive to implement the Final Report’s recommendations. We resolve to work in cooperation with Congress, Parliament, states, provinces and stakeholders to ensure that North America’s electric grid is robust and reliable. We would like to specifically thank the members of the Task Force and its Working Groups for their efforts and support as we investigated the blackout and moved toward completion of the Final Report. All involved have made valuable contributions. We submit this report with optimism that its recommendations will result in better electric service for the people of both our nations.

Sincerely,

U.S. Secretary of Energy Minister of Natural Resources Canada

# Contents
Page

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  iii 1. Introduction ..................................................................... 1 2. Overview of the North American Electric Power System and Its Reliability Organizations ....... 5 The North American Power Grid Is One Large, Interconnected Machine .................... 5 Planning and Reliable Operation of the Power Grid Are Technically Demanding .............. 6 Reliability Organizations Oversee Grid Reliability in North America . . . . . . . . . . . . . . . . . . . . . . . 10 Key Parties in the Pre-Cascade Phase of the August 14 Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3. Causes of the Blackout and Violations of NERC Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 The Causes of the Blackout in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Linking Causes to Specific Weaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Institutional Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4. Context and Preconditions for the Blackout: The Northeastern Power Grid Before the Blackout Began . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Electric Demands on August 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Generation Facilities Unavailable on August 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Unanticipated Outages of Transmission and Generation on August 14 . . . . . . . . . . . . . . . . . . . . . . 27 Key Parameters for the Cleveland-Akron Area at 15:05 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Power Flow Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Voltages and Voltage Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Past System Events and Adequacy of System Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Model-Based Analysis of the State of the Regional Power System at 15:05 EDT, Before the Loss of FE’s Harding-Chamberlin 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 System Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5. How and Why the Blackout Began in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Chapter Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Phase 1: A Normal Afternoon Degrades: 12:15 EDT to 14:14 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Phase 2: FE’s Computer Failures: 14:14 EDT to 15:59 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Phase 3: Three FE 345-kV Transmission Line Failures and Many Phone Calls: 15:05 EDT to 15:57 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Phase 4: 138-kV Transmission System Collapse in Northern Ohio: 15:39 to 16:08 EDT . . . . . . . . . 68 6. The Cascade Stage of the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Phase 5: 345-kV Transmission System Cascade in Northern Ohio and South-Central Michigan . . 77 Phase 6: The Full Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Phase 7: Several Electrical Islands Formed in Northeast U.S. and Canada: 16:10:46 EDT to 16:12 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Why the Blackout Stopped Where It Did . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Under-Frequency and Under-Voltage Load-Shedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Why the Generators Tripped Off . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 7. The August 14 Blackout Compared With Previous Major North American Outages . . . . . . . . . . . . . 103 Incidence and Characteristics of Power System Outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Outage Descriptions and Major Causal Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Common or Similar Factors Among Major Outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Comparisons With the August 14 Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110**Contents (Continued)** Page

**Appendixes**

**Tables**

**Figures**

iv  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

A. Members of the U.S.-Canada Power System Outage Task Force and Its Three Working Groups . . . 175 B. Description of Outage Investigation and Process for Development of Recommendations . . . . . . . . 179 C. List of Commenters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 D. NERC Actions to Prevent and Mitigate the Impacts of Future Cascading Blackouts . . . . . . . . . . . . . 193 E. List of Electricity Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 F. Electricity Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 G. Transmittal Letters from the Three Working Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 8. Performance of Nuclear Power Plants Affected by the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Findings of the U.S. Nuclear Working Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Findings of the Canadian Nuclear Working Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Perspective of Nuclear Regulatory Agencies on Potential Changes to the Grid . . . . . . . . . . . . . . . . 129 9. Physical and Cyber Security Aspects of the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Summary and Primary Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 SWG Mandate and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Cyber Security in the Electricity Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Information Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Cyber Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 10. Recommendations to Prevent or Minimize the Scope of Future Blackouts . . . . . . . . . . . . . . . . . . . 139 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

4.1. Generators Not Available on August 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2. Benchmarking Model Results to Actual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.3. Comparison of Voltage Criteria (Percent) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 7.1. Changing Conditions That Affect System Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 8.1. U.S. Nuclear Plant Trip Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 8.2. Summary of Events for U. S. Nuclear Power Plants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 8.3. Summary of Shutdown Events for Canadian Nuclear Power Plants . . . . . . . . . . . . . . . . . . . . . . . . 128

2.1. Basic Structure of the Electric System ............................................... 5 2.2. North American Interconnections .................................................. 6 2.3. PJM Load Curve, August 18-24, 2003 ............................................... 7 2.4. Normal and Abnormal Frequency Ranges ............................................ 7 2.5. NERC Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.6. NERC Regions and Control Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.7. NERC Reliability Coordinators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.8. Reliability Coordinators and Control Areas in Ohio and Surrounding States . . . . . . . . . . . . . . . . . 13 4.1. August 2003 Temperatures in the U.S. Northeast and Eastern Canada . . . . . . . . . . . . . . . . . . . . . . 25 4.2. Load Forecasts Below Actuals, August 11 through 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.3. MW and MVAr Output from Eastlake Unit 5 on August 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.4. Generation, Demand, and Interregional Power Flows on August 14, 2003, at 15:05 EDT . . . . . . . . 29**Figures (Continued)** Page

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  v 4.5. Scheduled Imports and Exports for the Northeast Central Region, June 1 through August 13, 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.6. Impacts of Transactions Flows on Critical Line Loadings, August 14, 2003 . . . . . . . . . . . . . . . . . . . 30 4.7. Actual Voltages Across the Ohio Area Before and On August 14, 2003 . . . . . . . . . . . . . . . . . . . . . . . 34 4.8. Reactive Reserves Around Ohio on August 14, 2003, for Representative Generators in the Area . . . 37 4.9. Loss of the Perry Unit Hurts Critical Voltages and Reactive Reserves: V-Q Analyses . . . . . . . . . . . . 42 4.10. Impact of Perry Unit Outage on Cleveland-Akron Area Voltage Stability . . . . . . . . . . . . . . . . . . . . . 43 4.11. Frequency on August 14, 2003, up to 16:09 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.12. Hourly Deviations in Eastern Interconnection Frequency for the Month of August 2003 . . . . . . . . 44

5.1. Timeline: Start of the Blackout in Ohio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.2. Timeline Phase 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.3. Eastlake Unit 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.4. Timeline Phase 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.5. FirstEnergy 345-kV Line Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.6. Voltages on FirstEnergy’s 345-kV Lines: Impacts of Line Trips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.7. Timeline Phase 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.8. Harding-Chamberlin 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.9. Hanna-Juniper 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.10. Cause of the Hanna-Juniper Line Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.11. Star-South Canton 345-kV Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.12. Cumulative Effects of Sequential Outages on Remaining 345-kV Lines . . . . . . . . . . . . . . . . . . . . . . 64 5.13. Timeline Phase 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.14. Voltages on FirstEnergy’s 138-kV Lines: Impact of Line Trips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.15. Simulated Effect of Prior Outages on 138-kV Line Loadings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6.1. Rate of Line and Generator Trips During the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.2. Area Affected by the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3. Sammis-Star 345-kV Line Trip, 16:05:57 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.4. Sammis-Star 345-kV Line Trip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.5. Line Flows Into Michigan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.6. Ohio 345-kV Lines Trip, 16:08:59 to 16:09:07 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.7. New York-Ontario Line Flows at Niagara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.8. First Power Swing Has Varying Impacts Across the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.9. Map of Zone 3 (and Zone 2s Operating Like Zone 3s) Relay Operations on August 14, 2003 . . . . . . 81 6.10. Michigan and Ohio Power Plants Trip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.11. Transmission and Generation Trips in Michigan, 16:10:36 to 16:10:37 EDT . . . . . . . . . . . . . . . . . . 83 6.12. Flows on Keith-Waterman 230-kV Ontario-Michigan Tie Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.13. Simulated 345-kV Line Loadings from 16:05:57 through 16:10:38.4 EDT . . . . . . . . . . . . . . . . . . . . 84 6.14. Simulated Regional Interface Loadings from 16:05:57 through 16:10:38.4 EDT . . . . . . . . . . . . . . . . 84 6.15. Michigan Lines Trip and Ohio Separates from Pennsylvania, 16:10:36 to 16:10:38.6 EDT . . . . . . . 85 6.16. Active and Reactive Power and Voltage from Ontario into Detroit . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.17. Measured Power Flows and Frequency Across Regional Interfaces, 16:10:30 to 16:11:00 EDT, with Key Events in the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.18. Cleveland and Toledo Islanded, 16:10:39 to 16:10:46 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.19. Generators Under Stress in Detroit, as Seen from Keith PSDR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.20. Western Pennsylvania Separates from New York, 16:10:39 EDT to 16:10:44 EDT . . . . . . . . . . . . . . 89 6.21. Northeast Separates from Eastern Interconnection, 16:10:45 EDT . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.22. PJM to New York Interties Disconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.23. New York and New England Separate, Multiple Islands Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.24. Generator Trips by Time and Cause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.25. Events at One Large Generator During the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96**Figures (Continued)** Page

vi  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  6.26. Measured Power Flows and Frequency Across Regional Interfaces, 16:10:45 to 16:11:30 EDT, with Key Events in the Cascade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.27. Frequency Separation Between Ontario and Western New York . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.28. Electric Islands Reflected in Frequency Plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.29. Area Affected by the Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.30. Cascade Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

7.1. North American Power System Outages, 1984-1997 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

# 1. Introduction

On August 14, 2003, large portions of the Midwest and Northeast United States and Ontario, Canada, experienced an electric power blackout. The out

age affected an area with an estimated 50 million people and 61,800 megawatts (MW) of electric load in the states of Ohio, Michigan, Pennsylva

nia, New York, Vermont, Massachusetts, Connect

icut, New Jersey and the Canadian province of Ontario. The blackout began a few minutes after 4:00 pm Eastern Daylight Time (16:00 EDT), and power was not restored for 4 days in some parts of the United States. Parts of Ontario suffered rolling blackouts for more than a week before full power was restored. Estimates of total costs in the United States range between $4 billion and $10 billion (U.S. dollars). 1 In Canada, gross domestic product was down 0.7% in August, there was a net loss of18.9 million work hours, and manufacturing shipments in Ontario were down $2.3 billion (Canadian dollars). 2 On August 15, President George W. Bush and then-Prime Minister Jean Chrétien directed that a joint U.S.-Canada Power System Outage Task Force be established to investigate the causes of the blackout and ways to reduce the possibility of future outages. They named U.S. Secretary of Energy Spencer Abraham and Herb Dhaliwal, Minister of Natural Resources, Canada, to chair the joint Task Force. (Mr. Dhaliwal was later suc

ceeded by Mr. John Efford as Minister of Natural Resources and as co-chair of the Task Force.) Three other U.S. representatives and three other Canadian representatives were named to the Task Force. The U.S. members were Tom Ridge, Secretary of Homeland Security; Pat Wood III, Chairman of the Federal Energy Regulatory Com

mission; and Nils Diaz, Chairman of the Nuclear Regulatory Commission. The Canadian members were Deputy Prime Minister John Manley, later succeeded by Deputy Prime Minister Anne McLellan; Kenneth Vollman, Chairman of the National Energy Board; and Linda J. Keen, Presi

dent and CEO of the Canadian Nuclear Safety Commission. The Task Force divided its work into two phases:

 Phase I: Investigate the outage to determine its causes and why it was not contained.

 Phase II: Develop recommendations to reduce the possibility of future outages and reduce the scope of any that occur. The Task Force created three Working Groups to assist in both phases of its work—an Electric Sys

tem Working Group (ESWG), a Nuclear Working Group (NWG), and a Security Working Group (SWG). The Working Groups were made up of state and provincial representatives, federal employees, and contractors working for the U.S. and Canadian government agencies represented on the Task Force. The Task Force published an Interim Report on November 19, 2003, summarizing the facts that the bi-national investigation found regarding the causes of the blackout on August 14, 2003. AfterNovember 19, the Task Force’s technical investigation teams pursued certain analyses that were not complete in time for publication in the InterimReport. The Working Groups focused on the drafting of recommendations for the consideration of the Task Force to prevent future blackouts and reduce the scope of any that nonetheless occur. In drafting these recommendations, the Working Groups drew substantially on information and insights from the investigation teams’ additional analyses, and on inputs received at three public meetings (in Cleveland, New York City, and Toronto) and two technical conferences (in Phila

delphia and Toronto). They also drew on com

ments filed electronically by interested parties on websites established for this purpose by the U.S. Department of Energy and Natural Resources Canada. Although this Final Report presents some new information about the events and circumstances before the start of the blackout and additional detail concerning the cascade stage of the black

out, none of the comments received or additional

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  1analyses performed by the Task Force’s investiga

tors have changed the validity of the conclusions published in the Interim Report. This report, however, presents findings concerning additional violations of reliability requirements and institu

tional and performance deficiencies beyond those identified in the Interim Report. The organization of this Final Report is similar to that of the Interim Report, and it is intended to update and supersede the Interim Report. It is divided into ten chapters, including this introduc

tory chapter:

 Chapter 2 provides an overview of the institu

tional framework for maintaining and ensuring the reliability of the bulk power system in North America, with particular attention to the roles and responsibilities of several types of reliabil

ity-related organizations.

 Chapter 3 identifies the causes of the blackout and identifies failures to perform effectively relative to the reliability policies, guidelines, and standards of the North American Electric Reliability Council (NERC) and, in some cases, deficiencies in the standards themselves.

 Chapter 4 discusses conditions on the regional power system on and before August 14 and identifies conditions and failures that did and did not contribute to the blackout.

 Chapter 5 describes the afternoon of August 14, starting from normal operating conditions, then going into a period of abnormal but still poten

tially manageable conditions, and finally into an uncontrollable blackout in northern Ohio.

 Chapter 6 provides details on the cascade phase of the blackout as it spread in Ohio and then across the Northeast, and explains why the sys

tem performed as it did.

 Chapter 7 compares the August 14, 2003, black

out with previous major North American power outages.

 Chapter 8 examines the performance of the nuclear power plants affected by the August 14 outage.

 Chapter 9 addresses issues related to physical and cyber security associated with the outage.  Chapter 10 presents the Task Force’s recom

mendations for preventing future blackouts and reducing the scope of any that occur. Chapter 10 includes a total of 46 recommenda

tions, but the single most important of them is that the U.S. Congress should enact the reliability pro

visions in H.R. 6 and S. 2095 to make compliance with reliability standards mandatory and enforce

able. If that could be done, many of the other rec

ommended actions could be accomplished readily in the course of implementing the legislation. An overview of the recommendations (by titles only) is provided on pages 3 and 4. Chapter 2 is very little changed from the version published in the Interim Report. Chapter 3 is new to this Final Report. Chapters 4, 5, and 6 have been revised and expanded from the corresponding chapters (3, 4, and 5) of the Interim Report. Chap

ters 7, 8, and 9 are only slightly changed from Chapters 6, 7, and 8 of the Interim Report. The Interim Report had no counterpart to Chapter 10. This report also includes seven appendixes:

 Appendix A lists the members of the Task Force and the three working groups.

 Appendix B describes the Task Force’s investigative process for developing the Task Force’s recommendations.

 Appendix C lists the parties who either commented on the Interim Report, provided suggestions for recommendations, or both.

 Appendix D reproduces a document released on February 10, 2004 by NERC, describing its actions to prevent and mitigate the impacts of future cascading blackouts.

 Appendix E is a list of electricity acronyms.

 Appendix F provides a glossary of electricity terms.

 Appendix G contains transmittal letters perti

nent to this report from the three Working Groups.

2  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  3 Overview of Task Force Recommendations: Titles Only Group I. Institutional Issues Related to Reliability

1. Make reliability standards mandatory and enforceable, with penalties for noncompliance. 2. Develop a regulator-approved funding mechanism for NERC and the regional reliability councils, to ensure their independence from the parties they oversee. 3. Strengthen the institutional framework for reliability management in North America. 4. Clarify that prudent expenditures and investments for bulk system reliability (including invest

ments in new technologies) will be recoverable through transmission rates. 5. Track implementation of recommended actions to improve reliability. 6. FERC should not approve the operation of new RTOs or ISOs until they have met minimum functional requirements. 7. Require any entity operating as part of the bulk power system to be a member of a regional reli

ability council if it operates within the council’s footprint. 8. Shield operators who initiate load shedding pursuant to approved guidelines from liability or retaliation. 9. Integrate a “reliability impact” consideration into the regulatory decision-making process. 10. Establish an independent source of reliability performance information. 11. Establish requirements for collection and reporting of data needed for post-blackout analyses. 12. Commission an independent study of the relationships among industry restructuring, competi

tion, and reliability. 13. DOE should expand its research programs on reliability-related tools and technologies. 14. Establish a standing framework for the conduct of future blackout and disturbance investigations. Group II. Support and Strengthen NERC’s Actions of February 10, 2004 15. Correct the direct causes of the August 14, 2003 blackout. 16. Establish enforceable standards for maintenance of electrical clearances in right-of-way areas. 17. Strengthen the NERC Compliance Enforcement Program. 18. Support and strengthen NERC’s Reliability Readiness Audit Program. 19. Improve near-term and long-term training and certification requirements for operators, reliability coordinators, and operator support staff. 20. Establish clear definitions for normal, alert and emergency operational system conditions. Clarify roles, responsibilities, and authorities of reliability coordinators and control areas under each condition. 21. Make more effective and wider use of system protection measures. 22. Evaluate and adopt better real-time tools for operators and reliability coordinators. 23. Strengthen reactive power and voltage control practices in all NERC regions. 24. Improve quality of system modeling data and data exchange practices. 25. NERC should reevaluate its existing reliability standards development process and accelerate the adoption of enforceable standards. 26. Tighten communications protocols, especially for communications during alerts and emergen

cies. Upgrade communication system hardware where appropriate. 27. Develop enforceable standards for transmission line ratings. 28. Require use of time-synchronized data recorders. 29. Evaluate and disseminate lessons learned during system restoration. 30. Clarify criteria for identification of operationally critical facilities, and improve dissemination of updated information on unplanned outages. 31. Clarify that the transmission loading relief (TLR) process should not be used in situations involv

ing an actual violation of an Operating Security Limit. Streamline the TLR process. (continued on page 142)

### Endnotes

4  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  1 See “The Economic Impacts of the August 2003 Blackout,” Electric Consumer Research Council (ELCON), February 2, 2004. 2 Statistics Canada, Gross Domestic Product by Industry

, August 2003, Catalogue No. 15-001; September 2003 Labour Force Survey; Monthly Survey of Manufacturing, August 2003, Catalogue No. 31-001. Overview of Task Force Recommendations: Titles Only (Continued) Group III. Physical and Cyber Security of North American Bulk Power Systems 32. Implement NERC IT standards. 33. Develop and deploy IT management procedures. 34. Develop corporate-level IT security governance and strategies. 35. Implement controls to manage system health, network monitoring, and incident management. 36. Initiate U.S.-Canada risk management study. 37. Improve IT forensic and diagnostic capabilities. 38. Assess IT risk and vulnerability at scheduled intervals. 39. Develop capability to detect wireless and remote wireline intrusion and surveillance. 40. Control access to operationally sensitive equipment. 41. NERC should provide guidance on employee background checks. 42. Confirm NERC ES-ISAC as the central point for sharing security information and analysis. 43. Establish clear authority for physical and cyber security. 44. Develop procedures to prevent or mitigate inappropriate disclosure of information. Group IV. Canadian Nuclear Power Sector 45. The Task Force recommends that the Canadian Nuclear Safety Commission request Ontario Power Generation and Bruce Power to review operating procedures and operator training associ

ated with the use of adjuster rods. 46. The Task Force recommends that the Canadian Nuclear Safety Commission purchase and install backup generation equipment.

# 2. Overview of the North American Electric Power System and Its Reliability Organizations

The North American Power Grid Is One Large, Interconnected Machine

The North American electricity system is one of the great engineering achievements of the past 100 years. This electricity infrastructure represents more than $1 trillion (U.S.) in asset value, more than 200,000 miles—or 320,000 kilometers (km) of transmission lines operating at 230,000 volts and greater, 950,000 megawatts of generating capability, and nearly 3,500 utility organizations serving well over 100 million customers and 283 million people. Modern society has come to depend on reliable electricity as an essential resource for national security; health and welfare; communications; finance; transportation; food and water supply; heating, cooling, and lighting; computers and electronics; commercial enterprise; and even entertainment and leisure—in short, nearly all aspects of modern life. Customers have grown to expect that electricity will almost always be avail

able when needed at the flick of a switch. Most customers have also experienced local outages caused by a car hitting a power pole, a construc

tion crew accidentally damaging a cable, or a lightning storm. What is not expected is the occur

rence of a massive outage on a calm, warm day. Widespread electrical outages, such as the one that occurred on August 14, 2003, are rare, but they can happen if multiple reliability safeguards break down. Providing reliable electricity is an enormously complex technical challenge, even on the most routine of days. It involves real-time assessment, control and coordination of electricity production at thousands of generators, moving electricity across an interconnected network of transmission lines, and ultimately delivering the electricity to millions of customers by means of a distribution network. As shown in Figure 2.1, electricity is produced atlower voltages (10,000 to 25,000 volts) at generators from various fuel sources, such as nuclear, coal, oil, natural gas, hydro power, geothermal, photovoltaic, etc. Some generators are owned by the same electric utilities that serve the end-use customer; some are owned by independent power producers (IPPs); and others are owned by cus

tomers themselves—particularly large industrial customers. Electricity from generators is “stepped up” to higher voltages for transportation in bulk over

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  5 **Figure 2.1. Basic Structure of the Electric System**transmission lines. Operating the transmission lines at high voltage (i.e., 230,000 to 765,000 volts) reduces the losses of electricity from conductor heating and allows power to be shipped economi

cally over long distances. Transmission lines are interconnected at switching stations and substa

tions to form a network of lines and stations called a power “grid.” Electricity flows through the inter

connected network of transmission lines from the generators to the loads in accordance with the laws of physics—along “paths of least resistance,” in much the same way that water flows through a network of canals. When the power arrives near a load center, it is “stepped down” to lower voltages for distribution to customers. The bulk power sys

tem is predominantly an alternating current (AC) system, as opposed to a direct current (DC) sys

tem, because of the ease and low cost with which voltages in AC systems can be converted from one level to another. Some larger industrial and com

mercial customers take service at intermediate voltage levels (12,000 to 115,000 volts), but most residential customers take their electrical service at 120 and 240 volts.While the power system in North America is commonly referred to as “the grid,” there are actually three distinct power grids or “interconnections” (Figure 2.2). The Eastern Interconnection includes the eastern two-thirds of the continental United States and Canada from Saskatchewan east to the Maritime Provinces. The Western Interconnection includes the western third of the continental United States (excluding Alaska), the Canadian provinces of Alberta and British Columbia, and a portion of Baja California Norte, Mexico. The third interconnection comprises most of the state of Texas. The three interconnections are electrically independent from each other except for a few small direct current (DC) ties that link them. Within each interconnection, electricity is pro

duced the instant it is used, and flows over virtu

ally all transmission lines from generators to loads. The northeastern portion of the Eastern Intercon

nection (about 10 percent of the interconnection’s total load) was affected by the August 14 blackout. The other two interconnections were not affected. 1


## Planning and Reliable Operation of the Power Grid Are Technically Demanding

Reliable operation of the power grid is complex and demanding for two fundamental reasons:

 First, electricity flows at close to the speed of light (186,000 miles per second or 297,600 km/sec) and is not economically storable in large quantities. Therefore electricity must be produced the instant it is used.

 Second, without the use of control devices tooexpensive for general use, the flow of alternating current (AC) electricity cannot be controlled like a liquid or gas by opening or closing a valvein a pipe, or switched like calls over a longdistance telephone network. 2 Electricity flowsfreely along all available paths from the generators to the loads in accordance with the laws of physics—dividing among all connected flow paths in the network, in inverse proportion to the impedance (resistance plus reactance) on each path. Maintaining reliability is a complex enterprise that requires trained and skilled operators, sophis

ticated computers and communications, and care

ful planning and design. The North American Electric Reliability Council (NERC) and its ten Regional Reliability Councils have developed sys

tem operating and planning standards for ensur

ing the reliability of a transmission grid that are based on seven key concepts:

 Balance power generation and demand continuously.

 Balance reactive power supply and demand to maintain scheduled voltages.

 Monitor flows over transmission lines and other facilities to ensure that thermal (heating) limits are not exceeded.

6  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  **Figure 2.2. North American Interconnections** Keep the system in a stable condition.

 Operate the system so that it remains in a reli

able condition even if a contingency occurs, such as the loss of a key generator or transmis

sion facility (the “N-1 criterion”).

 Plan, design, and maintain the system to oper

ate reliably.

 Prepare for emergencies. These seven concepts are explained in more detail below. 1. Balance power generation and demand contin

uously. To enable customers to use as much electricity as they wish at any moment, produc

tion by the generators must be scheduled or “dispatched” to meet constantly changing demands, typically on an hourly basis, and then fine-tuned throughout the hour, sometimes through the use of automatic generation con

trols to continuously match generation to actual demand. Demand is somewhat predictable, appearing as a daily demand curve—in thesummer, highest during the afternoon and evening and lowest in the middle of the night, and higher on weekdays when most businesses are open (Figure 2.3). Failure to match generation to demand causesthe frequency of an AC power system (nominally 60 cycles per second or 60 Hertz) to increase (when generation exceeds demand) or decrease (when generation is less than demand) (Figure 2.4). Random, small variations in fre

quency are normal, as loads come on and off and generators modify their output to follow the demand changes. However, large deviations in frequency can cause the rotational speed of gen

erators to fluctuate, leading to vibrations that can damage generator turbine blades and other equipment. Extreme low frequencies can trigger automatic under-frequency “load shedding,” which takes blocks of customers off-line in order to prevent a total collapse of the electric system. As will be seen later in this report, such an imbalance of generation and demand can also occur when the system responds to major disturbances by breaking into separate “islands”; any such island may have an excess or a shortage of generation, compared to demand within the island. 2. Balance reactive power supply and demand to maintain scheduled voltages. Reactive power sources, such as capacitor banks and genera

tors, must be adjusted during the day to main

tain voltages within a secure range pertaining to all system electrical equipment (stations, trans

mission lines, and customer equipment). Most generators have automatic voltage regulators that cause the reactive power output of genera

tors to increase or decrease to control voltages to scheduled levels. Low voltage can cause electricsystem instability or collapse and, at distribution voltages, can cause damage to motors andthe failure of electronic equipment. High voltages can exceed the insulation capabilities of equipment and cause dangerous electric arcs (“flashovers”).
3. Monitor flows over transmission lines and other facilities to ensure that thermal (heating) limits are not exceeded. The dynamic interactions between generators and loads, combined with the fact that electricity flows freely across all interconnected circuits, mean that power flow is ever-changing on transmission and dis

tribution lines. All lines, transformers, and other equipment carrying electricity are heated by the flow of electricity through them. The

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  7 **Figure 2.3. PJM Load Curve, August 18-24, 2003 Figure 2.4. Normal and Abnormal Frequency**  **Ranges**flow must be limited to avoid overheating and damaging the equipment. In the case of over

head power lines, heating also causes the metal conductor to stretch or expand and sag closer to ground level. Conductor heating is also affectedby ambient temperature, wind, and other factors. Flow on overhead lines must be limited toensure that the line does not sag into obstructions below such as trees or telephone lines, or violate the minimum safety clearances between the energized lines and other objects. (A short circuit or “flashover”—which can start fires or damage equipment—can occur if an energizedline gets too close to another object). Most transmission lines, transformers and other currentcarrying devices are monitored continuously to ensure that they do not become overloaded or violate other operating constraints. Multiple ratings are typically used, one for normal condi

tions and a higher rating for emergencies. The primary means of limiting the flow of power on transmission lines is to adjust selectively the output of generators.
4. Keep the system in a stable condition. Because the electric system is interconnected and dynamic, electrical stability limits must be observed. Stability problems can develop very quickly—in just a few cycles (a cycle is 1/60th of a second)—or more slowly, over seconds or minutes. The main concern is to ensure that generation dispatch and the resulting power flows and voltages are such that the system is stable at all times. (As will be described later in this report, part of the Eastern Interconnection became unstable on August 14, resulting in a cascading outage over a wide area.) Stability limits, like thermal limits, are expressed as a maximum amount of electricity that can be safely transferred over transmission lines.There are two types of stability limits: (1) Voltage stability limits are set to ensure that the unplanned loss of a line or generator (whichmay have been providing locally critical reactive power support, as described previously) will not cause voltages to fall to dangerously low levels. If voltage falls too low, it begins tocollapse uncontrollably, at which point automatic relays either shed load or trip generatorsto avoid damage. (2) Power (angle) stability limits are set to ensure that a short circuit or an unplanned loss of a line, transformer, or genera

tor will not cause the remaining generators and loads being served to lose synchronism with one another. (Recall that all generators and loads within an interconnection must operate at or very near a common 60 Hz frequency.) Loss of synchronism with the common frequency means generators are operating out-of-step with one another. Even modest losses of synchro

nism can result in damage to generation equip

ment. Under extreme losses of synchronism, the grid may break apart into separate electrical islands; each island would begin to maintain its own frequency, determined by the load/genera

tion balance within the island. 5. Operate the system so that it remains in a reli

able condition even if a contingency occurs, such as the loss of a key generator or transmis

sion facility (the “N minus 1 criterion”). The central organizing principle of electricity reli

ability management is to plan for the unex

pected. The unique characteristics of electricity

8  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Local Supplies of Reactive Power Are Essential to Maintaining Voltage Stability A generator typically produces some mixture of “real” and “reactive” power, and the balance between them can be adjusted at short notice to meet changing conditions. Real power, measured in watts, is the form of electricity that powers equipment. Reactive power, a characteristic of AC systems, is measured in volt-amperes reac

tive (VAr), and is the energy supplied to create or be stored in electric or magnetic fields in and around electrical equipment. Reactive power is particularly important for equipment that relies on magnetic fields for the production of induced electric currents (e.g., motors, transformers, pumps, and air conditioning.) Transmission lines both consume and produce reactive power. At light loads they are net producers, and at heavy loads, they are heavy consumers. Reactive power consumption by these facilities or devices tends to depress transmission voltage, while its production (by generators) or injection (from storage devices such as capacitors) tends to sup

port voltage. Reactive power can be transmitted only over relatively short distances during heavy load conditions. If reactive power cannot be sup

plied promptly and in sufficient quantity, volt

ages decay, and in extreme cases a “voltage collapse” may result.mean that problems, when they arise, can spread and escalate very quickly if proper safe

guards are not in place. Accordingly, through years of experience, the industry has developed a network of defensive strategies for maintain

ing reliability based on the assumption that equipment can and will fail unexpectedly upon occasion. This principle is expressed by the requirement that the system must be operated at all times to ensure that it will remain in a secure condition (generally within emergency ratings for current and voltage and within established stability limits) following the loss of the most important generator or transmission facility (a “worst sin

gle contingency”). This is called the “N-1 crite

rion.” In other words, because a generator or line trip can occur at any time from random fail

ure, the power system must be operated in a preventive mode so that the loss of the most important generator or transmission facility does not jeopardize the remaining facilities in the system by causing them to exceed their emergency ratings or stability limits, which could lead to a cascading outage. Further, when a contingency does occur, the operators are required to identify and assess immediately the new worst contingencies, given the changed conditions, and promptly make any adjustments needed to ensure that if one of them were to occur, the system would still remain operational and safe. NERC operat

ing policy requires that the system be restored as soon as practical but within no more than 30 minutes to compliance with normal limits, and to a condition where it can once again with

stand the next-worst single contingency with

out violating thermal, voltage, or stability limits. A few areas of the grid are operated to withstand the concurrent loss of two or more facilities (i.e., “N-2”). This may be done, for example, as an added safety measure to protect

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  9 Why Don’t More Blackouts Happen? Given the complexity of the bulk power system and the day-to-day challenges of operating it, there are a lot of things that could go wrong— which makes it reasonable to wonder why so few large outages occur. Large outages or blackouts are infrequent because responsible system owners and opera

tors practice “defense in depth,” meaning that they protect the bulk power system through lay

ers of safety-related practices and equipment. These include: 1. A range of rigorous planning and operating studies, including long-term assessments,year-ahead, season-ahead, week-ahead, dayahead, hour-ahead, and real-time operational contingency analyses. Planners and operators use these to evaluate the condition of the sys

tem, anticipate problems ranging from likely to low probability but high consequence, and develop a good understanding of the limits and rules for safe, secure operation under such contingencies. If multiple contingencies occur in a single area, they are likely to be interde

pendent rather than random, and should have been anticipated in planning studies. 2. Preparation for the worst case. The operating rule is to always prepare the system to be safe in the face of the worst single contingency that could occur relative to current conditions, which means that the system is also prepared for less adverse contingencies. 3. Quick response capability. Most potentialproblems first emerge as a small, local situation. When a small, local problem is handled quickly and responsibly using NERC operating practices—particularly to return the system to N-1 readiness within 30 minutes or less— the problem can usually be resolved and contained before it grows beyond local proportions. 4. Maintain a surplus of generation and trans

mission. This provides a cushion in day-today operations, and helps ensure that small problems don’t become big problems. 5. Have backup capabilities for all critical func

tions. Most owners and operators maintain backup capabilities—such as redundant equipment already on-line (from generation in spinning reserve and transmission operating margin and limits to computers and other operational control systems)—and keep an inventory of spare parts to be able to handle an equipment failure.a densely populated metropolitan area or when lines share a common structure and could be affected by a common failure mode, e.g., a sin

gle lightning strike. 6. Plan, design, and maintain the system to oper

ate reliably. Reliable power system operation requires far more than monitoring and control

ling the system in real-time. Thorough plan

ning, design, maintenance, and analysis are required to ensure that the system can be oper

ated reliably and within safe limits. Short-term planning addresses day-ahead and week-ahead operations planning; long-term planning focuses on providing adequate generation resources and transmission capacity to ensure that in the future the system will be able to withstand severe contingencies without experi

encing widespread, uncontrolled cascading outages. A utility that serves retail customers must esti

mate future loads and, in some cases, arrangefor adequate sources of supplies and plan adequate transmission and distribution infrastructure. NERC planning standards identify a range of possible contingencies and set correspondingexpectations for system performance under several categories of possible events, ranging from everyday “probable” events to “extreme” events that may involve substantial loss of customer load and generation in a widespread area. NERC planning standards also address requirementsfor voltage support and reactive power, disturbance monitoring, facility ratings, system mod

eling and data requirements, system protection and control, and system restoration. 7. Prepare for emergencies. System operators are required to take the steps described above to plan and operate a reliable power system, but emergencies can still occur because of external factors such as severe weather, operator error, or equipment failures that exceed planning, design, or operating criteria. For these rare events, the operating entity is required to have emergency procedures covering a credible range of emergency scenarios. Operators must be trained to recognize and take effective action in response to these emergencies. To deal with a system emergency that results in a blackout, such as the one that occurred on August 14, 2003, there must be procedures and capabilities to use “black start” generators (capable of restarting with no external power source) and to coordinate operations in order to restore the system as quickly as possible to a normal and reliable condition. 

## Reliability Organizations Oversee Grid Reliability in North America

NERC is a non-governmental entity whose mis

sion is to ensure that the bulk electric system in North America is reliable, adequate and secure. The organization was established in 1968, as a result of the Northeast blackout in 1965. Since its inception, NERC has operated as a voluntary orga

nization, relying on reciprocity, peer pressure and the mutual self-interest of all those involved to ensure compliance with reliability requirements. An independent board governs NERC. To fulfill its mission, NERC:

 Sets standards for the reliable operation and planning of the bulk electric system.

 Monitors and assesses compliance with standards for bulk electric system reliability.

 Provides education and training resources to promote bulk electric system reliability.

 Assesses, analyzes and reports on bulk electric system adequacy and performance.

 Coordinates with regional reliability councils and other organizations.

 Coordinates the provision of applications (tools), data and services necessary to support the reliable operation and planning of the bulk electric system.

 Certifies reliability service organizations and personnel.

 Coordinates critical infrastructure protection of the bulk electric system.

 Enables the reliable operation of the intercon

nected bulk electric system by facilitating infor

mation exchange and coordination among reliability service organizations. Recent changes in the electricity industry have altered many of the traditional mechanisms, incentives and responsibilities of the entities involved in ensuring reliability, to the point that the voluntary system of compliance with reliabil

ity standards is generally recognized as not ade

quate to current needs. 3 NERC and many other electricity organizations support the development of a new mandatory system of reliability standards

10  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations and compliance, backstopped in the United States by the Federal Energy Regulatory Commission. This will require federal legislation in the United States to provide for the creation of a new electric reliability organization with the statutory author

ity to enforce compliance with reliability stan

dards among all market participants. Appropriate government entities in Canada and Mexico are prepared to take similar action, and some have already done so. In the meantime, NERC encour

ages compliance with its reliability standards through an agreement with its members. NERC’s members are ten regional reliability councils. (See Figure 2.5 for a map showing the locations and boundaries of the regional councils.) In turn, the regional councils have broadened their membership to include all segments of the electric industry: investor-owned utilities; federal power agencies; rural electric cooperatives; state, municipal and provincial utilities; independent power producers; power marketers; and end-use customers. Collectively, the members of the NERCregions account for virtually all the electricity supplied in the United States, Canada, and a portion of Baja California Norte, Mexico. The ten regional councils jointly fund NERC and adapt NERC standards to meet the needs of their regions. The August 14 blackout affected three NERC regional reliability councils—East Central Area Reliability Coordination Agreement (ECAR), Mid-AtlanticArea Council (MAAC), and Northeast Power Coordinating Council (NPCC). “Control areas” are the primary operational enti

ties that are subject to NERC and regional council standards for reliability. A control area is a geo

graphic area within which a single entity, Inde

pendent System Operator (ISO), or Regional Transmission Organization (RTO) balances gener

ation and loads in real time to maintain reliable operation. Control areas are linked with each other through transmission interconnection tie lines. Control area operators control generation directly to maintain their electricity interchange schedules with other control areas. They also operate collectively to support the reliability of their interconnection. As shown in Figure 2.6, there are approximately 140 control areas in North America. The control area dispatch centers have sophisticated monitoring and control systems and are staffed 24 hours per day, 365 days per year. Traditionally, control areas were defined by utility service area boundaries and operations were largely managed by vertically integrated utilities that owned and operated generation, transmis

sion, and distribution. While that is still true in some areas, there has been significant restructur

ing of operating functions and some consolidation of control areas into regional operating entities. Utility industry restructuring has led to an unbundling of generation, transmission and dis

tribution activities such that the ownership and operation of these assets have been separated either functionally or through the formation of independent entities called Independent System Operators (ISOs) and Regional Transmission Organizations (RTOs).

 ISOs and RTOs in the United States have been authorized by FERC to implement aspects of the Energy Policy Act of 1992 and subsequent FERC policy directives.

 The primary functions of ISOs and RTOs are to manage in real time and on a day-ahead basis the reliability of the bulk power system and the operation of wholesale electricity markets within their footprint.

 ISOs and RTOs do not own transmission assets; they operate or direct the operation of assets owned by their members.

 ISOs and RTOs may be control areas themselves, or they may encompass more than one control area.

 ISOs and RTOs may also be NERC Reliability Coordinators, as described below. Five RTOs/ISOs are within the area directly affected by the August 14 blackout. They are:

 Midwest Independent System Operator (MISO)

 PJM Interconnection (PJM)

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  11 **Figure 2.5. NERC Regions** New York Independent System Operator (NYISO)

 New England Independent System Operator (ISO-NE)

 Ontario Independent Market Operator (IMO) Reliability coordinators provide reliability over

sight over a wide region. They prepare reliability assessments, provide a wide-area view of reliabil

ity, and coordinate emergency operations in real time for one or more control areas. They may oper

ate, but do not participate in, wholesale or retail market functions. There are currently 18 reliabil

ity coordinators in North America. Figure 2.7 shows the locations and boundaries of their respective areas. 

## Key Parties in the Pre-Cascade Phase of the August 14 Blackout
The initiating events of the blackout involved two control areas—FirstEnergy (FE) and American Electric Power (AEP)—and their respective reli

ability coordinators, MISO and PJM (see Figures 2.7 and 2.8). These organizations and their reli

ability responsibilities are described briefly in this final subsection. 1. FirstEnergy operates a control area in north

ern Ohio. FirstEnergy (FE) consists of seven electric utility operating companies. Four of these companies, Ohio Edison, Toledo Edison, The Illuminating Company, and Penn Power, operate in the NERC ECAR region, with MISO serving as their reliability coordinator. These four companies now operate as one integrated control area managed by FE. 4 2. American Electric Power (AEP) operates a con

trol area in Ohio just south of FE. AEP is both a transmission operator and a control area operator. 3. Midwest Independent System Operator (MISO) is the reliability coordinator for FirstEnergy. The Midwest Independent System

12  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  **Figure 2.6. NERC Regions and Control Areas**Operator (MISO) is the reliability coordinator for a region of more than 1 million square miles (2.6 million square kilometers), stretching from Manitoba, Canada in the north to Kentucky in the south, from Montana in the west to western Pennsylvania in the east. Reliability coordina

tion is provided by two offices, one in Minne

sota, and the other at the MISO headquarters in Indiana. Overall, MISO provides reliability coordination for 37 control areas, most of which are members of MISO. 4. PJM is AEP’s reliability coordinator. PJM is one of the original ISOs formed after FERC orders 888 and 889, but was established as a regional power pool in 1935. PJM recently expanded its footprint to include control areas and transmis

sion operators within MAIN and ECAR (PJMWest). It performs its duties as a reliability coor

dinator in different ways, depending on the control areas involved. For PJM-East, it is both the control area and reliability coordinator for ten utilities, whose transmission systems span the Mid-Atlantic region of New Jersey, most of Pennsylvania, Delaware, Maryland, West Virginia, Ohio, Virginia, and the District ofColumbia. The PJM-West facility has the reliability coordinator desk for five control areas (AEP, Commonwealth Edison, Duquesne Light,Dayton Power and Light, and Ohio Valley Electric Cooperative) and three generation-only control areas (Duke Energy’s Washington County (Ohio) facility, Duke’s Lawrence County/Hanging Rock (Ohio) facility, and Alle

gheny Energy’s Buchanan (West Virginia) facility. 

### Reliability Responsibilities of Control Area Operators and Reliability Coordinators
1. Control area operators have primary responsi

bility for reliability. Their most important responsibilities, in the context of this report,

are:
N-1 criterion. NERC Operating Policy 2.A— Transmission Operations:
“All C ONTROL AREAS shall operate so that instability, uncontrolled separation, or cas

cading outages will not occur as a result of the most severe single contingency.” Emergency preparedness and emergency response. NERC Operating Policy 5—Emer

gency Operations, General Criteria: “Each system and C ONTROL AREA shall promptly take appropriate action to relieve any abnormal conditions, which jeopardize reliable Interconnection operation.” “Each system, C ONTROL AREA , and Regionshall establish a program of manual and automatic load shedding which is designed to arrest frequency or voltage decays that couldresult in an uncontrolled failure of components of the interconnection.” NERC Operating Policy 5.A—Coordination with Other Systems: “A system, C ONTROL AREA , or pool that is experiencing or anticipating an operating emergency shall communicate its current and future status to neighboring systems, C ONTROL AREAS , or pools and throughout the interconnectio n....A system shall inform

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  13 **Figure 2.7. NERC Reliability Coordinators Figure 2.8. Reliability Coordinators and Control**  **Areas in Ohio and Surrounding States**other system s... whenever . . . the system’s condition is burdening other systems or reducing the reliability of the Interconnec

tion . . . [or whenever] the system’s line load

ings and voltage/reactive levels are such that a single contingency could threaten the reli

ability of the Interconnection.” NERC Operating Policy 5.C—Transmission System Relief:

“Action to correct an O PERATING S ECURITY L IMIT violation shall not impose unaccept

able stress on internal generation or transmis

sion equipment, reduce system reliability beyond acceptable limits, or unduly impose voltage or reactive burdens on neighboring systems. If all other means fail, corrective action may require load reduction.” Operating personnel and training: NERC Oper

ating Policy 8.B—Training:
“Each O PERATING AUTHORITY should period

ically practice simulated emergencies. The scenarios included in practice situations should represent a variety of operating condi

tions and emergencies.” 2. Reliability Coordinators such as MISO and PJM are expected to comply with all aspects of NERC Operating Policies, especially Policy 9, Reliability Coordinator Procedures, and its appendices. Key requirements include: NERC Operating Policy 9, Criteria for Reliabil

ity Coordinators, 5.2: Have “detailed monitoring capability of the RELIABILITY AREA and sufficient monitoring

14  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations 

Institutional Complexities and Reliability in the Midwest The institutional arrangements for reliability in the Midwest are much more complex than they are in the Northeast—i.e., the areas covered by the Northeast Power Coordinating Council (NPCC) and the Mid-Atlantic Area Council (MAAC). There are two principal reasons for this complexity. One is that in NPCC and MAAC, the independent system operator (ISO) also serves as the single control area operator for the individual member systems. In comparison, MISO provides reliability coordination for 35 control areas in the ECAR, MAIN, and MAPP regions and 2 others in the SPP region, and PJM provides reliability coor

dination for 8 control areas in the ECAR and MAIN regions (plus one in MAAC). (See tablebelow.) This results in 18 control-area-tocontrol-area interfaces across the PJM/MISO reli

ability coordinator boundary. The other is that MISO has less reliability-related authority over its control area members than PJM has over its members. Arguably, this lack of authority makes day-to-day reliability operations more challenging. Note, however, that (1) FERC’s authority to require that MISO have greater authority over its members is limited; and (2) before approving MISO, FERC asked NERC for a formal assessment of whether reliability could be maintained under the arrangements proposed by MISO and PJM. After reviewing proposed plans for reliability coordination within and between PJM and MISO, NERC replied affirmatively butprovisionally. FERC approved the new MISOPJM configuration based on NERC’s assessment. NERC conducted audits in November and December 2002 of the MISO and PJM reliability plans, and some of the recommendations of the audit teams are still being addressed. The ade

quacy of the plans and whether the plans were being implemented as written are factors in NERC’s ongoing investigation.

**Reliability Coordinator (RC) Control**  **Areas in RC Area**  **Regional Reliability Councils Affected and**  **Number of Control Areas Control Areas of Interest in RC Area** MISO 37 ECAR (12), MAIN (9), MAPP (14), SPP (2) FE, Cinergy, Michigan Electric Coordinated System PJM 9 MAAC (1), ECAR (7), MAIN (1) PJM, AEP, Dayton Power & Light ISO New England 2 NPCC (2) ISONE, Maritime Provinces New York ISO 1 NPCC (1) NYISO Ontario Independent Market Operator 1 NPCC (1) IMO Trans-Energie 1 NPCC (1) Hydro Québeccapability of the surrounding RELIABILITY AREAS to ensure potential security violations are identified.” NERC Operating Policy 9, Functions of Reliabil

ity Coordinators, 1.7: “Monitor the parameters that may have sig

nificant impacts within the RELIABILITY AREA and with neighboring RELIABILITY AREAS with respect t o... sharing with other RELIABILITY C OORDINATORS any information regarding potential, expected, or actual criti

cal operating conditions that could nega

tively impact other RELIABILITY AREAS . The RELIABILITY C OORDINATOR will coordinate with other RELIABILITY C OORDINATORS and C ONTROL AREAS as needed to develop appro

priate plans to mitigate negative impacts of potential, expected, or actual critical operat

ing condition s . . . .” NERC Operating Policy 9, Functions of Reliabil

ity Coordinators, 6: “Conduct security assessment and monitor

ing programs to assess contingency situa

tions. Assessments shall be made in real time and for the operations planning horizon at the C ONTROL AREA level with any identified problems reported to the RELIABILITY C O

ORDINATOR . The RELIABILITY C OORDINATOR is to ensure that C ONTROL AREA , RELIABILITY AREA , and regional boundaries are suffi

ciently modeled to capture any problems crossing such boundaries.”

Endnotes

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  15 What Constitutes an Operating Emergency? An operating emergency is an unsustainable condition that cannot be resolved using theresources normally available. The NERC Operating Manual defines a “capacity emergency” as when a system’s or pool’s operating generationcapacity, plus firm purchases from other systems, to the extent available or limited by transfer capability, is inadequate to meet its demand plus its regulating requirements. It defines an “energy emergency” as when a load-serving entity has exhausted all other options and can no longer provide its customers’ expected energy requirements. A transmission emer

gency exists when “the system’s line loadings and voltage/ reactive levels are such that a single contingency could threaten the reliability of the Interconnection.” Control room operators and dispatchers are given substantial latitude to determine when to declare an emergency. (See pages 66-67 in Chapter 5 for more detail.) 1 The province of Québec, although considered a part of the Eastern Interconnection, is connected to the rest of the East

ern Interconnection only by DC ties. In this instance, the DC ties acted as buffers between portions of the Eastern Intercon

nection; transient disturbances propagate through them less readily. Therefore, the electricity system in Québec was not affected by the outage, except for a small portion of the prov

ince’s load that is directly connected to Ontario by AC trans

mission lines. (Although DC ties can act as a buffer between systems, the tradeoff is that they do not allow instantaneousgeneration support following the unanticipated loss of a generating unit.)
2 In some locations, bulk power flows are controlled throughspecialized devices or systems, such as phase angle regulators, “flexible AC transmission systems” (FACTS), and highvoltage DC converters (and reconverters) spliced into the AC system. These devices are still too expensive for general application.
3 See, for example, Maintaining Reliability in a Competitive Electric Industry (1998), a report to the U.S. Secretary of Energy by the Task Force on Electric Systems Reliability; National Energy Policy (2001), a report to the President of the United States by the National Energy Policy Development Group, p. 7-6; and National Transmission Grid Study (2002), U.S. Dept. of Energy, pp. 46-48.

4 The remaining three FE companies, Penelec, Met-Ed, and Jersey Central Power & Light, are in the NERC MAAC region and have PJM as their reliability coordinator. The focus of this report is on the portion of FE in the ECAR reliability region and within the MISO reliability coordinator footprint.

# 3. Causes of the Blackout and Violations of NERC Standards

Summary
This chapter explains in summary form the causes of the initiation of the blackout in Ohio, based on the analyses by the bi-national investigation team. It also lists NERC’s findings to date concerning seven specific violations of its reliability policies, guidelines, and standards. Last, it explains how some NERC standards and processes were inade

quate because they did not give sufficiently clear direction to industry members concerning somepreventive measures needed to maintain reliability, and that NERC does not have the authority to enforce compliance with the standards. Clearstandards with mandatory compliance, as contemplated under legislation pending in the U.S. Congress, might have averted the start of this blackout. Chapters 4 and 5 provide the details that support the conclusions summarized here, by describing conditions and events during the days before and the day of the blackout, and explain how those events and conditions did or did not cause or con

tribute to the initiation of the blackout. Chapter 6 addresses the cascade as the blackout spread beyond Ohio and reviews the causes and events of the cascade as distinct from the earlier events in Ohio.



## The Causes of the Blackout in Ohio
A dictionary definition of “cause” is “something that produces an effect, result, or consequence.” 1 In searching for the causes of the blackout, the investigation team looked back through the pro

gression of sequential events, actions and inac

tions to identify the cause(s) of each event. The idea of “cause” is here linked not just to what hap

pened or why it happened, but more specifically to the entities whose duties and responsibilities were to anticipate and prepare to deal with the things that could go wrong. Four major causes, or groups of causes, are identified (see box on page 18). Although the causes discussed below produced the failures and events of August 14, they did not leap into being that day. Instead, as the following chapters explain, they reflect long-standing insti

tutional failures and weaknesses that need to be understood and corrected in order to maintain reliability. 

## Linking Causes to Specific Weaknesses
Seven violations of NERC standards, as identified by NERC, 2 and other conclusions reached by NERC and the bi-national investigation team are aligned below with the specific causes of theblackout. There is an additional category of conclusions beyond the four principal causes—the failure to act, when it was the result of preceding conditions. For instance, FE did not respond to the loss of its transmission lines because it did not have sufficient information or insight to reveal the need for action. Note: NERC’s list of violations has been revised and extended since publication of the Interim Report. Two violations (numbers 4 and 6, as cited in the Interim Report) were dropped, and three new violations have been identified in this report (5, 6, and 7, as numbered here). NERC continues to study the record and may identify additional violations. 3 Group 1: FirstEnergy and ECAR failed to assess and understand the inadequacies of FE’s system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria and remedial measures.

 FE did not monitor and manage reactive reserves for various contingency conditions as required by NERC Policy 2, Section B, Require

ment 2.

 NERC Policy 2, Section A, requires a 30-minute period of time to re-adjust the system to prepare to withstand the next contingency.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  1718  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  

## Causes of the Blackout’s Initiation
The Ohio phase of the August 14, 2003, blackout was caused by deficiencies in specific practices, equipment, and human decisions by various organizations that affected conditions and out

comes that afternoon—for example, insufficient reactive power was an issue in the blackout, but it was not a cause in itself. Rather, deficiencies in corporate policies, lack of adherence to industry policies, and inadequate management of reactive power and voltage caused the blackout, rather than the lack of reactive power. There are four groups of causes for the blackout:

Group 1: FirstEnergy and ECAR failed to assess and understand the inadequacies of FE’s system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria. (Note: This cause was not identified in the Task Force’s Interim Report. It is based on analysis completed by the investigative team after the publication of the Interim Report.) As detailed in Chapter 4:A) FE failed to conduct rigorous long-term planning studies of its system, and neglected to conduct appropriate multiple contingency or extreme condition assessments. (See pages 37-39 and 41-43.) B) FE did not conduct sufficient voltage analyses for its Ohio control area and used operational voltage criteria that did not reflect actual volt

age stability conditions and needs. (See pages 31-37.) C) ECAR (FE’s reliability council) did not con

duct an independent review or analysis of FE’s voltage criteria and operating needs, thereby allowing FE to use inadequate prac

tices without correction. (See page 39.) D)Some of NERC’s planning and operational requirements and standards were sufficiently ambiguous that FE could interpret them to include practices that were inadequate for reli

able system operation. (See pages 31-33.) Group 2: Inadequate situational awareness at FirstEnergy. FE did not recognize or understand the deteriorating condition of its system. As discussed in Chapter 5: A) FE failed to ensure the security of its transmis

sion system after significant unforeseen con

tingencies because it did not use an effective contingency analysis capability on a routine basis. (See pages 49-50 and 64.) B) FE lacked procedures to ensure that its opera

tors were continually aware of the functional state of their critical monitoring tools. (See pages 51-53, 56.) C) FE control center computer support staff and operations staff did not have effective internal communications procedures. (See pages 54, 56, and 65-67.) D) FE lacked procedures to test effectively the functional state of its monitoring tools after repairs were made. (See page 54.)E) FE did not have additional or back-up monitoring tools to understand or visualize the status of their transmission system to facilitate its operators’ understanding of transmissionsystem conditions after the failure of their primary monitoring/alarming systems. (See pages 53, 56, and 65.)

Group 3: FE failed to manage adequately tree growth in its transmission rights-of-way. This failure was the common cause of the outage of three FE 345-kV transmission lines and one 138-kV line. (See pages 57-64.)

Group 4: Failure of the interconnected grid’s reliability organizations to provide effective real-time diagnostic support. As discussed in Chapter 5: A) MISO did not have real-time data from Dayton Power and Light’s Stuart-Atlanta 345-kV line incorporated into its state estima

tor (a system monitoring tool). This precluded (continued on page 19) NERC is lacking a well-defined control area(CA) audit process that addresses all CA responsibilities. Control area audits have generally not been conducted with sufficient regularity and have not included a comprehensive audit of the control area’s compliance with all NERC and Regional Council requirements. Compliance with audit results is not mandatory.

 ECAR did not conduct adequate review or analyses of FE’s voltage criteria, reactive power management practices, and operating needs.

 FE does not have an adequate automatic undervoltage load-shedding program in the Cleve

land-Akron area.
Group 2: Inadequate situational awareness at FirstEnergy. FE did not recognize or understand the deteriorating condition of its system. Violations (Identified by NERC):
 Violation 7: FE’s operational monitoring equip

ment was not adequate to alert FE’s operators regarding important deviations in operating conditions and the need for corrective action as required by NERC Policy 4, Section A, Require

ment 5.
 Violation 3: FE’s state estimation and contin

gency analysis tools were not used to assess system conditions, violating NERC Operating Policy 5, Section C, Requirement 3, and Policy 4, Section A, Requirement 5. Other Problems:

 FE personnel did not ensure that their Real-Time Contingency Analysis (RTCA) was a functional and effective EMS application asrequired by NERC Policy 2, Section A, Requirement 1.
 FE’s operational monitoring equipment was not adequate to provide a means for its operators to evaluate the effects of the loss of significant transmission or generation facilities as required by NERC Policy 4, Section A, Requirement 4.

 FE’s operations personnel were not provided sufficient operations information and analysis tools as required by NERC Policy 5, Section C, Requirement 3.
 FE’s operations personnel were not adequately trained to maintain reliable operation under emergency conditions as required by NERC Pol

icy 8, Section 1.
 NERC Policy 4 has no detailed requirements for: (a) monitoring and functional testing of critical EMS and supervisory control and data acquisi

tion (SCADA) systems, and (b) contingency analysis.

 NERC Policy 6 includes a requirement to plan for loss of the primary control center, but lacks specific provisions concerning what must be addressed in the plan.

 NERC system operator certification tests for basic operational and policy knowledge.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  19 

## Causes of the Blackout’s Initiation (Continued)
MISO from becoming aware of FE’s system problems earlier and providing diagnostic assistance or direction to FE. (See pages 49-50.) B) MISO’s reliability coordinators were using non-real-time data to support real-time “flowgate” monitoring. This prevented MISO from detecting an N-1 security violation in FE’s system and from assisting FE in neces

sary relief actions. (See pages 48 and 63.) C) MISO lacked an effective way to identify the location and significance of transmission line breaker operations reported by their Energy Management System (EMS). Such informa

tion would have enabled MISO operators to become aware earlier of important line out

ages. (See page 48.) D) PJM and MISO lacked joint procedures or guidelines on when and how to coordinate a security limit violation observed by one of them in the other’s area due to a contingency near their common boundary. (See pages 62-63 and 65-66.) In the chapters that follow, sections that relate to particular causes are denoted with the following symbols: **Cause 2** Inadequate Situational Awareness

**Cause 3** Inadequate Tree Trimming **Cause 4** Inadequate RC Diagnostic Support **Cause 1** Inadequate System UnderstandingSignificant additional training is needed to qualify an individual to perform system opera

tion and management functions.

Group 3: FE failed to manage adequately tree growth in its transmission rights-of-way. This failure was the common cause of the outage of three FE 345-kV transmission lines and affected several 138-kV lines.

 FE failed to maintain equipment ratings through a vegetation management program. A vegetation management program is necessary to fulfill NERC Policy 2, Section A, Requirement 1 (Control areas shall develop, maintain, and implement formal policies and procedures to provide for transmission security... including equipment ratings.)

 Vegetation management requirements are not defined in NERC Standards and Policies.

Group 4: Failure of the interconnected grid’s reliability organizations to provide effective diagnostic support. Violations (Identified by NERC):

 Violation 4 : MISO did not notify other reliability coordinators of potential system problems asrequired by NERC Policy 9, Section C, Requirement 2.

 Violation 5 : MISO was using non-real-time data to support real-time operations, in violation of NERC Policy 9, Appendix D, Section A, Criteria 5.2.

 Violation 6 : PJM and MISO as reliability coordi

nators lacked procedures or guidelines between their respective organizations regarding the coordination of actions to address an operating security limit violation observed by one of them in the other’s area due to a contingency near their common boundary, as required by Policy 9, Appendix C. Note : Policy 9 lacks specifics on what constitutes coordinated procedures and training.
Other Problems:
 MISO did not have adequate monitoring capa

bility to fulfill its reliability coordinator respon

sibilities as required by NERC Policy 9, Appendix D, Section A.

 Although MISO is the reliability coordinator for FE, on August 14 FE was not a signatory to the MISO Transmission Owners Agreement and was not under the MISO tariff, so MISO did not have the necessary authority as FE’s Reliability Coordinator as required by NERC Policy 9, Sec

tion B, Requirement 2.

 Although lacking authority under a signed agreement, MISO as reliability coordinator nev

ertheless should have issued directives to FE to return system operation to a safe and reliable level as required by NERC Policy 9, Section B, Requirement 2, before the cascading outages occurred.

 American Electric Power (AEP) and PJM attempted to use the transmission loading relief (TLR) process to address transmission power flows without recognizing that a TLR would not solve the problem.

 NERC Policy 9 does not contain a requirement for reliability coordinators equivalent to the NERC Policy 2 statement that monitoring equipment is to be used in a manner that would bring to the reliability coordinator’s attention any important deviations in operating conditions.

 NERC Policy 9 lacks criteria for determining thecritical facilities lists in each reliability coordinator area.

 NERC Policy 9 lacks specifics on coordinatedprocedures and training for reliability coordinators regarding “operating to the most conservative limit” in situations when operating conditions are not fully understood.

Failures to act by FirstEnergy or others to solve the growing problem, due to the other causes. Violations (Identified by NERC):

 Violation 1: Following the outage of the Cham

berlin-Harding 345-kV line, FE operating per

sonnel did not take the necessary action to return the system to a safe operating state as required by NERC Policy 2, Section A, Standard 1.

 Violation 2: FE operations personnel did not adequately communicate its emergency operat

ing conditions to neighboring systems as required by NERC Policy 5, Section A.
Other Problems:
 FE operations personnel did not promptly take action as required by NERC Policy 5, General

20  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations Criteria, to relieve the abnormal conditionsresulting from the outage of the HardingChamberlin 345-kV line.

 FE operations personnel did not implement measures to return system operation to within security limits in the prescribed time frame of NERC Policy 2, Section A, Standard 2, follow

ing the outage of the Harding-Chamberlin 345-kV line.

 FE operations personnel did not exercise the authority to alleviate the operating security limit violation as required by NERC Policy 5, Section C, Requirement 2.

 FE did not exercise a load reduction program to relieve the critical system operating conditions as required by NERC Policy 2, Section A, Requirement 1.2.

 FE did not demonstrate the application of effective emergency operating procedures as required by NERC Policy 6, Section B, Emer

gency Operations Criteria.

 FE operations personnel did not demonstrate that FE has an effective manual load shedding program designed to address voltage decaysthat result in uncontrolled failure of components of the interconnection as required by NERC Policy 5, General Criteria.

 NERC Policy 5 lacks specifics for Control Areason procedures for coordinating with other systems and training regarding “operating to the most conservative limit” in situations when operating conditions are not fully understood.



## Institutional Issues
As indicated above, the investigation team identi

fied a number of institutional issues with respect to NERC’s reliability standards. Many of the insti

tutional problems arise not because NERC is an inadequate or ineffective organization, but rather because it has no structural independence from the industry it represents and has no authority to develop strong reliability standards and to enforce compliance with those standards. While many in the industry and at NERC support such measures, legislative action by the U.S. Congress is needed to make this happen. These institutional issues can be summed up generally: 1. Although NERC’s provisions address many of the factors and practices which contributed to the blackout, some of the policies or guidelines are inexact, non-specific, or lacking in detail, allowing divergent interpretations among reli

ability councils, control areas, and reliability coordinators. NERC standards are minimum requirements that may be made more stringent if appropriate by regional or subregional bodies, but the regions have varied in their willingness to implement exacting reliability standards. 2. NERC and the industry’s reliability community were aware of the lack of specificity and detail in some standards, including definitions of Operating Security Limits, definition of planned outages, and delegation of Reliability Coordinator functions to control areas, but they moved slowly to address these problems effectively. 3. Some standards relating to the blackout’s causes lack specificity and measurable compli

ance criteria, including those pertaining to operator training, back-up control facilities, procedures to operate when part or all of the EMS fails, emergency procedure training, system restoration plans, reactive reserve requirements, line ratings, and vegetation management.4. The NERC compliance program and regionbased auditing process has not been comprehensive or aggressive enough to assess the capability of all control areas to direct the operation of their portions of the bulk power system. The effectiveness and thoroughness of regional councils’ efforts to audit for compliance with reliability requirements have varied signifi

cantly from region to region. Equally important, absent mandatory compliance and penalty authority, there is no requirement that an entity found to be deficient in an audit must remedy the deficiency. 5. NERC standards are frequently administrative and technical rather than results-oriented. 6. A recently-adopted NERC process for develop

ment of standards is lengthy and not yet fully understood or applied by many industry partic

ipants. Whether this process can be adapted to support an expedited development of clear and auditable standards for key topics remains to be

seen.

 U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  217. NERC has not had an effective process to ensure that recommendations made in various reports and disturbance analyses are tracked for accountability. On their own initiative, some regional councils have developed effective tracking procedures for their geographic areas. Control areas and reliability coordinators operate the grid every day under guidelines, policies, and requirements established by the industry’s reli

ability community under NERC’s coordination. If those policies are strong, clear, and unambiguous, then everyone will plan and operate the system at a high level of performance and reliability will be high. But if those policies are ambiguous and do not make entities’ roles and responsibilities clear and certain, they allow companies to perform at varying levels and system reliability is likely to be compromised. Given that NERC has been a voluntary organiza

tion that makes decisions based on member votes,if NERC’s standards have been unclear, nonspecific, lacking in scope, or insufficiently strict,that reflects at least as much on the industry community that drafts and votes on the standards as it does on NERC. Similarly, NERC’s ability to obtain compliance with its requirements through its audit process has been limited by the extent to which the industry has been willing to support the audit program.


### Endnotes

22  U.S.-Canada Power System Outage Task Force  August 14th Blackout: Causes and Recommendations  1 Webster’s II New Riverside University Dictionary, Riverside Publishing Co., 1984.
2 A NERC team looked at whether and how violations of NERC’s reliability requirements may have occurred in the events leading up to the blackout. They also looked at whether deficiencies in the requirements, practices and pro

cedures of NERC and the regional reliability organizations may have contributed to the blackout. They found seven spe

cific violations of NERC operating policies (although some are qualified by a lack of specificity in the NERC requirements).

The Standards, Procedures and Compliance Investigation Team reviewed the NERC Policies for violations, building on work and going beyond work done by the Root Cause Analy

sis Team. Based on that review the Standards team identified a number of violations related to policies 2, 4, 5, and 9.

Violation 1: Following the outage of the ChamberlinHarding 345-kV line, FE did not take the necessary actions to return the system to a safe operating state within 30 minutes. (While Policy 5 on Emergency Operations does not address the issue of “operating to the most conservative limit” when coordinating with other systems and operating conditions are not understood, other NERC policies do address this matter: Policy 2, Section A, Standard 1, on basic reliability for single contingencies; Policy 2, Section A, Standard 2, to return a sys

tem to within operating security limits within 30 minutes; Policy 2, Section A, Requirement 1, for formal policies and procedures to provide for transmission security; Policy 5, General Criteria, to relieve any abnormal conditions that jeop

ardize reliable operation; Policy 5, Section C, Requirement 1, to relieve security limit violations; and Policy 5, Section 2, Requirement 2, which gives system operators responsibility and authority to alleviate operating security limit violations using timely and appropriate actions.)

Violation 2: FE did not notify other systems of an impend

ing system emergency. (Policy 5, Section A, Requirement 1, directs a system to inform other systems if it is burdening oth

ers, reducing system reliability, or if its lack of single contin

gency coverage could threaten Interconnection reliability. Policy 5, Section A, Criteria, has similar provisions.)

Violation 3: FE’s state estimation/contingency analysis tools were not used to assess the system conditions. (This is addressed in Operating Policy 5, Section C, Requirement 3, concerning assessment of Operating Security Limit viola

tions, and Policy 4, Section A, Requirement 5, which addresses using monitoring equipment to inform the system operator of important conditions and the potential need for corrective action.)

Violation 4: MISO did not notify other reliability coordinators of potential problems. (Policy 9, Section C, Requirement 2, directing the reliability coordinator to alert all control areasand reliability coordinators of a potential transmission problem.)

Violation 5: MISO was using non-real-time data to supportreal-time operations. (Policy 9, Appendix D, Section A, Criteria For Reliability Coordinators 5.2, regarding adequate facilities to perform their responsibilities, including detailedmonitoring capability to identify potential security violations.)

Violation 6: PJM and MISO as Reliability Coordinators lacked procedures or guidelines between themselves on when and how to coordinate an operating security limit violation observed by one of them in the other’s area due to a contin

gency near their common boundary (Policy 9, Appendix 9C, Emergency Procedures). Note : Since Policy 9 lacks specifics on coordinated procedures and training, it was not possible for the bi-national team to identify the exact violation that occurred.

Violation 7: The monitoring equipment provided to FE operators was not sufficient to bring the operators’ attention to the deviation on the system. (Policy 4, Section A, System Monitoring Requirements regarding resource availability and the use of monitoring equipment to alert operators to the need for corrective action.)
3 NERC has not yet completed its review of planning stan

dards and violations.