ISO/RTO PERFORMANCE METRICS 

Commission Staff Report 

AD10-5-000 

Federal Energy Regulatory Commission 

October 21, 2010 
Docket No. AD10-5 

Table of Contents 

Section Page

I. Background ...................................................................................................... 5

II. Notice of Filing and Responsive Pleadings ..................................................... 7

III. Discussion .................................................................................................... 7

A. Procedural Issues ......................................................................................... 7

1. Comments ................................................................................................ 7

2. Response .................................................................................................. 9

B. Policy Issue ................................................................................................ 10

1. Comments .............................................................................................. 10

2. Response ................................................................................................ 12

C. Metrics Issues............................................................................................. 12

1. General Issues ........................................................................................ 12

a. Comments ........................................................................................ 12

2. Response ................................................................................................ 15

3. Discussion of Individual Metrics........................................................... 17

a. National and Regional Reliability Standards Compliance Metrics. 17

i. ISO/RTO Performance Metric......................................................... 17

ii. Comments .................................................................................... 17

iii. Response ...................................................................................... 18

b. Dispatch Reliability Metrics ............................................................ 19

i. ISO/RTO Performance Metric......................................................... 19

ii. Comments .................................................................................... 20

iii. Response ...................................................................................... 21

c. Load Forecast Accuracy Metric....................................................... 21

i. ISO/RTO Performance Metric......................................................... 21

ii. Comments .................................................................................... 22

iii. Response ...................................................................................... 22

d. Wind Forecasting Accuracy Metric................................................. 23

i. ISO/RTO Performance Metric......................................................... 23

ii. Comments .................................................................................... 23

iii. Response ...................................................................................... 23

e. Unscheduled Flows Metric .............................................................. 24

f. Transmission Outage Coordination Metric ......................................... 24

i. ISO/RTO Performance Metrics ....................................................... 24

ii. Response ...................................................................................... 25

g. Long-Term Reliability Planning – Transmission Metric ................ 26

i. ISO/RTO Performance Metrics ....................................................... 26

ii. Comments .................................................................................... 26

iii. Response ...................................................................................... 27

h. Long-Term Reliability Planning – Resources Metric...................... 27

2
Docket No. AD10-5 

i. ISO/RTO Performance Metrics ....................................................... 28

ii. Comments .................................................................................... 28

iii. Response ...................................................................................... 29

i. Infrastructure Investment – Interconnection and Transmission Process 

Metrics ....................................................................................................... 31

i. ISO/RTO Performance Metrics ....................................................... 31

j. Comments............................................................................................ 31

i. Response .......................................................................................... 31

k. Special Protection Systems.............................................................. 31

l. Backup Facility.................................................................................... 32

i. ISO/RTO Performance Metric......................................................... 32

ii. Comments .................................................................................... 32

iii. Response ...................................................................................... 32

m. Market Competitiveness .................................................................. 32

i. ISO/RTO Performance Metrics ....................................................... 32

ii. Comments .................................................................................... 33

iii. Response ...................................................................................... 34

n. Market Pricing Metrics .................................................................... 34

i. ISO/RTO Performance Metrics ....................................................... 34

ii. Comments .................................................................................... 35

iii. Response ...................................................................................... 35

iv. System Lambda............................................................................ 36

o. Energy Market Price Convergence.................................................. 36

i. ISO/RTO Performance Metrics ....................................................... 36

ii. Comment...................................................................................... 36

iii. Response ...................................................................................... 36

p. Congestion Management Metrics .................................................... 37

i. ISO/RTO Performance Metrics ....................................................... 37

ii. Comments .................................................................................... 37

iii. Response ...................................................................................... 38

q. Resource Availability Metric........................................................... 39

i. ISO/RTO Performance Metric......................................................... 39

ii. Comments .................................................................................... 39

iii. Response ...................................................................................... 40

iv. Fuel Diversity............................................................................... 41

r. Renewables Metrics............................................................................. 41

i. ISO/RTO Performance Metrics ....................................................... 41

ii. Comments .................................................................................... 42

iii. Response ...................................................................................... 42

s. Administrative Costs Metrics.............................................................. 43

i. ISO/RTO Performance Metrics ....................................................... 43

ii. Comments .................................................................................... 43

iii. Response ...................................................................................... 44

3
Docket No. AD10-5 

t. Customer Satisfaction Metric.............................................................. 45

i. ISO/RTO Performance Metrics ....................................................... 45

ii. Comments .................................................................................... 45

iii. Response ...................................................................................... 46

iv. Billing Controls Metric ................................................................ 46

IV. APPENDIX A............................................................................................ 47

V. APPENDIX B ................................................................................................ 50

4
Docket No. AD10-5 

 

Federal Energy Regulatory Commission Staff Report on ISO/RTO 

Performance Metrics 

Consistent with the recommendations of the report issued by the 

Government Accountability Office, FERC Could Take Additional Steps to Analyze 

Regional Transmission Organizations’ Benefits and Performance,1

 and the 

Federal Energy Regulatory Commission’s (Commission) FY2010-2014 Strategic 

Plan, Commission Staff has worked with Independent System Operators (ISO), 

Regional Transmission Organizations (RTO), stakeholders and other experts to 

develop standardized measures or metrics designed to track the performance of 

ISO/RTO operations and markets. The Chairman plans to use the metrics 

developed by Commission Staff to submit a report to Congress that will explain 

what the measures and reported performance metrics communicate about the 

benefits of ISOs/RTOs and, where appropriate, to identify changes that may need 

to be made to address any performance concerns. 

The purpose of this Commission Staff Paper is to describe the performance 

metrics that have been developed to track the performance of RTO/ISO operations 

and markets and to outline the process that will be taken to complete the report. 

Commission Staff appreciates the public comments that have been provided and is 

taking them into account in developing the final list of metrics to be used in 

tracking the performance of ISO/RTO operations and markets. 

I. Background

Responding to a request for an investigation into ISO/RTO costs, structure, 

processes and operations,2

 the Government Accountability Office, in a September 

 1

 U.S. Gov’t Accountability Office, FERC Could Take Additional Steps to 

Analyze Regional Transmission Organizations’ Benefits and Performance (2008) 

(Government Accountability Office Report). 

2

 This request was made on May 21, 2007, by Senator Joseph I. Lieberman, 

Chairman, and Senator Susan M. Collins, Ranking Minority Member, of the U.S. 

Senate Committee on Homeland Security and Governmental Affairs, in a letter to 

5
Docket No. AD10-5 

2008 Report to the U.S. Senate Committee on Homeland Security and 

Governmental Affairs, recommended that the Chairman take the following 

actions: (1) work with RTOs, stakeholders, and other experts to develop 

standardized measures that track the performance of RTO operations and markets; 

and (2) report the performance results to Congress and the public annually, while 

also providing an interpretation of: (a) what the measures and reported 

performance communicate about the benefits of RTOs and, where appropriate, (b) 

changes that need to be made to address any performance concerns.3

 The 

Government Accountability Office Report also suggested that performance 

metrics be explored for non-ISOs/RTOs. 

 As noted in the Government Accountability Office report, ISO/RTO 

performance has been an issue of interest to stakeholders and the public since the 

inception of ISOs and RTOs. The Government Accountability Office and 

stakeholders acknowledge that ISOs/RTOs provide a significant amount of 

information that measures the ISO or RTOs’ performance, and the ISOs/RTOs 

communicate their results regularly. But the information being provided is not 

supplied in a standardized format that allows for a comprehensive perspective of 

all ISOs/RTOs. The effort described in this paper is intended to address these 

gaps. 

 The Performance Metrics effort is also part of the Commission’s Strategic 

Plan, which includes a Metrics Initiative. The first step of that effort is to explore 

and develop appropriate operational and financial metrics for ISOs/RTOs in fiscal 

year 2010. The next steps in the Metrics Initiative are as follows: (1) explore and 

develop appropriate operational and financial metrics for non-ISOs/RTO regions 

in fiscal year 2011; (2) establish appropriate common metrics between ISOs/RTOs 

and non-ISO/RTO regions in fiscal year 2012; (3) monitor implementation and 

performance in fiscal year 2013; and (4) evaluate performance and seek changes 

as necessary in fiscal year 2014. 

 To begin the process of developing ISO/RTO performance metrics, 

Commission Staff initiated a process with the ISOs/RTOs, stakeholders and other 

experts to develop a set of consensus performance measures. As a first step, 

Commission Staff developed a broad range of metrics designed to track the 

 

the U.S. Government Accountability Office. The letter expressed the Senators’ 

concern that ISOs/RTOs may not be living up to their full potential with respect to 

improving efficiencies and reducing costs, and that they might not have adequate 

incentives to minimize costs. 

3

 Government Accountability Office Report at 59. 

6
Docket No. AD10-5 

performance of ISOs/RTOs in three specific areas, i.e., reliability, markets and 

organizational effectiveness. Next, Commission Staff met with the ISOs/RTOs to 

discuss Commission Staff’s proposed metrics and to determine the ability of each 

ISO/RTO to compute the proposed metrics. Commission Staff then held focused 

outreach meetings with a variety of industry, consumer, and state regulatory 

associations4

 to receive their input. As a follow-up to that outreach, Commission 

Staff’s proposed performance metrics were noticed for public comment and reply 

comment in Docket No. AD10-5-000 on February 3, 2010. 

II. Notice of Filing and Responsive Pleadings

Notice of the February 3, 2010 filing was published in the Federal Register, 

75 Fed. Reg. 7581 (2010) with comments due on or before March 5, 2010 and 

reply comments due on or before March 19, 2010. Fifty-nine parties filed 

comments and reply comments. The parties filing comments are listed in 

Appendix A. 

III. Discussion

A. Procedural Issues 

1. Comments 

Some commenters5

 argue that the process used by Commission Staff to 

develop the ISO/RTO performance metrics inappropriately allowed ISOs/RTOs to 

develop the proposed metrics. They fault Commission Staff for not starting the 

process with their recommendations, or basing the proposed metrics on their 

recommendations. For these reasons, these commenters urge the Commission to 

start with a clean slate6

 and to develop, in conjunction with stakeholders, the 

 4

 APPA, AWEA, EEI, ELCON, EPSA, NARUC, NASUCA, NECPUC and 

NRECA. Appendix A to this paper includes a list of intervenors and commenters. 

5

 APPA/ELCON, EMCOS, Industrial Customers, NRECA, ODEC and 

TAPS. 

6

 Comments of EMCOS at 12 (“For the foregoing reasons, EMCOS request 

that the Commission reject the proposed RTO Performance Metrics that 

accompanied its Notice in this proceeding. EMCOS further suggest that the 

Commission undertake a further proceeding[.]”); Comments of NRECA at 3 

(“NRECA concurs and believes that any metrics used to evaluate RTO/ISO 

benefits and performance should be developed, in the first instance, through the 

RTOs’/ISOs’ own stakeholder processes.”). 

7
Docket No. AD10-5 

metrics by which the ISOs/RTOs will be evaluated.7

 Public Systems endorses the 

issuance of a Notice of Proposed Rulemaking by the Commission and NARUC 

and DEMEC recommend that the Commission hold technical conferences. 

Commenters8

 also assert that the Commission must obligate ISOs/RTOs to issue 

reports and data, and that either ISOs/RTOs or the Commission must be obligated 

to take steps to improve performance. EEI expresses its understanding that the 

Commission has no plans to make the metrics a regulatory requirement; it asserts 

that the metrics should be for informational purposes only and not used to assess 

penalties.9

 

 With respect to the process going forward, some commenters10 recommend 

an ongoing process that allows stakeholders the opportunity for informed review 

and input before the metrics are finalized or revised from time to time. Others11

suggest that ISOs/RTOs be required to file plans for remediation of deficiencies in 

their metrics reports, and that these plans should be open for comment. Public 

Systems asserts that the Commission should be obligated to issue a report and then 

to issue orders based on comments stating whether the metrics will be retained or 

revised. Industrial Customers recommend that the Commission, on its own 

initiative, take all necessary steps to improve ISO/RTO performance. Ohio 

Consumer Counsel asserts that the Commission should provide a neutral 

interpretation of the data reported. 

 7

 Comments of APPA/ELCON at 8 (“To ensure due process, the 

Commission must allow all interested parties an equivalent opportunity to 

participate in the development of these metrics.”); Comments of Public Systems at 

2 (“Public Systems urge that the Commission view the current commenting 

opportunities as the commencement, rather than completion, of a process of 

performance metric identification, review, and refinement.”). 

8

 APPA/ELCON, Industrial Customers, NEPOOL, New Jersey Rate 

Counsel and Public Systems. 

9

 Duke, PHI Companies and National Grid also express their concern that 

the metrics may be used for reliability infractions, and request clarification that the 

metrics not be used for this purpose. 

10 NEPOOL, NESCOE, Public Systems, SoCal Edison and TAPS. 

11 APPA/ELCON, Industrial Customers, NECPUC, NESCOE and New 

Jersey Rate Counsel. 

8
Docket No. AD10-5 

COMPETE argues for establishing non-ISO/RTO metrics now, rather than 

next year, to more quickly enable better informed decisions. 

2. Response 

 As outlined above, Commission Staff initiated a process with ISOs/RTOs, 

stakeholders and other experts to develop standardized performance measures. As 

discussed below, Commission Staff is taking into account stakeholder 

recommendations in developing the list of metrics that will be submitted by each 

ISO/RTO to the Commission. Commission Staff considers this process to be an 

interactive and transparent development of performance metrics, and does not 

believe that any benefits from restarting the process would justify the attendant 

delay in using the draft metrics to gather performance data. Commission Staff’s 

process to develop metrics incorporated a wide range of ISO/RTO and stakeholder 

perspectives and input. 

Some commenters suggest that the Commission must obligate ISOs/RTOs 

to issue reports and data, and that either ISOs/RTOs or the Commission must be 

obligated to take steps to improve performance. These suggestions are more 

appropriate to a rulemaking proceeding than to this proceeding, in which the 

Chairman and Commission Staff are responding to Government Accountability 

Office recommendations. It is appropriate to (and sufficient for) this proceeding 

that the ISOs/RTOs are providing their performance data voluntarily. 

Commission Staff clarifies for commenters that the Chairman plans to send 

Congress a report in 2011 that responds to the Government Accountability Office 

Report and includes the following information: 

 three to five years of historical ISO/RTO performance data, provided by the 

ISOs and RTOs, on common and standardized metrics for the ISOs/RTOs 

under the jurisdiction of the Commission;12 

 an explanation of the metrics and an interpretation of what the measures 

and reported performance communicate about the benefits of ISOs/RTOs; 

 where appropriate, a discussion of possible changes that need to be made to 

address any performance concerns; and 

 a description of the metrics development process, including a description of 

how the Commission Staff worked with ISOs/RTOs, stakeholders, and 

 12 These ISOs/RTOs are ISO New England, Inc., New York Independent 

System Operator, Inc., PJM Interconnection, L.L.C., Midwest Independent System 

Operator, Inc., Southwest Power Pool, Inc., and the California Independent 

System Operator Corporation. 

9
Docket No. AD10-5 

other experts to develop the metrics. The report would commit to filing a 

follow-up report in the following year. 

Commission Staff also clarifies that in fiscal year 2011 it will initiate a 

voluntary and collaborative process – similar to the process used with ISOs/RTOs 

-- for developing performance metrics in non-ISO/RTO regions. Inasmuch as the 

primary focus of the Government Accountability Office recommendations was on 

ISO/RTO performance measures, Commission Staff allocated its resources to 

developing performance measures for these first. 

B. Policy Issue 

1. Comments 

Certain municipal power interests, industrial customers and state 

commissions13 question whether organized markets are a better market design than 

traditional cost-of-service regulation and request detailed cost and revenue data 

from each generator to test their hypothesis. For example, APPA/ELCON contend 

that “[a] fundamental argument underlying restructuring of wholesale electricity 

markets is that competition will be more effective than cost-based price regulation 

in achieving just and reasonable prices and benefits for consumers. . . . [A] 

primary measure of the efficacy of competition in a market is whether the price of 

a good, through competition, is driven down to the marginal cost of producing that 

good.”14

 APPA/ELCON, Consumer Coalition and DEMEC assert that it is necessary 

to obtain revenue, cost and profit information on individual generators and 

demand response resources, in recognition of the fact that the price-cost mark-up 

data does not reflect data in all ISO/RTO markets, such as those for ancillary 

services and capacity markets. 

Other commenters15 observe that the Government Accountability Office 

report raises the question of whether the ISO/RTO construct has provided the 

promised benefits to customers. They conclude that the only way to answer that 

question is to compare costs under organized regional markets to costs that would 

have occurred absent the organized markets. 

 13 APPA/ELCON, California PUC, EMCOS, Ohio PUC and Public 

Systems. 

14 Comments of APPA/ELCON at 13. 

15 Industrial Customers, Maryland PSC and TANC. 

10
Docket No. AD10-5 

APPA/ELCON, California PUC, TAPS and Commissioner Tyrone 

Christy16 assert that organized regional energy markets may be allowing 

participants to exercise potentially inappropriate market power, i.e., prices in 

organized regional markets may be exceeding marginal costs, and therefore the 

resulting rates are not just and reasonable. In supplemental comments, 

APPA/ELCON provided return on equity data for unregulated generation 

providers in the PJM Interconnection that shows a return on equity range of 12 to 

31 percent for these suppliers.17 APPA/ELCON considers these returns to be high 

and a strong indicator of the need for further investigation to ensure rates are just 

and reasonable. 

 Opposing these positions, the Federal Trade Commission considers an 

evaluation of the justness and reasonableness of an ISO/RTO market design using 

a single-clearing-price, market-based rate regime to be beyond the scope of the 

performance metric initiative. 

EPSA points out that it is a collateral attack on the legality of market-based 

pricing to assume that the just and reasonable standard requires marginal cost 

pricing. EPSA also cites to court precedent affirming that the Commission may 

rely upon market-based prices in lieu of cost-of-service regulation where the 

Commission has found that buyers and sellers lack market power. IPPNY and P3 

support these comments. EPSA, MidAmerican and IRC note that the Commission 

conducts triennial reviews to determine whether markets are sufficiently 

competitive to support market-based rates, and these markets are monitored on a 

daily basis by market monitors and the Commission’s Office of Enforcement. 

EPSA and MidAmerican consider measuring a variety of generator-specific 

cost and revenue information to be inappropriate since this information does not 

reflect market success or ISO/RTO benefits. The ISO New England Market 

Monitor opposes the gathering and publishing of data on the financial performance 

of individual market participants, since these data are not good measures of market 

performance. FirstEnergy considers the cost-of-service preference of 

APPA/ELCON and other parties to be short-sighted since it will result in low 

short-term prices to the detriment of future resource investment and sustainable 

electric markets. 

 16 Vice Chairman, Pennsylvania PUC. 

17 This data is provided in 2009 Financial Performance of Owners of 

Unregulated Generation, issued in May 2010. 

11
Docket No. AD10-5 

2. Response 

Commission Staff considers the APPA/ELCON position to be a criticism of 

the Commission’s ISO/RTO market design using a single-clearing-price marketbased rate policy, and not an ISO/RTO performance issue. ISOs/RTOs can only 

implement the Commission-approved market-based rate design for their organized 

markets. Therefore, it is appropriate to evaluate the performance of ISOs/RTOs 

based on the market design that the Commission has approved. Commission Staff 

agrees with the point made by the Federal Trade Commission in its reply 

comments: 

This proposal confuses policy performance with 

organizational performance metrics. FERC’s use of a 

single market-clearing price approach for 

compensating generators – rather than a pay-as-bid 

approach combined with plant-specific price controls – 

reflects an energy policy decision. The results of that 

policy decision are not under the control of RTOs. 

Thus, using generator costs compared to revenues as a 

performance metric for RTOs would hold those 

organizations accountable for results that reflect FERC 

policy choices. For this reason, we recommend that 

FERC not adopt generator costs compared to revenues 

as a performance metric for RTOs.”18 

At the same time, however, Commission Staff recognizes that generator 

profits may be reflective of competitive market conditions; therefore, metrics on 

the price-cost mark-up19 and generator net revenues are being added, as discussed 

in the Market Competitiveness Metrics discussion below. 

C. Metrics Issues 

1. General Issues 

a. Comments 

Most commenters consider Commission Staff’s proposed performance 

metrics to be a good start toward developing useful, meaningful metrics. 

 18 Federal Trade Commission Reply Comments at 8. 

19 California PUC, EMCOS, and NRECA endorse this metric. 

12
Docket No. AD10-5 

NEPOOL notes that, for New England, much of the specific data and information 

identified in the proposed metrics is already reported by ISO New England or 

otherwise publicly available, as required by the NEPOOL Participants Agreement. 

At the same time, a number of commenters20 urge the Commission to reevaluate the purpose of developing metrics, particularly in terms of their ability to 

provide meaningful indicators of performance. They assert that the only useful 

metrics are those that measure activities over which the ISOs/RTOs have control, 

or that represent key functions of ISOs/RTOs. These commenters consider a 

number of the draft performance metrics to be inappropriate, since they are 

influenced primarily by other market participants or are subject to a variety of 

market forces, such as transmission outage coordination and generator availability. 

In addition, Midwest ISO Transmission Owners and PHI Companies assert 

that the quantitative information provided in the ISO/RTO Performance Metrics 

will be meaningless without any context as to the underlying circumstances that 

affected the metric. For this reason, NECPUC recommends that ISOs/RTOs be 

required to submit a narrative on each metric that explains and interprets the data. 

Many of these commenters urge the Commission to recognize regional differences 

among ISOs/RTOs. 

Finally, commenters raise concerns that Commission Staff’s proposed 

performance metrics do not include important metrics or are not designed 

appropriately to provide meaningful measures, as follows. 

 The Federal Trade Commission recommends a grid efficiency performance 

metric to measure an RTO’s contribution to the effective functioning of the 

broader power system through economic dispatch efficiency, cooperation 

with interconnection-wide planning efforts and constructive work to resolve 

seams issues. Similarly, NY Transmission Owners endorse measuring 

ISO/RTO effectiveness in eliminating seams between markets and a 

measure of the inefficiencies caused by unauthorized loop flows. The 

Federal Trade Commission also asserts that there should be metrics on the 

minimum characteristics and functions specified in Order No. 200021 in 

 20 Midwest ISO Transmission Owners, National Grid, NECPUC, NEPOOL 

and United Illuminating. 

21 Regional Transmission Organizations, Order No. 2000, FERC Stats. & 

Regs. ¶ 31,089 (1999), order on reh’g, Order No. 2000-A, FERC Stats. & Regs. ¶ 

31,092 (2000), aff’s sub nom. Pub. Util. Dist. No. 1 of Snohomish County 

Washington v. FERC, 272 F.3d 607 (D.C. Cir. 2001). 

13
Docket No. AD10-5 

order to avoid the worst problems that can arise from using incomplete 

performance metrics. 

 Commenters22 recommend that ISOs/RTOs be required to undertake costbenefit studies on their major functions, to determine whether the functions 

can be justified and to ensure accountability for ISO/RTO projections. 

 NARUC endorses the development of a baseline by which to compare the 

ISOs/RTOs to regions outside these organized markets. 

 The ISO New England Market Monitor recommends that the metrics 

include the following information: capacity market prices, capacity market 

quantities offered and cleared, capacity required, the amount of existing 

capacity and the amount of capacity added and retired each year. The New 

England Market Monitor asserts that this information is required to indicate 

whether the market is supporting new investment and efficient turnover of 

assets. Dominion endorses a similar list of metrics. Ohio Consumer 

Counsel also supports capacity price metrics and OMS supports metrics on 

the cost of capacity and the method of compensating capacity in each 

ISO/RTO. 

 ITC asserts that ISO/RTO performance should be measured based on 

whether the regional entity has established clear cost allocation policies for 

new transmission. AWEA recommends a metric to measure the percent of 

transmission projects that qualify for cost sharing. 

 Certain commenters23 suggest metrics to measure a wide range of market 

structure data such as virtual bid and offer prices, credit defaults, liquidity 

of exchange-traded energy, dollars transacted in each market (energy, 

ancillary services, FTRs, virtual transactions) and lists of ISO/RTO 

products. 

In response to comments on the metrics, IRC asserts that the primary 

purpose of the metrics is to provide a common database for the collection of 

metrics in a uniform format and that metrics are not intended to substitute for 

 22 EMCOS, New Jersey Rate Counsel, Ohio Consumer Counsel and TAPS. 

23 Allegheny, APPA/ELCON, California DWR, DC Energy, NRECA, Ohio 

PUC and SoCal Edison. 

14
Docket No. AD10-5 

review and adjudication of tariffs under sections 205 and 206 of the Federal Power 

Act.24

2. Response 

Based on the comments received, Commission Staff has revised the metrics 

listed in the February 3, 2010 list of proposed performance metrics. The full list 

of revised metrics (ISO/RTO Performance Metrics) is provided in Appendix B. 

 

The ISO/RTO Performance Metrics discussed below measure ISOs/RTOs’ 

performance in terms of their ability to administer their markets efficiently and 

ensure reliability. ISO/RTO operations and administration are designed to provide 

benefits in an efficient manner that provides value to consumers. A number of 

metrics are influenced by factors beyond the control of ISOs/RTOs; nonetheless, 

they provide useful information on trends in the ISO/RTO markets that reflect on 

ISO/RTO operations, as discussed with regard to the individual metrics, below. 

Commission Staff is recommending that the ISO/RTO reports include 

narrative explanations to accompany the ISO/RTO data on performance metrics on 

a number of issues of concern to commenters. In these narratives Commission 

Staff encourages the ISOs/RTOs to quantify their information to the extent data is 

available and to provide information on metrics that the individual ISOs/RTOs are 

finding useful to track, even if they are not on the list of metrics in Appendix B. 

For example, PJM performs a perfect dispatch analysis to provide insights into 

how its organized markets are performing. This type of quantification and 

exploration of potential metrics will facilitate the evolution of the performance 

metrics. 

Commission Staff agrees with the Federal Trade Commission that 

measurement of ISO/RTO performance should consider the regional entities’ 

efforts to constructively resolve interconnection-wide issues and seams issues. 

Commission Staff expects that this information can be communicated most 

effectively in a narrative discussion since there is not a standardized product from 

these efforts that can be easily put into a metric. The metrics discussions that 

follow address the performance of ISOs/RTOs in carrying out the other minimum 

functions25 specified in Order No. 2000 (excluding the open-access same-time 

 24 16 U.S.C. §§ 824d, 824e (2006). 

25 The Order No. 2000 minimum functions are tariff administration and 

design, congestion management, parallel path flow, ancillary services, open access 

same-time information system, total transmission capability, available 

15
Docket No. AD10-5 

information system and available transmission capability calculation function, 

which do not bear directly on ISO/RTO performance).26 The ISO/RTO 

Performance Metrics do not address the minimum RTO characteristics that are 

prescribed in Order No. 2000 (independence, scope and configuration, operational 

authority and short-term reliability) since these characteristics explain what the 

RTOs are, and not what they do; as such, the minimum characteristics do not 

provide information on the performance of ISOs/RTOs. 

 

 Commission Staff considers a number of other commenters’ 

recommendations to be beyond the scope of this metrics effort. For example, 

recommendations for cost-benefit studies and baseline analyses would provide 

analyses of interest to stakeholders. However, these analyses are intended to serve 

other purposes, such as a benchmark for industry practices, that go beyond the 

purpose of this effort. Other recommendations are measures of Commission 

policies, rather than measures of ISO/RTO performance. These include ITC’s 

recommendation for a metric on whether an ISO/RTO has established a clear cost 

allocation policy for new transmission and AWEA’s proposed metric on cost 

sharing. 

 Certain commenters recommend metrics that would be redundant of the 

proposed metrics. For example, recommendations for capacity market metrics to 

measure investment in capacity and capacity offers, bids and prices are intended to 

measure resource adequacy performance, which is already being measured in the 

Long-Term Reliability Planning metrics in the reserve margin metrics. While 

Commission Staff will not include these additional metrics, Commission Staff 

expects that the narrative discussion accompanying the reserve margin metric will 

address investment and capacity market impacts on resource adequacy. 

 Finally, there are a number of recommendations for metrics that provide 

market information, but have no relation to ISO/RTO performance, such as 

recommendations for market structure data, virtual bid and offer prices, credit 

defaults, liquidity of exchange-traded energy, dollars transacted in each market 

(energy, ancillary services, FTRs, virtual transactions) and lists of ISO/RTO 

products. Commission Staff does not consider this data relevant to ISO/RTO 

performance metrics and therefore will not request the development of metrics 

based on this information. 

 

transmission capability, market monitoring, planning and expansion and 

interregional coordination. 

26 The parallel path flow function will be included in the narrative 

explanation of interconnection-wide issues and seams issues. 

16
Docket No. AD10-5 

3. Discussion of Individual Metrics 

 The three major categories of performance metrics are reliability, market 

benefits and organizational effectiveness. The reliability performance metrics, 

discussed first, were chosen to measure the reliability of day-to-day operations in 

metrics such as compliance with national and regional reliability standards, 

dispatch, forecasting and Special Protection Schemes as well as to measure longterm reliability in metrics such as long-term generation and transmission planning. 

The market benefits metrics were chosen to measure the performance of 

ISOs/RTOs based on market prices, congestion management costs and resource 

availability and to measure the efficiency of ISO/RTO markets in price 

convergence and competition metrics. Finally, the organization effectiveness 

metrics were chosen to measure ISO/RTO performance in accomplishing their 

objectives in a cost-effective manner that provides value to market participants. 

 The individual metrics are discussed in the order they are listed in 

Appendix B. 

 

a. National and Regional Reliability Standards 

Compliance Metrics 

i. ISO/RTO Performance Metric 

This metric measures the number of violations of national and regional 

reliability standards and provides additional information on how these violations 

were reported (self-reported or reported in audits) and the severity of the 

violations. It thereby indicates both the significance of the violation and the 

effectiveness of self-reporting procedures in identifying violations in a timely 

manner. This metric is intended to be a comprehensive measure of reliability 

performance based on functions undertaken by ISOs/RTOs.27 The metric also 

includes unserved energy (or load shedding) caused by violations. 

ii. Comments 

EEI contends that measuring reliability performance on the basis of 

aggregate violations will not provide an accurate representation of system 

performance and that it is not possible initially to determine whether an increase or 

 27 A full listing of the reliability standards is provided at 

http://www.nerc.com/page.php?cid=2|20. 

17
Docket No. AD10-5 

decrease in self-reported violations is good or bad. EEI also observes that a selfreported violation metric may discourage self-reporting. EPSA asserts that 

violation data should be supporting information only since increases/decreases in 

these measures are not indications of performance. 

The California PUC recommends that ISOs/RTOs report the number of 

violations regarding the NERC standard on operating reserve requirements. 

NRECA recommends that this performance measure include megawatt-hours of 

energy, and the number of customers, unserved due to violations. OMS states that 

making an explicit recognition of the difference in ISO/RTO size would be 

beneficial for the interpretation of relative ISO/RTO performance. 

iii. Response 

Responding to commenters that question whether these metrics are accurate 

representations of system performance, Commission Staff considers the metrics to 

be basic measures of reliability in ISOs/RTOs and therefore they must be included 

in a report on ISO/RTO operating performance.28 Commission Staff considers 

this information to be the basic building block from which a meaningful 

assessment of ISO/RTO reliability performance must start. Responding to EEI’s 

comments on self-reporting violations, Commission Staff notes that self-reported 

violations can indicate an entity is improving its own compliance and catching 

problems early. The Commission reaffirmed this in its Revised Policy Statem

on Penalty Guidelines.

ent 

 

29 Consistent with the Commission’s position on self28 Commission Staff expects, per the Commission’s regulations on the 

enforcement of reliability standards, that the report would not identify any 

nonpublic violations or any other non-public information related to cyber security 

incidents or violations that could similarly jeopardize the bulk power system until 

the information had been released by the Commission. See 18 C.F.R. § 39.7(b)(4) 

(the disposition of each violation or alleged violation that relates to a 

Cybersecurity Incident, or that would otherwise jeopardize the security of the Bulk 

Power System if publicly disclosed shall be nonpublic unless the Commission 

directs otherwise.) 

29 See Enforcement of Statutes, Orders, Rules and Regulations, 132 FERC ¶ 

61,216 (2010) at P 141 (Self-reports, for example, add significant value to overall 

industry compliance, and the Commission will continue to place great importance 

on self-reporting. As we stated in the 2005 Policy Statement, “[c]ompanies are in 

the best position to detect and correct violations of our orders, rules and 

regulations, both inadvertent and intentional, and should be proactive in doing so.” 

Providing credit for self-reporting gives organizations an incentive to detect and 

18
Docket No. AD10-5 

reporting, Staff does not expect that a self-reported violation metric will 

discourage self-reporting. In addition, NERC already makes public filings 

Commission of self-reported violations, so the total number of self-reported 

violations will not be new information. RTOs will have the opportunity to explain 

specific circumstances in the narrative ac

to the 

companying the metrics. 

 

 Commission Staff agrees that the number of violations, alone, does not 

provide meaningful information. Rather, the significance of the violations and the 

context of their occurrence will be communicated in the narrative summaries that 

will accompany the metrics. Based on comments received, metrics on compliance 

with operating reserves standards and unserved energy (or load shedding) caused 

by violations are included in the ISO/RTO Performance Metrics. Consistent with 

the Revised Policy Statement on Penalty Guidelines, Commission Staffs 

recommends that the unserved energy metric be calculated as the quantity of load 

lost, in MWh, as a measure of the seriousness of the violation, rather than a 

specific, individualized assessment of the value of losses of load that result from 

reliability violations.30 Responding to OMS, Commission Staff expects that the 

narrative explanation that accompanies the metrics will provide the necessary 

context as to the scale and configuration of each ISO/RTO to allow for a 

meaningful interpretation of the metrics. 

b. Dispatch Reliability Metrics 

i. ISO/RTO Performance Metric 

Dispatch reliability is measured by three metrics. The first metric measures 

the performance of ISO/RTO dispatch operations in maintaining interconnection 

steady-state frequency within defined limits by balancing power demand and 

supply in real time. This is evaluated using Control Performance Standards 1 and 

correct violations early. Self-reporting also assists the Commission’s review of 

violations and facilitates the process of providing remedies to affected parties.) 

30 Id. at P 4. 

19
Docket No. AD10-5 

2.31 An alternative method of measurement is using the Balancing Authority Area 

Control Error Limit.32 

The second metric measures the number of transmission loading relief or 

unscheduled flow events that required ISO/RTO actions to manage transmission 

flows in response to congestion for level three severity events and higher. This 

metric indicates how often ISOs/RTOs must resort to manual actions to redirect 

physical flows and therefore is an indication of the frequency of events that cannot 

be managed through ISO/RTO dispatching and unit commitment. 

The third metric measures the availability of energy management system 

hardware and software, such as state estimators, for performing real-time 

monitoring and security analysis functions. Availability is measured as the 

percentage of hours that the energy management system is operationally available. 

ii. Comments 

Duke considers the value of the transmission loading relief/unscheduled 

flow metric to be questionable since ISOs/RTOs have different load profiles, 

resources and other variables. For this reason, Duke contends that the 

transmission loading relief/unscheduled flow metric must factor in different 

operating practices and system configurations of ISOs/RTOs to be an effective 

metric. Midwest ISO TOs and Mid-American also consider the transmission 

loading relief/unscheduled flow metric to be of little value unless compared to the 

number of similar events in regions with no organized markets. NEPOOL states 

that the number of transmission loading relief events does not evaluate efficient 

outcomes and may not be wholly applicable to New England 

 Midwest ISO TOs recommend that the transmission loading 

relief/unscheduled flow metric also include information on the magnitude of 

transmission loading reliefs and the amount of energy curtailed via transmission 

 31 Control Performance Standard 1 is a statistical measure of Area Control 

Error variability. This standard measures Area Control Error in combination with 

the Interconnection’s frequency error. It is based on an equation derived from 

frequency-based statistical theory. Control Performance Standard 2 is a statistical 

measure of Area Control Error magnitude. The standard is designed to limit a 

control area’s unscheduled power flows. 

32 Procedures for the Balancing Authority Area Control Error Limit 

methodology can be found at 

http://www.nerc.com/docs/standards/sar/Supporting_Documents.pdf. 

20
Docket No. AD10-5 

loading reliefs relative to the total amount of megawatt-hours transferred across an 

ISO or RTO region. The Pennsylvania PUC also recommends that the metric 

report transmission loading relief events by severity and category and separately 

categorize transmission loading relief events called due to events within the 

ISO/RTO and those called by outside coordinators. EPSA supports a metric to 

measure energy management system availability. 

iii. Response 

Commission Staff agrees with commenters that events requiring 

transmission loading relief are a function of a wide range of factors outside 

ISO/RTO control, such as system configuration, weather, and load shifts. The 

purpose of ISO/RTO systems such as Security Constrained Economic Dispatch, 

however, is to resolve system constraint problems and thereby to avoid reliance on 

physical and manual procedures such as transmission loading relief that are more 

costly, less efficient and less reliable. For example, Security Constrained 

Economic Dispatch economically dispatches resources that have the greatest 

impact on a constrained facility first and thereby minimizes out-of-merit dispatch 

and the associated loss of system efficiency. Also, Security Constrained 

Economic Dispatch reduces flows on constrained transmission lines much more 

quickly and reliably than transmission load relief. For these reasons this metric, 

with appropriate narrative explanations and evaluated over a number of years, will 

provide an important indication of the trends of the effectiveness and efficiency of 

each ISO/RTO’s operating performance. Commission Staff agrees with 

commenters that the transmission loading relief/unscheduled flow metric can be 

improved with severity information and energy curtailment data and for this 

reason this information should be included in the metrics, to the extent the 

information is available. For ISO New England, which does not use transmission 

loading relief, a narrative explanation should be provided on its management of 

transmission flows. Responding to EPSA, the ISO/RTO Performance Metrics 

include an energy management system availability metric. 

c. Load Forecast Accuracy Metric 

i. ISO/RTO Performance Metric 

Actual load as a percentage variance from forecasted load, for both peak 

and off-peak periods, measures the effectiveness of the load forecasting function 

of ISOs/RTOs. Since load forecasting provides the basis for resource 

commitment, this metric impacts the incurrence of resource costs. The more 

accurate an ISO/RTO is in forecasting load, the greater the likelihood that it can 

commit sufficient resources in a cost effective manner that avoids overcommitment of resources, inefficient commitment of short lead-time resources or 

21
Docket No. AD10-5 

under-utilization of available resources. This metric measures the percentage 

difference between actual load and forecasted load and the mean absolute 

percentage error of day-ahead forecasts for all hours, the peak hour and the valley 

hour. The mean absolute percentage error will be measured as the yearly average 

error for all hours, the peak hour and the valley hour. 

ii. Comments 

Midwest ISO TOs and Duke question whether cross-ISO/RTO comparisons 

are meaningful because of regional differences. They also recommend: (1) that 

the metric be designed to account for the use of interruptible and other demand 

response resources during periods of high energy prices resulting in changes to 

actual load figures, and (2) that the metric be limited to day-ahead forecasting and 

thereby minimize the impact of weather variability. Ohio Consumer Counsel 

requests clarification on the time-frames to be measured and recommends a metric 

to measure the fees assessed for system imbalances caused by inaccurate load 

forecasting. NECPUC recommends additional specificity as to whether the 

forecasts are made on a weather-normalized basis. The New Jersey Rate Counsel 

endorses metrics to measure long-term peak load forecast error, defined to be the 

difference between the most recent forecast and the weather-normalized actual 

peak load. 

iii. Response 

Commission Staff clarifies for Midwest ISO TOs, Duke and Ohio 

Consumer Counsel that the metric is based on the day-ahead forecast and that the 

metric is calculated as the mean average of the absolute difference between the 

forecasted load and the actual load divided by the forecasted load for all relevant 

hours. Calculations will be made for the yearly average for all hours, the yearly 

average for the peak hour (the highest load hour) of each day, and the yearly 

average for the valley hour (the lowest load hour) of each day. The forecasts will 

account for the impact of interruptible load and demand response resources. 

Inasmuch as this metric is a day-ahead forecast measure, the measure minimizes 

the impact of weather variability that may result in regional differences in forecast 

accuracy.33 

 Commission Staff understands New Jersey Rate Counsel’s concern to be 

with the load forecasting process associated with resource adequacy planning. 

Commission Staff expects that these long-term forecasts will be constantly shifting 

 33 We discuss the issue raised by Ohio Consumer Counsel of costs 

associated with system imbalances in the Resource Availability section below. 

22
Docket No. AD10-5 

over time, to account for a number of factors such as the impact of economic 

trends, new technologies and population shifts. Therefore, Commission Staff does 

not consider comparisons of long-term forecasts or forecast errors to be 

appropriate as an ISO/RTO performance metric. At the same time Commission 

Staff recognizes that under-forecasting load can be a factor that leads to actual 

reserve margins being below the forecast reserve margin, to the detriment of 

reliability. To the extent under-forecasted load plays a role in achieving the target 

reserve margin, this information will be addressed in the narrative explanation 

accompanying the long-term generation reliability metric.34

d. Wind Forecasting Accuracy Metric 

i. ISO/RTO Performance Metric 

This metric measures the percentage accuracy of actual wind availability 

compared to forecasted wind availability as of the close of the prior day’s dayahead market. Improving the accuracy of the wind forecast will facilitate the 

timely commitment and dispatch of sufficient supplemental resources. 

ii. Comments 

California DWR asserts that this information should be a performance 

metric and should be expanded to include solar, tidal and other intermittent 

resources. California PUC recommends that the metric be standardized for 

comparative purposes. National Grid endorses making this information a future 

metric once wind monitoring and forecasting systems are in place. EEI considers 

proposals to make this information into a performance metric to be premature 

since the Commission is still investigating the benefits of gathering meteorological 

data in Docket No. RM10-11.35

iii. Response 

Responding to California DWR and California PUC, Commission Staff is 

including wind forecasting as a standardized ISO/RTO Performance Metric in 

recognition of the growing significance of this resource for nearly all of the 

ISOs/RTOs. In recognition of the fact that forecasting for other variable energy 

 34 Information in the narrative on over-forecasting of load would also have 

value. Over-forecasting can also be detrimental to ISO/RTO market efficiency 

since it leads to over-procuring of capacity and higher than necessary costs. 

35 Integration of Variable Energy Resources, 130 FERC ¶ 61,053 (2010). 

23
Docket No. AD10-5 

resources is either not performed by all ISOs/RTOs or is not performed according 

to a standardized process that allows for comparison, Commission Staff will not 

include the forecasts of other variable energy resources as performance metrics 

and instead Commission Staff considers it appropriate that the narrative 

discussions that accompany the metric include a discussion and quantification of 

forecast accuracy for those variable energy resources that ISOs/RTOs forecast. 

Responding to National Grid and EEI, Commission Staff considers it appropriate 

to include wind forecasting accuracy as a metric since most ISOs/RTOs have wind 

forecasting processes in place. Commission Staff does not find any basis to 

conclude that the data and analysis developed in this report will impact the 

Commission’s inquiry into eliminating barriers to the integration of variable 

energy resources in Docket No. RM10-11. Our report simply reports the accuracy 

of wind forecasts, whereas the inquiry has a different focus, namely to explore, 

among other issues, various techniques to improve data gathering and forecasting 

for variable energy resources. 

e. Unscheduled Flows Metric 

Unscheduled flows are defined as the difference between net actual 

interchange (actual power flow measured in real time) and the net scheduled 

interchange. The two components of unscheduled flows are inadvertent energy, 

defined to be the difference between actual and scheduled interchange for all 

interties, and parallel flow (or loop flow), defined to be the actual power flow on a 

contract path within an interconnection from one Balancing Authority Area to a 

second Balancing Authority Area via “parallel” transmission lines through a third 

Balancing Authority Area.36 Parallel flows are a function of the interconnection’s 

operating configuration, line resistance and physics. When unscheduled flows 

exceed system operating limits, curtailments may occur and efficient scheduling of 

the grid could be hindered. Accordingly, unscheduled flows provide information 

relevant to operational planning that is part of a comprehensive assessment of the 

reliability and efficiency of ISO/RTO markets. This metric is measured by the 

absolute value of megawatts of unscheduled flows over a year and as a percentage 

of total flows. No comments were submitted on unscheduled flows. 

f. Transmission Outage Coordination Metric 

i. ISO/RTO Performance Metrics 

 36 Parallel flows net out in an ISO/RTO region and therefore they do not 

contribute to inadvertent interchange. 

24
Docket No. AD10-5 

Effective transmission outage coordination is defined as early notification 

of planned outages of five days or longer – i.e., notification at least one month 

prior to the outage commencement date – and timely review of outage impacts, per 

ISO/RTO-established timeframes. Effective transmission outage coordination is 

also measured by the percentage of planned outages that are cancelled due to 

conflicting planned outages as well as forced (unscheduled) outages that could 

cause reliability issues and additional congestion costs.37 Effective transmission 

outage coordination by ISOs/RTOs ensures that outages do not threaten system 

reliability and that additional, and potentially more expensive, resources do not 

need to be committed. 

ii. Comments 

 Some commenters38 argue that this metric is not reasonably related to any 

function that an ISO/RTO can perform because timely reporting of an outage is 

under exclusive control of transmission owners. NEPOOL recommends that the 

metric be modified to differentiate between reasons for outages, and suggests that 

a better metric would be transmission outages that affect generation availability. 

Midwest ISO TOs state that measuring an ISO/RTO response to planned outages 

better demonstrates the effectiveness of ISO/RTO transmission outage 

coordination. They also recommend revising the percentage of 200 kV and above 

outages (both planned and unplanned) with less than two days’ notice by replacing 

“unplanned” with “emergency” since “unplanned” is too broad a term. DC 

Energy suggests measuring the one-month notification metric based on outages of 

one day, instead of the proposed five day outage metric. 

iii. Response 

Although transmission outage notices are the responsibility of transmission 

owners, and not ISOs/RTOs, Commission Staff expects that this measure will 

illuminate the effectiveness of coordination between transmission owners and 

ISOs/RTOs. Commission Staff expects that the narrative discussion 

accompanying the metric will identify the impact of transmission outages on 

generation availability and on declared emergencies, as requested by NEPOOL 

and Midwest ISO TOs. Responding to DC Energy, Commission Staff agrees that 

outages of less than five days can be significant events that require advance 

notification and therefore this information should be included. Accordingly, 

 37 The proposed metrics will measure outages for major transmission 

facilities, which are defined, for purposes of the metrics, as 200 kV and higher. 

38 CAISO, EEI, Mid-American, Midwest ISO and Midwest ISO TOs. 

25
Docket No. AD10-5 

Commission Staff would consider the addition of this information to be 

appropriate when the ISOs/RTOs have collected the necessary information, for 

possible inclusion in a follow-up report. 

g. Long-Term Reliability Planning – Transmission 

Metric 

i. ISO/RTO Performance Metrics 

This metric tracks the number of facilities approved to be constructed for 

reliability purposes, the percentage of approved construction projects on schedule 

and completed and the number of completed reliability and economic studies. 

This information measures the effectiveness of the ISO/RTO expansion planning 

process in identifying system reliability and economic needs in advance, thereby 

providing sufficient time for market participants to develop either a market-based 

solution (e.g., merchant transmission line, power plant or demand response) or a 

regulated solution (e.g., transmission line) to system reliability and economic 

requirements. The metric also measures the impact of demand response on the 

type and cost of infrastructure investment. 

ii. Comments 

 Certain commenters39 recommend additional metrics to measure project 

costs. ITC recommends that this metric be developed based on all transmission 

planning, not just reliability planning, and proposes a metric to measure the extent 

to which efficient and innovative technologies are deployed. National Grid 

recommends that the metric be based on transmission planning for both reliability 

and economics. 

 Other commenters40 assert that simply measuring the number of facilities 

approved does not provide an accurate assessment of ISO/RTO effectiveness to 

address reliability issues. TAPS recommends an additional metric for the status of 

approved transmission construction projects. Commenters41 also note that the 

percentage of approved construction on schedule and completed is often beyond 

the control of the ISO/RTO. NEPOOL states that the metric needs to distinguish 

ISO/RTO performance from transmission owner performance. NECPUC notes 

 39 California DWR, NECPUC, NEPOOL and OMS. 

40 Duke, EEI, EPSA and Midwest ISO TOs. 

41 Duke, EPSA, Mid-American, Midwest ISO TOs and NECPUC. 

26
Docket No. AD10-5 

that ISOs/RTOs do not approve project construction, and therefore the metric 

needs to be revised. 

 COMPETE recommends measuring the penetration of innovative resources 

such as flywheels and battery storage. Ohio Consumer Counsel endorses a metric 

on storage technologies. 

iii. Response 

The primary purpose of this metric is to assess the extent to which 

transmission solutions are deployed to meet reliability requirements, and therefore 

the cost, congestion and innovative technology measures that commenters suggest 

would go beyond the scope of the metric. With regard to OMS’s interest in a 

congestion relief metric, the narrative discussion that accompanies the Congestion 

Management metric, discussed below, will explain the impact of transmission 

planning on congestion trends. Responding to ITC and National Grid, the third 

metric listed in Appendix B measures both reliability and economic studies, and 

therefore addresses all project planning, including economic projects. Considering 

their growing importance in ISO/RTO operations, Commission Staff agrees with 

commenters that information should be provided on the integration of innovative 

technologies such as flywheels and battery storage into ISO/RTO markets in the 

narrative explanations. 

Commission Staff agrees with commenters that ISOs/RTOs should include 

a narrative discussion with this metric that explains the significance of the metrics 

data and provides a context to evaluate the benefit of ISO/RTO expansion 

planning. Commission Staff also expects that the supplemental information 

provided by ISOs/RTOs and the narrative discussion will address the status of 

approved construction projects. While Commission Staff agrees with commenters 

that the construction of planned projects is beyond the control of ISOs/RTOs, 

Commission Staff notes that most of the metrics measure project approvals in the 

expansion planning process42 and the completion of expansion studies, all of 

which are under the control of the ISOs/RTOs. Since project construction is the 

end result of these processes, Commission Staff includes this metric as one of the 

measures of the effectiveness of transmission planning. 

h. Long-Term Reliability Planning – Resources 

Metric 

 42 We clarify for NECPUC that the project approval metrics measure the 

number of reliability projects added to ISO/RTO expansion plans. 

27
Docket No. AD10-5 

i. ISO/RTO Performance Metrics 

Four metrics are employed to measure the effectiveness of long-term 

reliability planning by ISOs/RTOs. The first metric, processing time for 

generation interconnection requests, measures the effectiveness of ISO/RTO 

processes in achieving timely interconnection of new resources that are needed to 

ensure reliability. The second metric, actual reserve margins43 compared to 

planned reserve margins, measures the extent to which ISO/RTO generation 

resource planning processes are ensuring long-term resource adequacy and 

reliability. The third metric, the percentage of planned outages that are cancelled 

by ISOs/RTOs, provides an indication of the effectiveness of ISO/RTO in 

administering generation outage schedules. For example, a low cancellation 

percentage indicates that generation owners were allowed to complete nearly all 

the maintenance they had planned without incurring rescheduling costs or delays. 

And the fourth metric, the number of generating units and their capacity under 

reliability-must-run contracts, provides a measure of the degree to which an 

ISO/RTO must depend on critical facilities to maintain reliability and the 

flexibility of an ISO/RTO system to respond to emergencies and other 

contingencies. 

Two metrics measure the impact of demand response on long-term 

reliability.44 These are (1) the demand response share of total capacity, measured 

in megawatts, and (2) the demand response share of ancillary services, measured 

in megawatt-hours. 

ii. Comments 

Dominion supports a metric that reports planned generation outages that are 

cancelled or rescheduled after previous RTO approval, as a measure of the 

effectiveness of ISO/RTO handling of generation outage schedules. EPSA 

recommends a metric that measures the number of generating facilities and 

megawatt capability of units retained under reliability must run contracts. The 

Massachusetts AG endorses a measure of the megawatts procured in excess of the 

regional installed capacity requirement and capacity costs incurred in excess of the 

market clearing price. New Jersey Rate Counsel considers long-term peak load 

data and forecasts to be an important input to resource adequacy plans, as well as 

 43 The reserve margin is the number of megawatts of resources available as 

system reserves divided by the number of megawatts of peak load. 

44 These metrics were originally listed in a separate demand response metric 

in the proposed performance metrics. 

28
Docket No. AD10-5 

transmission and generation planning, and for this reason recommends including 

metrics to measure long-term load forecast error. New Jersey Rate Counsel also 

recommends metrics to measure deviations between actual reserve margins and 

projected reserve margins, with discussion of reasons for deviations. To recognize 

issues such as future carbon regulation and renewable portfolio standards, Ohio 

Consumer Counsel recommends that ISOs/RTOs be required to report additional 

information on the carbon contributions and generation fuels of ISO/RTO 

generation portfolios, the intended use of the generation additions and the target 

load or constraint the generation additions will serve. Ohio Consumer Counsel 

also endorses a metric to assess the actual reserve margin from an economic 

perspective using the projected value of lost load. 

With respect to the demand response metrics, EPSA supports a metric to 

measure the impact of demand response in peak conditions. Steel Producers 

support a metric to measure the impact of demand response in providing ancillary 

services. Steel Producers also argue for a more specific breakdown of the types of 

demand response used in each ISO/RTO to include categories such as regulation 

service, and for more refined metrics on the activity of these resources. 

Other commenters45 consider these metrics to be beyond ISO/RTO control 

and therefore not suitable as performance metrics. NESCOE notes that these 

metrics are developed within the states’ energy and environmental planning and 

policy frameworks. EEI and NEPOOL assert that the metrics are not meaningful 

since there are differences in resource mix, fuel prices, congestion and demand 

response efficiency capability in each ISO/RTO. 

iii. Response 

Commission Staff notes that the generation outage and reliability-must-run 

metrics supported by Dominion and EPSA are included in the metrics. Since the 

purpose of the metric is to measure reliability impacts, Commission Staff does not 

consider the capacity market efficiency and cost metrics proposed by the 

Massachusetts AG to be appropriate. At the same time, Commission Staff agrees 

with the Massachusetts AG that the efficiency of ISO/RTO capacity markets 

should be evaluated in a performance review. Since several ISOs/RTOs are not 

required to purchase capacity for their region and therefore these efficiency 

measures do not reflect their performance, a performance metric would not be 

appropriate. For those ISOs/RTOs that do perform this function, this capacity 

efficiency and cost information can be discussed in the narrative explanation that 

accompanies the total power cost metric, discussed below, for the capacity cost 

 45 Allegheny, Midwest ISO TOs, and NESCOE. 

29
Docket No. AD10-5 

component. Commission Staff agrees with New Jersey Rate Counsel that reserve 

margin estimates are important ISO/RTO functions that should be evaluated as 

part of an ISO/RTO performance review. The primary focus of this metric, 

however, is to measure the performance of ISOs/RTOs in achieving resource 

adequacy. Accordingly, Commission Staff agrees that it is appropriate that there 

be a discussion of a number of factors that impact the adequacy of the actual 

reserve margin vis-à-vis the projected reserve margin, including load forecasts and 

energy efficiency trends, in the narrative explanation that accompanies the metric, 

rather than creating an additional metric.46 Commission Staff also notes that 

reserve margin analysis varies from one ISO/RTO to another, as New Jersey Rate 

Counsel also recognizes, and therefore metric information would not be 

comparable across ISOs/RTOs. 

The additional information regarding fuel mix and carbon emissions that 

Ohio Consumer Counsel recommends collecting does not measure ISO/RTO 

performance with respect to long-term reliability planning. The metrics relating to 

fuel diversity, demand response, and renewables, however, will provide 

information relevant to the issues of concern to Ohio Consumer Counsel, as 

discussed below. Commission Staff does not agree with the recommendation to 

require information on the value of lost load since Commission Staff does not 

consider this information to be pertinent to measuring the long-term reliability 

planning performance of ISOs/RTOs. 

 Regarding the demand response metrics, Commission Staff recognizes that 

the amount of demand response in a market is not under the direct control of 

ISOs/RTOs, and that they are influenced by a number of factors, such as state 

demand response initiatives. Nonetheless, these metrics may be useful indicators 

of the impact on long-term reliability planning. Commission Staff recognizes that 

regional differences among the ISOs/RTOs will result in differing integration 

results for demand response resources. 

With respect to comments requesting more granularity in the data and more 

refined metrics, Commission Staff considers demand response data for separate 

ancillary services markets to be appropriate, as recommended by commenters, to 

the extent it is available. 

 46 We note that we also consider it appropriate to include information on the 

load forecasting process for resource adequacy in the narrative explanation, as 

discussed in the Load Forecasting Accuracy section above. 

30
Docket No. AD10-5 

i. Infrastructure Investment – Interconnection and 

Transmission Process Metrics 

i. ISO/RTO Performance Metrics 

These metrics track the progress made by ISOs/RTOs in completing their 

reliability reviews – namely, feasibility, system impact and facility studies – of 

interconnection and transmission service requests in a timely and efficient manner. 

The metrics track the number of requests, the time required to complete the 

reliability reviews and the costs of completing each of the three types of studies. 

ii. Comments 

ITC and EPSA support this performance metric. California PUC 

recommends that the information be clarified and normalized to facilitate 

benchmarking or comparative assessments. Ohio Consumer Counsel endorses 

reporting the information by utility and by independent power provider as well as 

providing information on formal complaints filed. Duke asserts that the 

interconnection and transmission service requests must be differentiated. 

iii. Response 

Responding to California PUC and Ohio Consumer Counsel, Commission 

Staff expects that the narrative explanation that accompanies this metric will 

address why there are differences between ISOs/RTOs in the completion and 

timing of transmission studies, as well as information on the types of entities being 

interconnected and the status of formal complaints. While Commission Staff 

agrees with Duke that the information would be more meaningful if it had separate 

information for interconnections and transmission service requests, this 

information is not available from the ISOs/RTOs. To address Duke’s concern, 

Commission Staff would consider it appropriate that the report provide narrative 

explanations on trends in interconnections and transmission service requests. 

j. Special Protection Systems 

Special Protection Systems47 are automatic protection systems designed to 

detect abnormal or predetermined system conditions and take corrective actions 

such as changing demand, generation or system configurations in order to 

 47 Special Protection Systems are also referred to as Special Protection 

Schemes, Remedial Action Schemes (RAS) or System Integrity Protection 

Schemes (SIPS). 

31
Docket No. AD10-5 

maintain system stability, acceptable voltage levels or power flows. The metrics 

measure both the frequency with which the ISO/RTO relies on these systems and 

their effectiveness, as measured by successful activations and the number of 

unintended activations. Duke considers this information to be more reflective of 

operations of transmission owners and not ISOs/RTOs. While Commission Staff 

agrees that Special Protection Systems are actions undertaken by transmission 

owners, these actions must be coordinated with ISO/RTO operators. Accordingly, 

this metric is a measure of ISO/RTO-wide performance in managing system 

stability, voltage levels and power flows and therefore it is appropriate that this 

information is included in an assessment of ISO/RTO reliability performance. 

k. Backup Facility 

i. ISO/RTO Performance Metric 

This metric measures the capability of the ISO/RTO to continue reliability 

operations in the event the primary control center is not functioning. The metric 

includes information on readiness training, communications capabilities and quick 

response capabilities. 

ii. Comments 

National Grid recommends that this information be deleted since similar 

NERC requirements already exist. EEI notes that this information is the same as 

the requirements of NERC EOP-008, and therefore its purpose is questionable. 

Duke asserts that NERC standards do not require that backup facilities be staffed 

at all times, and therefore metrics should be revised to report on whether the 

facility can be staffed in a short amount of time. 

iii. Response 

Upon further consideration, Commission Staff concludes that this 

information is sensitive in nature. It includes procedures and information that are 

critical to maintaining the integrity of the electric gird and for this reason 

recommends not requiring that ISOs/RTOs release this information to the public. 

Therefore, Commission Staff deletes this metric. 

l. Market Competitiveness 

i. ISO/RTO Performance Metrics 

By design, the Security Constrained Economic Dispatch and Security 

Constrained Unit Commitment processes in ISO/RTO markets are intended to 

32
Docket No. AD10-5 

foster competition among suppliers and to maximize production for the least-cost 

suppliers per least-cost unit commitment, thereby benefiting consumers. The 

following metrics are intended to evaluate how well ISO/RTO markets are 

performing in achieving competitive results: 

 The first metric, the price-cost mark-up, compares the system 

marginal price to the system marginal cost,48 assuming no system 

constraints. The difference between the marginal price and marginal 

cost indicates the degree of competition in ISO/RTO markets. 

ISO/RTO markets are more competitive the closer prices are to 

marginal costs. This metric is measured as the percentage mark-up 

for each year. 

 The second metric, generator net revenue, measures the revenue that 

a new generator would earn above its variable production costs if it 

were to operate only when its variable production costs were less 

than the energy price. This metric can be an indicator of whether 

generator net revenues are sufficient to ensure new investment, if 

needed, and are consistent with competitive markets. This metric is 

measured on an annual basis. 

 The third metric provides an indication of the magnitude of 

mitigation occurring in ISO/RTO markets, as measured by the 

dollars and megawatt hours mitigated, the percentage of dollars and 

megawatt hours mitigated and the percentage of unit hours prices are 

capped on an annual basis. This metric will be supplemented by a 

discussion of the significance of the actions for market power and 

competition. 

ii. Comments 

These metrics are endorsed by a number of commenters.49 Public Systems 

and Ohio Consumer Counsel recommend a break-down of price-cost mark-up data 

 48 To clarify the difference between this metric and the System Lambda 

metric discussed below, the price-cost mark-up compares the difference between 

marginal prices and marginal costs, whereas system lambda only measures 

marginal cost. 

49 California DWR, California PUC, DEMEC, EMCOS, EPSA, 

Massachusetts AG, NECPUC, NEPOOL, NRECA, New York PSC and NY 

Transmission Owners 

33
Docket No. AD10-5 

by fuel type, asset class and sub-market. Ohio Consumer Counsel and OMS also 

recommend metrics on market concentration and market power. California PUC, 

California DWR and the Missouri Commission recommend metrics to measure 

price volatility. 

iii. Response 

 Commission Staff will not request the break-down of price-cost mark-up 

data by fuel type, asset class or sub-market, as Public Systems and Ohio Consumer 

Counsel recommend. In ISO/RTO markets, the price that clears the market 

applies to all types of resources offering into the market at the price node, so there 

is no price applicable only to a fuel type or asset class. With respect to sub-market 

competition, the report should include a narrative discussion accompanying the 

metric that will evaluate competitive issues in sub-markets. The narrative 

discussion should also address market concentration and market power to the 

extent they have a bearing on market competitiveness, and therefore additional 

metrics recommended by Ohio Consumer Counsel and OMS are not needed. With 

respect to the price volatility measures recommended by California PUC, 

California DWR and the Missouri Commission, the narrative discussion should 

address those price movements that reflect anti-competitive behavior. 

m. Market Pricing Metrics 

i. ISO/RTO Performance Metrics 

The four market pricing metrics measure the customer cost impact of 

ISO/RTO markets. The first measure, the load-weighted locational marginal price 

metric, measures the cost to load of energy purchased in ISO/RTO markets. The 

second metric, components of total power costs, breaks out each element of all 

costs paid by load, thereby providing a comprehensive assessment of all ISO/RTO 

market costs.50 The third measure, the load-weighted, fuel-adjusted locational 

marginal price, is derived by holding fuel costs constant over a defined time period 

to show the trend of non-fuel customer costs over this period. This metric isolates 

the customer cost impact of cost elements such as transmission costs, congestion 

and losses, thereby providing a measure of the effectiveness of ISO/RTO market 

management. This metric also reflects the impact of load growth, investments in 

resources and the retirement of uneconomic facilities, and therefore it measures 

factors that are not entirely within the control of ISOs/RTOs. The fourth metric 

 50 The cost break-down includes the following cost categories: ISO/RTO 

costs and regulatory fees, operating reserve costs, ancillary services costs, 

transmission costs, capacity costs and energy costs. 

34
Docket No. AD10-5 

measures the impact of demand response on market prices, including impacts 

associated with voluntary curtailments by demand response during heat waves and 

other emergency conditions. All four metrics are measured on an annual basis. 

ii. Comments 

 Certain commenters51 recommend measuring additional costs, such as the 

cost of reliability-must-run units, reliability unit commitment charges and 

payments, high voltage access charges, excess cost payments, default interest 

charges, lost opportunity costs and FERC annual charges. Commenters52 also 

argue for greater granularity in the reported costs and the time periods analyzed, as 

well as an analysis of factors driving costs. The ISO New England Market 

Monitor considers the fuel-adjusted metric to be of limited usefulness since there 

is no standard methodology to calculate the fuel-adjusted price and it is not 

possible to capture shifts in relative costs of fuels. 

iii. Response 

The cost categories covered by the total power cost metric include the 

major costs of ISO/RTO markets to consumers, and therefore Commission Staff 

considers this metric to be a comprehensive measure of the costs of ISO/RTO 

markets. Commission Staff believes that the narrative explanation of the causes of 

cost trends that accompany each ISO/RTO metrics submission will be responsive 

to the concerns of commenters that recommend more granularity in the cost 

categories, down to the level of costs for individual ISO/RTO services so that the 

root causes of cost trends can be better identified. The narrative discussion should 

also provide information on peak price trends, as requested by OMS. 

Responding to commenters’ interest in tracking Commission annual 

charges, Commission Staff considers these costs to be beyond the control of 

ISOs/RTOs and therefore do not reflect ISO/RTO performance. With respect to 

the fuel-adjusted cost metric concern raised by the ISO New England Market 

Monitor, Commission Staff recognizes that each ISO/RTO uses a different year as 

its starting point for holding fuel prices constant and the ISOs/RTOs have differing 

fuel mixes, making comparisons across ISOs/RTOs difficult. Nonetheless, 

analysis of long-term trends is a useful measure of the effectiveness of ISOs/RTOs 

 51 California PUC, DC Energy, Dominion, NECPUC, NEPOOL, SoCal 

Edison and Westar. 

52 California PUC, ISO New England Market Monitor, New Jersey Rate 

Counsel and OMS. 

35
Docket No. AD10-5 

in managing non-fuel costs, and therefore the metric will provide useful 

performance information. 

Commission Staff notes that a fourth metric has been added to the proposed 

list of metrics provided in the February 3, 2010 notice to measure the impact of 

demand response on market prices. The purpose of this metric is to ensure the 

Market Pricing analysis addresses the impact of integrating demand response in 

ISO/RTO markets on customer costs. 

n. System Lambda 

System lambda is the incremental cost of energy of the marginal unit 

assuming no system constraints. This metric tracks the trend in marginal fuel 

costs and is an important metric since fuel costs represent the largest component of 

wholesale energy costs. APPA/ELCON recommends that this information be 

based on the single incremental cost of energy per FERC Form 714, rather than as 

a component of the locational marginal price as proposed. APPA/ELCON 

believes that with a correct definition, system lambda could provide useful 

information. Commission Staff agrees with APPA/ELCON that FERC Form 714 

provides an appropriate basis for measuring system lambda, and therefore this 

information should be included in the report. 

o. Energy Market Price Convergence 

i. ISO/RTO Performance Metrics 

Convergence of day-ahead and real-time energy prices provides an 

indication of the efficiency of ISO/RTO markets. Since the large majority of 

energy settlements and generator commitments occur in the day-ahead market, 

day-ahead price convergence with the real-time market ensures efficient dayahead commitments that reflect real-time operating needs. Energy market price 

convergence is measured by the absolute value and percentage of the annual 

difference between real-time energy market prices and day-ahead market prices. 

ii. Comment 

California PUC recommends adding information on virtual bidding, market 

liquidity (total participants in physical and virtual trading markets) and quarterly 

information on energy traded and ISO/RTO costs allocated to physical trading and 

virtual trading in total dollars and per megawatt-hour costs. 

iii. Response 

36
Docket No. AD10-5 

Commission Staff does not consider trader capabilities and market trends 

that support profitable participation to be reflective of ISO/RTO market efficiency. 

Likewise, Commission Staff considers the bidding and trading activity of virtual 

traders to be based primarily on assessments by traders as to the profitability of 

participation as well as their financial capabilities to take advantage of market 

shifts. Since the ability of ISOs/RTOs to influence these factors is either nonexistent or extremely limited, Commission Staff does not think it necessary to 

gather this information as part of an ISO/RTO performance assessment. 

Responding to California PUC’s recommendation for quarterly data, Commission 

Staff considers annual information and long-term trends to be the appropriate basis 

to evaluate this information. Information for shorter time periods are heavily 

influenced by seasonal shifts and operational factors that make an evaluation of 

the efficiency of ISO/RTO day-ahead and real-time markets difficult. 

Commission Staff does not consider information on ISO costs allocated to 

physical and virtual trading to be appropriate, as recommended by California 

PUC, since it is not relevant to an evaluation of the efficiency of prices in 

ISO/RTO markets. 

p. Congestion Management Metrics 

i. ISO/RTO Performance Metrics 

Congestion represents the cost to customers of paying for more expensive 

energy because physical transmission line limits do not allow full delivery of leastcost energy. The first congestion management metric, annual congestion costs 

divided by the megawatt hours of load served, tracks congestion cost trends 

relative to load growth thereby providing an indication of the efficiency of the 

overall ISO/RTO system as well as the effectiveness of ISO/RTO efforts to 

manage congestion costs through transmission expansion planning and other 

efficiency measures. This metric is also influenced by other factors, such as load 

trends, and as such its inputs are not entirely within the control of the ISO/RTO. 

The second metric, congestion revenues paid divided by congestion charges 

expressed as a percentage, tracks the ability of market participants to hedge these 

congestion costs, and thereby manage their costs. 

ii. Comments 

Duke considers the value of the metric to be limited because of differences 

in size/configuration of ISOs/RTOs. NRECA and APPA/ELCON recommend 

adding a metric that analyzes the costs of transmission investment and the 

associated value of congestion that would be relieved by such investments. The 

ISO New England Market Monitor notes that higher congestion costs may make 

sense if transmission costs are high and transmission is difficult to build. Various 

37
Docket No. AD10-5 

commenters53 recommend a number of additional and more granular metrics 

including metrics on day-ahead versus real-time congestion, congestion at 

interfaces, hedging shortfalls, changes in Narrow Constrained Areas, congestion 

charges at different times and FTR prices. EPSA asserts that the metric needs 

context in order to be meaningful and useful. APPA/ELCON endorses measuring 

congestion based on the difference between the maximum and minimum 

locational marginal price for the hour and calculated as a total, average and 

maximum difference for each month and year as well as the change from year to 

year. TAPS recommends a metric to assess whether ISOs/RTOs are enabling 

load-serving entities to secure long-term transmission rights for new long-term 

power supply arrangements. It recommends that the measures reveal the degree to 

which load-serving entities are being denied long-term rights for existing 

resources for which long-term rights were previously allocated, or to what extent 

load-serving entities have been unable to collect the full value of the long-term 

rights. 

 

iii. Response 

Commission Staff agrees with commenters that these metrics have limited 

usefulness in evaluating data across different ISOs/RTOs since the size and 

configuration of each ISO/RTO region differs, sometimes significantly, from each 

of the other regions. However, Commission Staff considers the trends in these 

metrics over time to be an indication of whether congestion management is 

improving in each ISO/RTO. Commission Staff agrees with commenters that the 

congestion cost metric will have more explanatory value if it is compared to the 

costs of relieving constraints. Therefore, the narrative discussions, with 

quantification to the extent possible, should include an analysis of the types of 

projects (e.g. transmission, generation, or demand response) and costs of relieving 

constraints to facilitate this comparison. The narrative explanations of trends 

should also address other relevant factors such as the sub-regional congestion and 

FTR prices recommended by commenters.54 

Commission Staff will not adopt the APPA/ELCON locational marginal 

price metric since the difference between the maximum and minimum locational 

marginal price in an ISO/RTO market reflects a number of factors unrelated to 

congestion, such as changes in the fuel mix, system operating conditions and load 

 53 California DWR, ITC, National Grid, NRECA, OMS and SoCal Edison. 

54 As discussed in the transmission planning metric section, the narrative 

discussions should include an assessment of the impact of transmission planning 

on congestion trends. 

38
Docket No. AD10-5 

shifts. The difference between maximum and minimum locational marginal price 

in an ISO/RTO market therefore will not reliably provide insight into ISO/RTO 

performance. Commission Staff believes that the hedging data, with narrative 

explanation, will provide TAPS with information on ISO/RTO performance with 

respect to providing the full value of the long-term transmission rights of loadserving entities for new and existing capacity. 

q. Resource Availability Metric 

i. ISO/RTO Performance Metric 

Resource availability is an indication of ISO/RTO market efficiency and 

cost management. Generator availability, measured as the unforced55 generator 

capacity available in the ISO/RTO markets, is a measure of generator 

responsiveness when the generator owner has indicated that the generation should 

be available. Higher generator availability can result in the commitment of fewer 

peak generators (or the importation of peak supplies) that potentially have high 

costs, thereby resulting in reduced costs. Demand resource availability measures 

the availability of demand response when called on to perform and when tested for 

its capability to meet capacity requirements. 

ii. Comments 

 Some commenters56 argue that the inputs to this metric are beyond 

ISO/RTO control and therefore that generator availability is not suitable as a 

performance metric. DC Energy proposes metrics to measure cost savings 

associated with improved outage statistics and more granular data on availability 

by unit type and years of service. California PUC endorses several metrics 

measuring forced outage rates and ratable time available for generators. Ohio 

Consumer Counsel and OMS support including generator availability information 

by generator type and fuel. EPSA recommends including additional information 

on out-of-merit dispatch and uplift payments, arguing that out-of-merit dispatch 

indicates the transparency and efficiency of the market and that uplift payments 

mask inadequate modeling by the ISO/RTO or inefficient pricing protocols. OMS 

and EPSA consider out-of-merit information to be a measure of the efficient daily 

operation of the ISO/RTO market. NEPGA supports these positions. Ohio 

 55 Unforced capacity represents the capacity of a generator adjusted for 

planned outages. 

56 Allegheny, CAISO, Mid-American, Midwest ISO TOs, NECPUC, and 

NEPOOL. 

39
Docket No. AD10-5 

Consumer Counsel recommends a metric to measure the amount of price 

responsive demand, by sector, that is attributed to advanced metering 

infrastructure. 

iii. Response 

While, as commenters note, generator availability is beyond the control of 

ISOs/RTOs, this metric is worth tracking for the following reasons: (1) it is a 

major factor impacting consumer costs, and therefore should be included in a 

comprehensive assessment of consumer costs; and (2) generator availability as 

measured by planned outages can be influenced by ISO/RTO incentives for 

greater availability, and therefore it provides an indication of the effectiveness of 

ISO/RTO programs. Commission Staff agrees that the cost savings information 

proposed by DC Energy provides useful information. This information should be 

included in each ISO/RTO narrative explanation since not all ISOs/RTOs have the 

data needed for the metric. Commission Staff will not request more granular 

information by unit type or age since the purpose of the metric is to measure the 

effectiveness of ISO/RTO programs to increase the availability of all generators. 

Commission Staff also does not consider metrics on forced outages and ratable 

time available of generation units to be appropriate, as recommended by California 

PUC and OMS. Such metrics measure generator activities over which ISOs/RTOs 

have no influence. 

 Responding to OMS, EPSA and NEPGA, Commission Staff considers outof-merit dispatches to be complex events caused by a number of factors; as such, 

they do not measure ISO/RTO performance unless these factors can be assessed 

and taken into consideration. For example, if out-of-merit dispatch is required 

when generators cannot change output quickly enough to accommodate changes in 

demand over five-minute dispatch intervals, and additional, more expensive units 

must be committed, both generator responsiveness and performance as well as 

ISO/RTO coordination play a role in managing these events. At the same time, 

however, Commission Staff agrees with EPSA and NEPGA that these events, 

particularly if they are frequent and result in the commitment of many generators, 

can subvert the efficiency benefits of ISO/RTO markets. For this reason, narrative 

explanations should be included in each section of each ISO/RTO document that 

discuss the trends in out-of-merit dispatch and progress made toward reducing 

these events and improving market efficiency. 

40
Docket No. AD10-5 

Commission Staff agrees with commenters57 that the performance of 

demand response resources needs to be addressed for a comprehensive assessment 

of resource availability, and therefore Commission Staff supports the inclusion of 

information and data, where available, on demand response performance. 

Commission Staff agrees with the Ohio Consumer Counsel that additional 

information on the status of advanced meter infrastructure programs and their 

impact on price responsive demand availability would be appropriate in the 

narrative discussions. However, Commission Staff does not consider the cost 

measures endorsed by these commenters to be appropriate since the cost of 

projects do not reflect ISO/RTO market performance. 

r. Fuel Diversity 

This metric is defined to be the percentage mix of fuel types installed and 

available (capacity fuel diversity) and consumed (generation fuel diversity) in the 

ISO/RTO region. Fuel diversity provides an indication of the capability in an 

ISO/RTO market to integrate fuels with different characteristics, such as lower 

costs or lower environmental impacts, and how this capability compares to the fuel 

mix of energy consumed in ISOs/RTOs. A higher percentage consumption of 

low-cost fuels, compared to their capability, would indicate that Security 

Constrained Economic Dispatch is maximizing output of lower cost fuels. SoCal 

Edison recommends adding information on the total MMBtus consumed in each 

fuel class. Commission Staff considers the percentage break-down of fuels to be a 

sufficient indication of fuel diversity and therefore additional information on the 

physical amounts of fuel consumed is not necessary. Commission Staff 

recognizes that some ISOs/RTOs may be resource-constrained due to factors 

beyond their control such as geographic locations and environmental regulations 

and therefore comparisons across ISOs/RTOs may reflect factors other than 

performance. Commission Staff expects that such constraints would be discussed 

in the narrative explanations, thereby providing context for the performance 

results. 

s. Renewables Metrics 

i. ISO/RTO Performance Metrics 

 

 57 DC Energy, Dominion, EPSA, ISO New England Market Monitor, New 

York PSC, Ohio PUC and Steel Producers. 

41
Docket No. AD10-5 

Renewable resource58 penetration in ISO/RTO markets is measured as the 

renewable MW share of total energy, measured in megawatt-hours, and the 

renewable MW share of total capacity, measured in megawatts. The renewables 

metrics indicate the diversity of an ISO/RTO resource mix to meet demand and 

capacity requirements. Greater diversity of resources can reduce consumer costs 

by providing more supply options and greater planning flexibility for ISOs/RTOs. 

ii. Comments 

 Commenters59 consider the metric to be beyond ISO/RTO control and 

therefore not suitable as a performance metric. NESCOE notes that the metric is 

developed within the states’ energy and environmental planning and policy 

frameworks. EEI and NEPOOL assert that the metric is not meaningful since 

there are differences in resource mix, fuel prices, congestion, renewable 

penetration and capacity factor in each ISO/RTO. AWEA recommends a number 

of metrics to measure the time and planning process steps required to integrate 

wind resources onto ISO/RTO systems. OMS endorses capacity factor metrics. 

 The ISO New England Market Monitor and EPSA recommend metrics to 

measure the operating performance of renewable resources and measurement of 

the payments made to these resources. Ohio Consumer Counsel endorses metrics 

on the costs of renewable interconnections and events during which renewable 

resources were backed down. Ohio Consumer Counsel also recommends a metric 

to record the amount of behind-the-meter distributed generation (renewable and 

combined heat and power) and the share of distributed generation in ISO/RTO 

energy and capacity markets. 

iii. Response 

Commission Staff recognizes that the amount of renewable resources in a 

market is not under the direct control of ISOs/RTOs, and that they are influenced 

by a number of factors, such as state renewable initiatives. Nonetheless, the 

metrics may be a useful indicator of the integration of renewable resources into the 

ISO/RTO markets. Commission Staff recognizes that regional differences among 

the ISOs/RTOs will result in differing penetration results for renewable resources. 

 58 Renewable resources are defined to be variable energy resources, 

including solar, wind, hydro, geothermal and biomass resources. 

59 Allegheny, Midwest ISO TOs, and NESCOE. 

42
Docket No. AD10-5 

Commission Staff agrees with commenters that additional information 

should be provided and quantified to the extent possible, to the extent it is 

available. For this reason, Commission Staff is adding the renewables percentage 

of capacity as a metric to the list of proposed metrics issued on February 3, 2010. 

Additional information would also be appropriate in narrative explanations that 

address the specific market products being provided by these resources and the 

most significant aspects of their participation including the status of 

interconnecting major renewable resources, as recommended by AWEA and Ohio 

Consumer Counsel, the frequency of events in which renewable resources were 

backed down, as recommended by Ohio Consumer Counsel and information on 

capacity factors as recommended by OMS. Commission Staff does not consider 

operating performance and cost measures for these resources to be appropriate 

since the operating performance and the cost of the projects do not measure 

ISO/RTO performance. 

t. Administrative Costs Metrics 

i. ISO/RTO Performance Metrics 

The administrative cost metrics measure the ability of ISOs/RTOs to keep 

costs within budgeted levels (actual versus budgeted administrative charges 

metric) and to manage the growth rate of administrative costs commensurate with 

the growth rate of system load (administrative charges cents per megawatt-hour of 

load served metric). The components of ISO/RTO administrative costs are capital 

costs – capital charges, debt service, interest expense and depreciation expense – 

and operating and maintenance costs net of miscellaneous income. By managing 

administrative costs, ISOs/RTOs can reduce customer costs. 

ii. Comments 

 

The New Jersey Rate Counsel notes that since costs are not directly 

comparable across ISOs/RTOs due to differences in size, function and capacity 

mix, ISO/RTO administrative cost metrics must be defined on a disaggregated 

basis.60 The Midwest ISO states that total administrative charges are most closely 

related to the geographic scope and load size of each ISO/RTO and that this figure 

would not be an appropriate mechanism for comparing ISO/RTO performance 

when there is such a wide variation in geographic scope and load size of the 

jurisdictional ISO/RTO. The Midwest ISO also notes that the metric fails to 

account for the different sets of services offered by each ISO/RTO to their 

 60 Ohio Consumer Counsel also supports a breakdown of major categories 

of administrative costs as well as weather-normalized data. 

43
Docket No. AD10-5 

members. OMS supports comparing actual and budgeted administrative charges 

with a break-down by category of costs. California DWR recommends a 

comparison of administrative and operating costs over time to determine 

streamlining and duplication in transmission operational and administrative costs 

between ISO/RTO and transmission owners. 

 The Federal Trade Commission endorses an internal operating efficiency 

metric that holds ISOs/RTOs accountable for making cost-effective use of 

resources such as staff and software to run the grid efficiently. DC Energy also 

supports a metric that measures whether administrative costs are justified by the 

projects undertaken. 

 TANC recommends that the Commission strengthen its oversight of the 

ISO/RTO budgeting process and makes the specific recommendation that 

ISOs/RTOs be required to submit their proposed budgets for Commission 

approval and explain any deviations, such as is required for NERC and the 

Regional Entities. ODEC asserts there is need for increased transparency in the 

budget process. OMS expresses concern that the stakeholder process for budget 

reviews requires considerable resources and does not allow for meaningful 

participation for some important stakeholders. 

iii. Response 

Commission Staff recognizes that administrative costs will vary by 

ISO/RTO and that comparisons between ISOs/RTOs may not be meaningful, as 

commenters point out. Commission Staff expects that the primary value of this 

metric will be in the analysis of long-term cost trends and the narrative 

explanations of major variances from budget and that this information will be 

responsive to the concerns of commenters. The narrative discussions should 

include discussions of major categories of costs61 as part of the variance analysis, 

thereby addressing recommendations of commenters. Also, the narrative 

discussions should address the cost-effectiveness of ISO/RTO administrative 

resource planning, as recommended by the Federal Trade Commission. Further, 

the narrative discussion could explain the effect that load reductions (due to 

demand resources, energy efficiency, or other factors) have had on reducing 

Mwhs, and thus the calculation of the second metric, so that these effects do not 

convey the misimpression that administrative costs per Mwh have increased. 

 61 We clarify for California PUC and Ohio Consumer Counsel that the 

administrative charges include capital investment. 

44
Docket No. AD10-5 

Commission Staff does not recommend a mandatory budgeting process. As 

discussed in the opening sections of this working paper, this is not a rulemaking 

proceeding and therefore mandatory requirements are not appropriate. 

u. Customer Satisfaction Metric 

i. ISO/RTO Performance Metrics 

The percentage of satisfied members metric and the independent 

assessment of customer satisfaction provide indications of the extent to which 

ISOs/RTOs provide value to their customers. The independent assessments of 

customer satisfaction are based on surveys undertaken by independent, third-party 

entities. These surveys analyze customer perspectives on a wide range of 

ISO/RTO activities. 

ii. Comments 

The Federal Trade Commission endorses a metric to measure 

responsiveness to grid users and the retail customers they serve. California DWR 

recommends an independent assessment of customer satisfaction with ISO/RTO 

performance and customer friendliness. NEPUC and NEPOOL recommend that 

satisfaction should be measured for separate functions; transmission planning, 

setting of installed capacity requirements, administration of market, market rule 

changes, market settlement, market development, and responsiveness to 

information requests. New Jersey Rate Counsel recommends assessing customer 

satisfaction by major functional area (e.g., operations, transmission planning, 

generation planning) and by customer group. Ohio Consumer Counsel supports 

assessing customer satisfaction by sector and Massachusetts AG recommends that 

information be provided on customer complaints broken down by topic area with 

the corresponding ISO/RTO response. 

 Several comments also recommend additional metrics. NY Transmission 

Owners recommend a metric to measure ISO/RTOs’ correct implementation of 

tariffs and avoidance of market errors. National Grid endorses a metric to measure 

the frequency of price flagging and tariff waiver requests by ISOs/RTOs. DC 

Energy recommends a metric to measure the frequency with which market reports 

are posted in a timely manner. NEPOOL endorses a metric to measure the 

responsiveness of ISOs/RTOs to participant or consumer requests for cost impact 

analyses and analyses of market rule revisions. ODEC has a similar 

recommendation for responsiveness per the requirements of Order No. 719.62 

 

62 Wholesale Competition in Regions with Organized Energy Markets, Order No. 

719, 73 Fed. Reg. 64,100 (Oct. 28, 2008), FERC Stats. & Regs. ¶ 31,281 (2008). 

45
Docket No. AD10-5 

NEPGA supports a metric to measure buyer and seller stakeholder support for 

ISO/RTO market rule changes. 

iii. Response 

 In response to comments, Commission Staff considers it appropriate to 

include information in the report from independent assessments of customer 

satisfaction and satisfaction surveys of specific ISO/RTO functions. The 

Customer Satisfaction Metric will be supplemented by a narrative discussion that 

should address responses by customer sector and by functional area as well as 

referencing key customer activities such as responsiveness to customer requests, 

the timeliness of market reports, time to resolve invoice disputes, correct 

implementation of tariffs, market errors/price flagging and stakeholder support for 

major rule changes, as recommended by commenters. 

v. Billing Controls Metric 

This metric indicates the accuracy and integrity of the ISO/RTO billing 

process, based on audits conducted according to the Statement on Auditing 

Standards No. 70 guidelines set by the American Institute of Certified Public 

Accountants. The audits describe the controls, the results of testing of the 

ISO/RTO controls, whether the controls were designed to achieve the control 

objectives, the auditor’s opinions on the audit and whether the controls that were 

tested were operating with sufficient effectiveness to provide reasonable assurance 

that the control objectives were achieved. NY Transmission Owners and 

Allegheny recommend metrics to measure billing errors. Commission Staff 

considers it appropriate that the narrative explanation that accompanies this metric 

include a discussion of major billing issues, including billing errors, rather than 

reporting an error metric. Such an explanation should provide a more meaningful 

assessment of the significant billing issues that may be the root cause of billing 

errors. 

 

46
Docket No. AD10-5 

 

IV. APPENDIX A

LIST OF INTERVENORS AND COMMENTERS 

 Allegheny Power and Allegheny Energy Supply Company, LLC 

(Allegheny) 

 American Public Power Association and Electricity Consumers Resource 

Council (APPA/ELCON) 

 American Wind Energy Association (AWEA) 

 California Department of Water Resources State Water Project (California 

DWR) 

 California Independent System Operator Corporation (CAISO) 

 California Public Utilities Commission (California PUC) 

 City of Redding, City of Santa Clara and M-S-R Public Power Agency 

(Cities/M-S-R) 

 COMPETE Coalition (COMPETE) 

 Consumer Commenters63 (Consumer Coalition) 

 DC Energy, LLC (DC Energy) 

 Delaware Municipal Electric Corporation (DEMEC) 

 Dominion Resources Services, Inc. (Dominion) 

 Duke Energy Corporation (Duke) 

 Edison Electric Institute (EEI) 

 Eastern Massachusetts Consumer-Owned Systems (EMCOS) 

 Electric Power Supply Association (EPSA) 

 63 AARP, American Forest & Paper Association, American Municipal 

Power, Inc., American Public Power Association, Blue Ridge Power Agency, 

Citizen Power, Citizens Utility Board, Coalition of Midwest Transmission 

Customers, Connecticut Office of Consumer Counsel, Delaware Municipal 

Electric Corporation, Inc., Electricity Consumers Resource Council, Illinois 

Attorney General, Industrial Energy Consumers of America, Kennebunk Light & 

Power, Maryland Office of People’s Counsel, Modesto Irrigation District, 

Municipal Electric Utilities Association of New York, National Consumer Law 

Center, NEPOOL Industrial Customer Coalition, New England Public Power 

Association, New York Association of Public Power, Office of the People’s 

Counsel for the District of Columbia, Ohio Partners for Affordable Energy, 

Pennsylvania Office of Consumer Advocate, PJM Industrial Customer Coalition, 

Public Citizen, Public Power Association of New Jersey, Public Utility Law 

Project of New York, Inc., and Virginia Citizens Consumer Council.

47
Docket No. AD10-5 

 Federal Trade Commission (Federal Trade Commission) 

 FirstEnergy Service Company (FirstEnergy) 

 Coalition of Midwest Transmission Customers and PJM Industrial 

Customer Coalition (Industrial Customers) 

 Internal Market Monitor of ISO New England Inc. (ISO New England 

Market Monitor) 

 ISO/RTO Council (IRC) 

 Independent Power Producers of New York, Inc., (IPPNY) 

 International Transmission Company (ITC) 

 Maine Public Utilities Commission (Maine Commission) 

 Public Service Commission of Maryland (Maryland PSC) 

 Attorney General of the Commonwealth of Massachusetts (Massachusetts 

AG) 

 MidAmerican Energy Company (MidAmerican) 

 Midwest Independent Transmission System Operator, Inc. (Midwest ISO) 

 Midwest ISO Transmission Owners (Midwest ISO TOs) 

 Missouri Public Service Commission (Missouri Commission) 

 Modesto Irrigation District (MID) 

 National Association of Regulatory Utility Commissioners (NARUC) 

 National Grid USA (National Grid) 

 National Rural Electric Cooperative Association (NRECA) 

 New England Conference of Public Utility Commissions (NECPUC) 

 New England Power Generators Association, Inc. (NEPGA) 

 New England Power Pool Participants Committee (NEPOOL) 

 New England States Committee on Electricity (NESCOE) 

 New Jersey Division of Public Advocate, Division of Rate Counsel (New 

Jersey Rate Counsel) 

 New Hampshire Public Utilities Commission (New Hampshire PUC) 

 New York State Public Service Commission (New York PSC) 

 Indicated New York Transmission Owners (NY Transmission Owners) 

 Public Utilities Commission of Ohio (Ohio PUC) 

 Office of the Ohio Consumers’ Counsel (Ohio Consumer Counsel) 

 Old Dominion Electric Cooperative (ODEC) 

 Organization of MISO States (OMS) 

 Pacific Gas and Electric Company (PG&E) 

 Pennsylvania Public Utilities Commission (Pennsylvania PUC) 

 Pepco Holdings, Inc. (PHI Companies) 

 PJM Power Providers Group (P3) 

 Connecticut Municipal Electric Energy Cooperative, Massachusetts 

Municipal Wholesale Electric Company and New Hampshire Electric 

Cooperative (Public Systems) 

48
Docket No. AD10-5 

 Retail Energy Supply Association (RESA) 

 Southern California Edison Company (SoCal Edison) 

 Steel Producers Association (Steel Producers) 

 Transmission Association of Northern California (TANC) 

 Transmission Access Policy Study Group (TAPS) 

 Tyrone Christy, Vice Chairman, Pennsylvania Public Utilities Commission 

(Commissioner Christy) 

 United Illuminating Company (United Illuminating) 

 Viridity Energy, Inc. (Viridity) 

 Westar Energy Inc. (Westar) 

49
Docket No. AD10-5 

50

V. APPENDIX B
1 

Appendix B ISO/RTO Performance Metrics 

Performance Metric Specific Metric(s) 

Reliability 

A. National or Regional 

Reliability Standards Compliance 

1. References to which Electricity Reliability Organization (ERO) and ROrganization (RRO) standards are applicable to each ISO/RTO 2. Number of violations self-reported 3. Number of violations identified as RRO or ERO audit findings 4. Total number of violations 5. Severity level of each violation 6. Compliance with operating reserve standards 7. Unserved energy (or load shedding) caused by violations 

B. Dispatch Reliability 1. Balance Authority Ace Limit (BAAL) OR// CPS1 and CPS2 

2. Number of hours of transmission load reliefs (of severity level 3 or higflows 3. Energy Management System (EMS) availability 

C. Operational Planning – 

Load Forecast Accuracy 

Actual load as a percentage variance from forecasted load (separate metri

valley metric) 

D. Wind Forecasting 

Accuracy 

Actual wind availability compared to forecasted wind availability 

E. Unscheduled Flows Difference between net actual interchange (actual measured power flow 

net scheduled interchange in megawatt hours (total system and major intpercentage of total flows 

F. Transmission Outage 

Coordination 

1. Percentage of > 200kV planned outages of 5 days or more that are sub

least 1 month prior to the outage commencement date 2. Percentage of planned outages studied in the respective ISO/RTO Tar
2 

Performance Metric Specific Metric(s) 

established timeframes 3. Percentage of > 200 kV outages cancelled by ISO/RTO after having b

approval 4. Percentage of > 200kV outages (both planned and unplanned) with le

G. Long-Term Reliability 

Planning – Transmission 

1. Number of facilities approved to be constructed for reliability purpose2. Percentage of approved construction on schedule and completed 3. Performance of Order 890 planning process related to: 

 a. Completion of reliability studies 

 b. Completion of economic studies 

4. Impact of Demand Response on Infrastructure Investment 

H. Long-Term Reliability 

Planning – Resources 

1. Processing time for generation interconnection requests 

2. Actual reserve margins compared with planned reserve margins 

3. Demand response megawatts as percentage of total capacity 4. Percentage of generation outages cancelled by ISOs/RTOs 5. Number and capacity of generation reliability must run contracts 6. Demand response megawatts as percentage of total ancillary services 

I. Infrastructure Investment 

– Interconnection and Transmission Process Metrics 

1. Number of requests 2. Number of studies completed 3. Average age of incomplete studies 4. Average time for completed studies 5. Average cost of each type of study completed (e.g., feasibility study, sand facility study) 
3 

Performance Metric Specific Metric(s) 

J. Special Protection 

Systems 

1. Number of special protection systems 2. Percentage of special protection systems that responded as designed w3. Number of unintended activations 

Performance Metric Specific Metric(s) 

Market Benefits 

A. Market Competitiveness 1. Price Cost Mark Up 

2. Generation Net Revenues 3. Percentage of hours offers are capped due to mitigation 

B. Market Pricing 1. Load-Weighted Locational Marginal Prices 

2. Components of Total Power Costs based on Load-Weighted Locationa(e.g. fuel costs, transmission charges, RTO costs, etc.) 

3. Load-Weighted, Fuel-Adjusted Locational Marginal Prices 4. Impacts of Demand Response on Market Prices 

C. System Lambda System Lambda (on marginal unit); unconstrained energy portion of systD. Energy Market Price 

Convergence 

1. Absolute dollar difference between day-ahead and real-time prices 

2. Percentage difference between day-ahead and real-time prices 

E. Congestion Management 1. Congestion charges per megawatt hour of load served 

2. Percentage of congestion dollars hedged through ISO/RTO-administermanagement markets 

F. Resource Availability 1. 1 – RTO forced outage rate 

2. Demand Response Availability 

G. Fuel Diversity Fuel diversity in terms of energy, installed capacity and actual production 
4 

Performance Metric Specific Metric(s) 

H. Renewables 1. Renewable megawatt hours as a percentage of total energy 

2. Renewable megawatts as a percentage of total capacity. 

Organizational Effectiveness 

A. Administrative Costs 1. Annual actual ISO/RTO administrative charges to members compared 

administrative charges 

2. Annual actual ISO/RTO administrative charges to members as cents peload served 

B. Customer Satisfaction 1. Percentage of satisfied stakeholders 

2. Independent assessment of stakeholder satisfaction 

C. Billing Controls SAS 70 audit scope (e.g. Type 1 or Type 2 audit) and results