U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512–1800; DC area (202) 512–1800

Fax: (202) 512–2104 Mail: Stop IDCC, Washington, DC 20402–0001

44–582 PDF 2009

S. Hrg. 110–817 

IMPROVING FEDERAL PROGRAM MANAGEMENT 

USING PERFORMANCE INFORMATION 

HEARING 

BEFORE THE

FEDERAL FINANCIAL MANAGEMENT, GOVERNMENT 

INFORMATION, FEDERAL SERVICES, AND 

INTERNATIONAL SECURITY SUBCOMMITTEE 

OF THE

COMMITTEE ON 

HOMELAND SECURITY AND 

GOVERNMENTAL AFFAIRS 

UNITED STATES SENATE 

ONE HUNDRED TENTH CONGRESS 

SECOND SESSION 

JULY 24, 2008 

Available via http://www.gpoaccess.gov/congress/index.html 

Printed for the use of the Committee on Homeland Security 

and Governmental Affairs 

( 

(II) 

COMMITTEE ON HOMELAND SECURITY AND GOVERNMENTAL AFFAIRS 

JOSEPH I. LIEBERMAN, Connecticut, Chairman 

CARL LEVIN, Michigan 

DANIEL K. AKAKA, Hawaii 

THOMAS R. CARPER, Delaware 

MARK L. PRYOR, Arkansas 

MARY L. LANDRIEU, Louisiana 

BARACK OBAMA, Illinois 

CLAIRE MCCASKILL, Missouri 

JON TESTER, Montana 

SUSAN M. COLLINS, Maine 

TED STEVENS, Alaska 

GEORGE V. VOINOVICH, Ohio 

NORM COLEMAN, Minnesota 

TOM COBURN, Oklahoma 

PETE V. DOMENICI, New Mexico 

JOHN WARNER, Virginia 

JOHN E. SUNUNU, New Hampshire 

MICHAEL L. ALEXANDER, Staff Director 

BRANDON L. MILHORN, Minority Staff Director and Chief Counsel 

TRINA DRIESSNACK TYRER, Chief Clerk 

FEDERAL FINANCIAL MANAGEMENT, GOVERNMENT INFORMATION, 

FEDERAL SERVICES, AND INTERNATIONAL SECURITY SUBCOMMITTEE 

THOMAS R. CARPER, Delaware, Chairman 

CARL LEVIN, Michigan 

DANIEL K. AKAKA, Hawaii 

BARACK OBAMA, Illinois 

CLAIRE MCCASKILL, Missouri 

JON TESTER, Montana 

TOM COBURN, Oklahoma 

TED STEVENS, Alaska 

GEORGE V. VOINOVICH, Ohio 

PETE V. DOMENICI, New Mexico 

JOHN E. SUNUNU, New Hampshire 

JOHN KILVINGTON, Staff Director 

KATY FRENCH, Minority Staff Director 

MONISHA SMITH, Chief Clerk 

(III) 

C O N T E N T S 

Opening statement: Page 

Senator Carper ................................................................................................. 1 

WITNESSES 

THURSDAY, JULY 24, 2008 

Hon. Martin O’Malley, Governor of Maryland ...................................................... 4 

Bernice Steinhardt, Director for Strategic Issues, U.S. Government Accountability Office ......................................................................................................... 18 

Marcus C. Peacock, Deputy Administrator, U.S. Environmental Protection 

Agency, and Former Official at U.S. Office of Management and Budget ........ 20 

Donald F. Kettl, Director, Fels Institute of Government, and Robert A. Fox 

Professor of Leadership, University of Pennsylvania ....................................... 23 

James Dyer, Chief Financial Officer, and Performance Improvement Officer, 

U.S. Nuclear Regulatory Commission ................................................................ 36 

Scott Pace, Associate Administrator for Program Analysis and Evaluation, 

and Performance Improvement Officer, National Aeronautics and Space 

Administration ..................................................................................................... 38 

Daniel Tucker, Performance Improvement Officer, and Deputy Assistant Secretary for the Budget, U.S. Department of Veterans Affairs ........................... 40 

ALPHABETICAL LIST OF WITNESSES

Dyer, James: 

Testimony .......................................................................................................... 36 

Prepared statement .......................................................................................... 92 

Kettl, Donald F.: 

Testimony .......................................................................................................... 23 

Prepared statement .......................................................................................... 86 

O’Malley, Hon. Martin: 

Testimony .......................................................................................................... 4 

Prepared statement .......................................................................................... 49 

Pace, Scott: 

Testimony .......................................................................................................... 38 

Prepared statement .......................................................................................... 97 

Peacock, Marcus C.: 

Testimony .......................................................................................................... 20 

Prepared statement .......................................................................................... 84 

Steinhardt, Bernice: 

Testimony .......................................................................................................... 18 

Prepared statement .......................................................................................... 55 

Tucker, Daniel: 

Testimony .......................................................................................................... 40 

Prepared statement .......................................................................................... 100 

APPENDIX 

Questions and Responses for the Record from: 

Governor O’Malley ............................................................................................ 107 

Ms. Steinhardt .................................................................................................. 112 

Mr. Peacock ....................................................................................................... 117 

Mr. Tucker ........................................................................................................ 128 

Prepared statement from Robert Shea, Associate Director for Government 

Performance, Office of Management and Budget .............................................. 131 

Charts submitted for the Record by Governor O’Malley ...................................... 136 

(1) 

IMPROVING FEDERAL PROGRAM MANAGEMENT USING PERFORMANCE INFORMATION 

THURSDAY, JULY 24, 2008 

U.S. SENATE, SUBCOMMITTEE ON FEDERAL FINANCIAL MANAGEMENT, GOVERNMENT INFORMATION, FEDERAL SERVICE, AND INTERNATIONAL SECURITY, OF THE COMMITTEE ON HOMELAND SECURITY

AND GOVERNMENTAL AFFAIRS, Washington, DC. 

The Subcommittee met, pursuant to notice, at 2:35 p.m., in room 

SD–342, Dirksen Senate Office Building, Hon. Thomas R. Carper, 

Chairman of the Subcommittee, presiding. 

Present: Senator Carper. 

OPENING STATEMENT OF SENATOR CARPER 

Senator CARPER. The Subcommittee will come to order. 

Governor, welcome. We are delighted that you are here. Thank 

you so much for taking time to join us. Senator Tom Coburn is the 

Ranking Republican of this Subcommittee. He will be here shortly. 

But given all that you have on your plate, Governor O’Malley, we 

are delighted that you are able to find time to be here and talk 

about something not only that you are interested in, but you know 

a whole lot about, and we thank you for your willingness to share 

your experiences, your perspectives, both as Mayor of Baltimore 

and now as Chief Executive of the State of Maryland. 

One of my favorite sayings is ‘‘I would rather see a sermon than 

hear one,’’ and I think we can look at what you have done, both 

as mayor and as governor of your State, to actually show us by 

your own behavior that this works and it is something that may 

just work for us, as well. 

In a little less than 6 months from now, we will have a new 

President standing on the West Portico of the Capitol, preparing to 

raise his right hand and take an oath to defend our country and 

Constitution. Our new Chief Executive will face a soaring Federal 

deficit at home, and security challenges abroad. Those are just a 

couple of the exceptional challenges that our Federal Government 

must be prepared to face. No matter who is elected President, 

whether it is Senator Obama or Senator McCain, we must make 

sure that person has the information and the tools that they need 

in order to keep our ship of state headed toward the horizon. 

And though our politics may differ here in Washington and 

across the country, I think we can all agree that the strength of 

our democracy hinges on the ability of our government to deliver 

2 

its promises to the people. We have a responsibility to be judicious 

stewards of the resources the taxpayers invest in America and ensure that those resources are managed honestly and that they are 

managed effectively. 

Just recently, this Subcommittee heard from a number of witnesses, including our former Comptroller General, David Walker, 

about the dire fiscal situation this country faces, not just this year, 

not just in this decade, but for decades to come. Over the next two 

decades, 80 million baby boomers—I am one of them—will become 

eligible for Social Security and Medicare. Today, these two programs already make up over 40 percent of our government’s total 

expenses. And as boomers like me start to draw benefits, some experts we have heard from say that the share of these programs 

could equal within 40 years not 40 percent of our government’s outlays, but all of our government’s outlays. 

Without any changes, we will not have extra funds to prepare a 

world-class workforce in the 21st Century, funds to make us energy 

independent, meet our transportation needs, clean our air, or address our pressing national security needs. There is only so much 

pie to go around, so how we manage who is getting the next slice, 

and how big it is going to be, becomes increasingly important. 

In 1993, the year that I became Governor of Delaware, the Government Performance and Results Act (GPRA), was enacted to help 

us better manage our Nation’s finite resources and improve the effectiveness and the delivery of Federal programs. Since that time, 

agencies have collected, and provided to Congress, a tremendous 

amount of performance data. However, Senator Coburn and I wondered if this information was being translated into results, or at 

least into improved results. Producing the information does not by 

itself improve performance, and we knew that there had been little 

increase in the actual use of performance data by agency managers. 

A lot of work has been done by GAO and others on how this data 

could be used, but nobody has really looked into whether agencies 

were putting theory into practice. A year or so ago, Senator Coburn 

and I asked the GAO to examine how performance information is 

being used to better manage Federal agencies, and if managers 

could be, and should be, employing it more often in their decisionmaking processes. We also asked the GAO to consider the recent 

efforts by the Administration to improve the usefulness of agency 

performance information and through initiatives such as the Program Assessment Rating Tool (PART), a key component of President Bush’s Management Agenda. 

I am eager to hear GAO’s findings and ask that we focus our discussion on the following crucial questions today. To what extent 

are Federal agencies using performance information to perform key 

management tasks, such as identifying performance problems and 

taking corrective actions or identifying and sharing best practices? 

Second, how can Federal agencies make better use of performance information to improve our results, and our outcomes? 

And finally, what lessons can be learned and how can we build 

on this Administration’s efforts to improve the usefulness and employment of performance information? 

3 

Today, with these questions in mind, I want us to do the following. One is to accurately assess how well and how often Federal 

agencies are using performance information to correct problems 

and promote best practices. 

Two, I would like for us to discuss how the use of performance 

information can be increased and improved in the next Administration. 

And three, I want us to solicit ideas about how Congress can play 

an effective role in the path forward. 

Today, it is my hope that we can begin to develop a blueprint for 

the next Administration. We must do all that we can to ensure that 

the transition is streamlined and that our next President has a 

clear picture of the strengths and weaknesses of agencies under his 

direction. 

We face, in this country, unparalleled challenges both here and 

abroad and these require a knowledgeable and nimble Federal Government that can respond effectively. With concerns growing about 

our mounting Federal deficit and national debt, the American people deserve to know that every dollar we send to Washington is 

being used to its utmost potential. Performance information is an 

invaluable tool that can do just that, if we use it. If used effectively, it can identify problems, help us find solutions, and develop 

approaches that improve results. 

We thank, again, not only our lead-off witness, but all of our witnesses for taking this opportunity, setting aside time in their own 

lives to be with us today to share with us some of their ideas about 

the challenges before us and how best to address those challenges. 

Our lead-off witness is Governor Martin O’Malley. He was elected Governor of Maryland in 2006, coming on the heels of an unusually successful tenure as Mayor of the City of Baltimore. Early in 

his life, as a young troubadour, he led his Irish group into the City 

of Wilmington and left the fans at O’Friels Irish Pub standing and 

cheering in his wake at an age when he was actually too young to 

get in legally to O’Friels, but came in and did a great job performing for us. 

In 2005, Time Magazine named him one of America’s Top Five 

Big City Mayors. As Mayor, he pioneered the CitiStat, a statisticsbased tracking system that focuses in on areas of under-performance and demands a results-driven government model. CitiStat has 

saved Baltimore residents more than $350 million and was awarded Harvard University’s prestigious Innovations in American Government Award in 2004. 

As governor, he has brought the program State-wide, implementing StateStat across all Maryland’s government services, and 

provided a model not just for mayors, I think not just for governors, 

but I think maybe for Presidents and for those of us who serve here 

in the Legislative Branch of our government. 

We are honored by your presence and we are especially honored 

to be able to serve here on a daily basis with your mother, who 

works, as some of the people here know, for Senator Mikulski. I am 

not sure who works for whom, and I don’t know if Senator Mikulski is actually here. She was going to try to stop by today. She may 

pop in. But if your mother is around and if she shows up, please 

4 

1The prepared statement of Governor O’Malley appears in the Appendix on page 49. 

introduce her to us. We are delighted that you are here and that 

your mom raised you so well. [Laughter.] 

Governor O’Malley, you are recognized to speak for as long as 

you wish. 

TESTIMONY OF HON. MARTIN O’MALLEY,1 GOVERNOR OF 

MARYLAND 

Governor O’MALLEY. Chairman Carper, thank you very much. 

Senator CARPER. Thank you. 

Governor O’MALLEY. Thank you for the opportunity to be able to 

join you at this important Subcommittee and as part of the discussion of this day. 

I am also joined by a number of members of my staff. Apropos 

to today’s discussion, Matt Gallagher is here who runs our 

StateStat office and is the Deputy Chief of Staff for Operations in 

our State government. 

Senator CARPER. Would you ask him to raise his hand? 

Governor O’MALLEY. There he is. 

Senator CARPER. All right. 

Governor O’MALLEY. He also before that ran the CitiStat office 

in the City of Baltimore before that. 

It is an honor to be here today before you to talk about an issue 

that, frankly, I believe is changing, for the better, the way that 

many of us look at the operations of our government, and it is our 

government. It is our belief that the same performance-based governing strategies that were so very valuable in igniting Baltimore’s 

come-back and have been so valuable to us at the State level in 

Maryland this past year-and-a-half can also work not only in the 

Federal Government, but in any large human organization. 

In our public life, we tend to be very good at measuring inputs. 

We typically refer to those inputs as the budget, and it is typically 

done on an annual basis. But we have often neglected to pay 

enough attention to outputs, to the product of government. We are 

constantly asking, what is our funding level. For example, for 

something like interoperable communications without asking 

whether or not the purchases that have been made by local, municipal, or State governments this year have been in accordance with 

the standard that moves us down the road to a point in time when 

all first responders throughout our Nation can actually talk with 

one another when responding to an emergency. 

Performance-based government, in its essence, is about measuring, tracking, and improving outputs. Inputs play a role, but only 

in the pursuit of outputs. 

Mr. Chairman, I was first introduced to this model of governing 

about 81⁄2 years ago when I began my first term as Mayor of the 

City of Baltimore. And when we were handed the keys to that 

16,000 person, $2 billion a year operation known as city government, we inherited our fair share of challenges, some of them very 

big challenges. More than 300 of our fellow citizens every year 

were being murdered in our city. The streets were too often littered 

with trash. Our schools were too often failing and people were 

abandoning our city, in essence, voting with their feet, and leaving 

5 

1The chart referred to appears in the Appendix on page 138. 

behind them buildings and homes that were vacant and themselves 

becoming nuisances. 

Quite understandably, the public at the time was demanding immediate results and immediate turn-around, and we rolled up our 

sleeves and got to work and found in beginning that work that 

there was very little in city government that was actually being 

measured in a consistent and real-time fashion. Now, don’t get me 

wrong. Oftentimes, information was being collected at the ground 

level, sometimes very faithfully, sometimes very dutifully, not always the right information, but rarely, if ever, was it being used 

in a timely manner by the appropriators and the policy makers and 

the administrators at the highest level in order to deliver better 

outputs and better outcomes for citizens on the ground. 

So we were very blessed to have met a gentleman by the name 

of Jack Maple, who was one of the great minds behind the performance-based strategies employed in the New York City Police Department for its turnaround. Jack was the Deputy Commissioner 

of Police under Commissioner Bratton and we felt, having observed 

ComStat in action, that if the NYPD could so successfully use simple off-the-shelf software, computer pin mapping, deploying police 

resources to where the crime was actually happening, that data collection and mapping technology could also work for the other 

things that government does, whether it is garbage collection or repairing streetlights or addressing complaints about potholes. 

And from this approach was expropriated the four main tenets 

of CitiStat, which were the main tenets of ComStat. One, timely, 

accurate information shared by all; two, rapid deployment of resources; three, effective tactics and strategies; and four, relentless 

follow-up, not on an annual basis, not on a biannual basis, but on 

a daily, weekly, biweekly, monthly basis in order to improve performance. 

So we started setting goals. We started measuring results. And 

we did so weekly. We began tracking outputs instead of just tracking inputs and we started geomapping every conceivable service. 

And in short time, we turned around the city where many neighborhoods were considered ungovernable and we started making our 

city government function again in order to improve the quality of 

life in every neighborhood. 

I go into some greater detail about some of the results we 

achieved in the written statement I submitted to the Subcommittee, but just a few examples. Most important of all is the 

primary responsibility of all governments and that is public safety. 

We were able to achieve a 40 percent reduction in violent crime, 

its lowest level, actually, in four decades. We were able to back up 

with 98 percent success a 48-hour guarantee to address complaints 

from citizens about potholes. 

We reduced the number of children exposed to dangerously high 

levels of lead from lead dust, lead paint poisoning from old homes 

and deteriorating homes by 65 percent in a relatively short period 

of time.1 We were able to identify and reclaim by clearing title 

more than 5,900 vacant homes and buildings which then enables 

them to be redeveloped and put back on the tax rolls. 

6 

1 Chart referred to appears in the Appendix on page 136. 

2 Chart referred to appears in the Appendix on page 137. 

We had a boarding and cleaning backlog of about some 8 months, 

which is how long it took when we began to address a boarding or 

cleaning complaint from any citizen. By the time we left, that was 

down to 14 days. Now, that didn’t happen overnight. It didn’t happen by measuring things annually. It happened by measuring them 

every single day and every week. 

And probably the most important outcome of all is that we were 

able by improving our quality of life with better-performing government to be able to reverse four decades of what had been seemingly 

insurmountable population loss, and the city started growing again. 

I brought two charts with us today. One of them on the far right 

is the combination of homicides, shootings,1 and you will see three 

kinds of death, and you can see them over time. Again, these are 

just measured annually, but we measured them every 2 weeks and, 

by golly, when you look over your shoulder, you see you are actually making progress shrinking those danger zones in our city. 

This next map is the cleaning and boarding measured not in 

terms of the reduction of the wait time but in terms of the improvement of the productivity.2 Again, these are annual outcomes, but 

the only way we were able to achieve the annual outcomes is because we developed systems so that we could measure every day, 

every week, and that information was then able to get back up to 

the policymakers, the administrators, the appropriators. 

In 2007, when we were given the keys to an even larger organization, namely the State government of the great people of Maryland, we took this model with us. We created a program called 

StateStat, which has allowed us to track and measure progress on 

a level that we have never before been able to achieve in our State 

government. And through our StateStat program, while only in its 

first year and a half, we have been able to identify problems in information sharing among various law enforcement agencies and 

across several jurisdictions and we work every day to improve 

them. 

Second, we were able to identify right off the bat the imperative, 

the need to close the House of Corrections, an old, dilapidated, and 

very dangerous prison in our State. We were able to close that 

without incident within 50 days, saving probably lives and also 

saving $3 million in overhead and overtime. 

We were also able to identify and fix more than 100 problems 

within our juvenile detention facilities, many of which had been 

under various consent decrees and court orders for a long time 

without ever improving those conditions. 

We are now using GPS technology and performance measures to 

target our resources that are geared at restoring the national treasure, which is the Chesapeake Bay, through a program that we call 

BayStat. You may notice that virtually anything, if you measure it, 

you can slap a ‘‘stat’’ on the end of it and you have a new way of 

saying whether what you are doing is making any impact on the 

problem you are trying to solve. 

We go through a few more accomplishments in the written statement I have submitted to you, but I wanted to close by just saying 

7 

a couple of words about CitiStat, StateStat, and why this model, 

I believe, can be and should be applied at the Federal level, particularly in the realm of homeland security. 

We believe the same approach can be relevant to governments 

and organizations anywhere and of any size. Recently, I had occasion to meet and to listen to Sir Michael Barbour from Tony Blair’s 

government who applied many of these principles to the administration of national government in the United Kingdom and also implemented a new innovation called the delivery unit to make sure 

that all of those along the chain of delivery, from policy maker to 

appropriator to administrator to implementor down to the level of 

citizen, were actually held accountable for their piece of delivering 

improved performance. 

The Environmental Protection Agency, I understand, has 

launched an EPAStat program, and there are governments worldwide that are working to implement this model. In fact, we have 

had several delegations from governments all over the world who 

have come to visit us in Maryland to learn about the workings, the 

early workings, of our program of statewide performance measurement. 

Government performance management, I believe, is really a nonpartisan issue. There is no Democratic or Republican way to fill a 

pothole, to make sure that you improve the outcomes, whether it 

is at a municipal, State, or National Government. The beauty of a 

map is that a map doesn’t know whether a neighborhood is black 

or white, or whether a neighborhood is rich or poor, or whether a 

neighborhood is Democratic or Republican. 

Most of us in government would say that it is the responsibility 

of every public official to provide the most effective government 

possible and to provide for continued progress and improvement. 

That is what CitiStat and StateStat are really about. 

Mr. Chairman, I want to close today with the words of Robert 

Kennedy, who once said, ‘‘There is no basic inconsistency between 

ideals and realistic possibilities, no separation between the deepest 

desires of the heart and of the mind, and the rational application 

of human effort to human problems.’’ The rational application, well 

and timely measured, of human effort to human problems. 

That is what performance-based government is about, and I 

thank you so very much for your interest and your leadership in 

bringing this to our Federal Government. Thanks very much. 

Senator CARPER. Governor, thank you very much. 

One of my finest memories of being governor was going to New 

Governors School every 2 years, right after the election. About the 

middle of November, current governors, would host the newly-elected governors and spouses and teach the new governors and spouses 

by virtue of our own experience the things we had done wrong, in 

many cases, and some of the things we had done right. 

We had a Center for Best Practices within the National Governors Association. I suspect we still do. I am just wondering, does 

your experience with CitiStat and what you are doing with 

StateStat, does any of that show up within the National Governors 

Association, either at a New Governors School like a forum, or 

through the Center for Best Practices so that other States can 

8 

learn from what you have done, both in Baltimore and in Maryland? 

Governor O’MALLEY. Yes. To some degree, other States have 

begun this in one way, shape, or form, and some of them started 

doing it years ago on the heels of seeing the success of the NYPD. 

In fact, Christine Gregoire, who began her first term as Governor 

of Washington State, visited us, actually before all the votes were 

totally counted in that close race—— 

Senator CARPER. I said to Christine, who had been Attorney General, I said, keep counting the votes and recounting the votes until 

you win. Then stop. [Laughter.] 

That is what they did. 

Governor O’MALLEY. Well, during the period of pre-transition, 

she sent a group of her people to Baltimore and has actually begun 

and does have a performance measurement program in Washington 

State. I know I have had conversations with Kathleen Sebelius in 

Kansas back at the time, and I believe she applied it to some 

things, as well. 

It is, I believe, part of the National Governors Association, part 

of their Office of Best Practices, and we were able to recruit from 

there not only Malcolm Wolf, our Energy Administrator with whom 

you had some discussions, but also our policy directors from that 

area. The best ideas are the ones that you find from other practitioners. That is what we believe. 

One of the exciting things about beginning this program in Baltimore years ago is that other cities have taken it up, and I have 

found myself learning from things that Mayor Cicilline in Providence was doing to apply this to juvenile justice, to be able to learn 

from Mayor Slay in St. Louis the things he was doing on lead paint 

abatement and the like. 

So I hope, and I believe, that there will be more of this going on 

at the State level that we can learn from. 

Senator CARPER. In Baltimore, I presume this initiative was 

something that you promoted as mayor, and as governor, I presume 

that this is an approach that you have been championing in your 

State. 

Governor O’MALLEY. Absolutely. One of the things I have noticed, Senator, is that newly-elected executives tend to have an 

easier time embracing this than those of us who have been in office 

for some time. The very uncomfortable part of this process is that 

when you begin measuring things that have never been measured 

before and start sharing that information widely and broadly, everyone comes to understand how poorly many things had been 

functioning in the past. The fresh start of a new executive is, I 

think, something that enables an organization to make that sort of 

culture shift in ways that others find more difficult. 

It has also been my experience that however uncomfortable that 

initial period is of the openness and the transparency, which are 

the hallmarks, I think, of performance measurement and a republican form of government, certainly, that openness and transparency over time pays tremendous dividends, however embarrassing the up-front moments are of, oh my goodness, I didn’t know 

that we were that bad at that particular service delivery. Over 

time, people come to respect and appreciate it. 

9 

Another innovation that we borrowed from Mayor Richard Daley 

in Chicago was the use of the 311 center for all city services. In 

the past, we would go around knocking on doors at campaign time 

and giving out the ‘‘where to call for help’’ card. The ‘‘where to call 

for help’’ card had some 300 various services listed alphabetically. 

Look under V for vacant houses. Look under R for rat abatement. 

And then we would have 300 different phone numbers. 

The 311 system on the front end and being able to have a call 

courteously answered and give every citizen, regardless of what 

neighborhood that the call came from, being able to give them a 

customer service number that was common to all citizens and a 

time frame within which to expect that service, whether it was a 

broken curb, a pothole, dead tree, or what have you, a time frame 

within which to expect that service to be delivered was something 

that gave our citizens a lot of optimism about the future and the 

fact that they still do, in fact, have the reigns of controlling and 

holding accountable that government into which they pay their 

hard-earned dollars. 

Senator CARPER. Well, you are going to have a whole crop of new 

governors being elected this November. A couple of weeks after 

that, they are going to show up at the NGA School for New Governors and Spouses and they are going to be looking for things, 

ways to provide good services, working within the constraints of 

tight budgets, as you know. I suspect that you can provide a real 

service for them as you are here today in sharing with them your 

successes in Baltimore and in Maryland. 

Actually, talking about taking good ideas, we used to steal the 

best ideas from one another. Sometimes you would attribute, sometimes not. But we have taken the ideas of New Governors School 

and we have actually incorporated it here in Washington. In the 

middle of November, 2 weeks after the election, we will have, I call 

it New Senators School, but it is actually this orientation for new 

Senators and their spouses, 3 days, very much like NGA. Faculty 

are current Senators and spouses, and basically learn from one another. 

Governor O’MALLEY. And I trust that they put former governors 

at the front of the class? 

Senator CARPER. Actually, three of the people who started it, 

Lamar Alexander, former governor, former NGA Chair from Tennessee, George Voinovich, former governor, former NGA Chair, 

Mark Pryor has very active as Attorney General from the State of 

Arkansas, and yours truly. So three out of the four are all governors. 

In Baltimore, you obviously have a city council, Legislative 

Branch. As governor, you have a General Assembly, Legislative 

Branch. Here, we have the Legislative Branch in which I am privileged to serve. What role in city, State, and maybe in the Federal 

Government, the National Government, can a Legislative Branch 

play? How is this relevant in the lives of the Legislative Branch in 

the city, State, and in Washington? 

Governor O’MALLEY. Initially, like many new ideas, it was greeted with a tremendous amount of skepticism and—— 

Senator CARPER. By your legislators? 

10 

Governor O’MALLEY. Yes. Let me talk first on the city council 

and then I will talk at the State level. In the city council, the concern was that this was going to undermine or somehow diminish 

the ability of council members to deliver for the constituents who 

called them for a variety of city services, and so over time, we were 

able to overcome that, primarily because of the openness and the 

transparency of the process and also the ability of city council staff 

to be able to log into the same system that our 311 operators were 

able to log into in order to give out those citizen service complaints 

and the time frame within which to expect that those services 

would be delivered. 

And once the council staff were trained in it and they went back 

and worked on their council members, a lot of the fears dissipated 

because everybody wants to be able to produce. Everybody wants 

to know that when their constituent calls, that they will be able to 

pick up the phone and deliver. 

On the State level, we recently created a new fund for the restoration and health of the Chesapeake Bay, work that has been 

going on for some time, work that needs to be accelerated and improved and—the tendency, I think, in most legislative bodies is to 

specifically designate any new dollars that are appropriated for a 

given purpose, namely the cleanup of the Chesapeake Bay. In a big 

public health challenge like the Chesapeake Bay, there are probably at least 100 different activities that could be funded that contribute to the health of the Chesapeake Bay, everything from 

stormwater upgrades to cover crops to expanding the forested buffer along streams. 

The legislature, because of the openness, because of the transparency, and because of the performance measurement that they 

saw in BayStat and their belief, their well-founded hope, I think, 

that the deployment of those dollars will be guided by the best 

science and the most effective use of those dollars, they chose after 

some deliberation not to specifically designate this first $2 million 

shall go to this. This next $3 million shall go here to that. 

I think that legislators that have seen the beginning of this process are encouraged that they will be able to get more timely, accurate reports on the things that are working, the things that are 

not, which then will make them much more effective in exercising 

their oversight of the things that we do fund, the things that we 

maybe should fund more of, and the things that perhaps we should 

not be funding as much of. 

Senator CARPER. In our State, and certainly in your State and 

the other 48 States, we have spent a lot of time and energy and 

money on raising student achievement. Most of the States have established academic standards of what we expect our kids to know 

and be able to do in math, science, English, social studies, and 

other curriculum. We designed these tests to measure student performance to try to figure out how to hold schools accountable, students accountable, educators accountable, even parents accountable. We have created enormous amounts of information from the 

results from our tests. Some States give tests on an annual basis 

to selective grades. Now more and more, we are giving at least annual tests to a broader cross-section of students and grades of students. 

11 

In my own State, we have a program called Principal for a Day. 

The State Chamber of Commerce does it in conjunction with our 

school districts. We have a little over 200 public schools. It is a program I participate in almost every year, so last year, I was Principal for a Day at a school called Stanton Middle School, about 10 

miles outside of the City of Wilmington. They had been taken 

under the wing of Selbyville Middle School. Selbyville is in southernmost Delaware, almost in Ocean City. 

Selbyville Middle School had figured out how to raise student 

achievement in ways most middle schools have not, and the results 

were dramatic and impressive and sustained, and the folks at 

Stanton had been struggling to figure out, what are we doing? 

What are they doing right? How can we learn from them? 

What they found out is that the folks at Selbyville Middle School 

in Southern Delaware had figured out how to, not to teach to the 

test, but to say these are our standards. We want our teachers to 

teach to the standards. Then we will have a test and the test will 

reflect the standards. 

But the school—Selbyville was able to figure out how to use the 

information gained from the tests to change the way they taught, 

to change the way they allocate resources, to change the way they 

work with individual students. Stanton Middle School last year— 

in fact, this year, was selected as one of the best, most improved 

schools in our State, and what they learned to do was to take the 

information and to use that to change behavior, not just of the students, but of the educators and the administrators, as well. 

Have you had the opportunity in your State to take this approach of performance information and using that to manage for 

better results and to put it to work in your schools? 

Governor O’MALLEY. To a degree. We attempted with some success to do this in the City of Baltimore and our schools there and 

it was embraced to a degree, especially when it came to some of 

the management functions of the school district. The school district 

in our State that has embraced the notion of performance management, performance measurement most wholeheartedly is Montgomery County under Dr. Weast, and I have had occasion to observe his SchoolStat meetings where he brings in principals, and 

oftentimes it is, as you would suggest, a principal from one of the 

higher-performing schools and a principal from one of the more 

struggling schools, and just as governors and mayors learn from 

one another, principals learn from one another the things they are 

doing and the things that work. 

Montgomery County has done it the best and the most wholeheartedly. Other school districts, I hope will follow on. I was reminded in listening to your story about the two school districts in 

Delaware, one of the leading governors in America on education reform is Mike Easley of North Carolina—— 

Senator CARPER. And he followed another great governor, Governor Hunt. 

Governor O’MALLEY. A good man. 

Senator CARPER. Oh, a terrific education governor. 

Governor O’MALLEY. I once heard him say, and it is true of all 

performance measurement, not just in schools but whether you talk 

about the potholes or the boarding or the other environmental 

12 

things, he said, ‘‘You are not going to improve the weight of the pig 

just by constantly weighing it.’’ 

Applying the standards and the performance measurement is important. Setting the goals is important. But along with that go the 

other tenets of the rapid deployment of resources, having real serious dialogue, conversation, and some experimentation to determine 

which tactics are effective, which strategies are effective, and having the guts to jettison some and embrace others, and doing so in 

a relentless follow-up way. 

There is always a temptation that if we wave the magic wand 

and simply weigh the pig more constantly, that somehow that is 

going to improve the weight of the pig, but the inputs matter. How 

you manage those inputs, where you deploy those inputs do matter. 

But the only way to determine that is by being wide awake and 

aware and having that dashboard that measures the performance 

over time—— 

Senator CARPER. All right. Thank you. 

I think there is a great opportunity in my State to build on what 

has happened at the two schools that I have mentioned. There are 

others, as well. But we are thinking about changing the way we actually administer our tests. Instead of being a test that is given 

out, pen and pencil test given once a year to a third grade, to a 

fourth grade, to a fifth grade, to have a computer-assisted test so 

the students could actually do the test from a computer. They 

would get the results almost immediately. 

And the test is something that if it is happening in the third 

grade and they are performing at a second or a first grade level, 

or a fourth or fifth grade level, that would be garnered and it 

would be a better tool to help the teachers and educators in that 

school to work with an individual student, and it is a test that 

could be given not just once a year, it could be given several times 

during the course of a year, and as you watch somebody who is an 

under-performer move from high first grade performance to low 

second grade, to middle second grade toward lower third grade during the course of a year, you actually see that progress. 

So we think about changing No Child Left Behind so there is 

more—it is there not to discredit or embarrass school districts, but 

to help and to really be a way to help move student performance. 

There is real potential here for applying what you have done in 

your State and city, I think. 

Governor O’MALLEY. May I say a word about the testing and the 

inputs? One of the real benefits that we have seen from the standardized tests over the last 7 years is the positive outcomes in 

achievement that followed almost immediately on the heels of a 

tremendous investment in full-day kindergarten for all children 

throughout our State. 

When I was first elected Mayor of Baltimore, not one of our 

grades scored majority proficient city-wide in reading or math, not 

one. Within the very first year after the first cohort of children in 

our city, which has a lot of challenges with poverty and adult literacy and other things, we saw the first graders score a majority 

proficient in reading and math, and some said, well, that was just 

a blip and once they are longer in the system, they will go down. 

13 

Well, quite to the contrary. As we look over our shoulders now 

from that point in time, that cohort has not only maintained its 

level of achievement, but every class and cohort of kids after them 

have come up to that rising level and it showed us that the investment that we made—we went from about, I think, $3.2 billion as 

a State to roughly $5 billion in State aid to local education, a big 

part of which was that full-day kindergarten, and we have seen 

that investment has brought about these results, including a narrowing of the achievement gap between minority—so-called minority students and non-minority students. 

High expectations are very important and I do believe that expectations become behavior if together we make the right choices 

to make it so. But the testing alone, I think the biggest part of the 

improvement of test scores in our State and certainly in our city 

was a result of the big investment, not necessarily a result of more 

tests. 

Senator CARPER. By having the testing results, when the testing 

results—actually, you trying to find out why are these kids doing 

better—— 

Governor O’MALLEY. Right. 

Senator CARPER [continuing]. In a particular school district, and 

then you look at the number of kids that maybe have the opportunity for full-day kindergarten, or you look at the kids in the 

school districts where they actually have an opportunity to get decent pre–K training, you are right. In my State and your State and 

across the country, one of the best indicators of better performance 

is quality—— 

Governor O’MALLEY. We are also about to borrow another idea 

from Governor Easley, the teacher surveys and learning from what 

the teachers are telling us—— 

Senator CARPER. Excellent. 

Governor O’MALLEY [continuing]. Almost in a sort of corporate 

360 approach. 

Senator CARPER. That is great. A couple questions and we will 

let you escape. Let me just ask, how do you, as chief executive of 

your State, convey your own personal commitment to addressing 

specific performance issues to the agency staff, the folks that are 

responsible for bringing about improvement? How do you do that? 

Governor O’MALLEY. We do it through—you mean actual means 

of keeping engaged as the executive in this process, Senator? 

Senator CARPER. Is this something you talk about in your State 

of the Union? Is it something that is reflected in your budget? How 

do you convey to the people that work in your agencies, not just 

the cabinet secretaries, not just the division directors, but also to 

rank-and-file employees why this is important, why you think it is 

working, why they should do it? 

Governor O’MALLEY. The relentlessness of our meetings—I was 

in a meeting today, for example, with the folks from our Public 

Safety Department, and in our State, Public Safety includes not 

only the prisons, but also parole and probation and all the agents 

that go into the sort of community supervision of people on the 

street. And at that meeting, there were probably some 30 or 40 

people present. Now, not everybody talks at the meetings, but we 

like to think everybody listens at the meetings. 

14 

The fact that at the meetings we have not only an executive summary that is distributed to all parties beforehand, but we also have 

minutes that are taken and sort of an after-action report as to the 

things that we need to prepare for by the next meeting, the questions that need to be answered by the next meeting. That is sort 

of our method, and the fact that we are relentless about it and that 

it is not subject to my schedule. I don’t go to all of them, but they 

always happen. And the fact that they are relentless and that the 

follow-up always happens, and fortunately, I have dedicated people 

like Mr. Gallagher and Michael Enright that continue to drive that 

train regardless of whether I am pulled out of town or to important 

committee hearings in Washington. 

Senator CARPER. I understand. Well, good. How important do you 

think it is to recognize good performance when it does occur? 

Governor O’MALLEY. Oh, critically important. A lot of times when 

people write about this and sometimes when journalists come to observe a session, they focus in on how uncomfortable these sessions 

are for the under-performers. The more significant change, I think, 

comes from the executive and leadership recognizing leadership at 

the department level, at the implementor level. 

Jack Maple had this theory of progress in a big organization. He 

said 80 percent of the people are in the middle, and then on one 

side are the leaders, and on the other 10 percent of the bell curve 

are the slackers. And if leadership and the executive recognize the 

leaders and the achievers, the whole organization tilts towards the 

leaders and the achievers. It improves morale. It improves outputs. 

It improves progress. It improves effectiveness. And that is what 

we try to do. 

So in our city, the methods we used for such things were, I don’t 

know, something as simple as a thank you note when the crews hit 

the 48-hour pothole guarantee. It seems like a little thing. It was 

probably the first time they were ever able to go home to their 

spouses and say, ‘‘Hey, look what the Mayor sent to me.’’ Sometimes we would give tickets to Oriole games or the Mayor’s Box at 

the Ravens game or other things, and we would do it in front of 

people at these events. 

And the people that are the leaders at one level of government 

were also the people that we promote in the course of time that 

hopefully imbues a sense of meritocracy to governments that quite 

honestly in the past, decisions are often made for reasons having 

little to do with the level of achievement, the leadership, or the industry or hard work of the individuals. 

Senator CARPER. You mentioned the Ravens games. The top draft 

pick for the Ravens at the NFL draft this year, they picked a quarterback from the University of Delaware, Joe Flacco. In Delaware, 

people tend to follow the Ravens or we tend to follow the Eagles. 

We will see how it turns out. 

One last thing. Six months from, it will be January 20. I believe 

that is the day we swear in a new President and Vice President. 

The Congress will be sworn in on January 3. So we are going to 

have in the space of a couple of weeks 435 new members, just 

sworn, newly minted House members and about a third of the Senate will be newly sworn in. 

15 

If you were going to give that new President and new Vice President, and maybe even the folks who serve in the Legislative 

Branch, just some friendly advice on how we can take your experience in Baltimore and your experience in Maryland and apply it to 

provide better service, better results, and some things that are important for all of our States and for our country, what advice might 

that be? 

Governor O’MALLEY. I think the most important advice is that 

beyond the guts and the courage that it takes to set goals and beyond the trust in the public that it takes to create a system that 

is open and transparent and performance measured, the third really important part of this sort of performance management is executive commitment. 

This cannot be something that an executive at any level—municipal, county, State, national—does for the sake of a one-day press 

conference or even for the sake of rolling into a budget address or 

a State of the Union. It requires a tremendous amount of executive 

commitment and relentless follow-up by that man or woman that 

is in charge of that government, because if there is not the commitment at the executive, it is very hard for it to transform that pyramid of command and control that is a big Federal Government. 

So those are the three things, Mr. Chairman. Have the guts to 

set goals. Have enough faith in the people we serve to create a system that is open, transparent, and measured for results. And third, 

stay committed as an executive to driving it every day and to never 

waver from that commitment to openness, transparency, and performance measurement. 

Senator CARPER. Good. Among the new governors who will be 

elected this year will be the Governor of Delaware, my State and 

your neighbor. The thought of coming in with a new chief executive, and as good as we think we are, we always know we can do 

better. But my hope is that our new governor will reach out to you 

and to see what he can learn from the experiences that you have 

had and the progress that you have made in your own city and 

your own State. 

One of the things that we are doing—we talked about this last 

week and again today—one of the things that we are trying to do 

in Delaware is endeavor to become the first State to deploy an offshore wind farm about 12 miles off the coast of Rehoboth Beach, 

as you know. Our hope is that we can find some partners to our 

south and to our west and maybe to our east and to our north who 

might like to partner with us. We talked a little bit about this, but 

my hope is that you and that Maryland might like to be a partner, 

at least explore that possibility with us. 

Governor O’MALLEY. Well, we are very interested in that. We 

have the renewable portfolio standard that was just updated, upgraded, increased by our legislature in, I believe, the most recent 

General Assembly session. We have a number of entities that are 

working on renewable sources and hats off to Delaware for leading 

the way on that offshore wind. I hope once we work through our 

request for proposals process, we will be able to join in that. We 

still have a long way to go, but I think it makes a whole lot of 

sense. 

16 

It is certainly needed as we move ourselves into more sustainable 

energy future, and we do so as a region. In fact, that might be the 

only way that this is able to be accomplished, or at least that could 

well be the leading edge of that progress, is when States join together and create the critical mass necessary to prime new endeavors like the wind project off Delaware’s coast. 

Senator CARPER. A couple of years from now, people will look at 

Rehoboth Beach on a clear day like today, and if they can look out 

12 miles to the east, they will see sticking above the horizon some 

small objects about the size of a thumbnail that will be 60 or 70 

windmills that collectively will provide about 15 percent of our electricity needs in Delaware. We believe, and I think you do, too, that 

to the extent that we can have more turbines and look for economies of scale, it is not just advantageous to us, but to our neighbors, as well. So I am encouraged with what you have said. 

Governor O’MALLEY. I am very interested. 

Senator CARPER. Last, going back to football again for just a moment, I will close with this. 

Governor O’MALLEY. I reserve the right to turn to staff. [Laughter.] 

Senator CARPER. Vince Lombardi, that legendary head coach of 

the Green Bay Packers, used to say that unless you are keeping 

score, you are just practicing. And it occurs to me that they didn’t 

just do more than keep score at Green Bay all those years that 

they played so well, but what they did is they kept not just score, 

but they kept the results from how each of the players did, the 

plays that worked, the plays that didn’t, and then they used that 

to change their game plans and really to change their defense and 

their offense. They actually applied the information that they collected. 

We are pretty good in government in keeping score, using the 

football analogy, how are we using the score that we are keeping, 

how are we using those scores and the information that we gather 

to actually change the plays that we call at the line of scrimmage, 

the plays that we call in the huddle, the way we deploy our defense 

or our offense, the kind of players that we draft? It actually does 

apply. It applies in the world of football and it applies very much 

in city government, State government, and I think in the Federal 

Government. 

You have been just a stand-out on this front, and I know the people of Baltimore have recognized that and rewarded you for that 

and the people of Maryland have done so, as well. Thank you so 

much for being here and helping to show us the way as a Nation, 

and we look forward to hearing our second and third panels, as 

well, but you have gotten us off on the right start. 

Governor O’MALLEY. Mr. Chairman, thank you very much. 

Senator CARPER. Thank you. 

Governor O’MALLEY. To the extent that we have been successful, 

it is because of the people we serve and the good people that I have 

been blessed to be able to work with in public service. Thank you 

very much. 

Senator CARPER. Thank you. 

Governor O’MALLEY. Thanks for your leadership. 

[Pause.] 

17 

Senator CARPER. Before I introduce our witnesses on our second 

panel, I want to take just a moment and thank all of our witnesses, 

from certainly the governor and this panel and our third panel, as 

well, for doing something that doesn’t happen every day, and that 

is for providing all of your testimony on time. You are to be commended for doing that and we are grateful. It is a notable achievement. 

Our three witnesses today, I will start off by providing just a little bit of background. Bernice Steinhardt is Director for Strategic 

Issues at the Government Accountability Office, where she is responsible for examining government-wide management issues. For 

over 9 years, she has led the GAO’s efforts in strategic planning 

and helped to develop the organization’s first strategic plan. She 

has held a number of positions at GAO, including Director of Public 

Health Issues Group and Associate Director for Energy, Natural 

Resources, and Science Issues, as well as Environmental Protection 

Issues. 

Before joining GAO, she served at the Department of the Interior 

at the President’s Council on Environmental Quality. Who was the 

President at that time when you were on the Council on Environmental Quality? 

Ms. STEINHARDT. The first time I was there, the Chairman was 

Russ Peterson and the President was Gerald Ford. 

Senator CARPER. You were just a child. Were you an intern then? 

Ms. STEINHARDT. You are very kind. 

Senator CARPER. Russ Peterson is still alive and well. 

Ms. STEINHARDT. Is he? 

Senator CARPER. He is 91 years old and he was a Republican 

then, now he is a Democrat, but I think he was always a Democrat 

if you want to know the truth. 

Ms. STEINHARDT. Well, he was an environmentalist, I can say 

that. 

Senator CARPER. He is a strong environmentalist. But he is still 

just doing incredible things at the age of 91. What an inspiration. 

Ms. STEINHARDT. He was a wonderful Chairman. 

Senator CARPER. He is one of my mentors. 

Ms. STEINHARDT. Mine, too. 

Senator CARPER. Good. Thank you. 

Marcus Peacock is the Deputy Administrator of the U.S. Environmental Protection Agency, where he currently is focused on improving EPA management systems. Previously, he served as an Associate Director in the U.S. Office of Management and Budget, 

where he was responsible for making budget decisions encompassing $160 billion in spending at various Federal agencies. While 

at OMB, he created the Performance Assessment Rating Tool 

(PART), which is used to rate the effectiveness of Federal programs. 

We want to thank Mr. Peacock for really filling two roles, wearing two hats here today, one speaking on behalf of EPA and also 

providing some insight into his former work at OMB. We had invited Clay Johnson of OMB to be with us today, but unfortunately, 

he is out of the country and OMB has kindly provided testimony 

18 

1The prepared statement of Mr. Shea appears in the Appendix on page 131. 

2The prepared statement of Ms. Steinhardt appears in the Appendix on page 55. 

3 Chart referred to appears in the Appendix on page 137. 

for the record from Robert Shea and we look forward to including 

it as a part of our hearing today.1

We are really grateful that you are here, Mr. Peacock. 

Dr. Don Kettl is Director of the Fels Institute of Government and 

the Robert A. Fox Leadership Professor at the University of Pennsylvania, where he specializes in management of public organizations. We consider him a neighbor, too, because he lives only about 

30 miles up the road from us in Philadelphia. 

In 2008, he was awarded the John Gaus Award for a lifetime of 

exemplary scholarship in political science and public administration. He is the author or editor of a dozen or so books and has won 

twice the Louis Brownlow Book Award for the best book published 

in public administration. His newest book, due out in November, is 

entitled ‘‘The Next Government of the United States: Why Our Institutions Fail Us and How to Fix Them.’’ He has consulted for governments at all levels, both here in the United States and abroad. 

Dr. Kettl is currently a nonresident Senior Fellow at The Brookings 

Institute and is a regular columnist for Governing Magazine. 

We are delighted that all of you are here. Your entire statement 

will be made part of our record. You are welcome to summarize 

that, and I would ask you to stick close to 5 minutes, but we won’t 

quarrel if you go a little bit beyond that. 

Ms. Steinhardt, I would welcome your leading this parade. 

Thank you so much for coming. 

TESTIMONY OF BERNICE STEINHARDT,2 DIRECTOR FOR STRATEGIC ISSUES, U.S. GOVERNMENT ACCOUNTABILITY OFFICE 

Ms. STEINHARDT. Thanks very much for having me. I am pleased, 

in turn, to be here today, especially in such distinguished company, 

and to share with you some of the lessons that we have learned in 

examining the progress that the Federal Government has made in 

managing for results. 

Starting at least 15 years ago with the Government Performance 

and Results Act, the Congress and each Administration have put 

into place a framework of strategic plans, performance measures, 

and reports that together have heightened focus on performance 

and accountability for results. So today, I would like to talk about 

where we now stand and to suggest some actions that the next Administration can take to sustain and build on what we have accomplished so far. 

Since 1997, GAO has conducted four surveys of Federal managers across the government in order to gauge the extent to which 

a performance culture has taken hold. Clearly, there has been 

progress compared to what we saw in our first survey, and I would 

call your attention to the chart up top there.3 Significantly more 

Federal managers that we surveyed in 2007 reported having performance measures for their programs, which is to say the basic information that they need to judge how well their programs are 

working. The blue bars are the 2007 results and the white bars, 

19 

1 Chart referred to appears in the Appendix on page 137. 

which you can see outlined there, are for 1997. So there is quite 

a bit of difference. 

But in order for this information to make a difference, it has to 

be useful and used. To borrow Governor Easley’s term, this is 

about weighing the pig. How are we making changes in the pig’s 

weight? So the second chart shows that there are still large percentages of managers who aren’t using performance information to 

a great extent in making decisions, like setting priorities or allocating resources.1

In fact, there has been significant change in only two of these 

nine areas, the ones that are highlighted there. One is an increase—and this is a good thing—in the use of performance information to reward employees. But at the same time, we also see a 

decrease in the use of performance information for making decisions about new program approaches or work processes, which does 

give us some concern, this decline, because this is where you would 

want to see managers using information to change direction when 

performance is not up to the standards. 

From our earlier work, and these latest survey results confirm 

this, we know that there are a number of practices that could spur 

some movement in these relatively static trend lines and lead to 

more widespread use of performance information. First and perhaps most importantly, and Governor O’Malley, I thought, underscored this very clearly, the agency leadership needs to demonstrate its commitment to achieving results. If the leaders don’t 

care or if they don’t show they care, then managers are not going 

to pay much attention, either. In fact, our survey results only underscore this report because those managers who reported using 

performance information to a greater extent also reported that 

their leadership showed a commitment to using the results of that 

information. 

Second, agencies need to link organizational results to individual 

performance. That is a kind of line of sight from the organization’s 

results to individual performance. And here, performance management systems can be used to let employees know how they can contribute to results and then to hold them accountable for doing so. 

Third, agencies need to make sure that they have the capacity 

to collect and to use performance information. Managers aren’t 

going to use the information if it is not timely, if it is not relevant, 

if it is not accurate, if it is difficult to use, or if they simply don’t 

know how to use it. 

Promoting these practices within agencies ought to be a first step 

for the next Administration. Beyond this, though, we would advise 

a few others, again based on insights that we have gained from our 

work over the years. 

First, we would urge a more strategic and cross-cutting approach 

to overseeing performance across government. Many of the challenges that government faces today, whether it is homeland security, emergency response, climate change, all of them involve multiple agencies and programs. While this Administration’s major 

performance improvement initiative, the PART process, has 

brought heightened attention to the performance of individual proVerDate Nov 24 2008 13:48 Jun 01, 2009 Jkt 044582 PO 00000 Frm 00023 Fmt 6633 Sfmt 6633 P:\DOCS\44582.TXT SAFFAIRS PsN: PAT
20 

1The prepared statement of Mr. Peacock appears in the Appendix on page 84. 

grams, we also need an approach that encompasses multiple programs and provides a more integrated view of what government is 

accomplishing. 

Second, performance information has to be more useful to the 

Congress. Whatever performance initiatives the next Administration adopts, the Congress should be engaged in helping to identify 

meaningful measures of success as well as the form in which the 

performance information will be useful to Congress in its oversight, 

legislative, and appropriations role, each of which could require different types of performance information. 

Finally, OMB can do more to build agency confidence in its performance assessments. OMB deserves full credit for its leadership 

in fostering a performance culture across government, but as our 

survey data suggests, many Federal managers still question the 

quality of the PART assessments. The more confidence that managers have in the quality of the assessments, the less they perceive 

them as paperwork exercises, the more likely they are to use the 

results. 

I want to close by noting that our study for you that is continuing, and we hope to be reporting back here early next year on 

additional actions that some agencies can take to increase their use 

of performance information. But in the meantime, we welcome the 

opportunity to work with you and your staff, who have been very 

helpful already in organizing this hearing, and we hope we can 

continue to work with you on these critical issues. 

With that, I will conclude my remarks and look forward to your 

questions. Thank you. 

Senator CARPER. Thank you so much for your coming. Thank you 

very much to you and your colleagues at GAO for your work in this 

arena and for helping us as we approach the change in administrations and the new Congress. Thank you. 

Ms. STEINHARDT. Our pleasure. 

Senator CARPER. Mr. Peacock, you have done good work, both at 

OMB and clearly at EPA on this front, in this and other endeavors, 

too. We thank you for that and we welcome your testimony today. 

Thanks for joining us. 

TESTIMONY OF MARCUS C. PEACOCK,1 DEPUTY ADMINISTRATOR, U.S. ENVIRONMENTAL PROTECTION AGENCY; AND 

FORMER OFFICIAL AT U.S. OFFICE OF MANAGEMENT AND 

BUDGET 

Mr. PEACOCK. Well, thank you, Mr. Chairman. Thank you for 

having this hearing. As you have noted, my name is Marcus Peacock and I serve as the Deputy Administrator at the Environmental Protection Agency. I do sincerely thank the Subcommittee 

for having this hearing. Congress has lots of hearings every year, 

but they tend to focus on how a particular government program 

can do a particular job better. I think this Subcommittee understands that you can come up with similar methods for not just improving one, two, or three Federal programs, but for improving all 

Federal programs. 

21 

The President certainly gets that idea. That is why in 2001, he 

directed that agencies integrate performance information into their 

budgeting decisions. This meant agencies had to identify performance measures for each Federal program and factor that when they 

did their budgets, the ultimate goal was to improve results across 

the government, and at that time the President directed me to lead 

this cross-agency effort. 

The result of the initial work we did was the Program Assessment Rating Tool (PART). PART is essentially a method for assessing the effectiveness of Federal programs, and more importantly, 

making recommendations regarding how those programs can be improved. And since its inception, the PART has won a number of 

awards. It has been copied by other governments. 

I am often given credit for creation of the PART, but I will tell 

you, I was very fortunate to have three things on my side. The first 

was support from the top, and that was from the President. The 

second was to have a very talented team of people at OMB, the Office of Management and Budget, who did the hard work of creating 

and fleshing out the PART. And finally, third, we had a very talented and smart advisory committee, which included Dr. Kettl, who 

kept looking at our work and reviewing it. 

In 2005, I moved from OMB to run the day-to-day operations of 

EPA, and in the last 3 years, I have spent a large amount of my 

time improving EPA’s performance management systems. A number of my predecessors, including Al Alm and Hank Habicht also 

spent—emphasized improving management systems, and hundreds 

of EPA staff over the last 20 years have spent time on this effort. 

So I know better than anyone else that I really stand on their 

shoulders. 

EPA has come a long way. As the governor mentioned, we just 

rolled out EPAStat. It is the first Federal agency to have a Stat 

program. But I would say the biggest problem EPA currently faces 

is to make sure our performance measures are actually used. You 

can treat a measurement system like a thermometer or a thermostat, and, of course, a thermometer measures the temperatures, but 

a thermostat not only measures the temperature, but allows you a 

way of changing it. And performance management systems should 

be thermostats, not thermometers. They should be used to produce 

change so that we become more effective at serving the public. 

Metrics for reporting don’t mean much, but metrics for management, I think, are vital. There are five barriers that I have found 

in trying to get performance information used more, and the first 

is a lack of fresh and frequent data. The Federal Government has 

lots of annual measures, but annual measures don’t work too well 

in running the day-to-day operations of an organization or program. 

There is this Enterprise Rent a Car commercial you may be familiar with where the car is all wrapped up in brown paper and 

I have always wondered how that car stays on the road. You can’t 

look out the windshield or the windows. And I think trying to use 

annual performance measures to manage the day-to-day operations 

of a program is similar to trying to drive that car around. With annual measures, you can’t really see where you are going. 

22 

The second barrier to using performance measures that I have 

run into is a focus on money and not results. Washington, DC still 

operates in a culture that asks how much did a program get rather 

than what did the program produce, and that is not a healthy way, 

I think, of looking at things. Just think about the incentives inherent in declaring an organization more successful simply by its 

spending more money. 

A third barrier is too many meaningless measures. What matters 

gets measured, but if everything is being measured, it is hard to 

tell what matters. So measures need to be meaningful and you 

need to be able to objectively evaluate where the measures came 

from. 

Fourth, too little access. The fewer people that have access to 

performance measures, the fewer people are able to actually use 

them. That is why I believe performance information should be 

made available to the widest possible audience. 

And then finally, the fifth barrier is resistance. Many people naturally worry about the consequences of not meeting performance 

targets. They think performance data may be used as a cover to either cut funding or punish people. And the fact is that these systems can be abused, but that is a very poor reason to avoid them. 

An organization cannot become excellent without measuring its 

performance. If you can’t see what you are doing right, then you 

won’t learn from that. 

In conclusion, I am fortunate to work at a place like EPA. The 

employees love the mission of the agency. They are very resultsdriven, so that if you can show them that you are going to be able 

to improve the results, they are willing to change what they are 

doing in order to get there. They understand that when EPA works 

better, public health and the environment improve faster. And EPA 

management initiatives that aren’t linked to results are just 

gobbledy-gook. You need to demonstrate that they will lead to 

cleaner air, water, and land. 

We need to get to the point where all Federal agencies practice 

good performance management. Whether the latest and hottest 

issue is homeland security or securities regulation or climate 

change, a functioning performance management system is invaluable to helping any Federal agency, no matter what its work, do 

its job better. We need to get to the point where there is always 

an answer to any Federal employee who asks the question, ‘‘How 

can I do my job better?’’ 

Thank you. 

Senator CARPER. Thank you very much. I want us to come back 

during the questions and answers. Mr. Peacock has just run 

through five reasons why we don’t always use information to help 

us improve our performance and I am going to ask our other witnesses to comment on those, if you would. 

Dr. Kettl, you were good enough to spend some time with Wendy 

Anderson and myself in our office earlier this week. We are grateful for that and for your years of work in this area and for all that 

you have written and lectured. We appreciate your being here 

today and helping us to do better, to do our job better. Thank you. 

You are recognized. 

23 

1The prepared statement of Mr. Kettl appears in the Appendix on page 86. 

TESTIMONY OF DONALD F. KETTL,1 DIRECTOR, FELS INSTITUTE OF GOVERNMENT, AND ROBERT A. FOX PROFESSOR 

OF LEADERSHIP, UNIVERSITY OF PENNSYLVANIA 

Mr. KETTL. Thank you, Mr. Chairman. It is a great pleasure to 

be here today. I want to thank the Subcommittee for this opportunity to explore what in many ways is the most important issue 

facing this government, which is how to make sure that it works, 

how to make sure it delivers results and value to citizens. It may 

very well be the most important question we never stop to ask 

until it is too late, and smart government means getting out ahead 

of that to make sure that we are prepared for the challenges that 

we know that we need to find a way to be able to master. 

I want to make several points here today as we try to explore 

that question, the first of which is that if you look from a 30,000- 

foot level at the question of government reform, we are, in all likelihood, at the end of a natural life cycle of Presidential management reforms. For the first time since the Eisenhower Administration, we are now in a place where the next thing that the Federal 

Government might do, the next thing a new President might do is 

obvious. 

President Eisenhower had the Hoover Commission. President 

Kennedy had his whiz kids. President Johnson had planning, programming, budgeting systems and budget reform. President Nixon 

had management by objectives. President Carter brought in zerobased budgeting from Georgia. The Reagan Administration brought 

in a whole wide privatization initiative. At this point in the campaign back in 1992, it was very clear that President Clinton was 

going to reinvent government based on the best seller. And by this 

point in the campaign in 2000, President Bush had already delivered a major speech on the management of the Federal Government. 

It is not clear what it is that is going to happen next, but it is 

clear that what we have done so far has accomplished a lot, but 

that we are now at the point where we are going to need to do 

something different, that more of the same kinds of reforms are not 

likely to get us to where we need to go. 

So the first point is that we are at a crucial turning point because we are at the end of a series of natural life cycles in the reform movement. 

The second thing is that we have to find a way to get very smart 

very fast about what kind of reform we need because we face a set 

of wicked problems that will prove especially punishing if we do not 

find a way to get ahead of these problems. We face problems 

whether we are talking about tomatoes in the supermarkets or 

hurricanes like Katrina that come up very quickly, that are very 

large, that provide little time to react, and that impose large consequences for failure. 

And so we are at a point where we need to find a way to do 

smart things fast to be able to deal with these issues that we know 

we are going to be facing, and we know that more of what we have 

been doing is simply not going to do the job. 

24 

The third point is I want to look in particular at what the reforms in the Clinton and Bush Administrations have produced. 

Both of these are tremendously importantly initiatives that in 

many ways, although they may not seem like it from a distance, 

are part of the same broad fabric of looking at results and trying 

to motivate managers to do better. 

The Clinton Administration brought a strong focus on results for 

the first time to the entire Federal Government. It encouraged Federal managers to innovate. It brought an expanded customer service initiative to the government, ranging from Social Security to the 

IRS, in ways that were truly breathtaking and important. But at 

the same time, one of the things the Administration did was to engage in substantial downsizing of the workforce without engaging 

in a right-sizing of what that workforce ought to look like. 

As a result of that, the Government Accountability Office has 

now named human resource management one of the high-risk 

areas that the Federal Government faces, the issues that are most 

likely to cause the biggest problems in management, and GAO has 

identified this and in many ways it is the consequence of trying to 

figure out now what kind of government do we need. We clearly 

can’t downsize any further without thinking about how to right-size 

what it is that we do, and that was a major question that the Clinton Administration left unanswered. 

In the Bush Administration, the PART process we have heard 

discussed here this afternoon was a tremendous breakthrough. For 

the first time, all Federal agencies had to try to assess goals for 

all their programs. They had to measure what it is that they were 

trying to accomplish and focus on activities, and it was government-wide in the scale that was anticipated in some ways by the 

Clinton Administration but never fully realized. And two other 

things happened tied to the Office of Management and Budget and 

tied to the budgetary process. Both of these were tremendously important and major breakthroughs. 

The piece that we have left unanswered here is trying to find 

ways of separating out the ideological commitments to some parts 

of the management agenda, for instance, increasing contracting 

out, from the question of what it is that we want the contracting 

out process and other management processes to accomplish, focusing on these goals and objectives, and ensuring that we have a 

focus on activities that cut across government agencies. 

The basic fact, for better or worse, is that there is no program 

that can adequately encompass any problem that matters. Or put 

differently, there is no problem that matters to anybody that any 

one Federal agency, any one level of government, any one unit can 

possibly control. That was a lesson from Hurricane Katrina and it 

strikes through every single important thing that the Federal Government does. 

So the problem is trying to figure out how to get leverage over 

these long and complex chains that typically require action across 

many agencies, and the piece that lies next after the President’s 

management agenda in this Administration is finding a way to get 

to that cross-cutting piece. What the PART process was terrific at 

was the vertical integration of knowledge. The next step has to be 

25 

the horizontal management coordination to try to ensure that kind 

of collaboration. 

What we need, in short, is my fourth point, actionable intelligence, trying to find ways of finding out the things that we can 

do something with, and once we figure out what it is that we need 

to know, getting action to happen. We need people—this is a lesson 

from the Clinton Administration—we need Federal managers who 

can lead. The difficulty is that we have a lot of very smart people 

who are doing very good things, but often out there feeling that 

they need to work against the odds. And what we have to do is find 

a way to help them to do what it is that they know they need to 

know and know what they need to do in a way that makes it easy. 

We need information that focuses government on results instead 

of collecting information within silos. We need to drive that results 

to try to improve the way the government works instead of simply 

producing, as we have been discussing here this afternoon, more 

weights on the pig without making the pig any better. 

We need, finally, a focus on collaboration that builds cooperation 

along a horizontal chain against all those people, all those organizations, all those agencies, to be part of a solution to ensure that, 

in the end, the problem gets solved instead of just the agency gets 

managed. 

We, in short, need to try to find ways of developing a new generation of reforms. More of what we have been doing is likely to 

produce diminishing returns because, as GAO is pointing out, we 

are collecting more and more information, but it is not like people 

are doing anything much with it. What we need to do is figure out 

how to make sure that the information that we have is information 

that improves the way that government works. 

We need results that matter to people, which gets to my fifth 

point about trying to figure out how to make that work. First, we 

need to talk about what it is that we are trying to do. Often, a 

focus on results within individual agencies has to do with how 

many checks have been mailed, how many grants have been processed, how many forms have been filed, but in the end, none of 

that matters unless the problems get solved. 

The fact is that most citizens don’t care about what it is that the 

FAA does, but what they want is when they go to the airport, they 

want to be able to take off to where they are going and land at the 

destination safely. They don’t care about the FDA, but they would 

like to be able to eat tomatoes safely without having to worry about 

it. Citizens care about results and we need a government that is 

driven toward those results. 

The second point within this, though, is that the chain of producing accountability toward those outcomes, which is what people 

care most about, is often long and complex, where individual agencies, individual units contribute pieces, but nobody controls the 

whole thing. That means we need to find a way to hold each individual player accountable for their contribution to the outcome as 

a whole. We need to know what we are trying to accomplish and 

understanding what each individual’s contribution to that is. 

This means ultimately, as my third point here, that the government needs to get leverage over the outcomes that it wants to try 

to accomplish. One of the great findings and discoveries about EPA 

26 

is that EPA is most effective not when it does it itself, but when 

it enlists its partners toward the common pursuit of making the environment clean. 

What is it this ought to look like in the next Administration? For 

the President, there are several things that could happen. The first 

is a new generation of reforms that understands the importance of 

what has happened before but the need primarily first to focus on 

outcomes and results that matter. 

Second, a kind of geographic information and performance stat 

for those parts of the Federal Government that make sense so that 

we can look at what it is that matters and measure in real time 

today how well we are doing it because if we wait until the end 

of the year for a GPRA report to produce some kind of results, by 

that point, it will be too late to change what it is that needs to be 

fixed. 

We need an Office of Personnel Management that is more robust 

and develops leadership in the Federal Government, and we need 

a focus in the White House in particular to have somebody in the 

White House who can focus on the issues of making the government work and making that important for the President. 

For Congress, we need a system of the Government Performance 

and Results Act that focuses on results that matter, hearings that 

on every occasion ask people what is it that we are trying to accomplish and how well are we doing it, and a focus on programs and 

agencies that put outcomes that matter for people at the forefront 

of things. 

We have plenty of information. We need to make it actionable 

and find ways of producing results for the citizens in the end, because that is what it is that the people expect. 

Senator CARPER. Excellent. Thank you very much for really three 

terrific testimonies. 

I want to go back to, as I said I would, to Mr. Peacock. You mentioned five barriers to agencies taking this approach and improving 

their outcomes. Would you briefly mention those again, and then 

I am going to ask our other two witnesses to comment on them. 

Mr. PEACOCK. Sure. The first one is lack of fresh and frequent 

data. The second is a focus on money, not results, and you could 

broaden that to inputs, not results or outcomes. Third is too many 

meaningless measures. Let me know if I am going too fast. Fourth 

is too little access. The governor talked about transparency. It is 

the same issue there, too little access to the measures. And then 

finally, just resistance, organizational resistance to change. 

Senator CARPER. Let me just ask either Ms. Steinhardt or Dr. 

Kettl, I want to ask both of you, to react to those five points, if you 

would. 

Ms. STEINHARDT. I think our work supports every single one of 

those points. In fact, where we have seen greater use, real live use 

of performance information is where agencies have been able to 

overcome some of those kinds of barriers. 

It is very important in agencies or organizations to be able to 

communicate the kind of performance information throughout the 

organization, making it transparent to everyone, and we have seen 

some good examples of how agencies do that, communicating frequently to all of its employees where they stand. It helps spur comVerDate Nov 24 2008 13:48 Jun 01, 2009 Jkt 044582 PO 00000 Frm 00030 Fmt 6633 Sfmt 6633 P:\DOCS\44582.TXT SAFFAIRS PsN: PAT
27 

petition. It provides good benchmarks for units across the organization. It has real benefits and a number of agencies have been able 

to do that. 

Let me consult my list again here. 

Senator CARPER. Go right ahead. 

Ms. STEINHARDT. Real-time information, absolutely. This was a 

point that I was making, I think, earlier, where the information 

isn’t timely, credible, accurate, it is not going to be used, and organizations that figure out how to make that information available on 

a daily or real-time basis are ones that are going to see that it is 

used, and Governor O’Malley talked about how that concept is applied and was applied in Baltimore and now in StateStat. We have 

seen some very good examples of that, as well. 

Too many measures, that is sort of a basic precept of performance measurement. The vital few. You can’t expect people to focus 

and really align themselves with what the organization is trying to 

accomplish if it is diffuse and scattered. 

Senator CARPER. Good. Thank you. Dr. Kettl. 

Mr. KETTL. I simply couldn’t agree more. That is exactly the core 

point here. The real issue here is that any time the process of performance descends into questions of measurement, we have in a 

sense lost the game because we get away from what it is that performance is all about. It is about communication. It is about talking 

about what it is that matters and figuring out how that communication gets people to act. And so as we tend to slice and dice 

things into ever smaller pieces and measure more things, we are 

likely to get to a point where we are unlikely to get people to act. 

Performance is about communication. It is not primarily about 

measurement. 

What is it that communication ought to be about? The key to 

overcoming resistance is primarily one of getting people to agree on 

what problem they want to try to solve and how to go about trying 

to do that. 

One of the examples that while mayor, Governor O’Malley 

worked on was something called the rat rub-out program in Baltimore. They had rats that they wanted to rub out. They had maps 

that showed where citizens were complaining about problems of 

rats. Which department in Baltimore is in charge of rats? The answer is, none of them are. On the other hand, it is a function of 

public health, of housing, of redevelopment, of transportation, of 

the people in charge of water and sewer. It is a cross-cutting collection of agencies which, put together, have to be responsible for solving the problem. 

If you ask everybody, do you want to collaborate? Everybody, of 

course, always says yes. But getting it to happen in real time and 

in real life is very hard to do. You put a map up on the board that 

shows, here is a problem with rats. Is anybody happy with that? 

No. What we are going to do about that? We want to get together 

to solve it. Who is going to solve it? Everybody understands their 

particular contribution to solving that problem, and that is what it 

is that drives it. It is communication about results that matter and 

getting people to put themselves to work to solve the problems that 

matter. 

28 

So the reason why so often performance management doesn’t 

work is it becomes issues of measurement and that conversation 

never happens. 

Senator CARPER. Ms. Steinhardt, and then Mr. Peacock, you may 

want to say something here when she is finished. 

Ms. STEINHARDT. I just wanted to add to this concept of visibility 

of information, because I think that is really also extremely key 

here. I came across this article a couple of days ago in the Wall 

Street Journal about how U.S. Airways has now gone from the bottom of the list of airlines in on-time performance to the top so far 

in 2008. What is really striking to me—the article is actually very 

interesting in discussing how it did that—is the fact that everybody 

knows about on-time performance. It is a very widely-used statistic. 

And in the airline industry, it is one of the key indicators of their 

viability as a company. It really matters to the CEO and everybody 

in the company, how their on-time performance is ranked. It is the 

visibility of the information, I think, that really makes such a big 

difference. 

Senator CARPER. Excellent. Mr. Peacock, do you want to add anything before we move to another question? 

Mr. PEACOCK. The governor mentioned that EPA had rolled out 

an EPAStat program and he also mentioned two other things. He 

said it is easy to take any measurement program and put the ‘‘stat’’ 

label on it, which is true. But he also mentioned that if you are 

going to do this seriously, you really have to be committed to it and 

have some core elements that you are committed to to make it 

work. 

Across these five barriers we have been talking about, EPA, 

through its EPAStat program, I think, is probably tackling—has 

tackled about three-and-a-half of them. But when I got to the agency, regarding lack of fresh and frequent information, I went in hoping to get monthly information. The most we could tolerate and are 

tolerating right now is quarterly information, which still took a lot 

of work to get. If it was any less frequent than that, it wouldn’t 

work. 

The number of measures we had was too many. We have been 

able to cut it by 15 percent. We will continue to decrease it so that 

it becomes more manageable. 

In terms of access, the relentless meetings, EPAStat meetings we 

have, and they do have to be relentless—this gets back to Dr. 

Kettl’s point, which I think is critical. This is all about communication. We have meetings—I have meetings personally with each assistant administrator and regional administrator every quarter on 

these statistics, but the richness of this is in discussing them and 

having a conversation about what is going right, what is going 

wrong, and what we are going to do about it. 

We now broadcast those meetings internally to any EPA employee who wants to view them, and our quarterly management report, as far as I know, we are the only Federal agency that on a 

quarterly basis puts out our performance measures to the public for 

anybody to look at. So we are very strong on the access. 

And then in terms of resistance, once again, I just happen to 

work in a culture where people are willing to see if changes results 

in us meeting our mission. There is just not a lot of resistance. 

29 

But we still have a problem, I think, with focus on money and 

inputs rather than results. It is the sort of culture we run into, not 

just at EPA but throughout the Federal Government and outside 

the Federal Government, where people want to know, well, how 

much money are you asking for, when really the first question 

should be, what are you planning on delivering next year? 

Senator CARPER. So true. Why do you suppose that is the case? 

Mr. PEACOCK. If I knew that, I wouldn’t be at three-and-a-half 

of these solved. I would be at four-and-a-half of them solved. I 

mean, I would be interested to hear why other people think that 

is the case. 

I will tell you, my experience in the OMB, where we realized this 

was a big problem, I mean, when the budget comes out every year, 

still, the first thing that is talked about is the dollars that are 

being proposed, not what we are going to get for it. So it is just 

ingrained in the way we think about things and I think it is going 

to be incremental, but it is a change that has started to take place, 

but it is going to have to keep taking place. 

Senator CARPER. Dr. Kettl, do you want to respond to that same 

question that I just asked? 

Mr. KETTL. I think your question is why is it we focus so much 

on inputs—— 

Senator CARPER. Yes. 

Mr. KETTL [continuing]. And it is like the old joke about the 

drunk looking for the keys underneath the lamppost. Is that where 

you lost them? No, I lost them over here. Why are you looking 

under the lamppost? Because that is where the light is. People 

focus on the things that are easy to look at and easy to measure 

because OMB at least publishes a budget that has all the numbers 

in it and it is easy to say, how do we know we care about something? Because we are spending a lot of money on it. Then we wonder why it is that we are not producing good results. Well, because 

that is not the function of what it is that we are driving the system 

to. 

But what we need is, in a sense, an alternative way of thinking 

about accountability. What citizens care about—what citizens know 

about is that they want government to work, and in the end, quick 

headline, increases, decreases, symbols that are valuable. But if it 

doesn’t make the government work better, citizens don’t care, and 

in fact, it makes them even more cynical. 

I think the main reason why we do it is, first, because the numbers are there and they are easy and there is no clear alternative 

otherwise, and second, it has to do with the fact that we don’t have 

an obvious alternative accountability mechanism that focuses on 

outcomes. Why? Because we have lots of measures, but we haven’t 

figured out what to do with them yet. That is the next generation 

of reform that I think we need. 

Senator CARPER. OK. Let us talk about the importance of leadership. I don’t think the City of Baltimore would have gone to 

CitiStat without the leadership of their mayor. My guess is that 

the State of Maryland wouldn’t have gone to StateStat without the 

leadership of Governor O’Malley. 

In Federal Government, we elect a President every 4 years, we 

elect or reelect. We have six-year terms here in the Senate and 

30 

two-year terms, as you know, in the House. Folks who serve in the 

Executive Branch as cabinet secretaries are—they can serve for 4 

years, occasionally 8 years, but more often than not they serve less 

than 4 years. Folks that are deputy secretaries and associate secretaries and under secretaries, they tend to come and go. 

How important is leadership and the continuity of leadership at 

the top and then down to different levels of government here in 

Washington? How important is the consistent message, that this is 

important, we have got to do this, and to hear that not just from 

the very top of the government, but also the top of each department 

and with the agencies themselves? How important is that? Please. 

Ms. STEINHARDT. I would say it is absolutely the first step. It is 

the precursor. Nothing else—none of the other recommendations, 

none of the other practices that we talk about would be successful 

if the leader of the organization doesn’t care about this because nobody else will if the leader of the organization does not. It doesn’t 

end there, but it starts there. 

Senator CARPER. All right. 

Mr. PEACOCK. I agree. It is a prerequisite, but there are two 

ways to institutionalize these systems and this way of thinking. 

One is through standard operating procedures and organizational 

structure, so for instance, at EPA, we now have put in place as 

part of a regular annual process we go through scrubbing our 

measures to make sure that they are meaningful, and that is actually the process by which we are reducing our measures. 

And second, in terms of an organization, for instance, we have 

now created a Program Analysis Division at EPA which essentially, 

with the Office of the Chief Financial Officer, makes the system 

run, produces the statistics. It is essentially like the EPAStat Office or the analogy which these cities and the State of Maryland, 

for instance, have. 

And then the second way to institutionalize this is the career 

people. Obviously, the policy officials, the political appointees aren’t 

there very long, but we have career officials and managers who, at 

least at EPA, have been there a very long time. Particularly as the 

value of these systems are demonstrated to them, they become believers and advocates for them, and whether or not you have what 

the political appointees may be doing, or even if the standard operating procedures are in place, if this is a value to good managers 

in terms of managing their programs, then these systems will be 

used. 

Mr. KETTL. Mr. Chairman, the people question is absolutely essential. There is a paradox in the way in which the Federal Government has evolved, I think. As it has gotten more complex, the 

role of people in leadership has become more important. We had 

this model for a long time that said we will create a very complex 

system and we will slide people in and out in standard operating 

procedures because they will be pretty much interchangeable. The 

idea of being able to create a civil service system and all kinds of 

standard operating procedures were not to make people unimportant, but to make it less important who was serving in those positions. 

It is increasingly important now, the more complex the system 

has gotten, to have the right people, smart people in the right 

31 

places to drive things forward or otherwise things just don’t happen, and the way in which it does happen, and, for example, the 

Coast Guard has proven this, is making that part of the organization’s culture. When the standard operating procedure becomes a 

culture that reinforces the pursuit of strong outcomes, that is what 

it is that produces this piece. 

Unfortunately, too often in the Federal Government, it is something that happens either by accident or by people having to do it 

despite the fact that the game somehow seems rigged against 

them. We have got to find a way to institutionalize strong performance-oriented cultures into the very core of the way government operates. 

Senator CARPER. In looking at our third panel and thinking 

about the agencies that we are holding up here for acclaim, for applause, because they have gotten it and they are really setting an 

example for the rest of us, EPA is one of those and I think it is 

fair to say that, Mr. Peacock, he probably wouldn’t admit this, but 

he is a big reason why EPA has moved down this road and has 

good results to show for it. 

But among the other agencies that are here, the Nuclear Regulatory Commission, NASA, the VA, and I wonder if there is any 

common thread with all of those or if you can think if maybe there 

is. I am not sure, but I think there might be, but any thoughts? 

Mr. KETTL. Mr. Chairman, I think it is in some ways a very simple answer to a very complex problem. They know what they are 

trying to do. They focus relentlessly on doing it. They reinforce people for doing it. They keep track of how well they are doing it. And 

it is a simple process that focuses on results and having a culture 

that reinforces what they are doing and they focus on what their 

outcome is. 

Their job is not simply to, at the VA, for example, taking a stack 

of papers on veterans’ benefits from Part A and moving them over 

to Part B. They focus on the veterans. They have had a system, for 

example, of veteran-centered health care for a long time that is 

what drives them and drives what it is that they do. They know 

what they are there for and they know who they are trying to serve 

and they focus the system on serving them. And developing standard operating procedures and leadership to try to do that is the key 

to making a high-performing organization. 

Senator CARPER. Anyone else? 

Mr. PEACOCK. I think Dr. Kettl is exactly right. A clear mission 

that everybody can see and believes in has got to be a prerequisite 

for this. I don’t remember who mentioned it, but this clear line of 

sight idea that everybody in the organization can see not only the 

mission of the agency, but how it connects to what they do on a 

day-to-day basis, and that is a prerequisite for this. 

There are other agencies that have a clear mission and I think 

everybody sees it but aren’t necessarily doing a great job at this. 

So while it is a prerequisite, I don’t think it is sufficient. 

Senator CARPER. Ms. Steinhardt. 

Ms. STEINHARDT. I agree completely with Dr. Kettl and Mr. Peacock’s comments. I would add, though, and I don’t necessarily apply 

this to VA and NRC, perhaps to NASA, but sometimes it is a crisis 

32 

that focuses an agency on what it is about and what its mission 

and what its desired goals ought to be. 

This happened to the IRS, actually, in the mid-1990s. There were 

oversight hearings, a lot of press attention, Congressional attention 

on allegations of taxpayer abuse and dismal customer-taxpayer 

service and IRS had no information, performance information, to 

refute all of these horror stories that were coming out. And the 

Congress itself, I think, played a very key role in setting some very 

specific performance targets for the agency, and the agency began 

to collect the information, became very performance focused and it 

really turned itself around. 

In the period certainly from the 2000 survey, when we started, 

when we were asking about managers’ use of information there to 

the most recent survey, they have showed a really positive improvement there, and more significantly in actually how the agency 

is performing, they have really made great strides. 

Senator CARPER. All right. Well, good. There is still plenty of 

work to be done. Every year, we hear the size of the tax gap, and 

we learned last year it is over $300 billion, monies that are owed, 

not being collected. We deal with this on this panel. We deal with 

the improper payments. Not everyone, not all of the Federal agencies are reporting their improper payments, but of those who do, 

we know that our improper payments are, I think, roughly $50 billion a year, mostly overpayments. And it occurs to me that if somehow there were no tax gap and if we weren’t making these improper payments, we would basically have a balanced budget. 

Ms. STEINHARDT. But today, we know the size of the tax gap, or 

at least now IRS is able to estimate the size of the tax gap. 

Senator CARPER. We weighed the pig. [Laughter.] 

Ms. STEINHARDT. Well, first step. 

Senator CARPER. There you go. A couple more questions and then 

we will excuse this panel and invite our last panel to the table. 

Ms. Steinhardt, in light of your survey results showing improvements in the amount and the types of performance measurements 

that agencies have at their disposal, why do you think so little 

change in managers’ use of that information is being displayed in 

their decisionmaking? 

Ms. STEINHARDT. Well, just to comment on the survey results, a 

lot of the results—the average, rather, is affected by the fact that 

we have a couple of large agencies like DOD and the Interior Department that are low, relatively low users of performance information. The smaller agencies like NASA, NRC, EPA, tend to be higher, but they don’t color the average so much. 

In terms of why we are not seeing so much progress or use in 

those other agencies like Interior, for example, that is the subject 

of the second part of our study, so we hope to be back here. So we 

hope to be back here with some more concrete information on what 

those agencies can do to improve. 

Senator CARPER. All right. Thank you. 

Mr. Peacock, how do you suppose the next Administration can 

get agency leaders to buy into such assessments and use a review 

process in resulting performance information to improve their management decisions? 

33 

Mr. PEACOCK. I mean, there are so many directions one could go 

in that would be worthwhile, although I think that Dr. Kettl hit 

on some of them. We now have a wealth of information that wasn’t 

there before, and looking at this information but from a cross-agency perspective, looking at it from the sense of the public who is 

more interested in protecting wetlands rather than whether it is 

the Corps of Engineers or the Department of Interior or EPA that 

is protecting the wetlands and being able to compare across agencies which programs are doing this the best, doing it the most efficiently, and then using that information in terms of making budget 

decisions or policy decisions. 

I mean, there would be any number of ways to do that, but there 

probably needs to be less emphasis now on trying to collect this information and more emphasis on actually bringing it in-house and 

someone sifting through it to actually use it and make decisions 

using it. 

Senator CARPER. OK. Thank you. 

And finally, one more for Dr. Kettl, if I could. In your work on 

how government responded to Hurricane Katrina, you have described the importance, and I am going to quote here, ‘‘the importance of working in horizontal networks instead of struggling over 

a vertical chain of command.’’ As GAO has testified, there are 

many programs that need to work together to accomplish common 

outcomes. I think Mr. Peacock has said as much, too. What are 

some approaches the next Administration could take to bring about 

a more cross-cutting focus on performance issues? 

Mr. KETTL. The first thing, Mr. Chairman, is to underline the 

importance of these cross-cutting approaches to begin with because 

increasingly, what it is that government does has to happen 

through those relationships. 

The second thing is to focus on a smaller number of more important outcome-based measures, less on how many applications for 

processing, more on what kind of results we are producing, getting 

the information in real time, getting the information displayed 

graphically. One of the things about Mayor and then Governor 

O’Malley’s work is that if you look at the maps, you just look at 

it and it talks to you. It tells you what the problem is and where 

to go about doing it. 

For example, he had a series of charts about the decline in the 

murder rate in the three kidney-shaped areas that he talked about. 

You look at that and you see it, you can see the change, you can 

see the progress, you see where the problem is. It tells you what 

it is that you need to work on and where you need to go. There 

is something about information that is focused on what matters, 

that is real time, that focuses on where it is that it happens that 

drives the collaborations that are required and then makes it possible to have the conversations about who it is who is going to contribute to the solutions to the problems that in the end matter. 

That is the secret to the successes in EPA’s work, of trying to figure out what are we trying to do. How can we make the air cleaner? Who has got a piece of the action? How can we measure whether or not we are doing it? And it is that kind of relentless cycle of 

follow-through, not once a year, once every 5 years collecting some 

interesting reports and putting it on a chart. It has got to be realVerDate Nov 24 2008 13:48 Jun 01, 2009 Jkt 044582 PO 00000 Frm 00037 Fmt 6633 Sfmt 6633 P:\DOCS\44582.TXT SAFFAIRS PsN: PAT
34 

time information that drives action and drives action the citizens 

care about. 

I can say that with confidence not because I know it to be true, 

but because I have watched others do it. The successful managers 

around the Federal Government are doing that and more of that 

is what it is that we need. 

Senator CARPER. I will just close with an observation. When I 

was privileged to serve as the chief executive of our State, there 

are a number of goals that we set. We wanted to raise student 

achievement, and we didn’t want to just put it all on the schools 

or on the Department of Education. We wanted to reduce the incidence of teenage pregnancy. We just didn’t want to put it all on the 

kids department or on the Department of Health and Social Services. We wanted to reduce the incidence of recidivism in our prisons, but we didn’t want to put it all on our prisons. We wanted to 

reduce the crime rate, but we didn’t want to put it all on our police. 

And we ended up establishing a Family Services Cabinet Council 

which included about half of the cabinet, and I would meet with 

them every month and we would set measurable goals and then try 

to hold one another accountable for progress. But I think the fact 

that I sat with the cabinet secretaries every month and then they 

would meet once a month without me with members of my governor’s office staff just to follow up had some pretty good effect. 

There is something to be said for trying to hold a particular 

agency responsible for getting, say, ‘‘A’’ done, but sometimes ‘‘A’’ 

needs to be done in collaboration with a bunch of other folks. So 

you need somebody saying, you guys and gals have to work together. We do have stovepipes in the Federal Government just like 

we did in State Government and they are hard to get rid of. 

I want to thank you so much for your testimony today, really for 

your terrific work. Ms. Steinhardt, to you and your colleagues at 

GAO who are just enormously helpful to this Subcommittee and 

really to our Executive Branch, too. To Mr. Peacock, who has 

worked in OMB and done great work there and now gone to EPA 

and helped to show as a practitioner what we can do and always 

is a source of inspiration, actually. And Dr. Kettl, we have a lot of 

people who testify before us. Few are as clear and concise, particularly academics—— [Laughter.] 

Who are as concise and able to put in terms that everybody, even 

I, can understand. It is a real gift. So we thank you. It is easy to 

see why you have had real success and are highly sought after. 

I want to leave the record open for a week or two so that folks 

who are not here can have a chance to submit some questions, and 

if you get those and would respond to them in a timely way, we 

would be most grateful. 

But thank you so much for joining us today. Thank you. 

Ms. STEINHARDT. Thank you. 

Mr. PEACOCK. Thank you. 

Mr. KETTL. Thank you, Mr. Chairman. 

Senator CARPER. To our third panel, welcome. Thank you for 

hanging in here until almost 4:30. We promise not to keep you 

much longer, but thanks a lot. Have you all been here for the presentations of our first two panels? 

Mr. TUCKER. Yes, sir. 

35 

Senator CARPER. Good. Well, a special thanks. 

One of the questions that I asked is, what are the common 

threads between the agencies that we are really holding up for acclaim here, who seem to get it and have been achieving better results because of that fact, in terms of measuring outcomes and actually using the information to change performance. That is one of 

the questions that I am going to be asking each of you, so be thinking about that, if you will. 

Jim Dyer is the Chief Financial Officer for the Nuclear Regulatory Commission. We were privileged in another committee that 

I serve on and chair, a subcommittee with Senator Voinovich, to 

have a chance to work with your agency a lot and have a very high 

regard for you. What is it, the best agency in the Federal Government in which to work? 

Mr. DYER. The best place to work. 

Senator CARPER. The best place to work, yes. But there, you 

serve as the person who is responsible for planning, for budgeting, 

and financial management of agency resources. You are also, I understand, currently the agency’s Performance Improvement Officer, 

responsible for leading the agency’s performance management activities. 

Prior to your position as Chief Financial Officer, I understand 

you served as Director of the Office of Nuclear Reactor Regulation, 

where you were responsible for agency safety programs for the 

commercial power, research, and test reactors of the United States. 

A couple of people suggested to me you also may have at one time 

served in the U.S. Navy and that you were a submarine officer 

from 1973 to 1977. I was on active duty from 1968 to 1973 looking 

for submarines and had a pretty good naval flight officer, P–3s, 

Orions, and our job was to hunt for Red October. We found plenty 

of them, but whenever we were looking for you guys, for our guys, 

we could almost never find you. You are mighty good at what you 

did. You served in the Naval Reserve until your retirement in 

1995, and I served in the Reserves until my wife made me quit in 

1991. But thank you for your service then and for your service 

today. 

Scott Pace is the Associate Administrator for Program Analysis 

and Evaluation at NASA. He is responsible for providing objective 

studies and analyses in support of policy, program, and budget decisions by the NASA Administrator. 

He previously served as Chief Technologist for Space Communications, where he participated in the negotiations that resulted 

in the 2004 GPS–Galileo agreement between the U.S. and the European Commission, and Mr. Pace also previously served as the 

Deputy Chief of Staff to NASA Administrator Sean O’Keefe. What 

is he doing these days? 

Mr. PACE. I believe he is working in industry these days. 

Senator CARPER. All right. Tell him the Junior Senator from 

Delaware was asking for him if you run into him, please. 

Mr. PACE. Certainly. 

Senator CARPER. Daniel Tucker, Deputy Assistant Secretary for 

Budget for the Department of Veterans Affairs. In this capacity, 

Mr. Tucker helps to oversee the $90 billion budget for the Department. In addition, Mr. Tucker serves as Performance Improvement 

36 

1The prepared statement of Mr. Dyer appears in the Appendix on page 92. 

Officer for the Department and is responsible for overseeing the development of Veterans Affairs’ annual performance plan and managing Program Assessment Rating Tool evaluations. Prior to his 

current position at the VA, he served as the Chief Financial Officer 

for the National Cemetery Administration. 

With those introductions behind us, let me just ask Mr. Dyer if 

you would go ahead and lead us off. Summarize, if you will, in 

roughly 5 minutes and then we will have some questions and call 

it a day. But thank you. Please proceed. 

TESTIMONY OF JAMES DYER,1 CHIEF FINANCIAL OFFICER, 

AND PERFORMANCE IMPROVEMENT OFFICER, U.S. NUCLEAR REGULATORY COMMISSION 

Mr. DYER. Thank you, Mr. Chairman. It is an honor to appear 

before you today to share our approach for using performance information to improve management of U.S. Nuclear Regulatory Commission programs. 

The mission of the NRC is to license and regulate the Nation’s 

civilian use of byproducts, source, and special nuclear materials to 

ensure adequate protection of public health and safety, promote the 

common defense and security, and protect the environment. 

The NRC is pleased that the Government Accountability Office 

recognized our improvements in the use of performance information 

and we believe it is the result of three factors: One, a strong commitment by the NRC commissioners and senior management to 

continuous improvement; two, the effective implementation of our 

planning, budgeting, and performance management process; and 

three, NRC openness with employees and our external stakeholders. 

We also believe that the recent government activities to create 

the Performance Improvement Officers and Performance Improvement Council can further improve the NRC performance management. 

We created the NRC planning, budgeting, and performance management process in response to the Government Performance and 

Results Act in 1997 and still use the four simple integrated components of the process to manage our program performance. 

First, our current strategic plan concisely identifies two strategic 

goals for safety and security to accomplish our mission and an organizational excellence objective which characterizes the manner in 

which we intend to achieve these goals. 

Second, our budget process involves multiple levels within the 

NRC organization, so we obtain staff’s commitment to complete the 

planned activities on a schedule and within budget. 

Third, NRC executives monitor performance using office operating plans that track budget expenditures and performance targets well beyond the detail in the NRC performance budget delivered to Congress to improve our accountability. 

And fourth, our program assessments integrate the inputs from 

several sources, including Congressional hearings and the OMB 

PART tool results to develop program improvements. Additionally, 

we have recently started using the Lean Six Sigma evaluation procVerDate Nov 24 2008 13:48 Jun 01, 2009 Jkt 044582 PO 00000 Frm 00040 Fmt 6633 Sfmt 6633 P:\DOCS\44582.TXT SAFFAIRS PsN: PAT
37 

ess to more systematically assess program performance. These program assessments provide a significant input for NRC’s senior executive and manager appraisals and awards. 

We also strive to effectively communicate NRC performance expectations and results to our staff and external stakeholders to promote openness and increased accountability. The NRC uses a strategic plan, the performance budget, Congressional reports, and our 

performance and accountability report to communicate with our 

stakeholders and receive their feedback. Commission meetings concerning NRC program performance reviews are some of our best attended public meetings. NRC offices publish quarterly operating 

plan results on their internal websites, routinely discuss the performance results during periodic staff meetings, and in newsletters, 

and recognize staff contributions to agency mission through the 

awards process. 

However, we do face challenges with effective performance management. As you are well aware, the NRC has had significant 

growth in the past few years, and this has increased the complexity 

of performance monitoring, with new programs, new organizations, 

and an increased level of work within the agency. Better performance management through expanded use of techniques such as the 

Lean Six Sigma and evaluation is needed for improved regulatory 

program consistency, efficiency, and effectiveness. 

Additionally, our current system of performance monitoring and 

assessments is very labor intensive. We need to modernize methods 

for processing performance information using the latest technology. 

Improving our budget structure and integrating cost and performance information in a more timely manner will facilitate improved 

performance management. 

Also, our performance metrics focus on the quantity and timeliness of our products over the quality because of the ease of measurement. Most often, quality issues are identified by schedule 

delays because of required rework. We need to develop methods for 

more timely measurement of the quality performance of our programs. 

The Office of Management and Budget-led Performance Improvement Council provides a forum to discuss these challenges through 

sharing agency best practices for improving the use of performance 

information. Presentations by other agencies on their experience 

with implementing Lean Six Sigma evaluations have been valuable 

to the NRC, and the Council’s planned agenda items offer the potential to further improve our performance management. 

In conclusion, we appreciate the Government Accountability Office’s recognition, but realize that we can further improve our use 

of performance information for managing our programs. We intend 

to continue to improve our performance through effective use of internal assessments, external oversight inputs, public feedback, and 

sharing experience with other Federal agencies. 

Thank you. I look forward to answering your questions. 

Senator CARPER. Mr. Dyer, thank you very much. 

Mr. Pace, your entire statement will be made part of the record 

for each of you, so if you want to summarize, feel free. 

38 

1The prepared statement of Mr. Pace appears in the Appendix on page 97. 

TESTIMONY OF SCOTT PACE,1 ASSOCIATE ADMINISTRATOR 

FOR PROGRAM ANALYSIS AND EVALUATION, AND PERFORMANCE IMPROVEMENT OFFICER, NATIONAL AERONAUTICS 

AND SPACE ADMINISTRATION 

Mr. PACE. Just the highlights. Good afternoon, Mr. Chairman. 

Thank you, and it is a pleasure to be here. 

As you mentioned my background at NASA, probably the most 

relevant point for today’s hearing is I am also the agency’s Performance Improvement Officer under Executive Order 13–450 and 

the lead for various program improvement initiatives under the 

President’s Management Agenda, which, of course, as you know, 

Sean O’Keefe had a lot to do with in writing. So I have had maybe 

a bit of an advantage. 

I have been able to observe the full spectrum of NASA’s activities, its performance, where we have excelled and where we can 

improve, and I would like to share some observations, I think, on 

why NASA generally receives good marks and continues to receive 

strong bipartisan support from Congress, for which we are very 

grateful. 

First of all, as some of the other speakers, I think, have mentioned, NASA is really fortunate in being charged by the Administration and Congress with missions that are well suited to performance management. We have a performance-based culture that 

values mission success as a central tenet, and through hard experiences, both good and bad, NASA has sought to put the right tools 

and governance processes in place to better ensure accountability, 

transparency, and oversight. These processes are not ends in and 

of themselves but means for accomplishing the missions that are 

assigned to us by the Administration and Congress. 

In our experience, the best tools for creating accountability and 

transparency are those that provide consistent external reporting 

requirements, that provide flexibility internally in the design of 

measurement techniques tailored to our unique missions. Consistent external reporting helps focus the agency on the most important measures. As was mentioned, if you measure too many 

things, you are probably deluding yourself. And flexibility is important to help avoid the trap of imposing simple one-size-fits-all performance measures that can really mask more than they can reveal. 

While our system is generally working well, we are not without 

challenges and we need the support of Congress in maintaining our 

commitment to the efficient and effective execution of agency missions. Our mission is very concrete and our goals readily flow down 

to every level of the agency. Fly each Space Shuttle mission safely 

until retirement. Complete the International Space Station. Develop the next generation of launch vehicles that will return us to 

the moon, Mars, and beyond. And the performance against concrete 

goals is, therefore, measurable, traceable, and thus actionable. 

Our budget is aligned to our goals. We have well-established program and project management policies driven by the need to deliver a wide portfolio of missions of many different sizes, many different destinations. 

39 

Our programs are typically milestone-driven and we face hard 

deadlines, such as planetary windows, which we just have to hit. 

Our schedules are complex and must be integrated. Many organizations have to come together to fly each mission. And as an agency, 

we are comparatively small and compact, and therefore the key 

strategic informed conversations can be held at the highest levels 

of the agency. So our internal decision loops can be fairly tight. 

We are currently focused on aligning all of our external reporting 

to a single set of measures. We are linking internal performance 

indicators directly to these measures and commitments, in particular GPRA and the PART tools, as has been discussed earlier. 

I could describe our processes in a lot of detail, but the real demonstration, I think, of performance management is mission success. 

We currently have 56 robotic spacecraft operating throughout, and 

in some cases beyond, the solar system. We continue to conduct 

groundbreaking scientific and aeronautics research. The Space Station is nearing completion. We have seven more assembly flights to 

go plus two logistics flights, which we hope we will be able to fly 

before Shuttle retirement in 2010. This October, we are looking toward a final Shuttle mission to the Hubble space telescope to finish 

its repair. 

In the risky business we are in, we are going to continue to face 

challenges. Nine projects have breached cost or schedule thresholds 

in fiscal year 2008 against performance-based lines that we established with the Congress in our major program annual reports. We 

are continuing aggressive effort to improve the fidelity of our upfront estimates of cost and applying more rigor to our life cycle cost 

estimates than really any other time in the agency. There are 

many sources of these cost growths, some we can control, some we 

can’t, and we are working on each one. 

We accept the need to improve performance and transparency, 

but at the same time, we are struggling with the issue of reporting 

complexity. Of course, if a project is in trouble, it is in trouble, but 

under current reporting requirements from the multiple stakeholders, such as the OMB, Congress, and the Government Accountability Office, we have a variety of different trigger points and 

thresholds that include life cycle costs, development costs, schedule 

growth, key milestone slips, and each one of them can be slightly 

different. 

So we are looking for some greater consolidation in how these 

various breaches are defined, greater consolidation of requirements 

for what is in and out of that reporting, not only to reduce our reporting transaction costs, but also to ensure that agency attention 

and effort is focused on mitigating the most significant project performance issues so we stay focused. 

I really want to thank you very much for your time this afternoon and really appreciate you drawing the Committee’s attention 

to this important topic. It is something that is near and dear to us 

and we appreciate your interest. Thank you. 

Senator CARPER. Thank you, and thank you for coming and for 

the example that you provide. Mr. Tucker. 

40 

1The prepared statement of Mr. Tucker appears in the Appendix on page 100. 

TESTIMONY OF DANIEL TUCKER,1 PERFORMANCE IMPROVEMENT OFFICER, AND DEPUTY ASSISTANT SECRETARY FOR 

THE BUDGET, U.S. DEPARTMENT OF VETERANS’ AFFAIRS 

Mr. TUCKER. Thank you, Mr. Chairman. I appreciate the opportunity to be here today to discuss how the Department of Veterans 

Affairs is using performance information to improve service delivery to veterans and accountability for results. 

Since the passage of GPRA 15 years ago, we have seen a major 

transformation in the manner in which performance information is 

used in budgeting and performance management throughout the 

VA. We have made noteworthy progress in using performance information to better justify our requests for resources, monitor our 

programs throughout the year, and document our accomplishments 

and challenges in a manner that is transparent to veterans, our 

stakeholders, and the general public. 

Every year, we reevaluate the performance measures included in 

our Congressional budget justifications and operating plans. We 

analyze each measure to ensure it is still appropriate and meaningful and we develop new or improved metrics that present a better 

gauge of program results. We use customer feedback to ensure that 

the performance metrics we employ are measuring those things 

that are most important to veterans and their families. 

Today, our budget request for every program contains a wealth 

of information on our strategic goals and objectives, our historical 

and projected performance levels, and the means and strategies we 

will use to achieve our goals. In other words, our budget is justified 

not only by the kinds of activities we will conduct, but more importantly, by the results we expect to achieve. 

VA employs a variety of mechanisms to monitor program performance. The most important of these tools is our monthly performance review. Chaired by the Deputy Secretary, these reviews 

involve senior leaders throughout the VA and focus on financial 

management and program performance as well as the execution of 

major construction and information technology projects. These 

meetings play a vital role in keeping the Department focused on its 

highest priorities, achieving key performance goals, and resolving 

operational challenges. 

On November 15 each year, we publish our annual Performance 

and Accountability Report. This report presents a detailed description of how well VA performed relative to the performance goals we 

established at the beginning of the year. 

Our commitment to transparency and reporting has been highlighted by the Mercatus Center of George Mason University. This 

independent research organization conducts an extensive evaluation of all agency performance reports. They have ranked VA 

among the top-rated reports all 9 years they have conducted their 

analysis. For the last 2 years, the Mercatus Center has presented 

an award to VA for achieving the highest score in the Federal Government in transparency in reporting and the Department is extremely proud of this recognition. 

Another one of our major accomplishments has been our success 

in establishing a performance culture in VA. This has largely inVerDate Nov 24 2008 13:48 Jun 01, 2009 Jkt 044582 PO 00000 Frm 00044 Fmt 6633 Sfmt 6633 P:\DOCS\44582.TXT SAFFAIRS PsN: PAT
41 

volved a gradual movement away from a focus on inputs toward a 

more meaningful discussion of program outputs and outcomes. And 

education and training within the Department for staff and managers is critical to this successful shift in emphasis. 

Our monthly performance reviews have also been instrumental 

in institutionalizing a performance culture at VA. Most importantly, these monthly meetings are a clear demonstration of the interest and support of VA’s top policy issues in using performance 

measurement to oversee departmental programs and operations. 

Another key strategy we employed to institutionalize a performance culture was to develop a set of key performance measures. 

Several years ago, we realized that we have been pretty successful 

in developing improved performance metrics, but we had created so 

many that it is hard to figure out what was most important. So to 

correct this problem, we identified a set of about 25 key measures 

that we considered critical to the success of the Department. These 

key measures formed the foundation of our budget request, our 

monthly performance reviews, and our annual performance report. 

While we have successfully tackled many of these challenges, we 

still have more that need further attention. We have a sound set 

of outcome measures for some of our programs, particularly medical care, but there are other program areas for which we still need 

better indicators of the extent to which VA programs improve quality of life for veterans and their families. 

An additional challenge all agencies face is how to more tightly 

link cost data to program performance. In particular, we need to 

strengthen our ability to demonstrate how performance could 

change if resource levels vary. As with all performance metrics, 

this will require the necessary information systems and analytical 

tools to produce valid and reliable data. 

That concludes my statement, Mr. Chairman. I will be happy to 

answer any questions. 

Senator CARPER. All right, Mr. Tucker. You are right on the 

money. That was exactly 5 minutes. Thank you. 

Thank you all for wonderful testimony. Why do you suppose your 

agencies have risen to the top, at least with respect to your use of 

information to be able to improve performance? What common 

threads are there with your agencies in terms of your experience 

and improvement? Mr. Dyer, when you retired from the Navy, 

what was your rank? 

Mr. DYER. I was a captain, sir. 

Senator CARPER. Captain Dyer, OK. I was a captain, too. 

Mr. DYER. Yes, sir. 

Senator CARPER. A recovering captain. A recovering governor. 

Mr. DYER. Senator, as I heard the other two presentations, that 

was the first time I had heard or seen their text, too, the thing that 

I heard—the three reasons I thought that we had improved in the 

past few years, I heard that come out in theirs, and that is a commitment to doing it—and we heard it from the earlier panels, too— 

a commitment to doing it, a mission statement that is simplified 

and that allows you to go to outcomes, which drive outputs, which 

drive the inputs to the budget, and that kind of a focus on executing to achieve the outcomes intended. 

Senator CARPER. Mr. Pace. 

42 

Mr. PACE. I would agree. I think the most important advantage, 

I remember the prior panel talking about as things become more 

and more complex, people become more and more important because they are not interchangeable and I think that the performance culture that we have at NASA, which is reinforced in so many 

different ways, has been actually really crucial, so that when you 

come in with a performance measurement system of some sort, initially, it is looked at askance. What is this, a new unfunded mandate dropping down on us? 

But when people get into it and see that you are going to use 

that information to make better decisions that then have some 

linkage to mission success, then there is a greater acceptance of it. 

It is something that actually reinforces the performance culture of 

the agency because it is not seen as alien to the agency but, in fact, 

is integral and consistent with the values of the agency, and that 

in turn is reflected by statements from the top management down 

through the system. 

So it is a very interwoven problem, but as you hear, it is a very 

consistent one that repeats in many organizations. 

Senator CARPER. All right. Mr. Tucker. 

Mr. TUCKER. I think from what the panel members here have 

stated as well as the members of the previous panels, the thing 

that you hear consistently is top management interest and top 

management involvement. You have to have the leadership engaged. If the leadership is engaged and focused on using performance information, that will drive change throughout the organization. 

Senator CARPER. Let me just interrupt for a second. I have been 

privileged to know three VA Secretaries during my time here in the 

Senate, and that was in 7 years, so you have had a number of Secretaries. Is it Tom Bowman? Who is the Chief of Staff—— 

Mr. TUCKER. Tom Bowman is. 

Senator CARPER. I think Mr. Bowman has been around for a 

while—— 

Mr. TUCKER. Yes, sir. 

Senator CARPER [continuing]. And maybe he provides the continuity. An old Marine, as I recall. 

Mr. TUCKER. Yes, sir. 

Senator CARPER. But you have had a fair amount of turnover at 

the top in the VA, but a continued commitment to improvement. 

Mr. TUCKER. I think the commitment from those that remain, 

like Tom Bowman is a good example as the Chief of Staff, he sits 

in on our monthly performance reviews every month. He is certainly a very active member of the management team. I think also 

what is important, especially at the VA, even though we are a very 

large agency and we have a great diversity of programs, we are all 

there to serve veterans and we are all there to serve their families 

and those that are eligible for the different benefits that we provide, and I think that is part of our performance and the success 

of our performance culture at the VA, is a real clear mission statement and we know who we are there to serve. 

Senator CARPER. If you think of it, let Tom Bowman know we 

just opened our second VA outpatient clinic in Delaware. We only 

have three counties. We have two of these clinics now. We just 

43 

opened two of them in the last 2 or 3 months. But he was a big 

help in getting that done, so tell him we are grateful for that and 

for his service. 

Mr. TUCKER. I will do that. 

Senator CARPER. You have mentioned performance culture. Mr. 

Pace has mentioned it. I think Mr. Dyer has mentioned it. One of 

my other hats is chairman of the subcommittee that has jurisdiction over nuclear safety. We talk a fair amount about safety culture, and during the time in my 23 years in Naval aviation, oftentimes our commanding officers would say to us, and again, our job 

was to hunt for Red October, track Soviet nuclear submarines, and 

do all kinds of surface surveillance and fly missions off the coast 

of Vietnam and Cambodia and other places. 

But our skippers would always say the most important thing you 

are doing in your mission today isn’t whether you are trying to 

mine some harbors or whether you are doing certain service missions where you are tracking the bad guys, the bad submarines, 

the Ruskies, whatever it was. He said your most important thing 

that you are doing is taking off safely, flying safely, coming back 

and landing safely, and walking away from it and going home. So 

that is the most important thing that you are doing. 

Almost everything that we did, whether it was in the aircraft, 

the folks who were maintaining our aircraft, training us, everything pointed toward safety. It was really a culture of safety and 

we tried to do it in a way so it wasn’t a ‘‘gotcha’’ system, so that 

if people observed things that were wrong, they would actually step 

forward and say, ‘‘This isn’t really a very safe practice,’’ and not be 

penalized or punished for having said that. 

So we try to take the same thing, the same approach with respect to our nuclear power plants and to encourage them, and I 

know that the NRC are very much involved in doing that for us. 

The safety culture, we have 104 nuclear power plants. We want 

every one of them to be as committed to safety as we were in my 

squadron, our squadrons in the Navy. 

Talk to me, if you will, about performance culture. How do we 

establish a kind of—obviously, you have a performance culture at 

your agencies and we have a bunch of agencies who don’t have it. 

You are the exception, not the rule. How do we establish the kind 

of performance culture at more agencies, much as we are trying to 

establish safety cultures at 104 nuclear power plants? And that 

would be a question for each of you. I don’t know who cares to go 

first. Mr. Dyer, feel free. 

Mr. DYER. Yes, sir, Mr. Chairman. Just like you talked about the 

safety culture at the nuclear plants and the guidelines that go 

along with them, I think starting with the performance culture at 

the NRC, it begins actually with the formulation of the budget. 

When we get in to recognizing the roles and responsibilities and of 

the individuals within the organization and get their commitment 

that we are going to expend this amount of resources to achieve 

this kind of results, we get that kind of commitment of them that 

they have bought in, that they sign up for it, and it is not something that we pass down from on top and just say, you are going 

to get X number of resources and get it done. It is they have an 

44 

input to it that starts, that contractor, that bond, if you will, to actually perform the work and get the results that are intended. 

Senator CARPER. All right. Thank you. Mr. Pace. 

Mr. PACE. Really, I want to also agree with that. One of the crucial issues we have is aligning the resources that content, that we 

make program commitment agreements and we say we want to do 

this vehicle, do this kind of mission, this is what it is going to cost, 

and if we change the resources, we change the content that we ask 

of people. And if we change what we ask of people, we also change 

the resources. The two in the budget process have to be very closely 

linked. So if the schedule slips or something else slips or new problems crop up, that we account for it. 

And I think that is part of having really almost an ethic, a moral 

duty, really, to the people we ask to do the work, that we are going 

to back them up and they, in turn, have an obligation to bring forward their best estimates about what it is going to take because 

we are both trying to achieve that mission success. So the budget 

process is at the center of it. 

The second thing I would add is that along with having a safety 

culture, it is important that our independent reviews and our assessments are done with the mindset of assisting the project to be 

successful, not of a mindset of auditing the project or trying to 

come up with a certain number of gigs. The mentality we try to 

bring to it is we are kind of like the graduate TAs. We are there 

to get the other students to pass. We may beat on them pretty 

hard, may review them pretty hard, but our goal is we want them 

to pass, we want them to succeed, and we are there to help them 

succeed, and having that mentality of an assist function also gets 

people to buy in, bring their problems forward because we are not 

out there to have them fail. We are there for mission success. 

Senator CARPER. Thank you. Mr. Tucker, anything you would 

like to add? 

Mr. TUCKER Just adding on top of what the other witnesses have 

said, I think one of the big changes that we have seen in establishing a performance culture at the VA is the shift from the discussion on inputs into outcomes. A lot of folks have said that today. 

I can think back years ago and when we would have monthly 

budget reviews in the Department. It was always just simply a discussion of, well, what is the variance from plan? Did you spend 

more than your monthly plan this month? Did you spend less? How 

much more? How much less? People were either happy or sad 

about the results, but the discussion didn’t really focus on program 

results. 

Those are the kinds of discussions that we have now. We have 

the data. We have the data systems in place to facilitate that discussion. And management from the very top all the way down to 

the medical center level use that information to improve performance. The measures are the same whether they are at the national 

level, the network level, or the local level. 

An example I can give you is on wait times for VA patients to 

get into our health care system. We have used performance information to really drive improvement in reducing the number of veterans that are waiting more than 30 days for an appointment 

from—it was over 250,000 in April 2006 and we have driven that 

45 

down to about 51,000 in June, just last month. So that is one example of how leadership is focused at the top all the way down to 

the facility level to drive performance improvement. 

Senator CARPER. OK. Thank you. Mr. Pace, maybe one last question for you. According to a GAO survey, I think it was done in 

2007, a large majority of your managers said that they use performance information to identify and solve program problems. Do 

you think that looking at program performance information has 

helped improve NASA’s programs, and if so, how? You have talked 

about this already, but do you want to add anything else? 

Mr. PACE. I think one of the major areas of differences where I 

have certainly seen a change in the time that I have been at NASA 

is when I first came there, we would go to program reviews and 

people would be talking about the status and in some cases how 

they felt about the program or project, as Mr. Tucker was just saying. What you see in the status reviews that we have monthly now 

and baseline performance reviews is much more toward quantitative. So programs are coming in with—here is where we are on 

the finances. Here we are on the human capital. This is what the 

state of the asset and facilities is looking like. This is where the 

program has hit various milestones. So monthly meetings, much 

more quantitative. 

That then sparked discussions. I mean, the most important thing 

that occurs in that quantitative data and where people see various 

disconnects is then a conversation occurs at the senior management level that says, why are those disconnects happening? What 

can we do about them? What is the resolution path forward? 

So I think that it is helping NASA quite a bit. We are doing 

things that are consistent with our culture, but we are doing them, 

I think, in a much more rigorous and consistent way. We are still 

learning. We still have plenty of other areas that are imperfect, but 

I think the intention on performance being quantitative and getting 

those discussions out in front of senior management has been very 

helpful. 

Senator CARPER. Thank you. Mr. Dyer, according to the GAO’s 

same 2007 survey results, the NRC’s managers report a significant 

increase in the use of performance information since responding, I 

think to a survey 7 years earlier, the 2000 survey. In fact, I am 

told that NRC managers are now among the most positive among 

their government-wide counterparts in their reported use of performance information. I just ask, in your view, how and why you 

think this change came about. 

Mr. DYER. Mr. Chairman, I believe we have had this same process in place since 1997 and every year we do a little bit better. We 

revise our performance indicators. We reach out and get them to 

lower levels in the organization. And I think the people can trend 

progress and we report out on it and we provide feedback to the 

staff on it and they recognize that these things are important and 

that they now accept them and embrace them. 

Senator CARPER. All right. One of the things that I find remarkable when we have testimony from the NRC and from the nuclear 

industry and others who are interested in the industry is the increase in the power plants that we have, their ability to provide 

power. If they are able to provide power 100 percent of the time, 

46 

that would be terrific. But it wasn’t that long ago when they were 

down around 60, 70 percent, and today they are up over 90, 91 percent. That is a remarkable achievement. 

Mr. DYER. Yes, sir. 

Senator CARPER. A lot of people deserve credit for that and I 

know among those people are the NRC. 

The last question would be really for all of you. Each of your 

agencies has had programs that were assessed by OMB through 

the PART process. What suggestions do you have for the next Administration, whoever might lead that, but the next Administration 

on how to design and structure a performance assessment process 

such as PART? 

Mr. TUCKER. I will go first. 

Senator CARPER. Yes, Mr. Tucker, feel free. 

Mr. TUCKER. I think what will be helpful for OMB to do is to 

work collaboratively with the agencies to look at the criteria that 

is used and really to continue the process and refine the process 

of the evaluation and performance measures. I mean, I think that 

has been the key benefit that the VA has obtained from going 

through the PART process is having a really hard look at the performance metrics that we use, whether they are outcome-focused or 

efficiency focused, and developing action plans and requiring the 

action plans to be updated on a regular basis so you are continually 

improving the metrics that the organization is using to assess performance. 

I think one other thing—I will just put in a plug—is recently in 

response to the Executive Order that the President signed back in 

November, OMB did establish an interagency Performance Improvement Council. I think that council should continue into the 

next Administration. It is just getting started. It is growing. It is 

getting traction, starting to get some legs under it. 

Senator CARPER. Who heads up that council? 

Mr. DYER. Robert Shea is leading the Performance Improvement 

Council at OMB right now. 

Senator CARPER. OK. Thank you. 

Mr. DYER. But I hope that it will be as successful in the future 

as the Budget Officers Advisory Council has been that OMB had 

established a long time ago where the budget officers from the different departments come together once a month to share best practices, to raise issues and concerns, and just to have that face-to-face 

once-a-month meeting and discussion. And I think the Performance 

Improvement Council can do the same thing. 

Senator CARPER. Good. I had not heard of the Performance Improvement Council. It sounds like a good idea. I hope it continues, 

as well. Thank you. 

Mr. DYER. Thank you. 

Senator CARPER. Mr. Pace. 

Mr. PACE. Well, I would also actually endorse some of the Performance Improvement Council meetings. I also go, as well. But I 

think one of the things I have found most valuable for it is actually 

the cross-agency discussions that have happened, so seeing what 

other agencies are doing, sharing notes with them, and comparing 

best practices. I thought that has been a useful forum. I know that 

other speakers have talked about cross-agency issues. 

47 

Regarding the PART, I think it obviously should continue. I 

think it has been helpful. We have been able to improve the measures of effectiveness for some of our programs. Particularly we 

have made some improvements, I think, in our education efforts 

that have been helpful and traceable to PART. 

But the other thing I would say in terms of maybe a particular 

parochial plug that we and some other agencies have is in measurements of R&D. We have a very project-oriented agency. Other 

agencies are very process-oriented. But some of our science agencies, NSF, maybe NOAA, ourselves, Geological Survey, when we 

are doing R&D activities that stretch over a longer period of time, 

it is often harder to see the results each year. You have to look 

over a longer time horizon to judge what the benefits of R&D are. 

So ourselves and NSF have had some good conversations about 

this. I know with DOE Science Office, we have also had some good 

conversations, and measuring effectiveness of R&D is one of those 

areas where a simple one-size-fits-all sometimes doesn’t work. And 

the conversations we have had in the PIC, I think have been helpful to us, coming up with some ideas we are going to hope to pursue. 

Senator CARPER. Thank you. Mr. Dyer, the last word? 

Mr. DYER. Mr. Chairman, I think the PART program, the PART 

is fundamentally sound and it is a good program. At the NRC, we 

have had seven PART evaluations. Six of them have been rated 

high as effective and one was moderately effective, and I owned, 

designed that one that was moderately effective, and it had a good 

therapeutic effect, I think, on the NRC that when we went back 

and pulled the string on it, what we really looked at and came up 

with was that we had a problem with our budget structure and 

that cascaded down into problems, and that wasn’t what we 

thought going in. 

So I have got a healthy respect for the PART. That being said, 

it is an extremely resource-intensive process. So as we go forward, 

I would hope that we streamline the process so we don’t have to 

dedicate the amount of energy that goes in in conducting the PART 

evaluations. 

Senator CARPER. All right. Thank you. 

We are at the end of our hearing. I was just reflecting on this. 

It is pretty clear I haven’t had a lot of company up here on this 

side of the rostrum. We have a lot of the agencies, not all the agencies have really emulated your success and your approach by taking seriously our ability to use the information that it has generated to improve our performance. Not everyone does that. 

Having said that, we are going to have a new President in 6 

months. We will have a new Congress in less than that. And we 

are going to still have a lot of problems to face here at home and 

around the world. We don’t have unlimited resources. We are finding out just how limited our resources are, financial resources are 

to deal with those challenges. 

And if we are smart, and I hope we will be, the more of us here 

in the Legislative Branch and the folks in the Executive Branch, 

including right at the top, will realize that if we want to provide 

better results, better outcomes for the folks who are paying the 

taxes in this country, there are a variety of ways to do it and one 

48 

of them is to take all this massive information that we have, try 

to figure out what of it we can learn from in order to provide cleaner air, in order to provide better transportation, in order to reduce 

our threats from within and without, and act on that. If we do 

those things, taxpayers will be happy and we will be better off as 

a Nation, too. 

We chose in this hearing today not to focus on the agencies that, 

frankly, weren’t doing a good job. We chose to hold up a handful 

that are doing a good job in this regard and to say by inference to 

other agencies that could learn from you, take a look at these folks. 

I said to Governor O’Malley, we would rather see a sermon than 

hear one, and with you, you have had a chance to give us a little 

bit of a sermonette, but we have also had an opportunity to watch 

your performance over a number of years and we like what we see. 

Hopefully, by holding you up for praise and acclaim, we will encourage some other agencies to emulate your good performance. 

With that having been said, we are going to leave the record 

open for 2 weeks so we will have the opportunity for some of my 

colleagues and to submit some questions, maybe some statements, 

as well, for the record. We would ask if you do receive questions 

that you just respond promptly and we will be better for it. 

Thank you so much for coming, for the good work that you and 

your colleagues back in your agencies are doing. Convey our congratulations and our thanks to them, if you will. Thank you so 

much. 

Mr. DYER. Thank you. 

Mr. PACE. Thank you. 

Mr. TUCKER. Thank you. 

Senator CARPER. This hearing is adjourned. 

[Whereupon, at 5:10 p.m., the Subcommittee was adjourned.] 

(49) 

A P P E N D I X 


44582.001
50 


44582.002
51 


44582.003
52 


44582.004
53 


44582.005
54 


44582.006
55 


44582.007
56 


44582.008
57 


44582.009
58 


44582.010
59 


44582.011
60 


44582.012
61 


44582.013
62 


44582.014
63 


44582.015
64 


44582.016
65 


44582.017
66 


44582.018
67 


44582.019
68 


44582.020
69 


44582.021
70 


44582.022
71 


44582.023
72 


44582.024
73 


44582.025
74 


44582.026
75 


44582.027
76 


44582.028
77 


44582.029
78 


44582.030
79 


44582.031
80 


44582.032
81 


44582.033
82 


44582.034
83 


44582.035
84 


44582.036
85 


44582.037
86 


44582.038
87 


44582.039
88 


44582.040
89 


44582.041
90 


44582.042
91 


44582.043
92 


44582.044
93 


44582.045
94 


44582.046
95 


44582.047
96 


44582.048
97 


44582.049
98 


44582.050
99 


44582.051
100 


44582.052
101 


44582.053
102 


44582.054
103 


44582.055
104 


44582.056
105 


44582.057
106 


44582.058
107 


44582.059
108 


44582.060
109 


44582.061
110 


44582.062
111 


44582.063
112 


44582.064
113 


44582.065
114 


44582.066
115 


44582.067
116 


44582.068
117 


44582.069
118 


44582.070
119 


44582.071
120 


44582.072
121 


44582.073
122 


44582.074
123 


44582.075
124 


44582.076
125 


44582.077
126 


44582.078
127 


44582.079
128 


44582.080
129 


44582.081
130 


44582.082
131 


44582.087
132 


44582.088
133 


44582.089
134 


44582.090
135 


44582.091
136 


44582.092
137 


44582.093
138 

Æ 


44582.094