Volume 1: Lessons From State Initiatives 

Inspector General - 

SEPTEMBER 1997 

OEI-05-9640260 
This report describes initiatives underway in 11 states to implement results-based systems 

for managing public health programs. It draws on the experiences of these States for 

insights into the development and implementation of such systems. 

Government-wide interest in enhancing the performance and accountability of 

government programs, including its grants programs, intensified with the publication, in 

1993, of the Vice President's Report on the National Performance Review. As a result 

of this and other governmental and nongovernmental influences, policymakers have been 

actively considering changes in Federal grantmaking and oversight authorities. Some 

proposals call for combining several categorical and block grants together and replacing 

them with performance partnerships. Performance partnership grants would be 

negotiated between Federal Government agencies and the States. 

The United States Department of Health and Human Services (HHS) has been 

considering performance partnership grants for some of its public health programs for 

several years. These arrangements would combine various categorical grants into 

performance partnerships and would reshape several of its block grant programs. 

The Assistant Secretary for Planning and Evaluation asked the Office of Inspector 

General to identify and examine State initiatives that use outcomes measures to assess 

the performance of their public health programs. 

This report is based on a review of results-based initiatives in the preventive health, 

maternal and child health, substance abuse and mental health programs of 11 States. We 

conducted onsite discussions in seven of these States: Florida, Illinois, Massachusetts, 

Nebraska, New York, North Carolina and Washington. We interviewed officials by 

telephone in four other States: Georgia, Minnesota, Ohio and Oregon. A companion 

volume contains brief descriptions of these States' initiatives. 

STATE INITIATIVES 

lThe results-based accountability initiatives we mrnined are generaw of two typ: 

(1) broad @om at Statewide strategic planning and priody setting, and (2) systems focused 

on target populations and spec@ program interventions. 

We States, in initiating thkir resulfi-based systems, have several characterirtics in common: 

public pressures for better government, top-level commitment, and atensive stakeholder 

involvement. 
State oficicak see mny ben@B fp-srn devebpmnt and kpapkmn~ation of rdb-based 

system ad are using data in vanbus ways. However, data from these system are ranlikely 

to becom the sole criteria or diving force for prpqamr~labic, budgemy, or contracting 

decisions. 

In hebophg their rd&-bmed accountabiliy systemy Stata face several Isignifiant 

chahnges: kues wibh the mbzs~6pes themselves, data con.cemY the use of rdy ad 

system cap~citk. 

Measurement: States must ensure that measures are related to program missions and 

goals, adopting a mix of different kinds of measures in order to balance the limitations of 

outcomes, output and input measures. They must also ensure that measures are 

appropriately focused on true priorities. 

Data: In considering measures, State officials struggle with whether the necessary data 

are available and reliable, as well as whether the data can be obtained in a timely 

manner in order to make necessary programmatic adjustments. 

Standards: States must consider whether, and how, they will adopt standards associated 

with the measures they select. If they adopt standards, what are the penalties or results 

if the standards are not met, and what conclusions can be drawn about program or 

contractor performance? 

Capacities: States must also fairly assess the capacities within their system to analyze and 

interpret the information they obtain through their measurement and data collection 

systems. 

7Iaei-e are Isigni@ant challenges facing HHS as it considern me directbnr for its 

perfomurnce paptnernhip grant initiative. 7lw abiliry ofH6-IS to address these chadlenges 

he& and flectively will aflect the smcess of irs gorts to impkmenr meanin@l 

perfomurnce partnepship with the States. These challenges include: 

Usefulness to States: How can HHS ensure that its performance partnership approach, 

relying on results-based systems for accountability, will be integrated with the State's own 

performance management efforts? 

Effectiveness of Partnerships: What kind of administrative infrastructure can best 

support the inter-agency partnerships? 

Information Exchange: How can HHS best support an exchange of information among 

Federal and State agencies and with the research and academic communities? 
Data Collection: How can HHS maximize the usefulness of its current data collection 

systems to enhance the effectiveness of its performance partnerships with the States? 

Research Agenda: How can HHS best leverage its resources to ensure a research 

agenda that addresses the major information needs of the States and its own agencies? 

Evaluation Capacitv: How can HHS ensure that its agencies and the State agencies with 

whom they partner have adequate capacity to evaluate the performance-based data 

emerging from these systems? 

CONCLUSION 

This report documents some important initiatives that are underway across the country to 

develop more meaningful approaches for managing public health programs. Developing 

these systems is inherently messy, difficult and time consuming. It involves multiple 

stakeholders with complex, sometimes competing, agendas. It is an expensive endeavor 

and can make heavy demands on limited resources. Progress is neither straightforward 

nor single-tracked. ;et, as is clear from the initiatives described here, States are making 

serious efforts to develop results-based systems and are moving forward. 

We document in this report a number of benefits State officials we interviewed see 

emerging from their efforts. They bear repeating here. These benefits include: 

(1) obtaining information efficiently, which can be used to improve program 

performance; (2) empirically demonstrating program results; and, (3) obtaining consensus 

among stakeholders on program missions and goals. We believe that these are significant 

benefits to program managers and strategic planners. 

The ultimate determinant of whether these systems succeed, however, may well be their 

usefulness to those most vested in the programs: legislators, administrators, providers, 

advocates, and consumers. The challenge facing these stakeholders will be to achieve an 

appropriate balance when using the data from these systems to enhance decisionmaking 

about program improvements, budgets and contracts, and feedback to providers and 

consumers. 

AGENCY COMMENTS 

We received very positive comments from the Assistant Secretary for Planning and 

Evaluation, the Substance Abuse and Mental Health Services Administration, the Centers 

for Disease Control, the Health Resources and Services Administration, and the 

Secretary for Health. Their comments offered further insights on such matters as: the 

impact of the Government Performance and Results Act in helping shape performance 

partnerships; the data and performance measurement problems facing States; the 

importance of information *exchange; and, the role of Federal agencies in performance 

partnerships. We were also informed that the National Research Council's Panel on 

Performance Measures and Data for Public Health Performance Partnership Grants has 
recently issued its first report entitled 12ssessment of Pe~omzance Measures for Public 

Health, Substance Abuse and Mental Health." 

Based on these comments, changes were made to the report as appropriate. The 

complete text of these comments can be found in Appendix B. We regard them as an 

integral part of the report and recommend them to the attention of the reader. 
I3 OF CONTENT 

PAGE 

E%EmSmY .......................................... i 

rnOD1UrnON ................................................. 1 

STATE .......................................... 5 

STATE .................... ; ...................... 10 

FEDERAL C33AILENGE ....................... .;. .............. 15 

CONcCkUSION .................................................. 18 

MPENDIrn 

A Federal Public Health Block Grant Programs ........................ A-1 

B: AgencyComents ........................................... B-1 

C Endnotes ................................................... C-1 
This report describes initiatives underway in 11 States to implement results-based systems 

for managing public health programs. draws on the experiences of these States for 

insights into the development and implementation of such systems. 

BACKGROUND 

The Federal Government currently administers both categorical and block grants to the 

States for the purposes of addressing public needs. Categorical grants are for specific 

and narrowly defined purposes, and generally include federally-specified eligibility and 

reporting requirements. Block grants are broader, encompass a larger range of purposes 

and goals, and contain fewer Federal prescriptions. 

Government-wide interest in enhancing the performance and accountability of 

government programs, including its grants programs, intensified with the publication, in 

1993, of the Vice President's Report on the National Performance Review. As a result 

of this and other governmental and nongovernmental influences, policymakers have been 

actively considering changes in Federal grantmaking and oversight authorities.' 

Some proposals call for combining several categorical and block grants together and 

replacing them with performance partnerships. These grants would be negotiated 

between Federal Government agencies and the States. They would allow States greater 

flexibility in meeting key national and State objectives. In return, States would provide 

the Federal Government with performance data and work with the Federal Government 

to establish goals. 

The United States Department of Health and Human Services (HHS) has been 

considering performance partnership grants for some of its public health programs for 

several years.2 (See Appendix A for a fuller description of the Federal block grants for 

preventive health, maternal and child health, substance abuse and mental health.) 

Over the past several years, the President has included provisions for performance 

partnership grants in his budget. Several legislative proposals to create such grants have 

also been introduced in the While not yet enacted, these proposals reflect 

the growing consensus about the need to improve the management and structure of 

Federal-State grant programs for health. 
The concept of performance partnerships for public health is also rooted in, and made 

possible by, activity at all levels of government and in the private sector to assess the 

impact of various health care interventions on health status and clinical outcomes. 

Spurred on by high costs, quality concerns, technological advancements in data storage 

and processing, increasing penetration of the health care marketplace by managed care 

organizations, and consumer demands for information, Federal, State, local, nonprofit 

and private organizations have been working to improve the way health status and 

outcomes are measured and used.4 This movement was given added impetus by the 

adoption of Total Quality Management principles by many healthcare organizations 

likewise seeking to introduce, within the confines of their own systems, the collection and 

use of meaningful data on patients and providers in order to identify and implement 

system improvements. 

These developments, along with other Federal initiatives, have formed a backdrop for 

HHS efforts on performance partnership grants in its public health program^.^ Federal 

initiatives, such as the Government Performance and Results Act, seeks to improve 

agency performance and results through results-based management. This technique uses 

the same process of ,,igaging stakeholders, identifying goals, and formulating 

performance measures as that employed by the States. 

In mid-1995, HHS requested the National Academy of Sciences to examine and make 

recommendations for specific performance measures that could be used in public health 

performance partnerships over the next few years. In 1996, HHS, in collaboration with 

several national health organizations, convened a series of regional meetings with States 

to discuss current activities in developing performance measurement systems for public 

health program^.^ The results of these meetings were provided to the Academy to assist 

in its analysis. The Academy issued its draft report in the fall of 1996.~ It is now 

assessing further developmental work needed in data systems to support performance 

measurement systems. 

The Assistant Secretary for Planning and Evaluation asked the Office of Inspector 

General to identify and examine State initiatives that use outcomes measures to assess 

the performance of their public health programs. Its primary interest lies in knowing 

more about the nature, extent, and uses of these outcomes measurement systems, and 

lessons learned and challenges faced by the States, building upon information supplied at 

the regional meetings sponsored by the Department. 

We contracted with Penny Thompson, a principal in Management Evaluation Training, 

LLC, an evaluation and management consulting firm, to undertake this study. The Office 

of Inspector General staff were project officers and participated in all phases of the 

study. 

In t,his report, we focus on 11 States' initiatives. We intend this report to serve as a 

resource document that offers insight into the experiences thus far of a number of States 
that have engaged in the process of developing outcomes measurement systems for 

public health. It is not an assessment of the outcomes measures themselves, their 

appropriateness, or adequacy. It is not exhaustive of all State efforts or even all efforts 

in the States we examined. 

A companion report, Results-Based Systems For Public Health Programs, Volume 2: State 

Case Studies, OEI-05-96-00261, contains short, descriptive summaries of the State 

initiatives we examined. 

This report, and its companion volume, are based on a review of results-based initiatives 

in the preventive health, maternal and child health, substance abuse and mental health 

programs of 11 States. We conducted onsite discussions in seven of these States: 

Florida, Illinois, Massachusetts, Nebraska, New York, North Carolina and Washington. 

We interviewed officials by telephone in four other States: Georgia, Minnesota, Ohio 

and Oregon. Two other States included in our first round of State contacts, Colorado 

and Texas, were dropped from our interviews due to resource constraints. 

We initially wrote to all officials in all the States mentioned above and requested 

available documents on their efforts. These documents served to confirm the presence of 

reported efforts and to form the basis for additional discussions with State officials. As 

we interviewed State officials onsite and by telephone, we often received or requested 

additional documentation which we then reviewed as well. 

The criteria we used in selecting the State initiatives described here included: 

(1) suggestions from government and non-government experts, researchers, and analysts 

identifymg States with positive experiences in developing and implementing results-based 

systems; (2) our own review of documents associated with individual State initiatives; 

(3) geographic representation and program balance to reflect the areas of maternal and 

child health, substance abuse, mental health, and preventive health; and, (4) our own 

judgments about whether particular initiatives were sufficiently different from, and less 

well known than, others. 

We conducted this study in accordance with the Quality Standards for Inspections issued 

by the President's Council on Integrity and Efficiency. 

As we reviewed the available literature on this subject, designed this study and conducted 

these interviews, we became increasingly sensitive to the importance of consistent, clear 

terms in describing efforts in these areas. Terms and definitions used for the purposes of 

this report and its companion volume include the following: 

Results-Based System: An initiative focused on using measurement systems to gauge 

program outcomes and effectiveness, with accountability attributes which might include 
public reporting, goal setting, and standards or requirements for meeting goals applied to 

program officials, providers, contractors or grantees. 

~utcomes' Measures: Measures reflecting ultimate programmatic results, including health 

status and risk behaviors. 

Performance Measures: Measures reflecting a program's more immediate effectiveness 

or efficiency, including activity levels, and direct accomplishments as a result of services 

rendered. 

Standard: A measure set as a requirement and expected threshold of performance for 

States, counties, providers or other partners in the results-based system. 

Goal: A measure representing the end to which efforts by States, counties, providers or 

other partners in the results-based system are aimed. 
The results-based acmuntability initiatives we exmind are gene* of two types: 

(1) broad efforts at Statewide strategic planning and priority setting, and (2) systems 

focused on target populations and specific program interventions. 

Some States are developing results-based systems to assess progress towards broad, 

health-related goals for the States' populations. The development of measures in these 

cases is usually connected with strategic planning efforts-designed to identify Statewide 

priorities and sometimes with reorganizations of health functions within a State. These 

efforts may also be duplicated at the local level, where officials adjust goals, priorities 

and measures based on their own assessments of local populations' health status and 

needs. We found examples of such efforts in Georgia, Florida, Minnesota, New York, 

North Carolina, Oregon and Washington State. 

Georgia's efforts are illustrative of State efforts of this type. In 1994, the State's 

Governor appointed a 21 member Policy Council for Children and Families. The 

following year, the Georgia legislature statutorily established the Council as a 19 member 

panel composed of leaders in business, child and family advocacy, local government and 

religion along with State directors of cognizant agencies. The Council issued a report in 

1994 outlining five results areas that it seeks for children and families, including healthy 

children. The Council established a Results Accountability Task Force which established 

26 benchmarks for the five results areas identified by the Council. Benchmarks for 

healthy children include: (1) reducing the percentage of children who have untreated 

vision, hearing or health problems at school entry; and, (2) reducing the pregnancy rate 

among school age girls. 

New York's efforts are ..lso illustrative of State efforts of this type. In the past year, New 

York's State Department of Public Health has initiated a process to develop resultsbased accountability systems for its Statewide public health programs. As a first step, it 

initiated a comprehensive review process that identified 12 public health areas of high 

priority for future action. Each priority area has measurable objectives to guide State 

and community actions for improving the health of New Yorkers. 

Program Spec@c Effor~s 

Some States are developing results-based systems to assess the effectiveness of particular 

interventions and programs. In these cases, measures are often developed in connection 

with efforts to develop consensus and agreement on program missions and expected 

results. Program specific efforts are generally focused on the target populations actually 

receiving some kind of service from the program. We found examples of such efforts in 

Florida, Illinois, Ohio, Massachusetts, New York, North Carolina and Washington State. 
Ohio's efforts in its Early Start Initiative are illustrative of State efforts of this type. 

Ohio's Early Start Initiative is focused on providing integrated, preventive services to 

families with infants and toddlers at high risk for abuse, neglect or developmental 

disabilities. Currently, 30 of Ohio's 88 counties are participating in the Early Start 

initiative. An essential part of the project was the development of performance goals 

and data collection protocols. 

The Massachusetts Department of Mental Health relies heavily on contracts with local 

provider agencies to deliver its service programs. Its Area Offices, with their citizen 

advisory boards, negotiate with and monitor these providers of both inpatient and 

community-based services. The Department uses a system of results-oriented 

performance measures with its local providers of services. 

New York's State Department of Health is working with Monroe, County health officials 

on an initiative to: (1) support child and family health services through a single funding 

stream of combined State and Federal funding sources, and (2) rely on results-oriented 

objectives and indicators for ensuring accountability for performance. The Department 

submitted the propowdl to the Federal Government for review in late summer 1996. 

As evident from the above, often both kinds of efforts are taking place at once, but they 

may or may not be explicitly linked. We found an example of an explicitly linked effort 

in Nebraska, where program officials have designed a system to use outcome 

measurements in conjunction with program specific performance measures. Nebraska 

has initiated an ambitious project in which, within 1 year's time, a complete 

reorganization of State health functions will have been designed and implemented. As 

part of this effort, a performance accountability system has been developed to identify 

outcomes, outcome indicators and performance measures for programs supported by the 

new Health and Human Services organization. 

In examining State efforts, we are able to categorize systems into two broad 

classifications. Beyond that, because of the limited scope and duration of our visits, 

document reviews and telephone interviews, we cannot present more refined models or 

typologies of States' efforts. Within the two broad classifications presented here, the 

variety of efforts reflects a wide range of choices available to State and local authorities 

in developing systems they consider appropriate to their needs, available resources, and 

other legislative and administrative imperatives. 

In describing the State efforts, the data presented here should be emphasized as a 

snapshot in time in a constantly changing environment in which State efforts will certainly 

evolve. Perhaps even dramatic changes in direction will occur with significant changes in 

State and local leadership, resources and other legislative and administrative imperatives. 

(For a more detailed description of these initiatives, see the companion report, ResultsBased Systems For Public Health Programs, Volume 2: State Case Studies, OEI-05-96- 

00261.) 
These States, in initiating their results-based system, have several characteristics in 

csmsn: public pressures for better governen4 toplevel csdtmenf and extensive 

stakeholder hvoPvement 

These results-based initiatives have been influenced by: the performance measurement 

movement that has become increasingly influential throughout government; the demands 

of State legislators and the public they represent for more evidence of results and 

program achievements; and, the interest in health reform at both State and Federal 

levels. Many State public health programs have been particularly influenced by the 

Healthy People 2000 initiative that established national public health objectives, goals, 

and standards for the nation to achieve by the year 2000. This document has sewed as a 

critical foundation and framework for several of the State efforts. described here. It has 

helped the States to refocus their public health programs to achieve specific outcomes. It 

has also served a larger purpose of familiarizing those outside the public health 

community with a different approach for thinking about accountability in these public 

services. 

Most of the State initiatives described here were either initiated or strongly backed by 

top-level State or agency officials. Sometimes, this commitment was in the form of a 

catalytic event, such as the major reorganization of a department or a major piece of 

legislation that required results-based systems. Other times, the commitment came from 

a newly elected Governor or a newly-appointed department head. 

All the State initiatives described here have involved a wide variety of stakeholders in the 

development of their systems and measures (e.g. providers, beneficiaries, program 

officials, advocates, and citizens). The development of partnerships with stakeholders 

was identified by respondents as one of the key aspects of their success, albeit one that 

was logistically difficult, time-consuming, and challenging. 

For example, New York's State Department of Public Health sponsored a series of six, 

l-day regional workshops across the State during May 1996. Participants identified 

serious health problems in their communities, causes of those problems, and effective 

interventions. An estimated 1,400 participants attended, representing local health care 

providers, government officials, community-based organizations, educators, advocates, and 

business and labor. 

Florida's Division of Alcohol, Drug Abuse and Mental Health used workgroups with 

stakeholders--providers, clients, legislative staff, oversight staff--to develop measures. 

These workgroups were arranged around target populations, consisted of 15 to 30 
people, and met according to their own needs and schedules (some for a full day; some 

for two full days). 

For the Massachusetts Department of Mental Health, developing standards and 

indicators for residential services involved participation of Department staff, providers, 

consumers, and family members. The Department used focus groups to solicit feedback 

from these stakeholders on draft proposals it had initially prepared. The Department 

then considered the recommendations of the stakeholders as it modified the drafts. Final 

drafts were ultimately reviewed by a senior Department policy group before being 

approved by the Commissioner for implementation. 

State officials see many benefits from development and .inolp1ernentation of results-based 

system, and are wing data in various ways. However, data from these system are 

unlikely to become the sole criteria or driving force for pragmatic, budgetary, or 

contracting decisions. 

State officials who have implemented results-based systems point to two key benefits 

from their efforts: (1) obtaining information efficiently, which can be used to improve 

program performance; and, (2) empirically demonstrating program results. State officials 

also point to significant benefits from the process of developing results-based systems. 

Officials frequently mentioned the value of obtaining consensus among stakeholders on 

program missions and goals, and establishing lines of communication with key 

stakeholders. 

Other benefits may become apparent from further experience. Some of the initiatives we 

examined are still in the early stages of implementation and are not fully operational. As 

a result, we can not yet draw conclusions for these programs as to how information 

obtained through results-based systems ultimately will be applied. Too, we did not 

interview other potential users of these data--planning staff, legislators, and stakeholders, 

among others--and so we cannot draw conclusions about how they might apply data from 

these systems. 

State officials we interviewed, who have had experience applying data from results-based 

systems, indicate that performance and outcome data are most often used as part of a 

larger mix of information and interests which determine how: (1) programs are 

managed; (2) providers, contractors, and local offices are assessed; (3) budgets are 

allocated; and, (4) contract decisions are made. For example, officials in Washington 

State's alcohol and drug abuse program indicated that they had used data from their 

results-based system to refocus priorities and interventions, but only after further 

research and analysis. In Massachusetts, the mental health program uses results-based 

data in its contracting process, but does not rely exclusively or even primarily on these 

data to make its decisions. 

One interesting development is the requirement by some State legislatures or executives 

for agencies to submit outcome and performance data with their budget requests. This 

suggests that executives and legislators might be increasingly inclined to make budgetary 
and allocation decisions which at least take results-based data into account. The 

systematic collection of this information across all State programs, in addition to spurring 

activity to develop goals and measures, also allows executives and legislators to compare 

mission statements, goals, outcomes and performance across agencies and programs. 

Washington State, for example, is requiring each agency to submit 6-year strategic plans 

as part of their 1997-99 biennium budget submissions. The plans must include the 

agency's mission description, strategies, goals, objectives, timeliness, and performance 

measures and standards. Florida is implementing its Government Performance and 

Accountability Act of 1994 over an &year period. Agencies must submit performance 

measures (input, output, and outcome) and standards in their budget submissions. In 

return, they will have flexibility to reallocate dollars among categories and programs. 

The use and focus of these results-based systems makes the case for a broad definition of 

accountability, in which mission definition, public reporting, consensus building, 

partnering, and the use of information to engage partners, target technical assistance and 

close performance gaps are the key ingredients. This type of use is different from a 

definition of accountability in which the results-based data are seen as a clear and precise 

road map by which program managers can make clear distinctions between good and 

poor program providers, effective or ineffective program interventions, or adequate or 

inadequate budgets. 
In developing their results-bad accountability systems, States face several si 

challenges: issues with the measures themselves, data concern, the use of results, and 

system capacities. 

These challenges are complex and not subject to quick solutions. Many overlap and 

reinforce one another. Yet the extent to which States will be able to translate their 

efforts into improved program effectiveness or more efficient targeting of public dollars 

will largely depend on how they address these challenges and answer the questions they 

pose. 

Measures 

In order to develop qpropriate measures, States must reflect on, and make decisions 

regarding, their primary missions and focus. In the areas of preventive health and 

maternal and child health, for example, the consideration of measurement systems has 

stimulated an ongoing debate about the role and mission of local public health programs. 

Is their mission one of providing primary care to the underserved; or assessment, policy 

development, and assurance oriented towards improving the infrastructure for an entire 

community; or some combination, and if so, with what balance? Likewise, in the 

substance abuse and mental health areas, the question is similarly put: is the orientation 

towards the community at large or towards target populations (e.g. those with current 

problems or those at risk of developing them) which are served by the programs? 

In addition, States must struggle with the questions of what outcomes are really desired 

and how desired outcomes relate to accountability. For example, is a provider 

responsible for treatment only, regardless of whether the mode of treatment is effective? 

Or, more than that, is the provider responsible for whether or not the client obtains 

employment and is able to become self-sufficient? 

What k the proper mix of dinerent masures? 

States must struggle through questions of the proper mix of measures: those that are 

end outcomes, others that are "intermediate" outcomes or risk behavior-oriented, and 

others that are capacity- or process-oriented performance measures. 

Many experts consider the use of systems that are purely outcomes-based to be 

inadequate. It can be difficult to hold program officials accountable for outcomes that 

can be influenced by a wide range of factors. Results obtained through outcomes 

measurement systems often raise a host of questions that require process and capacity 

oriented data to help answer. Indeed, the use of outcomes measures in some of the 

States we visited has increased the visibility and need for research, evaluation, and 
analytical support to interpret the data and conduct follow-up inquiries, rather than 

supplanting the need for such analytical support.' 

Appreciating limitations of systems relying solely on outcomes measures, the National 

Academy of Science recommended that HHS combine outcomes measures with process 

and capacity measures in its performance partnership grants with the state^.^ The 

States, too, are adopting approaches which include a mix of measures. 

In Nebraska, for example, State officials have made an explicit distinction between 

performance measures and outcomes measures, where the former is programmatically 

oriented towards measures of effectiveness, efficiency and quality; and the latter is 

oriented to broad health status measures. They believe-this mix of measures will help 

focus on end results while providing more direct goals and measurements towards which 

program officials can work. 

In North Carolina, at the Department of Environment, Health and Natural Resources, 

planners developed "Level I" indicators (health status indicators) and "Level 11" indicators 

(output, capacity, risk behavior and process measures) which augment the information 

available through the Level I indicators. 

A long list of measures may cause those within the system to lose focus on what really 

matters and to confuse priorities. A short list can provide too little data on which to 

assess performance or make funding decisions. In Oregon, State officials are being 

criticized in some quarters for measuring too much and losing sight of real priorities. In 

New York, an earlier effort in strategic planning in the late 1980s was derailed by too 

many measures and data points that prevented adequate tracking and accountability. 

Officials now express concern that they have focused too much on a few goals and a few 

measures and are not capturing all the health priorities of the State. 

Data Concerns 

Are the data available? 

Most State efforts are intended to be practical and, therefore, use data availability as a 

key criterion for the acceptance of a measure. However, this practice may compromise 

the reliability and validity of the data. If States select measures primarily for the current 

convenience and accessibility of data, important results may not be measured and 

decisionmaking based on the measures will be inadequate and flawed. Yet, if States 

identify important measures for which data are not available, or cannot be made 

available, the system will be theoretical only. 

Many States are facing a paucity of needed data, particularly at county and local levels. 

In these situations, States must ask hard questions about whether funds are available to 

collect additional data and who will collect them. If the States use funds to develop 
Statewide management systems or impose data collection responsibilities on providers, 

they must assess whether or not the collection of additional data will be seen as an 

administrative burden or as a crucial responsibility of program management. 

Our conversations with State officials coniirmed the urgency of investing in focused 

systems that can provide the core data necessary to support State-, county-, and providerlevel measurement efforts. In Illinois and Washington, officials responsible for alcohol 

and substance abuse prevention and treatment programs emphasized the importance of 

systems which can be altered easily, and unique identifiers to facilitate data analysis, 

validation and matching. 

A particularly thorny issue facing States is the lack of data available to properly assess 

the impact of managed care arrangements on enrollee health and States' budgets. 

Almost all of the States we contacted have yet to address this question, though many are 

taking steps to do so. North Carolina does include requirements for data reporting in 

managed care contracts and several other States mentioned that they intend to do the 

same. 

Even when data are available, the timeliness of obtaining them can present significant 

problems. While the usefulness of data may be dependent on the rate of change of the 

problem or environment, often data that are 2 or 3 years old cannot be used effectively 

to assess current program performance, to sanction providers, or to allocate scarce funds. 

In part, for this reason, Florida's alcohol, drug abuse and mental health program office 

requires providers to collect and report on data. These data could be obtained by 

matching data files with other State agencies, but would take considerably more time to 

obtain using this method. 

Use of Results 

How will States develop and use standar.? 

The States we examined are generally far away from the use of standards to require 

accountability among providers or contractors. States have not been reluctant to develop 

goals for broad health status measures, but have proceeded much more cautiously in 

developing standards for measures applicable to specific programs or interventions. Even 

when included in provider contracts or requests for proposals, standards tend to be 

broad, flexible, used to trigger further discussions and analysis when not met, and used to 

promote emphasis on results. 

For example, Washington's Division of Alcohol and Drug Abuse has not implemented 

standards for its performance accountability system. This decision was very purposeful. 

The Division believes that without baseline data and more experience with the system, it 

is unwise to develop standards. State officials report that providers there continue to be 

wary about the development and use of standards. 
Florida's alcohol, drug abuse and mental health providers are given two standard 

deviations as a range of acceptable performance. In the integrated services proposal 

being prepared between New York State and its Monroe County, county officials are 

expected to obtain goals with ranges of acceptable performance. In both of these cases, 

performance outside the range would trigger additional validation, analysis, research and 

discussions, rather than automatic sanctions. 

Officials in several States expressed concern about their exposure in reporting outcomes 

and performance data. They are particularly concerned-about public and legislative 

reaction when goals are not met. They believe the data can be used inappropriately by 

stakeholders to assume that program managers have simply mismanaged or misdirected 

their efforts, when, in fact, failure might be attributable to unanticipated or 

uncontrollable external influences. These concerns of the State officials in many ways 

mirror those of the programs' providers. In Nebraska, for example, the development of 

standards was purposely set aside in order to make program officials feel "safe" as they 

worked on developing a performance measurement system. 

Some officials noted the difficulty of responding appropriately to the data in the following 

way: does failure to achieve goals mean that the program should receive more money 

(in order to help it achieve the goals) or less money (because it is a poorly performing 

organization)? To answer this question, significant analysis and interpretation of data 

must take place with a depth of understanding of environmental complexities that these 

officials believe State legislatures and the public may not possess. 

How do States balance accomtabilirp, wah pampsh@? 

The above examples also reflect the difficulty States are facing in developing 

measurement systems with the assistance and active participation of those whose 

performance is being or will be measured. In Illinois, for example, officials observe that 

their partnership with providers has not evolved as they would like because of concern 

about how standards might be applied in evaluating their performance, particularly how 

external factors beyond the providers' control might be considered. We found that this 

concern was not exclusive to health care providers. To deal with this problem, some 

States have downplayed standards, as discussed above. 

System Capacities 

WilI resource investments in data systems and analytical evaluation be adequate? 

As discussed above, the collection of key data on outcomes and performance only begins 

a process of assessing performance, not ends it. In Nebraska, for example, officials 

believe their future success depends on whether the necessary investments in these 

capabilities will be made. And Washington's alcohol and drug abuse program officials 
attribute their success in large part to having made adequate investments in nonmainframe data systems and a research capability to support focused reviews based on 

results. 

Many States noted in their conversations with us the wide differences among their local 

partners in staffing and expertise, as well as in mission definition and demographics. In 

Florida, New York, North Carolina, Ohio, Washington, and Nebraska, officials pointed 

out the variation in funding and density among the most populated and the least 

populated counties in their States. These State officials pointed out that the ability and 

interest within each of these counties to collect data, interpret it, and use it to manage 

programs varies widely. 
Our inquiry has examined the initiatives of several States in implementing results-based 

accountability systems in their public health and mental health programs. Our primary 

focus has been on the States rather than on any efforts of HHS as it moves forward with 

its performance partnership initiative. 

At the same time, from our conversations with many State officials, we have come to 

appreciate the challenges facing HHS as it considers future directions for its performance 

partnership grant initiative. The ability of the Department to address these challenges 

timely and effectively will affect the success of its efforts to implement meaningful 

performance partnerships with the States. 

Since we have not examined Federal initiatives, we can not say to what extent agencies 

are already addressing these challenges. In some cases agencies may have moved well 

beyond some of these challenges or are well underway in addressing them. Nonetheless, 

we present them here as important questions for Federal agencies that are working with 

States to develop performance partnerships. 

To varying degrees, States raised these questions with us as indicators they will use in 

assessing the success of Federal-State partnerships. We do not intend the mere 

exposition of these challenges to reflect on the progress Federal agencies are making in 

pursuing performance partnerships. Rather, we intend them to reflect only the 

complexity of the enterprise. 

Usefulness to the States 

4-Iow can K16-IS ewe that its parh~emhip approach, relying on results-based system for 

accountability, will be integrated with the States' own perfomme management @om? 

The initiatives we examined confirm the seriousness and intensity of the States in 

implementing results-based systems for their programs. The State programs described 

here, for example, have already moved beyond the early talking stages and are deeply 

immersed in the nitty-gritty work of planning, implementation, and refinement. 

As with all intergovernmental initiatives, performance partnerships require 

accommodation and consensus, as well as recognition of both common and different 

needs, goals missions, capacities, and Federal and State environmental pressures. As 

HHS works with States in developing performance partnerships, what are the lessons 

HHS can learn from States ongoing experiences with results-based accountability systems. 

Given the numerous efforts' underway in the States, how can the Federal-State 

partnership accommodate and even support the variation in State approaches? 
To what extent will State and local officials be receptive to Federal leadership in light of 

their own unique and established methods, partnerships, resources, capabilities, and 

commitments? 

Effectiveness of Partnerships 

A critical underpinning of the partnership grant approach is effective working 

relationships among numerous government stakeholders. The complexities of these 

relationships go far deeper than a casual reference to "Federal-State partnerships" may 

suggest. The experiences of the States described here indicate the essential cross-cutting 

nature of many important public health goals and the need for clear roles, consistency of 

requirements, and appropriate resources among the Federal agency partners, and 

regional and headquarters offices. What are the roles for the Federal agencies involved? 

Should one Department or agency have the lead, or should all those affected be equally 

involved? What are the roles for regional offices? What staffing resources are necessary 

for the technical assistance and dialogue essential for this new partnership approach? 

Information Exchange 

As HHHS and the States move forward with results-based accountability systems, they will 

have a continuing need for information and expertise. Open dialogue among 

stakeholders will greatly benefit the performance measurement efforts within the 

Department. Such exchanges could lead, for example, to standard definitions to be used 

across HHS, as well as models and best practices on working with States on performance 

partnerships. 

In its leadership role for performance partnership grants, the Department will need to 

address many issues. For example, how can HHS learn from others about emerging 

developments that can enhance continuous improvement among Federal agencies and 

those of the States? What mechanisms, such as clearinghouses, advisory groups, or 

Federal-State meetings, are appropriate for giving the Department essential feedback 

and for informing itself, the States, and others with evolving knowledge, new ideas, and 

new approaches? 

Data Collection 

hlbw can IHMS mmihize the usejkkss of its cwent data collection system to enhance the 

ejJiectiveness of its peflomrtce prtnenhip grants with the St~ta? 

The HHS supports extensive, significant collections of health and health-related data that 

are critical to the success of States' efforts to develop results-based systems for 
accountability. The States already rely heavily on Federal survey and epidemiological 

data for their own purposes as well as for comparisons across States based on aggregated 

data. Because these systems are complex and expensive to operate, it is important that 

the Department ensure that its efforts, in so far as practical, meet the major, common 

needs for data among the States. 

Agenda for Research 

Significant questions, both theoretical and practical, about results-based systems remain 

unanswered. These have been identified and described in other sources.1° There is 

obviously a clear role here for MHS in the evaluation of these efforts. From the 

perspective of the States, however, it is important for the Department to appreciate the 

significance of the States' research needs for informing their ongoing efforts to implement 

effective systems. 

Adequate Evaluation Ca~acitv 

It will not be sufficient for HHS, or others, to rely primarily on performance data 

reported by the States for its critical decisions about program effectiveness, allocation of 

resources, and policy changes. These data should be considered only the raw materials 

for subsequent evaluation and analysis, essential for informing deeper understandings of 

what is actually going on. The data help frame the questions to ask; they, in themselves, 

do not provide the answers. A significant threat to the long-term viability of 

performance partnership grants and other results-based systems will be the failure to link 

them with sufficient systems for analysis and evaluation that will adequately provide for 

informed decisionmaking by the Federal Government, the States, and the general public. 
This report documents some important initiatives that are underway across the country to 

develop more meaningful approaches for managing public health programs. Developing 

these systems is inherently messy, difficult and time consuming. It involves multiple 

stakeholders with complex, sometimes competing, agendas. It is an expensive endeavor 

and can make heavy demands on limited resources. Progress is neither straightforward 

nor single-tracked. Yet, as is clear from the initiatives described here, States are making 

serious efforts to develop results-based systems and are moving forward. 

We document in this report a number of benefits State officials we interviewed see 

emerging from their efforts. They bear repeating here. These benefits include: 

(1) obtaining information efficiently, which can be used to improve program 

performance; (2) empirically demonstrating program results; and, (3) obtaining consensus 

among stakeholders on program missions and goals. We believe that these are significant 

benefits to program managers and strategic planners. 

The ultimate determinant of whether these systems succeed, however, may well be their 

usefulness to those most vested in the programs: legislators, administrators, providers, 

advocates, and consumers. The challenge facing these stakeholders will be to achieve an 

appropriate balance when using the data from these systems to enhance decisionmaking 

about program improvements, budgets and contracts, and feedback to providers and 

consumers. 

We received very positive comments from the Assistant Secretary for Planning and 

Evaluation, the Substance Abuse and Mental Health Senices Administration, the Centers 

for Disease Control, the Wealth Resources and Services Administration, and the 

Secretary for Health. Their comments offered further insights on such matters as: the 

impact of the Government Performance and Results Act in helping shape performance 

partnerships; the data and performance measurement problems facing States; the 

importance of information exchange; and, the role of Federal agencies in performance 

partnerships. We were also informed that the National Research Council's Panel on 

Performance Measures and Data for Public Health Performance Partnership Grants has 

recently issued its first report entitled '!Assessment of Peformance Measures for Public 

Health, Substance Abuse and Mental Health." 

Based on these comments, changes were made to the report as appropriate. The 

complete text of these comments can be found in Appendix B. We regard them as an 

integral part of the report ind recommend them to the attention of the reader. 
Federal Public rant Programs 
Among the current block grants administered by the United States Department of Health 

and Human Services (HHS) for public health purposes are the Community Mental 

Health Services Block Grants, Substance Abuse Block Grant, Preventive Health and 

Health Services Block Grants, and Maternal and Child Health Services Block Grant. 

Consistent with the character of block grants, the legislation establishing these grants 

specifies purposes, populations to be sewed, specific activities or processes to be carried 

out, reporting requirements, and limits or minimums on the uses of funds for particular 

purposes. A brief description of each of these block grants follows below. 

The Public Health Service Act provides for block grants for community mental health 

services. The Secretary of MHS, through the Director of the Center for Mental Health 

Services, makes grants to the States each fiscal year for the purposes of providing 

comprehensive community health services under a State plan, evaluating programs and 

services carried out under the plan, and planning, administration and educational services 

under the plan. State plans provide for the establishment and implementation of an 

organized community-based system of care for adults with serious mental illness or 

children with serious emotional disturbance. The plan must contain quantitative targets 

to be achieved, although the Act only specifies that such targets include the numbers of 

such individuals residing in the areas to be served under the system. Among other 

things, specific attention must be given to: (1) reducing the rate of hospitalization for 

eligible individuals, (2) providing outreach and services to eligible homeless individuals, 

and, (3) estimating incidence and prevalence in the State of the conditions of eligibility. 

States must establish mental health planning councils to review State plans, serve as an 

advocate for individuals with mental illnesses or emotional problems, and monitor, review 

and evaluate the allocation and adequacy of mental health services within the State. 

No more than 5 percent of the total grant to a State may be used for administrative 

expenses. 

Substance Abuse Block Grants 

The Public Health Service Act provides for block grants for the prevention and treatment 

of substance abuse. The HHS Secretary, through the Director of the Center for 

Substance Abuse, makes grants to the States each fiscal year. States must allocate a 

specific minimum percentage of funds for alcohol prevention and treatment and for 

prevention and treatment activities regarding other drugs. States must also spend a 

specific portion of their funds to educate and counsel individuals not requiring treatment 

about abuse and provide for activities to reduce their risk of such abuse. 

States must submit an assessment of needs to the Secretary in order to receive a grant. 

This assessment must include information by localities within the State on the incidence 

and prevalence of drug abuse and alcohol abuse; current prevention and treatment 
activities in the State; the need of the State for technical assistance; efforts by the State 

to improve such activities; and, the extent to which availability of such activities matches 

the needs. 

States must provide for independent peer review to assess the quality, appropriateness, 

and efficacy of treatment services provided in the State to individuals under the program. 

No more than 5 percent of the total grant to a State may be used for administrative 

expenses. 

Preventive Health and Health Services B1mk Grants 

The Public Health Service Act provides for block grants-for preventive health and health 

services. Allotments under this section are for preventive health services, comprehensive 

public health services, and emergency medical services. Payments to States may be used 

for activities designed to make progress towards Healthy People 2000 objectives. 

States must develop their plans in consultation with State Preventive Health Advisory 

Committees. They must specify strategies for making progress toward improving the 

health status of the population and establish reasonable criteria to evaluate the 

performance of entities receiving payments from the State. The statute also specifies 

that States must report to the Federal Government on progress in meeting the year 2000 

health objectives and use the uniform collecting and reporting formats developed by the 

Secretary for this purpose. 

No more than 1.0 percent of the total grant to a State may be used for administrative 

expenses. 

Maternal and Child Health Services Block Grant 

The Public Health Service Act provides for funds to improve the health of all mothers 

and children consistent with the applicable health status goals and national health 

objectives established by the Secretary for the year 2000. The grants are to be used to 

provide and assure mothers and children access and quality health services, reduce infant 

mortality and incidence of preventable disease and handicapping conditions among 

children, reduce the need for inpatient and long-term care services, increase the number 

of immunized children, and other objectives. 

States must report on a number of measures specified within the statute or by the 

Secretary, including rate of infant mortality, rate of low birthweight births, rate of 

maternal mortality, rate of neonatal deaths, and other such health status indicators. 
Agency Comments 
DEPARTMENT OF HEALTH & HUMAN SERVICES Office of the Secretary 

Washington, D.C. 20201 

AuG - 7' 1997 

TO: June Gibbs Brown 

Inspector General 

FROM: David F. Garrison 

Principal Deputy Assistant Secretary 

for Planning and Evaluation \ 

SUBJECT: Comments on the OIG Draft Report: "Results-Based Systems for Public Health 

Programs," OEI-05-96-00260 

Thank you for the opportunity to review this draft report. We continue to believe in the 

importance of establishing performance partnerships for greater flexibility, improved 

accountability, strengthened relationships with States and other partners, and, ultimately, 

improved outcomes for our customers. 

The report provides a comprehensive description of current state efforts to implement resultsbased systems and the challenges involved, both at the State and Federal levels. We were 

pleased to see that the report now includes examples of performance measures in each state case 

study. The report reinforces many of the challenges that we have tried to address over the past 

several years in our performance partnership efforts. For example, CDC and SAMHSA have 

each spent the last year working to integrate some aspects of performance measurement into their 

existing grants, have recognized the importance of conducting extensive stakeholder 

consultations as an integral part of this process, and have modified their proposals to be 

responsive to states concerns. The Department also recently commissioned the National 

Academy of Sciences to develop a report on performance partnerships, Assessment of 

Performance Measures for Public Health, Substance Abuse, and Mental Health, which provides 

guidance for Departmental efforts by laying out many of the same challenges with 

recommendations for addressing them. 

We appreciate the opportunity to review and comment on this report. We believe that it will 

provide valuable guidance as programs at the Federal and State levels implement systems of 

outcome-based management, particularly as we move fonvard with our State partners on 

implementation of the Government Performance and Results Act. 
DEPARTMENT OF HEALTH & NUW SERVICES Public Health Service 

.- 

Substance Abuse and Mental 

AUG 1 1997 

Health Services Administration 

Rockville MD 20857 

TO:' June Gibbs Brown 

Inspector General 

FROM: Administrator, SAMHSA 

SUBJECT: Comments on Draft Inspector General RGort, Results-Based Systemsfor 

Public Health Programs 

Thank you for the opportunity to review this very valuable report. Not only did we find its 

conclusions sound, but the information on the States' efforts will be extremely useful to all of us 

-- Federal staff and the States -- working on SAMHSA's pilots on State performance 

measurement. 

Our general comments on Volume I will be followed by specific comments on each section, as 

well as a few comments on Volume 11. Also, attached are more detailed comments on States' 

concerns about the type of data required and the nature of accountability under a results-based 

sys tern. 

Volume 1. General Comments 

As noted, we find the Report thorough and sound. We are particularly pleased that the Report 

stresses the importance of top-level commitment to and stakeholder involvement in results-based 

systems. States have indicated repeatedly that these two factors are critical to the success of 

these efforts. We are also pleased that, on page 9, the authors state that "the use and focus of 

results-based systems" calls for a "broad definition of accountability", which would lead to 

targeting technical assistance and closing gaps, as opposed to a more narrow definition that 

would provide clear distinctions between good and poor programs. 

The Report could be even stronger, however, if it were updated in the following ways: 

Reference to the National Research Council's Panel om Performance Measures and 

Data for Public Health Performance Partnership Grants: This Panel recently issued 

its first report, Assessment ofPerformance Measures for Public Health, Substance Abuse 

and Mental Health, which provides a "framework" for performance measurement which 

clarifies that, often, a combination of process, capacity, and outcome measures is 

necessary as State and Federal officials seek to determine the contribution made by their 

resources, but that the link between process and capacity measures to outcomes must be 

well-documented. It atso provides a set of "assessment guidelines" to assess the 

appropriateness of individual measures, including that they be: results-oriented; 
AUG-04-97 MON 09:29 AM 

Page 2 - June Gibbs Brown 

meaningful and understandable; valid, reliable and responsive; and that data are available 

to "support the measure." It also calls upon the Department to work with the States "to 

identify and develop common definitions and methods that will contribute to 

standardizing measurements". Finally, we urge the Report's authors to adopt the 

National Academy of Sciences Panel definitions, which include terms such as 

"performance measure" ("a qualitative indicator that can be used to track progress towird 

an objective"). 

a Clearer link to the Government Performance and Results Act: Although GPRA is 

referred to in the Report's introduction (p.2), its conceptual link to State efforts'(i.e., the 

Federal government is engaged in the same process of engaging stakeholders, identifying 

goals, and formulating measures as the States) and its impact on Federal receptiveness to 

State needs (i.e., better data on Federal programs) md "lessons learned" due to resultsbased management should be emphasized. 

State Challenges 

Attached is a background document prepared by the SAMHSA Center for Substance Abuse 

Prevention concerning data and measurement issues faced by States in implementing resultsbased systems. Three key points are made: 

Level sf aceounftability (i.e., client, program, system): "Most States appear more 

comfortable with accountability being tied to changes at the program level .... The major 

problems associated with collecting this level of data include cost/mmpower, programs 

that receive funding from multiple sources, and the fear of the providers that they will 

lose funding depending on the results." 

rn National vs. State Data: "...while national surveys can resolve the issues of variation in 

definition, design .ad methodology across States, the data are of little practical use at the 

State and sub-state level. They need data at the lowest common denominator possible to 

be usehl for planning and funding. One suggestion is to provide States with suflticient 

hds to conduct surveys (much as in State needs assessment contracts) that provide data 

at a level usehl for States, but require some common data collection (and provide 

consistent methodologies with training) so that they can be aggregated to the national 

level." 

"State data infrastructure capacity is a major issue. Most simply don't have sufficient 

funds, manpower, and or equipment to develop, implement, and maintain a performance 

measurement system." 
AUG-04-97 MON 09: 30 AM 

Page 3 - June Gibbs Brown 

Federal (HHS) Challenges 

We are pleased that language has been added, on page 15, that acknowledges that "agencies may 

. .. be well underway in addressing [these challenges]" because SAMHSA has been engaged in a 

true partnership with the States over the past 1 112 years to shape the performance partnership 

effort related to its block grants. As a result of this partnership, we have proposed legislation to 

provide State flexibility (waivers of certain conditions in exchange for measuring performance) 

and to strengthen State data infrastructure. 

We have three minor comments on the challenges: 

e Under "Information Exchange", we urge the authors to note the importance of 

information exchange on performance measurement efforts within the Department. Suck 

exchange could lead, for example, to standard definitions to be used across IKKHS, as well 

as models an^ Lest practices on working with States on performance partnerships. 

e Under "Agenda for Research", authors should note the role for HHS in evaluating and 

assessing these efforts as well. 

@ Under "Adequate Evaluation Capacity", we recommend that the specific importance of 

linking these efforts to GPRA be clarified (for example, through the block grants by 

ultimately usin: State performance indicators 3s block grant GPRA measures). 

Vohme 2. State Case Studies 

Page 33 - Substance Abuse treatment is not noted for Massachusetts. 

Page 43 - Minnesota has an outcomes monitoring system in place for substance abuse, but this 

was not noted. 

Again, thank you for the opportunity to comment on this very worthwhile and timely Report. 

We look forward to receiving a copy of the Final Report. 

Nelba Chavez, Ph.D. 

Attachment 
Attachment 

CENTER FOR SUBSTANCE ABUSE PREVENTION 

DISCUSSIONS WITH STATES ON RESULTS-BASED SYSTEMS 

(FL, LA, GA, IL, MA, MN, NE, NY, NC, OH, OR, WA) 

TYPES OF DATA 

Statewide or program 

States made it clear that, while it's easier to obtain State level data (e.g., using social indicators, 

or national or Statewide surveys), they have no confidence in their usefulness re: Performance 

Partnership Grants (PPGs) because the contribution of the effects of the programs they hnd and 

the limited populations they serve on changes in State level data is likely to be nonsignificant. In 

addition, there is no controlling for the effects of other substance abuse prevention programs on 

those data. Therefore, it would seem futile to hold the single State agency accountable for 

changes in results at that level. This holds true, to a lesser extent, even at a county level for most 

States. The exceptions may be those States that have established State inter-agency coordination 

of programs and even pooling of resources. This is by far the exception, rather than the rule. 

Most States appear more comfortable with accountability being tied to changes at the program 

level. The major problems associated with collecting this level of data include costhanpower, 

programs that receive fknding from multiple sources, and the fear of the providers that they will 

lose funding depending on the results. 

Another prob!em is that, typically, States are unable (as opposed to treatment) to do long term 

follow-up. Therefore, while changes in risk factors/status may be detected after involvement in 

prevention programs, they will be unlikely to track long-term outcomes such as use 5-10 years 

later (e.g., programs for elementary aged children). 

In the CSAP Minimum Data Set pilot, interested States have focused on data at the provider 

level. CSAP is currently modifying the system tested by eleven States in the Phase I pilot which 

was completed May 3 1, 1997, and pursuing the refinement of definitions, methodologies and 

procedures for the collection of five intermediate and outcome performance measures agreed 

upon by 27 States. These were identified using a multiple focus group approach and using 

selection criteria SO identified via consensus of the participating States. It is important to 

recognize however, that participation is voluntary and, by the time the data collection phase 

begins, CSAP expects some States to drop out. 
AUG-04-97 MON 09: 32 AM 

STATE ISSUES 

Measures are mother problem. Some valid data sources andfor instruments exist in some areas. 

There is much diversity among the States re: which measures to use, how they are defined etc. 

This is, therefore, a problem at the Federal level for the collection of the common core data. In 

fact, in our pilot, States used the National Academy of Sciences (NAS) criteria with the addition 

of the criterion "cheap and easy to use". Other considerations (discussed in NAS) of importance 

to States are the frequency of the data collection (especially if national survey); the speed with 

which results are obtained (a frequent criticism ); and the ability of the measure to be impacted 

by an activity (e.g., not likely to affect divorce rates). 

On the other hand, while national surveys can resolve the issues of variation in definition, design 

and methodology across States, the data are of little practical use at the State and sub-state level. 

They need data at the lowest common denominator possible to be useful for planning and 

funding. One suggestion is to provide States with sufficient funds to conduct surveys (much as 

in State needs assessment contracts) that provide data at a level useful for States, but require 

some common data collection (and provide consistent methodologies with training) so that they 

can be aggregated to the national level. State data infkastmcture capacity is a major issue. Most 

simply don't have sufficient funds, manpower, and/or equipment to develop, implement, and 

maintain a performance measurement system. 

States, while anxious, appear to see the importance and utility of performance measurement, 

especially for planning and resource allocation, and to tie to needs assessments. The issue here is 

whether the results will be used to provide technical assistance to those programs and States 

where results are disappointing, or whether the consequences will be punitive in nature. 

Although several States have Statewide performance measurement models with required 

performance measures, more use a menu approach and several permit local development and 

identification of local performance measures as long as they address Statewide goals andfor 

objectives. The more the variation, however, the less likely the data can be supportive of the 

PPG concept. 

Other related issues concern applying Statewide models to programs which, although targeting 

the same performance measure, often vary in intensity and scope. 
Public Heatth Service 

DEPARTMENT OF HEALTH & HUMAN SERVICES Centers for Disease antrd 

and Prevention (CDC) 

Date 

From 

Subjec 

To 

-AUG 0 6 1997 

Associate Director for Management and Operations, CDC 

CDC Comments on Office of Inspector General (OIG) Draft Report, "Results-Based 

Systems for Public Health Programs" (OEI-05-96-00260) 

June Gibbs Brown 

Inspector General 

Thank you for the opportunity to review the draft report. Attached are CDC's comments. 

If you should have questions concerning these comments, please have your staff contact 

Carolyn Russell, Director, Management Analysis and Services Office, at (404) 639-0440. 

Attachment 
CDC Comments om the Office of the Inspector General Draft Report, 

"Results-Based Systems for Public Health Programsy9 

Vol. 1, Pg. i, paragraph 3, 

We recommend restructuring the last sentencc as follows: These arrangements would 

combine various categorical grants into performance partnerships and would reshape 

several of its bIockgrantprograms. 

Vol. 1, Pg. ii, Challenges for the States, Paragraph 3, 

We recommend that the data objective refer to the need to take into consideration the 

availability of a system necessary to validate the data. 

Change the first sentence to read, " ... tle necessary data are available ..." 

Vol. 1, Pg. 8, paragraph 5, last sentence, 

Change " . . . this data. . . " to " . . . these data. . . " The same change should be made in 

the last sentence of paragraph 6. 

Vol. 1, Pg. 10-1 7, State and Federal Challenges, 

There are many questions and issues raised regarding the implementation of 

performance-based measures. The format in which the sections were written is 

confusing, for example, questions are answered by other questions. We recommend 

explaining that the section only outlines the challenges and does not attempt to provide 

solutions. 

Vol. 1, Pg. 1 1, paragraph 5, 

The reliability and validity of the measures are a major issues not addressed here. 

Vol. 1, Pg. 12, paragraph 3, 

The paragraph states that the major problem facing states is the lack of data for persons 

enrolled in managed care; however, the real issue facing states is the need to provide 

health care coverage and the lack of data available to properly assess the impact of 

managed care. This point should be clarified. 
Vol. 1, Pg. 12, paragraph 4, 

The paragraph entitled, "Are the data timely?'refers to the reconsideration of the 

usefulness of data that are two or three years old. It is also important to consider the rate 

of change of the problem and the environment before making a judgement on the 

usefulness of data. 

Vol. 1, Pg. 13, paragraph 4, 

The problem seems overly simplified by focusing solely on health care providers. This 

paragraph does not reflect that problems resulting from the attempt to balance 

accountability with partnership, extend beyond health care providers participating in the 

development of measurement systems. 

Vol. 1, Pg. 15, last paragraph, 

Change the word, "extendv in the last sentence, to "extent." 
DEPARTMENT OF HEALTH & HUW SERVICES Public Health Service 

Health Resources .and 

Services Administration 

Rockville MD 20857 

AUG 2 5 1997 

TO: Inspector General, OS, DHHS - 

FROM : Acting Deputy Administrator 

SUBJECT: Office of Inspector General Draft Reports "ResultsBased Systems For Public Health Programs Volume 1: 

Lessons From State Initiativesf1 OEI-05-96-00260 and 

8oResults-Based Systems For Public Health Programs 

Volume 2: State Case Studies" OEI-05-96-00261 

HRSA has reviewed the subject draft reports and has no comments. 

Staff questions may be referred to Michael Herbst on (301) 

443-5256. n 
Endnotes 
1. For a detailed discussion of general concepts of accountability and specific issues of accountability in 

Federal block grant programs, see President's Council on Integrity and Efficiency, Committee on Inspection 

and Evaluation, Accountability for Block Grants, July 1%. 

2 . For a discussion of Federal grant approaches to funding public health activities, see U.S. Department of 

Health and Human Services, Office of Inspector General, Federal Approaches to Funding Public Health 

Programs, OEI-01-94-00160, July 1W5. 

3 . The President's Fiscal Year (FY) 1996 budget included a proposal for 16 Performance Partnerships in 

the Public Health Service, including the consolidation of various grant programs within Centers for Disease 

Control (CDC), Health Resources and Services Administration (HRSA), and the Substance Abuse and Mental 

Health Services Administration (SAMHSA). In the FY 1997 budget,ihe Administration continued to propose 

performance partnerships for CDC, HRSA and SAMHSA In the FY 1998 budget, the Administration 

emphasized performance partnerships within SaMHSA only. 

Both Representative John Dingell and former Senator Nancy Kassenbaum introduced bills in the 104th 

Congress that would have implemented the performance partnership model for public health service programs, 

and specifically for mentrl health and substance abuse block grants. 

4. The National Research Council's Panel on Performance Measures and Data for Public Health 

Performance Partnership Grants has recently issued its first report: "Assessment of Peformance Measures for 

Public Health, Substance Abuse and Mental Health." 

, 5. A recent report by the Institute of Medicine, summarizing a workshop on performance monitoring for 

improving community health, identified a wide range of activities by Federal and non-Federal organizations. 

For example: 

The National Committee for Quality Assurance has developed the Health Plan Employer Data and 

Information Set to collect standardized information from health plans for their own use in assessing 

their performance, and to help purchasers make decisions among health plans. The Committee is also 

active in assisting health plans to improve their capacities for data collection, analysis and reporting. 

The Joint Commission on Accreditation of Healthcare Organizations is moving beyond standards to 

develop indicators for assessing the reliability of the standards in predicting actual performance. The 

Indicator Measurement System was originally developed for hospitals and is now being expanded. 

The HHS is responsible for Healthy People 2000, a set of objectives regarding the health promotion 

and disease prevention with goals to be accomplished by the issued model standards for translating 

those national year 2000. The American Public Health Association has set goals into community 

action plans. 

The HHS, through the Agency for Health Care Policy and Research, funds research on health care 

outcomes and quality measurement. 

The HHS, through the National Center for Health Statistics, assembles vital statistics and conducts 

significant health related surveys. 

For further information, see National Academy of Sciences, Institute of Medicine, Using Performance 

Monitoring to Improve Contntunir)) Health: Exploring the Issues, Workshop Summary, 1996. 
6. Among the national health organizations working with HHS were: the Association of State and 

Territorial Health Officers, the National Association of State Alcohol and Drug Abuse Directors, the National 

Association of State Mental Health Program Directors, and the National Association of City and County 

Health Officials. 

7 . National Academy of Sciences, Committee on National Statistics, Assessment of Performance Measures 

in Public Health, Phase I Report, Draft: for Co@ntent, September 1996. The report proposed performance 

measures in the areas of chronic disease, HIV/STD/Tl3, mental health, immunization, substance abuse, sexual 

assault prevention, disability prevention, and emergency medical services. 

8 . The U.S. General Accounting Office, which has written extensively on this subject, noted in a September 

1995 report: 

While State efforts will certainly be closely tied to block grant results, outcomes will just as certainly 

be affected by factors outside the control of state administrators. Because of the role that these 

variables may play, evaluation will need to kolate the effect of outside factors on state programs. 

[Emphask added.] For example, the incidence of low birth weight infants depends not only on the 

efforts of a particular state and local agency to fill the gaps in prenatal care, but also on many other 

demographic and situational factors, such as regional employment trends and demographic patterns ... 

See U.S. General Accounting Office, Block Grants: Issues in DesigningAccountability Provisions, GAOIAIMD95-226, September 1995. 

9 . National Academy of Sciences, Committee on National Statistics, Assessment of Performance Measures 

in Public Health, Phase I Report. 

10. See, for example, Department of Health and Human Services, Office of Inspector General, Federal 

Approaches to Funding Public Health Progrants, OEI-01-91-00160, July 1995.