Volume 1:  Lessons From State Initiatives 
Ins-p ector General 
SEPTEMBER 1997 
OEI-05-9640260  This report describes initiatives underway in 11 states to implement results-based systems 
for managing public health programs.  It draws on the experiences of these States for 
insights into the development and implementation of such systems. 
Government-wide interest in enhancing the performance and accountability of 
government programs, including its grants programs, intensified with the publication, in 
1993, of the Vice President's Report on the National Performance Review.  As a result 
of this and other governmental and nongovernmental influences, policymakers have been 
actively considering changes in Federal grantmaking and oversight authorities.  Some 
proposals call for combining several categorical and block grants together and replacing 
them with performance partnerships.  Performance partnership grants would be 
negotiated between Federal Government agencies and the States. 
The United States Department of Health and Human Services (HHS) has been 
considering performance partnership grants for some of its public health programs for 
several years.  These arrangements would combine various categorical grants into 
performance partnerships and would reshape several of its block grant programs. 
The Assistant Secretary for Planning and Evaluation asked the Office of Inspector 
General to identify and examine State initiatives that use outcomes measures to assess 
the performance of their public health programs. 
This report is based on a review of results-based initiatives in the preventive health, 
maternal and child health, substance abuse and mental health programs of 11 States.  We 
conducted onsite discussions in seven of these States:  Florida, Illinois, Massachusetts, 
Nebraska, New York, North Carolina and Washington.  We interviewed officials by 
telephone in four other States:  Georgia, Minnesota, Ohio and Oregon.  A companion 
volume contains brief descriptions of these States' initiatives. 
STATE INITIATIVES 
lThe results-based accountability initiatives we mrnined are generaw of two typ: 
(1) broad @om  at Statewide strategic planning and priody setting, and (2) systems focused 
on target populations and spec@ program interventions. 
W eSta tes, in initiating thkir resulfi-based systems, have several characterirtics in common: 
public pressures for better government, top-level commitment, and atensive stakeholder 
involvement.  State oficicak see mny ben@B fp-srn devebpmnt and kpapkmn~ationo f rdb-based 
system a d a re using data in vanbus ways.  However, data from these system are ranlikely 
to becom the sole criteria or diving force for prpqamr~labic,b udgemy, or contracting 
decisions. 
In hebophg their rd&-bmed accountabiliy systemy Stata face several Isignifiant 
chahnges:  kues wibh the mbzs~6pesth emselves, data con.cemY the use of r d ya d  
system cap~citk. 
Measurement:  States must ensure that measures are related to program missions and 
goals, adopting a mix of different kinds of measures in order to balance the limitations of 
outcomes, output and input measures.  They must also ensure that measures are 
appropriately focused on true priorities. 
Data:  In considering measures, State officials struggle with whether the necessary data 
are available and reliable, as well as whether the data can be obtained in a timely 
manner in order to make necessary programmatic adjustments. 
Standards:  States must consider whether, and how, they will adopt standards associated 
with the measures they select.  If they adopt standards, what are the penalties or results 
if the standards are not met, and what conclusions can be drawn about program or 
contractor performance? 
Capacities:  States must also fairly assess the capacities within their system to analyze and 
interpret the information they obtain through their measurement and data collection 
systems. 
7Iaei-e are Isigni@ant  challenges facing HHS as it considern m e dir ectbnr for its 
perfomurnce paptnernhip grant initiative.  7lw abiliry ofH6-IS to address these chadlenges 
h e &  and flectively will aflect the smcess of irs gorts to impkmenr meanin@l 
perfomurnce partnepship with the States.  These challenges include: 
Usefulness to States:  How can HHS ensure that its performance partnership approach, 
relying on results-based systems for accountability, will be integrated with the State's own 
performance management efforts? 
Effectiveness of Partnerships:  What kind of administrative infrastructure can best 
support the inter-agency partnerships? 
Information Exchange:  How can HHS best support an exchange of information among 
Federal and State agencies and with the research and academic communities?  Data Collection:  How can HHS maximize the usefulness of its current data collection 
systems to enhance the effectiveness of its performance partnerships with the States? 
Research Agenda:  How can HHS best leverage its resources to ensure a research 
agenda that addresses the major information needs of the States and its own agencies? 
Evaluation Capacitv:  How can HHS ensure that its agencies and the State agencies with 
whom they partner have adequate capacity to evaluate the performance-based data 
emerging from these systems? 
CONCLUSION 
This report documents some important initiatives that are underway across the country to 
develop more meaningful approaches for managing public health programs.  Developing 
these systems is inherently messy, difficult and time consuming.  It involves multiple 
stakeholders with complex, sometimes competing, agendas.  It is an expensive endeavor 
and can make heavy demands on limited resources.  Progress is neither straightforward 
nor single-tracked.  ;et,  as is clear from the initiatives described here, States are making 
serious efforts to develop results-based systems and are moving forward. 
We document in this report a number of benefits State officials we interviewed see 
emerging from their efforts.  They bear repeating here.  These benefits include: 
(1) obtaining information efficiently, which can be used to improve program 
performance; (2) empirically demonstrating program results; and, (3) obtaining consensus 
among stakeholders on program missions and goals.  We believe that these are significant 
benefits to program managers and strategic planners. 
The ultimate determinant of whether these systems succeed, however, may well be their 
usefulness to those most vested in the programs:  legislators, administrators, providers, 
advocates, and consumers.  The challenge facing these stakeholders will be to achieve an 
appropriate balance when using the data from these systems to enhance decisionmaking 
about program improvements, budgets and contracts, and feedback to providers and 
consumers. 
AGENCY COMMENTS 
We received very positive comments from the Assistant Secretary for Planning and 
Evaluation, the Substance Abuse and Mental Health Services Administration, the Centers 
for Disease Control, the Health Resources and Services Administration, and the 
Secretary for Health.  Their comments offered further insights on such matters as:  the 
impact of the Government Performance and Results Act in helping shape performance 
partnerships; the data and performance measurement problems facing States; the 
importance of information *exchange;a nd, the role of Federal agencies in performance 
partnerships.  We were also informed that the National Research Council's Panel on 
Performance Measures and Data for Public Health Performance Partnership Grants has  recently issued its first report entitled 12ssessmento f  Pe~omzanceM  easures for  Public 
Health, Substance Abuse and Mental Health." 
Based on these comments, changes were made to the report as appropriate.  The 
complete text of these comments can be found in Appendix B.  We regard them as an 
integral part of the report and recommend them to the attention of the reader.  I 3  O F  C O N T E N T  
PAGE 
.......................................... 
E % E m S m Y   i 
................................................. 
rnOD1UrnON  1 
.......................................... 
STATE  5 
.................... ...................... 
STATE  10 
; 
.......................  .............. 
FEDERAL C33AILENGE  15 
.;. 
.................................................. 
CONcCkUSION  18 
MPENDIrn 
........................ 
A  Federal Public Health Block Grant Programs  A-1 
........................................... 
B:  AgencyComents  B-1 
................................................... 
C Endnotes  C-1  This report describes initiatives underway in 11 States to implement results-based systems 
for managing public health programs.  draws on the experiences of these States for 
insights into the development and implementation of such systems. 
BACKGROUND 
The Federal Government currently administers both categorical and block grants to the 
States for the purposes of addressing public needs.  Categorical grants are for specific 
and narrowly defined purposes, and generally include federally-specified eligibility and 
reporting requirements.  Block grants are broader, encompass a larger range of purposes 
and goals, and contain fewer Federal prescriptions. 
Government-wide interest in enhancing the performance and accountability of 
government programs, including its grants programs, intensified with the publication, in 
1993, of the Vice President's Report on the National Performance Review.  As a result 
of this and other governmental and nongovernmental influences, policymakers have been 
actively considering changes in Federal grantmaking and oversight authorities.' 
Some proposals call for combining several categorical and block grants together and 
replacing them with performance partnerships.  These grants would be negotiated 
between Federal Government agencies and the States.  They would allow States greater 
flexibility in meeting key national and State objectives.  In return, States would provide 
the Federal Government with performance data and work with the Federal Government 
to establish goals. 
The United States Department of Health and Human Services (HHS) has been 
considering performance partnership grants for some of its public health programs for 
several years.2  (See Appendix A for a fuller description of the Federal block grants for 
preventive health, maternal and child health, substance abuse and mental health.) 
Over the past several years, the President has included provisions for performance 
partnership grants in his budget.  Several legislative proposals to create such grants have 
also been introduced in the  While not yet enacted, these proposals reflect 
the growing consensus about the need to improve the management and structure of 
Federal-State grant programs for health.  The concept of performance partnerships for public health is also rooted in, and made 
possible by, activity at all levels of government and in the private sector to assess the 
impact of various health care interventions on health status and clinical outcomes. 
Spurred on by high costs, quality concerns, technological advancements in data storage 
and processing, increasing penetration of the health care marketplace by managed care 
organizations, and consumer demands for information, Federal, State, local, nonprofit 
and private organizations have been working to improve the way health status and 
outcomes are measured and used.4  This movement was given added impetus by the 
adoption of Total Quality Management principles by many healthcare organizations 
likewise seeking to introduce, within the confines of their own systems, the collection and 
use of meaningful data on patients and providers in order to identify and implement 
system improvements. 
These developments, along with other Federal initiatives, have formed a backdrop for 
HHS efforts on performance partnership grants in its public health  program^.^  Federal 
initiatives, such as the Government Performance and Results Act, seeks to improve 
agency performance and results through results-based management.  This technique uses 
the same process of ,,igaging  stakeholders, identifying goals, and formulating 
performance measures as that employed by the States. 
In mid-1995, HHS requested the National Academy of Sciences to examine and make 
recommendations for specific performance measures that could be used in public health 
performance partnerships over the next few years.  In 1996, HHS, in collaboration with 
several national health organizations, convened a series of regional meetings with States 
to discuss current activities in developing performance measurement systems for public 
health  program^.^  The results of these meetings were provided to the Academy to assist 
in its analysis.  The Academy issued its draft report in the fall of 1996.~ It is now 
assessing further developmental work needed in data systems to support performance 
measurement systems. 
The Assistant Secretary for Planning and Evaluation asked the Office of Inspector 
General to identify and examine State initiatives that use outcomes measures to assess 
the performance of their public health programs.  Its primary interest lies in knowing 
more about the nature, extent, and uses of these outcomes measurement systems, and 
lessons learned and challenges faced by the States, building upon information supplied at 
the regional meetings sponsored by the Department. 
We contracted with Penny Thompson, a principal in Management Evaluation Training, 
LLC, an evaluation and management consulting firm, to undertake this study.  The Office 
of Inspector General staff were project officers and participated in all phases of the 
study. 
In t,his report, we focus on 11 States' initiatives.  We intend this report to serve as a 
resource document that offers insight into the experiences thus far of a number of States  that have engaged in the process of developing outcomes measurement systems for 
public health.  It is not an assessment of the outcomes measures themselves, their 
appropriateness, or adequacy.  It is not exhaustive of all State efforts or even all efforts 
in the States we examined. 
A companion report, Results-Based Systems For Public Health Programs,  Volume 2:  State 
Case Studies, OEI-05-96-00261, contains short, descriptive summaries of the State 
initiatives we examined. 
This report, and its companion volume, are based on a review of results-based initiatives 
in the preventive health, maternal and child health, substance abuse and mental health 
programs of 11 States.  We conducted onsite discussions in seven of these States: 
Florida, Illinois, Massachusetts, Nebraska, New York, North Carolina and Washington. 
We interviewed officials by telephone in four other States:  Georgia, Minnesota, Ohio 
and Oregon.  Two other States included in our first round of State contacts, Colorado 
and Texas, were dropped from our interviews due to resource constraints. 
We initially wrote to all officials in all the States mentioned above and requested 
available documents on their efforts.  These documents served to confirm the presence of 
reported efforts and to form the basis for additional discussions with State officials.  As 
we interviewed State officials onsite and by telephone, we often received or requested 
additional documentation which we then reviewed as well. 
The criteria we used in selecting the State initiatives described here included: 
(1) suggestions from government and non-government experts, researchers, and analysts 
identifymg States with positive experiences in developing and implementing results-based 
systems; (2) our own review of documents associated with individual State initiatives; 
(3) geographic representation and program balance to reflect the areas of maternal and 
child health, substance abuse, mental health, and preventive health; and, (4) our own 
judgments about whether particular initiatives were sufficiently different from, and less 
well known than, others. 
We conducted this study in accordance with the Quality Standards for Inspections issued 
by the President's Council on Integrity and Efficiency. 
As we reviewed the available literature on this subject, designed this study and conducted 
these interviews, we became increasingly sensitive to the importance of consistent, clear 
terms in describing efforts in these areas.  Terms and definitions used for the purposes of 
this report and its companion volume include the following: 
Results-Based System:  An initiative focused on using measurement systems to gauge 
program outcomes and effectiveness, with accountability attributes which might include  public reporting, goal setting, and standards or requirements for meeting goals applied to 
program officials, providers, contractors or grantees. 
~utcomesM' easures:  Measures reflecting ultimate programmatic results, including health 
status and risk behaviors. 
Performance Measures:  Measures reflecting a program's more immediate effectiveness 
or efficiency, including activity levels, and direct accomplishments as a result of services 
rendered. 
Standard:  A measure set as a requirement and expected threshold of performance for 
States, counties, providers or other partners in the results-based system. 
Goal:  A measure representing the end to which efforts by States, counties, providers or 
other partners in the results-based system are aimed.  The results-based acmuntability initiatives we exmind are gene*  of two types: 
(1) broad efforts at Statewide strategic planning and priority setting, and (2) systems 
focused on target populations and specific program interventions. 
Some States are developing results-based systems to assess progress towards broad, 
health-related goals for the States' populations.  The development of measures in these 
cases is usually connected with strategic planning efforts-designed to identify Statewide 
priorities and sometimes with reorganizations of health functions within a State.  These 
efforts may also be duplicated at the local level, where officials adjust goals, priorities 
and measures based on their own assessments of local populations' health status and 
needs.  We found examples of such efforts in Georgia, Florida, Minnesota, New York, 
North Carolina, Oregon and Washington State. 
Georgia's efforts are illustrative of State efforts of this type.  In 1994, the State's 
Governor appointed a 21 member Policy Council for Children and Families.  The 
following year, the Georgia legislature statutorily established the Council as a 19 member 
panel composed of leaders in business, child and family advocacy, local government and 
religion along with State directors of cognizant agencies.  The Council issued a report in 
1994 outlining five results areas that it seeks for children and families, including healthy 
children.  The Council established a Results Accountability Task Force which established 
26 benchmarks for the five results areas identified by the Council.  Benchmarks for 
healthy children include:  (1) reducing the percentage of children who have untreated 
vision, hearing or health problems at school entry; and, (2) reducing the pregnancy rate 
among school age girls. 
New York's efforts are ..lso illustrative of State efforts of this type.  In the past year, New 
York's State Department of Public Health has initiated a process to develop results- 
based accountability systems for its Statewide public health programs.  As a first step, it 
initiated a comprehensive review process that identified 12 public health areas of high 
priority for future action.  Each priority area has measurable objectives to guide State 
and community actions for improving the health of New Yorkers. 
Program Spec@c Effor~s 
Some States are developing results-based systems to assess the effectiveness of particular 
interventions and programs.  In these cases, measures are often developed in connection 
with efforts to develop consensus and agreement on program missions and expected 
results.  Program specific efforts are generally focused on the target populations actually 
receiving some kind of service from the program.  We found examples of such efforts in 
Florida, Illinois, Ohio, Massachusetts, New York, North Carolina and Washington State.  Ohio's efforts in its Early Start Initiative are illustrative of State efforts of this type. 
Ohio's Early Start Initiative is focused on providing integrated, preventive services to 
families with infants and toddlers at high risk for abuse, neglect or developmental 
disabilities.  Currently, 30 of Ohio's 88 counties are participating in the Early Start 
initiative.  An essential part of the project was the development of performance goals 
and data collection protocols. 
The Massachusetts Department of Mental Health relies heavily on contracts with local 
provider agencies to deliver its service programs.  Its Area Offices, with their citizen 
advisory boards, negotiate with and monitor these providers of both inpatient and 
community-based services.  The Department uses a system of results-oriented 
performance measures with its local providers of services. 
New York's State Department of Health is working with Monroe, County health officials 
on an initiative to:  (1) support child and family health services through a single funding 
stream of combined State and Federal funding sources, and (2) rely on results-oriented 
objectives and indicators for ensuring accountability for performance.  The Department 
submitted the propowdlt o the Federal Government for review in late summer 1996. 
As evident from the above, often both kinds of efforts are taking place at once, but they 
may or may not be explicitly linked.  We found an example of an explicitly linked effort 
in Nebraska, where program officials have designed a system to use outcome 
measurements in conjunction with program specific performance measures.  Nebraska 
has initiated an ambitious project in which, within 1 year's time, a complete 
reorganization of State health functions will have been designed and implemented.  As 
part of this effort, a performance accountability system has been developed to identify 
outcomes, outcome indicators and performance measures for programs supported by the 
new Health and Human Services organization. 
In examining State efforts, we are able to categorize systems into two broad 
classifications.  Beyond that, because of the limited scope and duration of our visits, 
document reviews and telephone interviews, we cannot present more refined models or 
typologies of States' efforts.  Within the two broad classifications presented here, the 
variety of efforts reflects a wide range of choices available to State and local authorities 
in developing systems they consider appropriate to their needs, available resources, and 
other legislative and administrative imperatives. 
In describing the State efforts, the data presented here should be emphasized as a 
snapshot in time in a constantly changing environment in which State efforts will certainly 
evolve.  Perhaps even dramatic changes in direction will occur with significant changes in 
State and local leadership, resources and other legislative and administrative imperatives. 
(For a more detailed description of these initiatives, see the companion report, Results- 
Based Systems For Public Health Programs, Volume 2:  State Case Studies, OEI-05-96- 
00261.)  These States, in initiating their results-based system, have several characteristics in 
csmsn:  public pressures for better governen4 toplevel csdtmenf and extensive 
stakeholder hvoPvement 
These results-based initiatives have been influenced by:  the performance measurement 
movement that has become increasingly influential throughout government; the demands 
of State legislators and the public they represent for more evidence of results and 
program achievements; and, the interest in health reform at both State and Federal 
levels.  Many State public health programs have been particularly influenced by the 
Healthy People 2000 initiative that established national public health objectives, goals, 
and standards for the nation to achieve by the year 2000.  This document has sewed as a 
critical foundation and framework for several of the State efforts. described here.  It has 
helped the States to refocus their public health programs to achieve specific outcomes.  It 
has also served a larger purpose of familiarizing those outside the public health 
community with a different approach for thinking about accountability in these public 
services. 
Most of the State initiatives described here were either initiated or strongly backed by 
top-level State or agency officials.  Sometimes, this commitment was in the form of a 
catalytic event, such as the major reorganization of a department or a major piece of 
legislation that required results-based systems.  Other times, the commitment came from 
a newly elected Governor or a newly-appointed department head. 
All the State initiatives described here have involved a wide variety of stakeholders in the 
development of their systems and measures (e.g. providers, beneficiaries, program 
officials, advocates, and citizens).  The development of partnerships with stakeholders 
was identified by respondents as one of the key aspects of their success, albeit one that 
was logistically difficult, time-consuming, and challenging. 
For example, New York's State Department of Public Health sponsored a series of six, 
l-day regional workshops across the State during May 1996.  Participants identified 
serious health problems in their communities, causes of those problems, and effective 
interventions.  An estimated 1,400 participants attended, representing local health care 
providers, government officials, community-based organizations, educators, advocates, and 
business and labor. 
Florida's Division of Alcohol, Drug Abuse and Mental Health used workgroups with 
stakeholders--providers, clients, legislative staff, oversight staff--to develop measures. 
These workgroups were arranged around target populations, consisted of 15 to 30  people, and met according to their own needs and schedules (some for a full day; some 
for two full days). 
For the Massachusetts Department of Mental Health, developing standards and 
indicators for residential services involved participation of Department staff, providers, 
consumers, and family members.  The Department used focus groups to solicit feedback 
from these stakeholders on draft proposals it had initially prepared.  The Department 
then considered the recommendations of the stakeholders as it modified the drafts.  Final 
drafts were ultimately reviewed by a senior Department policy group before being 
approved by the Commissioner for implementation. 
State officials see many benefits from development and .inolp1ernentation of results-based 
system, and are wing data in various ways.  However, data from these system are 
unlikely to become the sole criteria or driving force for pragmatic, budgetary, or 
contracting decisions. 
State officials who have implemented results-based systems point to two key benefits 
from their efforts:  (1) obtaining information efficiently, which can be used to improve 
program performance; and, (2) empirically demonstrating program results.  State officials 
also point to significant benefits from the process of developing results-based systems. 
Officials frequently mentioned the value of obtaining consensus among stakeholders on 
program missions and goals, and establishing lines of communication with key 
stakeholders. 
Other benefits may become apparent from further experience.  Some of the initiatives we 
examined are still in the early stages of implementation and are not fully operational.  As 
a result, we can not yet draw conclusions for these programs as to how information 
obtained through results-based systems ultimately will be applied.  Too, we did not 
interview other potential users of these data--planning staff, legislators, and stakeholders, 
among others--and so we cannot draw conclusions about how they might apply data from 
these systems. 
State officials we interviewed, who have had experience applying data from results-based 
systems, indicate that performance and outcome data are most often used as part of a 
larger mix of information and interests which determine how:  (1) programs are 
managed; (2) providers, contractors, and local offices are assessed; (3) budgets are 
allocated; and, (4) contract decisions are made.  For example, officials in Washington 
State's alcohol and drug abuse program indicated that they had used data from their 
results-based system to refocus priorities and interventions, but only after further 
research and analysis.  In Massachusetts, the mental health program uses results-based 
data in its contracting process, but does not rely exclusively or even primarily on these 
data to make its decisions. 
One interesting development is the requirement by some State legislatures or executives 
for agencies to submit outcome and performance data with their budget requests.  This 
suggests that executives and legislators might be increasingly inclined to make budgetary  and allocation decisions which at least take results-based data into account.  The 
systematic collection of this information across all State programs, in addition to spurring 
activity to develop goals and measures, also allows executives and legislators to compare 
mission statements, goals, outcomes and performance across agencies and programs. 
Washington State, for example, is requiring each agency to submit 6-year strategic plans 
as part of their 1997-99 biennium budget submissions.  The plans must include the 
agency's mission description, strategies, goals, objectives, timeliness, and performance 
measures and standards.  Florida is implementing its Government Performance and 
Accountability Act of 1994 over an &year period.  Agencies must submit performance 
measures (input, output, and outcome) and standards in their budget submissions.  In 
return, they will have flexibility to reallocate dollars among categories and programs. 
The use and focus of these results-based systems makes the case for a broad definition of 
accountability, in which mission definition, public reporting, consensus building, 
partnering, and the use of information to engage partners, target technical assistance and 
close performance gaps are the key ingredients.  This type of use is different from a 
definition of accountability in which the results-based data are seen as a clear and precise 
road map by which program managers can make clear distinctions between good and 
poor program providers, effective or ineffective program interventions, or adequate or 
inadequate budgets.  In developing their results-bad accountability systems, States face several si 
challenges:  issues with the measures themselves, data concern, the use of results, and 
system capacities. 
These challenges are complex and not subject to quick solutions.  Many overlap and 
reinforce one another.  Yet the extent to which States will be able to translate their 
efforts into improved program effectiveness or more efficient targeting of public dollars 
will largely depend on how they address these challenges and answer the questions they 
pose. 
Measures 
In order to develop qpropriate measures, States must reflect on, and make decisions 
regarding, their primary missions and focus.  In the areas of preventive health and 
maternal and child health, for example, the consideration of measurement systems has 
stimulated an ongoing debate about the role and mission of local public health programs. 
Is their mission one of providing primary care to the underserved; or assessment, policy 
development, and assurance oriented towards improving the infrastructure for an entire 
community; or some combination, and if so, with what balance?  Likewise, in the 
substance abuse and mental health areas, the question is similarly put:  is the orientation 
towards the community at large or towards target populations (e.g. those with current 
problems or those at risk of developing them) which are served by the programs? 
In addition, States must struggle with the questions of what outcomes are really desired 
and how desired outcomes relate to accountability.  For example, is a provider 
responsible for treatment only, regardless of whether the mode of treatment is effective? 
Or, more than that, is the provider responsible for whether or not the client obtains 
employment and is able to become self-sufficient? 
What k the proper mix of dinerent masures? 
States must struggle through questions of the proper mix of measures:  those that are 
end outcomes, others that are "intermediate" outcomes or risk behavior-oriented, and 
others that are capacity- or process-oriented performance measures. 
Many experts consider the use of systems that are purely outcomes-based to be 
inadequate.  It can be difficult to hold program officials accountable for outcomes that 
can be influenced by a wide range of factors.  Results obtained through outcomes 
measurement systems often raise a host of questions that require process and capacity 
oriented data to help answer.  Indeed, the use of outcomes measures in some of the 
States we visited has increased the visibility and need for research, evaluation, and  analytical support to interpret the data and conduct follow-up inquiries, rather than 
supplanting the need for such analytical support.' 
Appreciating limitations of systems relying solely on outcomes measures, the National 
Academy of Science recommended that HHS combine outcomes measures with process 
and capacity measures in its performance partnership grants with the  state^.^  The 
States, too, are adopting approaches which include a mix of measures. 
In Nebraska, for example, State officials have made an explicit distinction between 
performance measures and outcomes measures, where the former is programmatically 
oriented towards measures of effectiveness, efficiency and quality; and the latter is 
oriented to broad health status measures.  They believe-this mix of measures will help 
focus on end results while providing more direct goals and measurements towards which 
program officials can work. 
In North Carolina, at the Department of Environment, Health and Natural Resources, 
planners developed "Level I" indicators (health status indicators) and "Level 11" indicators 
(output, capacity, risk behavior and process measures) which augment the information 
available through the Level I indicators. 
A long list of measures may cause those within the system to lose focus on what really 
matters and to confuse priorities.  A short list can provide too little data on which to 
assess performance or make funding decisions.  In Oregon, State officials are being 
criticized in some quarters for measuring too much and losing sight of real priorities.  In 
New York, an earlier effort in strategic planning in the late 1980s was derailed by too 
many measures and data points that prevented adequate tracking and accountability. 
Officials now express concern that they have focused too much on a few goals and a few 
measures and are not capturing all the health priorities of the State. 
Data Concerns 
Are the data available? 
Most State efforts are intended to be practical and, therefore, use data availability as a 
key criterion for the acceptance of a measure.  However, this practice may compromise 
the reliability and validity of the data.  If States select measures primarily for the current 
convenience and accessibility of data, important results may not be measured and 
decisionmaking based on the measures will be inadequate and flawed.  Yet, if States 
identify important measures for which data are not available, or cannot be made 
available, the system will be theoretical only. 
Many States are facing a paucity of needed data, particularly at county and local levels. 
In these situations, States must ask hard questions about whether funds are available to 
collect additional data and who will collect them.  If the States use funds to develop  Statewide management systems or impose data collection responsibilities on providers, 
they must assess whether or not the collection of additional data will be seen as an 
administrative burden or as a crucial responsibility of program management. 
Our conversations with State officials coniirmed the urgency of investing in focused 
systems that can provide the core data necessary to support State-, county-, and provider- 
level measurement efforts.  In Illinois and Washington, officials responsible for alcohol 
and substance abuse prevention and treatment programs emphasized the importance of 
systems which can be altered easily, and unique identifiers to facilitate data analysis, 
validation and matching. 
A particularly thorny issue facing States is the lack of data available to properly assess 
the impact of managed care arrangements on enrollee health and States' budgets. 
Almost all of the States we contacted have yet to address this question, though many are 
taking steps to do so.  North Carolina does include requirements for data reporting in 
managed care contracts and several other States mentioned that they intend to do the 
same. 
Even when data are available, the timeliness of obtaining them can present significant 
problems.  While the usefulness of data may be dependent on the rate of change of the 
problem or environment, often data that are 2 or 3 years old cannot be used effectively 
to assess current program performance, to sanction providers, or to allocate scarce funds. 
In part, for this reason, Florida's alcohol, drug abuse and mental health program office 
requires providers to collect and report on data.  These data could be obtained by 
matching data files with other State agencies, but would take considerably more time to 
obtain using this method. 
Use of Results 
How will States develop and use standar.? 
The States we examined are generally far away from the use of standards to require 
accountability among providers or contractors.  States have not been reluctant to develop 
goals for broad health status measures, but have proceeded much more cautiously in 
developing standards for measures applicable to specific programs or interventions.  Even 
when included in provider contracts or requests for proposals, standards tend to be 
broad, flexible, used to trigger further discussions and analysis when not met, and used to 
promote emphasis on results. 
For example, Washington's Division of Alcohol and Drug Abuse has not implemented 
standards for its performance accountability system.  This decision was very purposeful. 
The Division believes that without baseline data and more experience with the system, it 
is unwise to develop standards.  State officials report that providers there continue to be 
wary about the development and use of standards.  Florida's alcohol, drug abuse and mental health providers are given two standard 
deviations as a range of acceptable performance.  In the integrated services proposal 
being prepared between New York State and its Monroe County, county officials are 
expected to obtain goals with ranges of acceptable performance.  In both of these cases, 
performance outside the range would trigger additional validation, analysis, research and 
discussions, rather than automatic sanctions. 
Officials in several States expressed concern about their exposure in reporting outcomes 
and performance data.  They are particularly concerned-about public and legislative 
reaction when goals are not met.  They believe the data can be used inappropriately by 
stakeholders to assume that program managers have simply mismanaged or misdirected 
their efforts, when, in fact, failure might be attributable to unanticipated or 
uncontrollable external influences.  These concerns of the State officials in many ways 
mirror those of the programs' providers.  In Nebraska, for example, the development of 
standards was purposely set aside in order to make program officials feel "safe" as they 
worked on developing a performance measurement system. 
Some officials noted the difficulty of responding appropriately to the data in the following 
way:  does failure to achieve goals mean that the program should receive more money 
(in order to help it achieve the goals) or less money (because it is a poorly performing 
organization)?  To answer this question, significant analysis and interpretation of data 
must take place with a depth of understanding of environmental complexities that these 
officials believe State legislatures and the public may not possess. 
How do States balance accomtabilirp, wah pampsh@? 
The above examples also reflect the difficulty States are facing in developing 
measurement systems with the assistance and active participation of those whose 
performance is being or will be measured.  In Illinois, for example, officials observe that 
their partnership with providers has not evolved as they would like because of concern 
about how standards might be applied in evaluating their performance, particularly how 
external factors beyond the providers' control might be considered.  We found that this 
concern was not exclusive to health care providers.  To deal with this problem, some 
States have downplayed standards, as discussed above. 
System Capacities 
WilI resource investments in data systems and analytical evaluation be adequate? 
As discussed above, the collection of key data on outcomes and performance only begins 
a process of assessing performance, not ends it.  In Nebraska, for example, officials 
believe their future success depends on whether the necessary investments in these 
capabilities will be made.  And Washington's alcohol and drug abuse program officials  attribute their success in large part to having made adequate investments in non- 
mainframe data systems and a research capability to support focused reviews based on 
results. 
Many States noted in their conversations with us the wide differences among their local 
partners in staffing and expertise, as well as in mission definition and demographics.  In 
Florida, New York, North Carolina, Ohio, Washington, and Nebraska, officials pointed 
out the variation in funding and density among the most populated and the least 
populated counties in their States.  These State officials pointed out that the ability and 
interest within each of these counties to collect data, interpret it, and use it to manage 
programs varies widely.  Our inquiry has examined the initiatives of several States in implementing results-based 
accountability systems in their public health and mental health programs.  Our primary 
focus has been on the States rather than on any efforts of HHS as it moves forward with 
its performance partnership initiative. 
At the same time, from our conversations with many State officials, we have come to 
appreciate the challenges facing HHS as it considers future directions for its performance 
partnership grant initiative.  The ability of the Department to address these challenges 
timely and effectively will affect the success of its efforts to implement meaningful 
performance partnerships with the States. 
Since we have not examined Federal initiatives, we can not say to what extent agencies 
are already addressing these challenges.  In some cases agencies may have moved well 
beyond some of these challenges or are well underway in addressing them.  Nonetheless, 
we present them here as important questions for Federal agencies that are working with 
States to develop performance partnerships. 
To varying degrees, States raised these questions with us as indicators they will use in 
assessing the success of Federal-State partnerships.  We do not intend the mere 
exposition of these challenges to reflect on the progress Federal agencies are making in 
pursuing performance partnerships.  Rather, we intend them to reflect only the 
complexity of the enterprise. 
Usefulness to the States 
4-Iow can K16-IS e w e t hat its parh~emhipa pproach, relying on results-based system for 
accountability, will be integrated with the States' own perfomme management @om? 
The initiatives we examined confirm the seriousness and intensity of the States in 
implementing results-based systems for their programs.  The State programs described 
here, for example, have already moved beyond the early talking stages and are deeply 
immersed in the nitty-gritty work of planning, implementation, and refinement. 
As  with all intergovernmental initiatives, performance partnerships require 
accommodation and consensus, as well as recognition of both common and different 
needs, goals missions, capacities, and Federal and State environmental pressures.  As 
HHS works with States in developing performance partnerships, what are the lessons 
HHS can learn from States ongoing experiences with results-based accountability systems. 
Given the numerous efforts' underway in the States, how can the Federal-State 
partnership accommodate and even support the variation in State approaches?  To what extent will State and local officials be receptive to Federal leadership in light of 
their own unique and established methods, partnerships, resources, capabilities, and 
commitments? 
Effectiveness of Partnerships 
A critical underpinning of the partnership grant approach is effective working 
relationships among numerous government stakeholders.  The complexities of these 
relationships go far deeper than a casual reference to "Federal-State partnerships" may 
suggest.  The experiences of the States described here indicate the essential cross-cutting 
nature of many important public health goals and the need for clear roles, consistency of 
requirements, and appropriate resources among the Federal agency partners, and 
regional and headquarters offices.  What are the roles for the Federal agencies involved? 
Should one Department or agency have the lead, or should all those affected be equally 
involved?  What are the roles for regional offices?  What staffing resources are necessary 
for the technical assistance and dialogue essential for this new partnership approach? 
Information Exchange 
As HHHS and the States move forward with results-based accountability systems, they will 
have a continuing need for information and expertise.  Open dialogue among 
stakeholders will greatly benefit the performance measurement efforts within the 
Department.  Such exchanges could lead, for example, to standard definitions to be used 
across HHS, as well as models and best practices on working with States on performance 
partnerships. 
In its leadership role for performance partnership grants, the Department will need to 
address many issues.  For example, how can HHS learn from others about emerging 
developments that can enhance continuous improvement among Federal agencies and 
those of the States?  What mechanisms, such as clearinghouses, advisory groups, or 
Federal-State meetings, are appropriate for giving the Department essential feedback 
and for informing itself, the States, and others with evolving knowledge, new ideas, and 
new approaches? 
Data Collection 
hlbw can IHMS mmihize the usejkkss of its cwent data collection system to enhance the 
ejJiectivenesso f its peflomrtce prtnenhip grants with the St~ta? 
The HHS supports extensive, significant collections of health and health-related data that 
are critical to the success of States' efforts to develop results-based systems for  accountability.  The States already rely heavily on Federal survey and epidemiological 
data for their own purposes as well as for comparisons across States based on aggregated 
data.  Because these systems are complex and expensive to operate, it is important that 
the Department ensure that its efforts, in so far as practical, meet the major, common 
needs for data among the States. 
Agenda for Research 
Significant questions, both theoretical and practical, about results-based systems remain 
unanswered.  These have been identified and described in other sources.1  There is 
obviously a clear role here for MHS in the evaluation of these efforts.  From the 
perspective of the States, however, it is important for the Department to appreciate the 
significance of the States' research needs for informing their ongoing efforts to implement 
effective systems. 
Adequate Evaluation Ca~acitv 
It will not be sufficient for HHS, or others, to rely primarily on performance data 
reported by the States for its critical decisions about program effectiveness, allocation of 
resources, and policy changes.  These data should be considered only the raw materials 
for subsequent evaluation and analysis, essential for informing deeper understandings of 
what is actually going on.  The data help frame the questions to ask; they, in themselves, 
do not provide the answers.  A significant threat to the long-term viability of 
performance partnership grants and other results-based systems will be the failure to link 
them with sufficient systems for analysis and evaluation that will adequately provide for 
informed decisionmaking by the Federal Government, the States, and the general public.  This report documents some important initiatives that are underway across the country to 
develop more meaningful approaches for managing public health programs.  Developing 
these systems is inherently messy, difficult and time consuming.  It involves multiple 
stakeholders with complex, sometimes competing, agendas.  It is an expensive endeavor 
and can make heavy demands on limited resources.  Progress is neither straightforward 
nor single-tracked.  Yet, as is clear from the initiatives described here, States are making 
serious efforts to develop results-based systems and are moving forward. 
We document in this report a number of benefits State officials we interviewed see 
emerging from their efforts.  They bear repeating here.  These benefits include: 
(1) obtaining information efficiently, which can be used to improve program 
performance; (2) empirically demonstrating program results; and, (3) obtaining consensus 
among stakeholders on program missions and goals.  We believe that these are significant 
benefits to program managers and strategic planners. 
The ultimate determinant of whether these systems succeed, however, may well be their 
usefulness to those most vested in the programs:  legislators, administrators, providers, 
advocates, and consumers.  The challenge facing these stakeholders will be to achieve an 
appropriate balance when using the data from these systems to enhance decisionmaking 
about program improvements, budgets and contracts, and feedback to providers and 
consumers. 
We received very positive comments from the Assistant Secretary for Planning and 
Evaluation, the Substance Abuse and Mental Health Senices Administration, the Centers 
for Disease Control, the Wealth Resources and Services Administration, and the 
Secretary for Health.  Their comments offered further insights on such matters as:  the 
impact of the Government Performance and Results Act in helping shape performance 
partnerships; the data and performance measurement problems facing States; the 
importance of information exchange; and, the role of Federal agencies in performance 
partnerships.  We were also informed that the National Research Council's Panel on 
Performance Measures and Data for Public Health Performance Partnership Grants has 
recently issued its first report entitled '!Assessment of Peformance Measures for  Public 
Health, Substance Abuse and Mental Health." 
Based on these comments, changes were made to the report as appropriate.  The 
complete text of these comments can be found in Appendix B.  We regard them as an 
integral part of the report ind recommend them to the attention of the reader.  Federal Public  rant Programs  Among the current block grants administered by the United States Department of Health 
and Human Services (HHS) for public health purposes are the Community Mental 
Health Services Block Grants, Substance Abuse Block Grant, Preventive Health and 
Health Services Block Grants, and Maternal and Child Health Services Block Grant. 
Consistent with the character of block grants, the legislation establishing these grants 
specifies purposes, populations to be sewed, specific activities or processes to be carried 
out, reporting requirements, and limits or minimums on the uses of funds for particular 
purposes.  A brief description of each of these block grants follows below. 
The Public Health Service Act provides for block grants for community mental health 
services.  The Secretary of MHS, through the Director of the Center for Mental Health 
Services, makes grants to the States each fiscal year for the purposes of providing 
comprehensive community health services under a State plan, evaluating programs and 
services carried out under the plan, and planning, administration and educational services 
under the plan.  State plans provide for the establishment and implementation of an 
organized community-based system of care for adults with serious mental illness or 
children with serious emotional disturbance.  The plan must contain quantitative targets 
to be achieved, although the Act only specifies that such targets include the numbers of 
such individuals residing in the areas to be served under the system.  Among other 
things, specific attention must be given to:  (1) reducing the rate of hospitalization for 
eligible individuals, (2) providing outreach and services to eligible homeless individuals, 
and, (3) estimating incidence and prevalence in the State of the conditions of eligibility. 
States must establish mental health planning councils to review State plans, serve as an 
advocate for individuals with mental illnesses or emotional problems, and monitor, review 
and evaluate the allocation and adequacy of mental health services within the State. 
No more than 5 percent of the total grant to a State may be used for administrative 
expenses. 
Substance Abuse Block Grants 
The Public Health Service Act provides for block grants for the prevention and treatment 
of substance abuse.  The HHS Secretary, through the Director of the Center for 
Substance Abuse, makes grants to the States each fiscal year.  States must allocate a 
specific minimum percentage of funds for alcohol prevention and treatment and for 
prevention and treatment activities regarding other drugs.  States must also spend a 
specific portion of their funds to educate and counsel individuals not requiring treatment 
about abuse and provide for activities to reduce their risk of such abuse. 
States must submit an assessment of needs to the Secretary in order to receive a grant. 
This assessment must include information by localities within the State on the incidence 
and prevalence of drug abuse and alcohol abuse; current prevention and treatment  activities in the State; the need of the State for technical assistance; efforts by the State 
to improve such activities; and, the extent to which availability of such activities matches 
the needs. 
States must provide for independent peer review to assess the quality, appropriateness, 
and efficacy of treatment services provided in the State to individuals under the program. 
No more than 5 percent of the total grant to a State may be used for administrative 
expenses. 
Preventive Health and Health Services B1mk Grants 
The Public Health Service Act provides for block grants-for preventive health and health 
services.  Allotments under this section are for preventive health services, comprehensive 
public health services, and emergency medical services.  Payments to States may be used 
for activities designed to make progress towards Healthy People 2000 objectives. 
States must develop their plans in consultation with State Preventive Health Advisory 
Committees.  They must specify strategies for making progress toward improving the 
health status of the population and establish reasonable criteria to evaluate the 
performance of entities receiving payments from the State.  The statute also specifies 
that States must report to the Federal Government on progress in meeting the year 2000 
health objectives and use the uniform collecting and reporting formats developed by the 
Secretary for this purpose. 
No more than 1.0 percent of the total grant to a State may be used for administrative 
expenses. 
Maternal and Child Health Services Block Grant 
The Public Health Service Act provides for funds to improve the health of all mothers 
and children consistent with the applicable health status goals and national health 
objectives established by the Secretary for the year 2000.  The grants are to be used to 
provide and assure mothers and children access and quality health services, reduce infant 
mortality and incidence of preventable disease and handicapping conditions among 
children, reduce the need for inpatient and long-term care services, increase the number 
of immunized children, and other objectives. 
States must report on a number of measures specified within the statute or by the 
Secretary, including rate of infant mortality, rate of low birthweight births, rate of 
maternal mortality, rate of neonatal deaths, and other such health status indicators.  Agency Comments  DEPARTMENT OF HEALTH & HUMAN SERVICES  Office of the Secretary 
Washington, D.C. 20201 
- 
AuG  7 ' 1997 
TO:  June Gibbs Brown 
Inspector General 
FROM:  David F. Garrison 
Principal Deputy Assistant Secretary 
for Planning and Evaluation  \ 
SUBJECT:  Comments on the OIG Draft Report: "Results-Based Systems for Public Health 
Programs," OEI-05-96-00260 
Thank you for the opportunity to review this draft report. We continue to believe in the 
importance of establishing performance partnerships for greater flexibility, improved 
accountability, strengthened relationships with States and other partners, and, ultimately, 
improved outcomes for our customers. 
The report provides a comprehensive description of current state efforts to implement results- 
based systems and the challenges involved, both at the State and Federal levels.  We were 
pleased to see that the report now includes examples of performance measures in each state case 
study.  The report reinforces many of the challenges that we have tried to address over the past 
several years in our performance partnership efforts. For example, CDC and SAMHSA have 
each spent the last year working to integrate some aspects of performance measurement into their 
existing grants, have recognized the importance of conducting extensive stakeholder 
consultations as an integral part of this process, and have modified their proposals to be 
responsive to states concerns. The Department also recently commissioned the National 
Academy of Sciences to develop a report on performance partnerships, Assessment of 
Performance Measures for Public Health, Substance Abuse, and Mental Health, which provides 
guidance for Departmental efforts by laying out many of the same challenges with 
recommendations for addressing them. 
We appreciate the opportunity to review and comment on this report. We believe that it will 
provide valuable guidance as programs at the Federal and State levels implement systems of 
outcome-based management, particularly as we move fonvard with our State partners on 
implementation of the Government Performance and Results Act.  DEPARTMENT OF HEALTH & N U W S ERVICES  Public Health Service 
.- 
Substance Abuse and Mental 
Health Services Administration 
Rockville MD 20857 
AUG  1  1997 
TO:'  June Gibbs Brown 
Inspector General 
FROM:  Administrator, SAMHSA 
SUBJECT:  Comments on Draft Inspector General RGort, Results-Based Systemsfor 
Public Health Programs 
Thank you for the opportunity to review this very valuable report. Not only did we find its 
conclusions sound, but the information on the States' efforts will be extremely useful to all of us 
--  -- 
Federal staff and the States  working on SAMHSA's pilots on State performance 
measurement. 
Our general comments on Volume I will be followed by specific comments on each section, as 
well as a few comments on Volume 11.  Also, attached are more detailed comments on States' 
concerns about the type of data required and the nature of accountability under a results-based 
syst ern. 
Volume 1. General Comments 
As noted, we find the Report thorough and sound. We are particularly pleased that the Report 
stresses the importance of top-level commitment to and stakeholder involvement in results-based 
systems. States have indicated repeatedly that these two factors are critical to the success of 
these efforts.  We are also pleased that, on page 9, the authors state that "the use and focus of 
results-based systems" calls for a "broad definition of accountability", which would lead to 
targeting technical assistance and closing gaps, as opposed to a more narrow definition that 
would provide clear distinctions between good and poor programs. 
The Report could be even stronger, however, if it were updated in the following ways: 
Reference to the National Research Council's Panel om Performance Measures and 
Data for Public Health Performance Partnership Grants: This Panel recently issued 
its first report, Assessment ofPerformance Measures for Public Health, Substance Abuse 
and Mental Health, which provides a "framework" for performance measurement which 
clarifies that, often, a combination of process, capacity, and outcome measures is 
necessary as State and Federal officials seek to determine the contribution made by their 
resources, but that the link between process and capacity measures to outcomes must be 
well-documented. It atso provides a set of "assessment guidelines" to assess the 
appropriateness of individual measures, including that they be: results-oriented;  AUG-04-97  MON  09:29 AM 
- 
Page 2  June Gibbs Brown 
meaningful and understandable; valid, reliable and responsive; and that data are available 
to "support the measure."  It also calls upon the Department to work with the States "to 
identify and develop common definitions and methods that will contribute to 
standardizing measurements".  Finally, we urge the Report's authors to adopt the 
National Academy of Sciences Panel definitions, which include terms such as 
"performance measure" ("a qualitative indicator that can be used to track progress towird 
an objective"). 
a  Clearer link to the Government Performance and Results Act: Although GPRA is 
referred to in the Report's introduction (p.2),i ts conceptual link to State efforts'(i.e., the 
Federal government is engaged in the same process of engaging stakeholders, identifying 
goals, and formulating measures as the States) and its impact on Federal receptiveness to 
State needs (i.e., better data on Federal programs) md  "lessons learned" due to results- 
based management should be emphasized. 
State Challenges 
Attached is a background document prepared by the SAMHSA Center for Substance Abuse 
Prevention concerning data and measurement issues faced by States in implementing results- 
based systems. Three key points are made: 
Level sf aceounftability (i.e., client, program, system): "Most States appear more 
comfortable with accountability being tied to changes at the program level .... The major 
problems associated with collecting this level of data include cost/mmpower, programs 
that receive funding from multiple sources, and the fear of the providers that they will 
lose funding depending on the results." 
rn  National vs. State Data: "...whilen ational surveys can resolve the issues of variation in 
definition, design .ad methodology across States, the data are of little practical use at the 
State and sub-state level. They need data at the lowest common denominator possible to 
be usehl for planning and funding. One suggestion is to provide States with suflticient 
h d st o conduct surveys (much as in State needs assessment contracts) that provide data 
at a level usehl for States, but require some common data collection (and provide 
consistent methodologies with training) so that they can be aggregated to the national 
level." 
"State data infrastructure capacity is a major issue. Most simply don't have sufficient 
funds, manpower, and or equipment to develop, implement, and maintain a performance 
measurement system."  AUG-04-97  MON  09: 30  AM 
- 
Page 3  June Gibbs Brown 
Federal (HHS) Challenges 
We are pleased that language has been added, on page 15, that acknowledges that "agencies may 
. .. be well underway in addressing [these challenges]" because SAMHSA has been engaged in a 
true partnership with the States over the past 1 112 years to shape the performance partnership 
effort related to its block grants. As a result of this partnership, we have proposed legislation to 
provide State flexibility (waivers of certain conditions in exchange for measuring performance) 
and to strengthen State data infrastructure. 
We have three minor comments on the challenges: 
e  Under "Information Exchange", we urge the authors to note the importance of 
information exchange on performance measurement efforts within the Department. Suck 
exchange could lead, for example, to standard definitions to be used across IKKHS, as well 
as models  Lest practices on working with States on performance partnerships. 
 an^ 
e  Under "Agenda for Research", authors should note the role for HHS in evaluating and 
assessing these efforts as well. 
Under "Adequate Evaluation Capacity", we recommend that the specific importance of 
@ 
linking these efforts to GPRA be clarified (for example, through the block grants by 
ultimately usin:  State performance indicators 3s block grant GPRA measures). 
Vohme 2.  State Case Studies 
- 
Page 33  Substance Abuse treatment is not noted for Massachusetts. 
- 
Page 43  Minnesota has an outcomes monitoring system in place for substance abuse, but this 
was not noted. 
Again, thank you for the opportunity to comment on this very worthwhile and timely Report. 
We look forward to receiving a copy of the Final Report. 
Nelba Chavez, Ph.D. 
Attachment  Attachment 
CENTER FOR SUBSTANCE ABUSE PREVENTION 
DISCUSSIONS WITH STATES ON RESULTS-BASED SYSTEMS 
(FL, LA, GA, IL, MA, MN, NE, NY, NC, OH, OR, WA) 
TYPES OF DATA 
Statewide or program 
States made it clear that, while it's easier to obtain State level data (e.g., using social indicators, 
or national or Statewide surveys), they have no confidence in their usefulness re: Performance 
Partnership Grants (PPGs) because the contribution of the effects of the programs they hnd and 
the limited populations they serve on changes in State level data is likely to be nonsignificant. In 
addition, there is no controlling for the effects of other substance abuse prevention programs on 
those data. Therefore, it would seem futile to hold the single State agency accountable for 
changes in results at that level. This holds true, to a lesser extent, even at a county level for most 
States. The exceptions may be those States that have established State inter-agency coordination 
of programs and even pooling of resources. This is by far the exception, rather than the rule. 
Most States appear more comfortable with accountability being tied to changes at the program 
level.  The major problems associated with collecting this level of data include costhanpower, 
programs that receive fknding from multiple sources, and the fear of the providers that they will 
lose funding depending on the results. 
Another prob!em is that, typically, States are unable (as opposed to treatment) to do long term 
follow-up. Therefore, while changes in risk factors/status may be detected after involvement in 
prevention programs, they will be unlikely to track long-term outcomes such as use 5-10 years 
later (e.g., programs for elementary aged children). 
In the CSAP Minimum Data Set pilot, interested States have focused on data at the provider 
level. CSAP is currently modifying the system tested by eleven States in the Phase I pilot which 
was completed May 3 1, 1997, and pursuing the refinement of definitions, methodologies and 
procedures for the collection of five intermediate and outcome performance measures agreed 
upon by 27 States. These were identified using a multiple focus group approach and using 
selection criteria  SO identified via consensus of the participating States. It is important to 
recognize however, that participation is voluntary and, by the time the data collection phase 
begins, CSAP expects some States to drop out.  AUG-04-97 MON  09:3 2  AM 
STATE ISSUES 
Measures are mother problem.  Some valid data sources andfor instruments exist in some areas. 
There is much diversity among the States re: which measures to use, how they are defined etc. 
This is, therefore, a problem at the Federal level for the collection of the common core data. In 
fact, in our pilot, States used the National Academy of Sciences (NAS) criteria with the addition 
of the criterion "cheap and easy to use".  Other considerations (discussed in NAS) of importance 
to States are the frequency of the data collection (especially if national survey); the speed with 
which results are obtained (a frequent criticism ); and the ability of the measure to be impacted 
by an activity (e.g., not likely to affect divorce rates). 
On the other hand, while national surveys can resolve the issues of variation in definition, design 
and methodology across States, the data are of little practical use at the State and sub-state level. 
They need data at the lowest common denominator possible to be useful for planning and 
funding. One suggestion is to provide States with sufficient funds to conduct surveys (much as 
in State needs assessment contracts) that provide data at a level useful for States, but require 
some common data collection (and provide consistent methodologies with training) so that they 
can be aggregated to the national level. State data infkastmcture capacity is a major issue. Most 
simply don't have sufficient funds, manpower, and/or equipment to develop, implement, and 
maintain a performance measurement system. 
States, while anxious, appear to see the importance and utility of performance measurement, 
especially for planning and resource allocation, and to tie to needs assessments. The issue here is 
whether the results will be used to provide technical assistance to those programs and States 
where results are disappointing, or whether the consequences will be punitive in nature. 
Although several States have Statewide performance measurement models with required 
performance measures, more use a menu approach and several permit local development and 
identification of local performance measures as long as they address Statewide goals andfor 
objectives. The more the variation, however, the less likely the data can be supportive of the 
PPG concept. 
Other related issues concern applying Statewide models to programs which, although targeting 
the same performance measure, often vary in intensity and scope.  Public Heatth Service 
DEPARTMENT OF HEALTH & HUMAN SERVICES  Centers for Disease antrd 
and Prevention (CDC) 
Date  -AUG  0 6  1997 
Associate Director for Management and Operations, CDC 
From 
Subjec  CDC Comments on Office of Inspector General (OIG) Draft Report, "Results-Based 
Systems for Public Health Programs" (OEI-05-96-00260) 
To  June Gibbs Brown 
Inspector General 
Thank you for the opportunity to review the draft report.  Attached are CDC's comments. 
If you should have questions concerning these comments, please have your staff contact 
Carolyn Russell, Director, Management Analysis and Services Office, at (404) 639-0440. 
Attachment  CDC Comments om  the Office of the Inspector General Draft Report, 
"Results-Based Systems for Public Health Programsy9 
Vol. 1, Pg. i, paragraph 3, 
We recommend restructuring the last sentencc as follows: These arrangements would 
combine various categorical grants into performance partnerships and would reshape 
several of its bIockgrantprograms. 
Vol. 1,  Pg. ii, Challenges for the States, Paragraph 3, 
We recommend that the data objective refer to the need to take into consideration the 
availability of a system necessary to validate the data. 
Change the first sentence to read, " ...t le necessary data are available. .." 
Vol. 1, Pg. 8, paragraph 5, last sentence, 
Change " . . . this data. . .  to " . . . these data. . . " The same change should be made in 
" 
the last sentence of paragraph 6. 
Vol. 1, Pg. 10-17 , State and Federal Challenges, 
There are many questions and issues raised regarding the implementation of 
performance-based measures. The format in which the sections were written is 
confusing, for example, questions are answered by other questions. We recommend 
explaining that the section only outlines the challenges and does not attempt to provide 
solutions. 
Vol. 1, Pg. 11 , paragraph 5, 
The reliability and validity of the measures are a major issues not addressed here. 
Vol. 1, Pg. 12, paragraph 3, 
The paragraph states that the major problem facing states is the lack of data for persons 
enrolled in managed care; however, the real issue facing states is the need to provide 
health care coverage and the lack of data available to properly assess the impact of 
managed care. This point should be clarified.  Vol. 1, Pg. 12, paragraph 4, 
The paragraph entitled, "Are the data timely?'refers to the reconsideration of the 
usefulness of data that are two or three years old. It is also important to consider the rate 
of change of the problem and the environment before making a judgement on the 
usefulness of data. 
Vol. 1, Pg. 13, paragraph 4, 
The problem seems overly simplified by focusing solely on health care providers.  This 
paragraph does not reflect that problems resulting from the attempt to balance 
accountability with partnership, extend beyond health care providers participating in the 
development of measurement systems. 
Vol. 1, Pg. 15, last paragraph, 
Change the word, "extendv in the last sentence, to "extent."  DEPARTMENT OF HEALTH & H U W S ERVICES  Public Health Service 
Health Resources .and 
Services Administration 
Rockville MD 20857 
AUG  2 5  1997 
TO:  Inspector General, OS, DHHS  - 
FROM :  Acting Deputy Administrator 
SUBJECT:  Office of Inspector General Draft Reports "Results- 
Based Systems For Public Health Programs Volume 1: 
Lessons From State Initiativesf1O EI-05-96-00260 and 
8oResults-BasedS ystems For Public Health Programs 
Volume 2: State Case Studies" OEI-05-96-00261 
HRSA has reviewed the subject draft reports and has no comments. 
Staff questions may be referred to Michael Herbst on (301) 
443-5256. 
n  Endnotes  1. For a detailed discussion of general concepts of accountability  and specific issues of accountability in 
Federal block grant programs, see President's Council on Integrity and Efficiency, Committee on Inspection 
and Evaluation, Accountability for Block Grants, July 1%. 
. 
2  For a discussion of Federal grant approaches to funding public health activities, see U.S. Department of 
Health and Human Services, Office of  Inspector  General, Federal Approaches to Funding Public Health 
Programs, OEI-01-94-00160, July 1W5. 
. 
3  The President's Fiscal Year (FY) 1996 budget included a proposal for 16 Performance Partnerships in 
the Public Health Service, including the consolidation of various grant programs within Centers for Disease 
Control (CDC), Health Resources and Services Administration (HRSA), and the Substance Abuse and Mental 
Health Services Administration (SAMHSA).  In the FY 1997 budget,ihe Administration continued to propose 
performance partnerships for CDC, HRSA and SAMHSA  In the FY 1998 budget, the Administration 
emphasized performance partnerships within SaMHSA only. 
Both Representative John Dingell and former Senator Nancy Kassenbaum introduced bills in the 104th 
Congress that would have implemented the performance partnership model for public health service programs, 
and specifically for mentrl health and substance abuse block grants. 
4.  The National  Research  Council's  Panel on Performance Measures and  Data  for  Public  Health 
Performance Partnership Grants has recently issued its first report:  "Assessment of Peformance Measures for 
Public Health, Substance Abuse and Mental Health." 
5.  A recent report by the Institute of Medicine, summarizing a workshop on performance monitoring for 
, 
improving community health, identified a wide range of activities by Federal and non-Federal organizations. 
For example: 
The National Committee for Quality Assurance has developed the Health Plan Employer Data and 
Information Set to collect standardized information from health plans for their own use in assessing 
their performance, and to help purchasers make decisions among health plans. The Committee is also 
active in assisting health plans to improve their capacities for data collection, analysis and reporting. 
The Joint Commission on Accreditation of Healthcare Organizations is moving beyond standards to 
develop indicators for assessing the reliability of the standards in predicting actual performance. The 
Indicator Measurement System was originally developed for hospitals and is now being expanded. 
The HHS is responsible for Healthy People 2000, a set of objectives regarding the health promotion 
and disease prevention with goals to be accomplished by the issued model standards for translating 
those national year 2000.  The American Public Health Association has set goals into community 
action plans. 
The HHS, through the Agency for Health Care Policy and Research, funds research on health care 
outcomes and quality measurement. 
The HHS, through the National Center for Health Statistics, assembles vital statistics and conducts 
significant health related surveys. 
For  further  information, see National  Academy  of  Sciences,  Institute  of  Medicine,  Using Performance 
Monitoring to Improve Contntunir))H  ealth:  Exploring the Issues, Workshop Summary, 1996.  6 .   Among the national health organizations working with HHS were:  the Association of State and 
Territorial Health Officers, the National Association of State Alcohol and Drug Abuse Directors, the National 
Association of State Mental Health Program Directors, and the National Association of City and County 
Health Officials. 
. 
7  National Academy of Sciences, Committee on National Statistics, Assessment of Performance Measures 
in Public Health, Phase I  Report, Draft:  for Co@ntent, September 1996.  The report proposed performance 
measures in the areas of chronic disease, HIV/STD/Tl3, mental health, immunization, substance abuse, sexual 
assault prevention, disability prevention, and emergency medical services. 
. 
8  The U.S. General Accounting Office, which has written extensively on this subject, noted in a September 
1995 report: 
While State efforts will certainly be closely tied to block grant results, outcomes will just as certainly 
be affected by factors outside the control of state administrators.  Because of the role that these 
variables may play,  evaluation  will need  to kolate the effect of  outside factors  on state programs. 
[Emphask added.]  For example, the incidence of low birth weight infants depends not only on the 
efforts of a particular state and local agency to fill the gaps in prenatal care, but also on many other 
... 
demographic and situational factors, such as regional employment trends and demographic patterns 
See U.S. General Accounting Office, Block Grants: Issues in DesigningAccountability Provisions, GAOIAIMD- 
95-226, September 1995. 
. 
9  National Academy of Sciences, Committee on National Statistics, Assessment of Performance Measures 
in Public Health, Phase I  Report. 
10.  See, for example, Department of Health and Human Services, Office of Inspector General, Federal 
Approaches to Funding Public Health Progrants, OEI-01-91-00160, July 1995.