identify which areas have priority for action in their communities and which measurable
objectives can be used to monitor progress.

Coordinating the Department’s planning and program efforts at the State level. Once
communities engage in local prioritization and planning activities, it will be incumbent on
the State to direct resources to support community-identified priorities rather than
Federal or State-identified priorities, as reflected by categorical funding streams.
Coordinating and integrating the results of community-based planning with ongoing
Statewide program planning by the Department will be a challenge. In particular,
providing communities with flexible funding while assuring some level of accountability
for outcomes will be key to the successful implementation of “Communities Working
Together."

Using results-based goal setting for decisionmaking. While the MCHSBG’s application

for the past 2 years has included results-based goals, objectives and implementation
objectives, as required by HRSA, State MCH staff indicate that the results-based goal
setting and reporting are not used as much as interventions. Often other criteria
influence program decisions (e.g. media-led, stakeholder-led, legislature-led). The
Department and local health units are likely to face similar challenges in implementing
"Communities Working Together."

Data issues. Among the significant issues are gaps in available data and a lack of valid
data important for practical, valid assessment of problems and needs and for monitoring
program accomplishments. As noted above, most data in New York are now available at
a State-level only. Further, the State lacks data on children and adolescents and no
population-based measures for assessing mental health problems, such as depression, at
the community level.

The potential for othe :s to misuse program performance data is another important issue
facing program administrators. The challenges are: (1) to educate other potential users
about the limits of the data so they avoid policy and budget decisions based on raw
numbers taken out of context; and, (2) to avoid penalizing programs without first having
had all partners working together to improve performance.

Lessons Learned

Department staff identified several factors important to the successful completion of the
review.

Process-related factors. Relying on the Public Health Council and its Priorities
Committee, which were knowledgeable, capable, well-respected, and external to the
Department, to give direction to the process enhanced the legitimacy of the effort.
Further, the Commissioner made the effort a top priority with a tight deadline for
completion and redirected Department resources to support it. Finally, the extensive
involvement of communities across the State encouraged their support for the overall
process and enhanced the relevance of the final plan to their local needs. For example,

60


11. Lack of education readiness and school success

12. Lack of child/family attachment to schools

13. Developmentally and physically hazardous school environments
14. Anti-social youth behavior

Prevention providers use the IRRA data in tandem with community input (e.g. parents,
teachers/schools, business owners, local organizations/groups, pastors/churches and local
law enforcement) when choosing the risk factor on which they will focus their prevention
efforts. Prevention providers in each of the 104 counties, Chicago’s 77 neighborhoods
and 177 municipalities with populations over 10,000 use the assessment tool in addition
to nine protective factors which are used to help identify existing community resources.
Based on the IRRA results, knowledge of their community's resources and feedback from
community stakeholders, prevention providers determine which outcomes to target, step
2.

In order to develop and implement programs, step 3, providers develop and test
strategies or program interventions that will lead to the desired outcomes or community
changes, ensure that those strategies or interventions build on community resources,
research and experience; and, finally, implement strategies and programs.

Following that, step 4, review program implementation, involves: evaluating the
implementation of the strategies or programs with key stakeholders; determining the
quality of implementation, customer satisfaction, resource availability and unresolved
issues; and, recommending modifications for improvement.

The University of Illinois’ Center for Prevention Research and Development has
developed 18 outcome measurement surveys that prevention providers can administer to
their clients to complete step 5 of the CPRR process. Prevention providers may also
develop their own evaluative tool should they wish to do so. The University wil’ provide
data entry and analysis support to prevention staff when they use one of the University’s
18 survey tools. The University will also provide technical assistance to those providers
that cannot use one of the 18 survey tools.

Process for Developing the Performance System

The DASA has come to view "prevention" as the long-term solution to ATOD abuse.
The impetus for such a philosophy began in 1986, sheparded through the State legislature
by, then Illinois Speaker of the House, George Ryan. In 1986, the State legislature
appropriated $12 million for DASA’s ATOD prevention activities and has continued to
support and fund such efforts.

During this time, a sort of "brain trust" was created involving key State officials within
DASA and among academia. These ties are still strong today. Prevention efforts at this
time, being activity driven, provided data based on outputs rather than results,
consequently, the impact of these programs was unknown. So it was decided that a new

26


TABLE OF CONTENTS

PAGE

INTRODUCTION .......-- 0c ccc cece eee nee nett ett e eet eenee 1
STATES INTERVIEWED

+ FLORIDA ... 0... cece eee nee een n ee eteeeee 5

Division of Alcohol, Drug Abuse and Mental Health .................... 6

Department of Health 22.0.0... ccc ccc ce eee cee eee nee 11

* GEORGIA ..... 2. cece eee ee tenet e tenn ne eees 15

Georgia Policy Council for Children and Families ...............-2..0.0- 16

© ILLINOIS ... 1... eect eee ee tee eee eee 24

Department of Alcoholism and Substance Abuse ...........600 200s eeee 25

* MASSACHUSETTS ....... 2.0... cece eee een eee eee neces 32

Bureau of Substance Abuse Services .. 00.0.6... c cece eee cence eee 33

Department of Mental Health «0... 0... cece cece ee cee eee e eee 38

+ MINNESOTA ....... 0.2.2 ccc cece ccc ee eee eee eee e ee eeeee 42

State Public Health Goals 2.6.0.2... cece cece cette eens 43

+ NEBRASKA ... 1.2... 02 eee cece eect ence tee eee e eens 46

Nebraska Partnership for Health and Human Services ...............044 47

¢ NEW YORK ........-. cece cece eee e eet e tenner eneee 54

Department of Public Health 2.0... 0... eee cece eee eee 55

Monroe County Department of Health .. 2.0.6... 0.0 see eee e eee eee 62


document as one that would serve as a starting point for discussion of a public health
accountability system, rather than the end result.

However, the recommendations of the Task Force were not immediately acted upon.
State legislators, pressed to reduce spending and cut taxes, were wary of appropriating
funds for new projects. Without additional resources to carry on this effort, development
of an accountability system took a back seat to the current needs of established DEHNR

programs.

A New Beginning

In early 1996, the Deputy State Health Director, refocused DEHNR energies once again
on the development of an accountability system. A second task force was formed,
informally known as Accountability Task Force JI, with the expectation that its efforts
would begin where the first Task Force left off. For the most part, this expectation was
met. However, many members of Task Force II had not served on Task Force I. As a
result, the DEHNR found that new members needed to be informed of, not only what
decisions were made, but the reasoning behind the decisions in order for new members
to concur with the first Task Force’s results and to proceed with the new endeavor. Task
Force II was smaller in number with more representation from the State’s health
divisions.

Task Force II pared down the number of Level I and II indicators developed by Task
Force I, ultimately selecting 8 Level I and 32 Level II indicators. Such documents as the
national Healthy People 2000 objectives and the Healthy Carolinians 2000 objectives were
reviewed to help identify possible indicators. Members determined that the outcome
measures for Level I and II indicators would be those for which: (1) data was already
being collected, and (2) data was accessible to the DEHNR at no extra cost.

The DEHNR is proceeding with implementation of its proposed accountability system,
though it is just now getting feedback from the counties and local health departments
regarding the selected Level I and II indicators. Level I and II indicators were sent out
to counties and local health departments in November 1996 for comment. In January
1997, a meeting of the Association of Local Health Directors was held and Level I and II
indicators were discussed. “About 45 to 50 of the 86 health directors attended this
meeting (there were 12 or 13 health director vacancies at this time). For the most part,
feedback to date has been largely accepting.

Sample Performance Measures

Data for Level I indicators are provided on a county basis. The 8 Level I indicators, and
an example of one of their population-based health outcome measures, are:

1. Maternal and Infant Health
Measure: Infant mortality rate

69


OHIO

80


GEORGIA

Georgia Policy Council for Children and Families

Overview

Georgia has decided to adopt results-based accountability to improve results for the
health and welfare of its children and families. The five results Georgia seeks for its
children and families are: (1) healthy children, (2) children ready for school,

(3) children succeeding in school, (4) strong families, and (5) self-sufficient families. The
corporate benchmarking approach is now being used to help measure the extent to which
these results are being achieved. The State is planning on implementing results-based
budgeting within the next 2 years.

Nature, Extent and Uses of Performance Measures

Since 1991 community and State partnerships have evolved, engaging in collaborative
planning and integrated service strategies designed to strengthen the family and improve
the well-being of children. The State has developed a broad framework for achieving the
above mentioned results, the essential elements of which are:

Results Accountability: Reform strategies will be designed to address specific
benchmarks (the ends), so that communities can develop strategies (the means) that
target their most prevalent problems.

Community-Driven Decisions: The State Policy Council and local Community
Partnerships have inclusive, legal structures that are recognized by both public and
private sectors as the change agents for children and families.

Innovative Strategies: Community Partnerships will develop new strategies focusing on:
early and ongoing contact with children and families to prevent problems from occurring,
improving access to family-friendly services, and expanding comprehensive services with a
bias toward prevention.

Government Streamlining: The State will "...reinvent its systems and the ways that
people and organizations think, behave, and apply their resources." For example,
processes and procedures will be streamlined for efficiency, new forms of frontline
practice and more effective ways to evaluate performance will be instituted.

Redirecting Resources: "State and community partners will develop specific agreements
on how dollars will be invested to support the agreed-upon results and the innovative
community strategies."

Definitions for the terms results-based and performance-based were developed as they
are often confused. In essence, results and benchmarks have to do with "ends," while

16


Outcome-Oriented: Service effectiveness will be determined through the measurement
of appropriate outcome indicators; and,

Simplicity: The quality improvement system will be clear and direct. It will not be added
to existing processes but will reduce or replace them.

With these guiding principles in mind, the Quality Improvement Committee conceived
the following five, very broad outcomes for their primary consumers:

Education/Habilitation: Each primary consumer should have age-appropriate
education/habilitation experiences.

Health: Each primary consumer should be physically and mentally healthy.

Housing/Support: Each primary consumer should have basic food, clothing and shelter in
his natural home with his own family or in similar, safe living arrangements in a
community of his own choice.

Social: Each primary consumer should have relationships which contribute to his well-
being and foster trust, self-esteem and social competence, Each primary consumer
should also exercise social responsibility.

Vocation/Avocation: Each primary consumer should have gainful employment or a
productive activity appropriate to his age and capabilities.

As the Quality Improvement Section of the DMHDDSAS and the Rules Reduction Task
Force began to address the challenge of developing such a system, the rapid transition of
the health care system to managed care and the increased pressure to reduce
government spending and regulation impacted the rules development process. In
addition, expected funding for this initiative failed to be passed. It took approximately 2
1/2 years, from the Quality Improvement Committee’s initial recommendations to the
Mental Health Study Commission in December 1992, to revise the State’s administrative
code and achieve rule reduction for mental health, developmental disabilities and
substance abuse facilities and services. In the new rules standards are largely
performance-oriented and specifically. directed. toward establishing and being accountable
for client outcomes and consumer satisfaction.

During this time, the DMHDDSAS was also developing its consumer satisfaction surveys.

November 1995 marked the first administration of the satisfaction surveys and the State
General Assembly’s vote to adopt the new administrative code.

Phase If
Once the administrative code had been amended, the DMHDDSAS proceeded to

identify program area goals and objectives. Finally, client outcome assessment survey
instruments were developed.

75


MASSACHUSETTS

Youth Programs Prevention Services
Bureau of Substance Abuse Services

Department of Public Health

Overview

The Massachusetts Department of Public Health is moving to develop results-based
accountability systems across its many programs. This effort is consistent with the
Commonwealth’s interest in moving all State agencies in this direction.

During the past year, the Prevention Unit of the Bureau of Substance Abuse Services
initiated a process to develop outcome measures for its Youth Programs. As such, it is
among the first of the Department’s programs to begin outcomes planning. The
Bureau’s initiative is currently known as the Pilot Outcome Project (POP). It is the first
phase of a larger effort to implement outcomes systems for all its Youth Programs.

The Pilot Outcome Program presently involves 18 local program sites among the
Bureau’s 50 Youth Programs. The Bureau contracts with these local programs, which
focus on substance abuse prevention services directed to youth in high-risk environments.
The programs provide services primarily in the areas of youth development, student
assistance, court diversion, or street outreach.

Nature, Extent, and Uses of Performance Measures

The Bureau notes that u_ing outcome measures for assessing the performance of
prevention programs poses particular difficulties. The development of reliable models
and valid ways of measuring results is in its infancy and therefore is not well estavlished.

The Bureau is focusing on the use of intermediate outcomes, defined as "...a specific
proposed program result relating to a risk or protective factor for substance abuse...[it]
should relate to your overall program goal of reducing-the percentage of youth who
engage in the use of alcohol and other drugs. The outcome does not need to be a large
part of your activities; it may represent a piece of your plans."

The Bureau has distributed to all programs a menu of standardized outcome measures
used in substance abuse prevention programs by other States and by academic
researchers. These measures focus on community as well as individual factors. In
addition, the Bureau plans to draw on outcomes materials being prepared by the Federal
Government as soon as they become available.

3 Kathleen Herr-Zaya and Homer Rahn-Lopez, Memorandum to Executive Director and BSAS Youth Program

Director, "Requirements and Highly Recommended Outcomes Planning,” May 30, 1996, 1.

33


Committee members believed that they had taken a "critical step" when they
differentiated "core functions" from "capacity". They defined capacity as: (1) the
presence of supportive organizational structures and policies, (2) a skilled workforce
equipped with necessary resources, (3) an effective information and communication
system, and (4) an active involvement with the general public, community providers and
elected officials. As outlined in the 1996 report:

Separating the concepts of core function and capacity allowed for the development
of a multi-dimensional tool--a matrix format--which could be used to measure the
current status of both with a manageable number of indicators.’®

Process for Establishing Performance Systems

In 1989, Washington State created a Department of Health. The divisions that comprise
today’s Department of Health were formerly divisions in a large umbrella agency which
dealt with many health and social services issues. The State legislature formed the
Department to bring some cohesion to health policy development and raise awareness of
health issues. Local health jurisdictions were a key constituency in the creation of the
Department.

In 1990, an association of local public health officials began work on reviewing the
recommendations of an influential 1988 Institute of Medicine (IOM) report, "The Future
of Public Health," and its applicability to Washington State. The IOM report identified
assessment, policy development, and assurance as the core public health functions. The
IOM report was used by the Health Care Study Commission whose recommendations
resulted in passage of the State’s Health Services Act in 1993.

The Health Services Act anticipated national health reform. The Act incorporated some
of the work of the local and State health officials in devising a new vision of puttlic health
responsibilities for the State and local health jurisdictions, and allocated money for
further work on the concept. The Health Services Act stated:

The Legislature finds that the public health functions of community assessment,
policy development, and assurance of service delivery are essential elements in

_ achieving the objectives of health reform in Washington State. The legislature
further finds that the population-based services are cost effective and are a critical
strategy for the long term containment of health care costs. The legislature
further finds that the public health system in the State lacks the capacity to fulfill
these functions... .

The department of health shall develop, in consultation with local health
departments and districts, the State board of health, the health services
commission, area Indian health service, and other State agencies, health services
providers, and citizens concerned about public health, a public health services

8 Washington State Department of Health, “Public Health Improvement Plan," December 1996, p. 36.

104


Formula: Number of individuals hospitalized who meet the DMH clinical
criteria for inpatient admission.
Total number of hospitalizations after evaluation by the emergency
screening team.

Indicator 2: Percent of individuals evaluated and appropriately admitted to the crisis
stabilization unit.

Formula: Number of individuals admitted to the crisis stabilization unit who
meet the DMH clinical criteria for crisis stabilization services
admission.

Total number of admissions to the crisis stabilization unit after
evaluation by the emergency screening team.

STANDARD: When a face-to-face evaluation is clinically indicated, the face to face
evaluation begins within one hour of initial contact.

Indicator 1: When a face-to-face evaluation is clinically indicated, the face-to-face
evaluation begins within one hour of initial contact.

Formula: Number of evaluations that were begun within one hour of referral.

Total number of face-to-face evaluations.

Management Challenges

The Department staff describe data-related issues as their most significant challenge.
The contractors have primary responsibility for data collection and reporting in this

system. Issues of data availability and validity, consensus on which data to collect and
how to collect them, definitions of terms, and changes to the contractors’ systems for
collection have been significant. Other complicating factors include orienting contractors
new to the system and continually changing the contractors’ data collection systems due
to the inevitable changes to the indicators themselves.

Another major challenge facing the Departmart will be changing contractors’ orientation
from one of simply collecting data to one of using data to improve their performance.

Lessons Learned

The Department identified several factors important to its progress to date in developing
results-based accountability systems. These include:

+ gaining sustained commitment from the Department’s top leadership;

40


learn lessons about what to do and what not do as they consider the feasibility of
pursuing this approach with other counties.

Lessons Learned

Among the important lessons learned by those involved in advancing this initiative are:

Be tenacious and prepared to seize the moment of opportunity. County officials have

pursued this idea for consolidation of funding for several years. They did their
homework and prepared administrative systems that would support a transition from
categorical to consolidated funding. They did not give up; they were ready to move when
the climate for change became hospitable.

Bring all partners into the picture sooner rather than later. Officials believe that earlier

communication with cognizant Federal agencies could have made the process more
efficient.

Plans should be locally conceived. This particular package of categorically funded grants
works well in a consolidated approach for Monroe County. Another county might select
a different package of children and family health grants or focus on a different public
health issue such as Disease Control and Prevention grants.

For Further Information

Geraldine Bunn, Associate Director
Division of Family and Local Health
New York Department of Health
Empire State Plaza, Corning Tower
Albany, New York 12237

65


The Bureau-sponsored school survey questions can be used to address substance
availability and frequency of use over time. For example, one section of the survey asks
about use of drugs other than alcohol:

How often have you used psychedelics
LSD, mescaline, PCP)? TIMES

Never 1-2 3-5 6-9 10-19 20-39 40+

In your lifetime (ever)...... A B C D E F G
During the last 12 months

(past year)........ A BC D £E F G
During the last 30 days

(past month)....... A B C D £E F G
Management Challenges

Among the challenges facing the Bureau, as it proceeds with its implementation of the
Pilot and subsequently the expansion of the system to other Youth Programs, are:

Defining outcomes measures for prevention programs. Outcome measures need to be
valid and consistent across programs; they need to reflect outcomes for individuals but

also ideally to address the programs’ basic prevention activities in the community.
ry prog) P

Planning to involve programs in data collection and analysis issues. Plans are being
developed to address the different levels of technological sophistication among the

program sites, and changing attitudes and practices towards using data as tools for
program improvement; and to identify resources for supporting data collection and
computer analysis.

Expanding the Pilot Outcome Project to additional Youth Programs sites. The Pilot

Outcome Project is regularly informing the non-pilot sites of the progress: of the project.
Also, the lessons learned from the ITA and GTA sites are being documented, while the
project is exploring methods for effectively using pilot sites to assist new sites and for
developing more formal systems.

5 Commonwealth of Massachusetts, Department of Public Health, “Tobacco, Alcohol, and Other Drug Use
Survey,” prepared by Health and Addiction Research, Inc., 3-5.

36


departments. The local health departments in the most populace counties will be ranked
against each other. The other health departments will be ranked against others in their
region. Each county and local health department will be informed of its ranking in
relation to the other counties and local health departments, respectively. (As of
February 1997 all counties and local health departments have received their ranking

status.)

The DEHNR plans to use the Level I and II data to help pinpoint deficiencies in public
health programs. The low ranking (bottom 10%) performers will be identified, and the
DEHNR will work with these poor performers to improve their public health programs.
The DEHNR sees itself in partnership with the counties and local health departments.

A "support" position within the State Health Director’s Office has been created and is
now being filled. This individual will have the day-to-day responsibility for overseeing the
accountability effort and will act as liaison between DEHNR and the local health
departments to improve public health programs. The plan is to send out a review team
composed of DEHNR staff to help the local health departments formulate corrective
action plans to improve the effectiveness of their program(s).

Process for Establishing Performance Measures

The Proposal

The DEHNR’s move towards a results-based accountability system began in late 1992
under the direction of the State Health Director, Dr. Ronald Levine. The impe-as for
this endeavor was the following two pieces of legislation passed by the State General
Assembly in 1991, which called for enhanced monitoring efforts by DEHNR:

+ House Bill 499, (An Act to Establish the Mission and Essential Services of the
Public Health System) "...directs DEHNR to: attempt to assure, within the
context of available resources, that a wide range of public health services are
available to all citizens of the State."

+ House Bill 183, (An Act to Require the Department of Environment, Health and
Natural Resources to Conduct Various Health Related Projects) directs DEHNR
to: "implement-a monitoring and evaluation program to measure local health
department progress in applying health outcome standards and achieving health
outcome objectives established by the Commission for Health Services...and to
provide assistance to local health departments that are having difficulty meeting
objectives."

The DEHNR appointed a 20 member Accountability Task Force to "...propose a public
health accountability system." Members of the Task Force were State and local officials
with backgrounds in a variety of public health disciplines. Meeting 12 to 13 times in
1993, the Task Force issued its report in January 1994 proposing an accountability system
that attempts to determine whether the resources appropriated to public health are being
used in an effective and efficient manner. The Task Force viewed the purpose of this

68


improvement plan. The plan shall provide a detailed accounting of deficits in the
core functions of assessment, policy development, assurance of the current public
health system, how additional public health funding would be used, and describe
the benefits expected from expanded expenditures.

In 1995, the legislature, responding in part to developments at the Federal level where
health reform had stalled, repealed health reform provisions for universal access and an
employer pay or play mandate. However, the legislature strongly supported the PHIP
initiative. The legislature appropriated funds to begin implementation of the PHIP.

The PHIP is a collaborative effort of State and local health jurisdictions. Throughout the
life of PHIP, a steering committee with broad stakeholder representation, along with
technical committees (one on performance measures) similarly constituted, have guided
the work. Leadership often came from the local health jurisdictions themselves.

As part of the PHIP process, a large group of individuals also identified 39 key public
health problems in the areas of infectious disease, noninfectious disease, violence and
injury, family and individual health, and environmental health. For each of these five
areas, a subcommittee composed a variety of stakeholders was formed to discuss and
research key problems, standards, and interventions.

These 39 key public health problems in infectious disease, noninfectious disease, violence
and injury, family and individual health, and environmental health laid the foundation for
developing The Health of Washington State. In each of these areas, outcome standards,
threshold standards, and interventions are detailed. For example, in the area of
noninfectious disease, six problems are identified: cardiovascular disease, female breast
cancer, uterine cervix cancer, diabetes, tobacco use, and chemical dependency. In the
area of female breast cancer, measures include: mammogram rates and breast cancer
rates.

To develop performance measures for the core function capacities, the technica:
committee charged with developing performance measures established 20 standards of
performance and 39 performance indicators. The assessment tool was sent to all local
health jurisdictions. Twenty-eight of 33 LHJs responded. They all used the instrument
to rate themselves on.each of the 39 performance indicators.

The assessment tool sent to all LHJs for self assessment on a Likert scale of 0 through 5
included such indicators as:

+ LHI uses data acquired through the information systems in the development of
health policy and the evaluation of prevention services.

9 Revised Code of Washington 43.70.520, contained in Washington State Department of Health, Public Health
Improvement Plan, November 29, 1994, p. 247.

105


NEBRASKA

46


Mother spontaneously vocalizes to child at least twice during visit (excluding
scolding). (Section I, interaction #1)

Father provides some caretaking every day. (Section VI, interaction #41)

In some cases, a direct question may need to be asked of the caregiver. For example,
"Does the family has a pet?" or "Does the child have a special place in which to keep his
toys and treasures?"

Management Challenges

Understanding the new mission. The principle of the Early Start Initiative required
counties/providers to think beyond the “quick fix" (e.g. providing money for food, rent or
gasoline) to the long-term goal of behavior modification and the services needed to affect
this change. One issue the Department has had to struggle with, is people’s (e.g.
legislators, providers and the public) need to simplify the Early Start program into a fixed
package of services and related costs. The Department plans to conduct a public
awareness campaign about the purpose of this initiative.

The Early Start Initiative is being administered by the State very much like a block grant.

Not all Department staff have grasped the concept of block grant administration which
involves a more "hands off approach." In the beginning stages of program development,
the counties floundered a bit, thinking they had to invent new and different programs.
They wished for the Department’s feedback on the viability of various program concepts.
However, the State, seeing its role in this initiative as primarily that of a funding agent,
instructed Department staff to be helpful but not directive. Instead, counties were
instructed to build on existing services and systems already in existence within the county
and only create new services where no service existed. The counties were given free rein
to develop their Early 3tart programs. Performance agreements, standard operating
procedure in the past, may still be used by the counties but the Department will neither
approve nor disapprove such agreements.

Expansion uncertain. Though the Ohio Family and Children First Birth to Three
Committee has submitted a half dozen proposals to expand Early Start, it does not
appear that the State legislature will increase funding for expansion. State resources are
limited and while the Governor and the Office of Budget and Management are in
general support of the Early Start Initiative, it is viewed as part of the State’s abuse and
neglect prevention effort. Also, the legislature has yet to see any data with which to
gauge the effectiveness of this initiative. (As of January 1997, counties have submitted
data but once to the Department. This first set of data will be used as a benchmark
against which county programs’ progress towards meeting their short-term goals and long-
term indicators may be assessed.)

88


Subpopulation: Parents abusing substances putting children at risk.

Measures: (number served, reduced substance abuse, arrests, individual
satisfaction, substance free newborns, employment)

Subpopulation: All other adults abusing substances.

Measures: (number served, substance abuse, arrests, individual satisfaction,
employment)

Example: Mental Health
Target Group: Children with, or at risk of, serious emotional disturbances.
Outcome: Will function appropriately and live in stable settings.
Subpopulation: Children at risk of developing severe emotional disturbances.

Measures: (number served, community days, functional level, family satisfaction,
percent school days attended, arrests)

Subpopulation: Children with severe emotional disturbances.

Measures: (amber served, community days, functional level, family satisfaction,
percent school days attended, arrests)

Management Challenges

Data challenges, as described above, remain significant. Contractor performance
standards are quite broad for several reasons, including program managers’
understanding that forces outside the control of a particular provider can impact results;
but it is also a reflection of the limitations of the data. In particular, the time lag in
contractor reporting and State response can cause the State to question performance that
occurred 2 years earlier, at a time when staff, management, and service delivery methods
at a provider could have been quite different than at present.

Perhaps the most significant question facing the Division in the future as it moves ahead
with performance-based budgeting is the response of the legislature to the Division’s
plans. In responding to its budget submission, the legislature could decide to allocate
funds based on strict categories of funding, without flexibility to the agency, outside the
Government Accountability and Results Act. It could appropriate the Division’s $500
million budget based on its own targets. It is not certain that the legislature is willing to
appropriate large amounts of dollars, and give program managers flexibility to move
resources, based on a few performance measurements; in fact, the Division expanded its
original list of 24 key measurements for 13 target populations to 52 measurements in
response to legislative concerns about sparse data.


impact of various health care interventions on health status and clinical outcomes.
Spurred on by high costs, quality concerns, technological advancements in data storage
and processing, increasing penetration of the health care marketplace by managed care
organizations, and consumer demands for information, Federal, State, local, nonprofit
and private organizations have been working to improve the way health status and
outcomes are measured and used. This movement was given added impetus by the
adoption of Total Quality Management principles by many healthcare organizations
likewise seeking to introduce, within the confines of their own systems, the collection and
use of meaningful data on patients and providers in order to identify and implement
system improvements.

These developments, along with other Federal initiatives such as the Government
Performance and Results Act, have formed a backdrop for HHS efforts on performance
partnership grants in its public health programs.*

In mid-1995, HHS requested the National Academy of Sciences to examine and make
recommendations for specific performance measures that could be used in public health
performance partnerships over the next few years. In 1996, HHS, in collaboration with
several national health organizations, convened a series of regional meetings with States
to discuss current activities in developing performance measurement systems for public
health programs.’ The results of these meetings were provided to the Academy to assist
in its analysis. The Academy issued its draft report in the fall of 1996.° It is now
assessing further developmental work needed in data systems to support performance
measurement systems.

This Inquiry

The Assistant Secretary for Planning and Evaluation asked the Office of Inspector
General to identify and examine State initiatives that use outcomes measures to assess
the performance of their public health programs. Its primary interest lies in knowing
more about the nature, extent, and uses of these outcomes measurement systems, and
lessons learned and challenges faced by the States, building upon information supplied at
the regional meetings sponsored by the Department.

We contracted with Penny Thompson,.a principal in Management. Evaluation Training,
LLC, an evaluation and management consulting firm, to undertake this study. The Office
of Inspector General staff were project officers and participated in all phases of the
study.

In this report, we describe selected initiatives in 11 States. We intend this report to serve
as a resource document that offers insight into the experiences thus far of a number of
States that have engaged in the process of developing outcomes measurement systems for
public health. It is not an assessment of the outcomes measures themselves, their
appropriateness, or adequacy. It is not exhaustive of all State efforts or even all efforts
in States we examined.


MINNESOTA
State Public Health Goals

Minnesota Department of Health

Overview

Minnesota’s Public Health Goals provide an overall framework for the many entities that
are working to improve the public health. Organizations (e.g. health plans and medical
providers) are encouraged to examine the goals to determine how they can contribute to
the achievement of selected goals and objectives.

Nature, Extent, and Uses of Performance Measures

Since the early 1980’s, Minnesota’s community health service districts have been
statutorily required to develop comprehensive health services plans every 4 years.
However, with the advent of State health reform and a push towards managed care, the
need for Statewide, comprehensive public health goals as an overall framework to guide
these activities soon became evident. Minnesota’s 1995 Public Health Goals are
"..intended to provide a common direction for the many entities that are working to
improve the public’s health... . [and] It is hoped...will stimulate and encourage further
voluntary efforts toward the development of healthy communities."

Each of the 17 public health goals (including 3 system development or infrastructure
goals) has one or more objectives that were selected to serve as indicators of progress
toward the goals over time. Some of the objectives were chosen, in part, based on the
availability of baseline ¢.1ta; some currently have no baseline data available.

The intended use of the data is to see what public health improvements have been
accomplished over time. The data will not be used for compliance purposes. Rather,
the State hopes that stakeholders (e.g. health plans, providers and private insurers) will
use this data to voluntarily initiate program improvements.

Process for Developing the Performance Systems

Development of the State’s public health goals was coordinated through the Minnesota
Department of Health. The process for developing these goals builds upon the mission
and strategy laid out in "Minnesota Milestones," the Governor’s initiative to develop a
long-range plan based on Minnesotans’ vision for their State. This strategic planning
document draws upon the work done by the State of Oregon in setting benchmarks and
encompasses all of Minnesota’s State agencies. The Federal Healthy People 2000 goals
were also canvassed by the Department of Health for inclusion in the State’s public
health goals. Input on the proposed public health goals was then sought from public
health agencies, medical care providers, health plan companies, voluntary organizations,

43


In March 1992 the DMHDDSAS formed a 22-member Quality Improvement Committee
charged with making recommendations to the Mental Health Study Commission for the
development of, and transition to, a quality improvement system for the DMHDDSAS.
The Quality Improvement Committee was composed of DMHDDSAS staff and
representatives from client advocacy groups, the Area. Programs and The Commission for
Mental Health, Developmental Disabilities and Substance Abuse Services. (Members of
this Commission were appointed by the Governor and General Assembly. Their mission
was to adopt rules to be followed in the conduct of State and local mental health,
developmental disabilities and substance abuse programs. The Commission also acted in
an advisory capacity to the State’s Secretary of Human Resources.)

Believing that the DMHDDSAS had put too much confidence in structure and process
inputs as adequate measures of quality, the Quality Improvement Committee determined
that a continual assessment of the performance of the service delivery system and its
impact on client care and client outcomes was needed. In addition, the Committee felt
that the streamlining of administrative rules was absolutely necessary to the success of
this endeavor. By 1992, the DMHDDSAS had lost any funding for research contracting
and most of its internal evaluation staff due to legislative budget cuts. Without this
expertise to draw on, the Quality Improvement Committee realized it needed to develop
a quality improvement process that was neither too costly or onerous to implement.

The Quality Plan was developed, approved by the Mental Health Study Commission in
December 1992 and subsequently endorsed as policy guidance for the DMHDDSAS in
February, 1993 by thc State Legislature. A significant portion of the plan involved
changing from a prescriptive, procedures-oriented regulatory system for the public
behavioral health system to a simpler, outcome-oriented approach that is focused on
consumers and reduces administrative complexity.

The Committee developed seven working principles by which the new quality
improvement system was to be guided. These principles are:

Accessibility: Consumers will have convenient and easy access to services;
Early Intervention & Family-Centered: Consumers will be more likely to succeed when
services are provided at the earliest possible-time in the onset of need and when family

consumers are involved from the onset;

Consumer-Driven: Services will be designed and delivered to meet the needs of
consumers;

Empowerment of Consumers: Consumers will be able to make choices concerning
services, lifestyle and supports;

Empowerment of Providers: Service providers will be able to remove service barriers;

74


In addition, The Alliance for the Mentally Ill of Ohio and The Ohio Advocates for
Mental Health will perform a meta analysis of the data collected by the three pilot
projects. This report is due to the State by the end of June 1997. These data are
expected to help facilitate policy, service planning and system improvement at the local
and State levels.

Process for Developing the Performance Systems

The impetus for these demonstration projects came from the State’s Office of Consumer
Services and the Community Support Program Advisory Committee. Under State
standards Ohio’s local mental health boards are required to do outcome evaluations of
the State mental health programs and are also responsible for gathering client feedback.
In the past, the State has never really enforced the outcome evaluation requirement and
client feedback usually consisted of client satisfaction surveys that have not been all that
substantive.

The interview instrument was developed inhouse by the Program Evaluation and
Research Office in Ohio’s Department of Mental Health. Consumer satisfaction surveys
used by the States of Georgia and Pennsylvania for their own clients of mental health
services were reviewed as a starting point for Ohio’s own model.

Management Challenges

Working together to effect improvements. The Department of Mental Health is
encouraging its local mental health boards to use the "continuous quality improvement

model." This effort will necessitate the local mental health boards working together to
make this happen. This may be problematic due to the fact that the local mental health
boards: do not all have the same service plan, are located in different geographic areas,
and do not necessarily tend to share information.

Understanding the data. The plan is to interview clients, effect improvements and then
interview clients to determine if improvements have indeed resulted. Determining how
to use the data, how to improve the system and how to ensure the data is used by local
boards to make a positive impact on consumers will be a challenge for the Department.

For Further Information

Ms. Dee Roth, Chief

Office of Program Evaluation and Research
Ohio Department of Mental Health

30 East Broad Street, Room 1170
Columbus, OH 43266-0414

83


Department of Health and Human Services

OFFICE OF
INSPECTOR GENERAL

Results-Based Systems For
Public Health Programs

Volume 2: State Case Studies

os SERVICES.
Sy J) JUNE GIBBS BROWN
z 7 Inspector General
% SEPTEMBER 1997

OEI-05-96-00261



"Healthy People 2000: National Health Promotion and Disease Prevention Objectives,"
published in 1990. This trend report is updated annually.

Process for Developing Performance Systems

A number of factors influenced the development of DASA’s performance accountability
system. The most significant and immediate factor was a critical report issued by the
Legislative Budget Committee (LBC), which conducted a routine review of DASA system
operations and found them deficient. In addition to problems of data reliability and
validity, the LBC criticized the systems for lacking the capacity to measure outcomes.
The results of this review were widely publicized in the front pages of the Seattle Times.
Around the same time, a new division director was experiencing his own difficulties with
DASA data systems, finding them cumbersome, inflexible and expensive to use because
they were mainframe based, and often lacked key data necessary to make good policy
decisions.

In addition to these DASA specific issues, the larger environment also influenced a move
to a more responsive, user friendly system. Beginning in 1988 and 1989, DASA officials
observed intense discussion at the Federal and State level about program effectiveness,
outcomes, and accountability. State economies were slumping, including Washington
State, creating more pressure for budget tightening in State government. DASA officials
believed performance information was critical for them to protect their program against
unwarranted assault.

Ironically, budget pressures also led to more scrutiny of State spending for research and
planning activities. State legislators and State budget officials were particularly sensitive
to management systems procurements, and in fact had created a data oversight group to
review and approve such investments. In a very concrete way, the problems identified by
the LBC with regard to DASA’s systems helped DASA officials convince legisla.ors and
budget officials to move forward on DASA’s management system improvements.
Advances in technology also made DASA’s vision for a non-mainframe, user-responsive
system possible.

TARGET Development

DASA officials submitted a plan to the data oversight committee, which was approved,
which called for a two year time frame to design and implement their new system. They
received approved for a $1.2 million investment in the new system.

After two false starts, DASA officials contracted with a small consulting firm with
expertise in organizational development, training, and hardware/software expertise, to
develop their data system. Their initial use of a departmental data team was
unsuccessful because the expertise was exclusively mainframe, and the design plans
submitted were not acceptable to DASA management. Their subsequent use of an
independent consultant recommended by another program office proved unsatisfactory as
well. By the time the final consulting team had been hired, a year had passed without

109


INTRODUCTION

PURPOSE

This report contains brief descriptions of selected initiatives underway in 11 States to
implement results-based systems for managing public health programs.

BACKGROUND
Federal Approaches to Grantmaking

The Federal Government currently administers both categorical and block grants to the
States for the purposes of addressing public needs, Categorical grants are for specific
and narrowly defined purposes, and generally include federally-specified eligibility and
reporting requirements. Block grants are broader, encompass a larger range of purposes
and goals, and contain fewer Federal prescriptions.

Government-wide interest in enhancing the performance and accountability of
government programs, including its grants programs, intensified with the publication, in
1993, of the Vice President’s Report on the National Performance Review. As a result
of this and other governmental and nongovernmental influences, policymakers have been
actively considering changes in Federal grantmaking and oversight authorities.!

Some proposals call for combining several categorical and block grants together and
replacing them with performance partnerships. These grants would be negotiated
between Federal Government agencies and the States. They would allow States greater
flexibility in meeting key national and State objectives. In return, States would provide
the Federal Governmem with performance data and work with the Federal Government
to establish goals.

Performance Partnerships and Public Health

The United States Department of Health and Human Services (HHS) has been
considering performance partnership grants for some of its public health programs for
several years.? (See Appendix A for a fuller description of the Federal block grants for
preventive health, maternal and child health, substance abuse and mental health.)

Over the past several years, the President has included provisions for performance
partnership grants in his budget. Several legislative proposals to create such grants have
also been introduced in the Congress.? While not yet enacted, these proposals reflect
the growing consensus about the need to improve the management and structure of
Federal-State grant programs for health.

The concept of performance partnerships for public health is also rooted in, and made
possible by, activity at all levels of government and in the private sector to assess the


significant progress on the project. Nonetheless, matters moved quickly once this final
team was on board.

The DASA also contracted with a quality assurance consultant, who was knowledgeable
about program issues and served as a credibie source of information about the project to
a wide variety of audiences, including the legislature, program officials and program
stakeholders. The DASA convened an executive management committee, composed of
the project sponsor (DASA’s director), the consulting team, the quality assurance
consultant, county officials, providers and other stakeholders, The executive management
team met regularly to discuss design and data plans, and defined how they would
measure the success of the project. Weekly meetings were also held with the program
staff, consulting team, quality assurance consultant, and sponsor.

To develop the items that would be collected within the new system, an advisory
committee was formed. This advisory committee was composed of counselors, county
staff, State researchers, and others. It met every month for a full day. The advisory
committee agreed that the system would meet the needs of the Federal Government,
State government, counties, legislature, and providers. Over 400 data items were
identified through the deliberations of the advisory committee. The committee
prioritized data needs and approved system screens.

Performance Accountability Plan

Parallel to its efforts on TARGET and the development of the report on Statewide
trends, DASA developed a performance accountability plan for residential treatment
service delivery. Using its quality assurance consultant, who facilitated and summarized
results of meetings and prepared a final report on the consensus achieved, DASA
outlined outcomes expected from residential treatment. Providers, who were reluctant
initially to join the effort because of fear that unrealistic benchmarks would be set for
which they had no control, were ultimately persuaded that the development of
performance measures was inevitable, because of the budget pressures and demand for
accountability discussed above. The DASA officials also believe that their own
longstanding credibility and constructive relationships with the provider community
assisted in creating the atmosphere of mutual regard and trust necessary for consensus
building. The DASA’s director commented, "We weren’t trying to build new
relationships at the same time we were trying to accomplish this task (of building a
performance measurement system)."

In addition to coordinating a stakeholder group of providers and counties, DASA
developed outcome measures by conducting surveys of other States and Federal agencies;
reviewing the recommendations of the LBC; and conducting a review of the negative
consequences of alcoholism.

110


measurement system. Unlike its own system, it should be an online, user-friendly system
with language that works in modules so that changes can be processed without undue
down-time. Also, minimum data entry and coding standards should be established at the
outset, provider staff trained in these standards and field tests should be conducted prior
to implementation. The DASA has found that there is a need to involve other State
departments’ in the development of the management information system and data
collection process, as well, so that various department data systems can interface.

Training provider staff, Some of the data DASA receives from its treatment providers is
unreliable. Generally, this occurs because of differences in interpretation of the data or
data entry errors. Officials commented that, to minimize this error, they would require
that each piece of data be individually confirmed.

For Further Information:
DASA Treatment Division

100 West Randolph, Suite 5-600
Chicago, IL 60601

31


OHIC
“Five Point Plan For Strengthening Public Health in Ohio"

Department of Health

Overview

The Ohio Department of Health’s Five Point Plan is a planning document offering a
systematic approach to developing five program initiatives that together will strengthen
public health leadership and collaborative relationships at the State and local community
level. The five interrelated program areas targeted for development are: (1) Community
Needs and Capacity Assessment, (2) Data System Development and Management,

(3) Leadership Skill Development, (4) Healthy People Ohio, and (5) Performance
Standards. Though not yet developed, the Department has decided that the performance
standards will be out-ome based.

Nature, Extent and Uses of Performance Measures

The Five Point Plan is still in draft as of February 1997. Performance standards will be
developed for the other four program areas. These four interrelated program areas, and
a brief explanation of what they are intended to accomplish, are:

Community Needs and Capacity Assessment: This initiative will provide
methodology(ies) and tools for assessing and prioritizing community health problems and
needs, and the community’s capacity to address priority needs. A component of this
program area, and the first step prior to the community assessment, is to examine the
organizational readiness of local health departments.

Data Systems Development and Management: This initiative will provide State and local
health departments with a Statewide, integrated public health data system.

Leadership Skill Development; This initiative will provide a continuing education
program for. public-health professionals that will focus on developing skill-improvement
through practical training programs. The interdisciplinary curriculum will include the
following skills: basic public health sciences, policy development/program planning,
analysis, communication, cultural sensitivity and financial planning and management.

Healthy People Ohio: This initiative is modelled after The Healthy People 2000:

National Health Promotion and Disease Prevention Objectives. It is intended to provide
State and local health departments with a tool to define public health priorities for policy
development and program planning, implementation and funding. A structured approach
will be developed which establishes objectives with measurable outcomes that are aimed
at improving health status, preventing disease/disability and improving access to care
while also controlling health care costs. The State plans to adapt the United States

90


¢ The Home Observation for Measurement of the Environment Scale (HOME).
This tool provides information on the quality and quantity of the social, emotional
and cognitive support available to young children in their home environment. All
30 Early Start counties are required to administer.

+ Individual Family Service Plan (IFSP).
+ Parent Satisfaction Survey.
+ Data Collection Tool.

Providers’ primary concern with the Early Start Initiative was its extensive data collection
requirements and associated costs, which would involve the administration of surveys to
all clients and data entry. Some of these concerns were assuaged when the Department
assured providers that they would not have to collect this large amount of data
indefinitely.

Currently, providers do not have the capability of submitting their data electronically.
The Department is now working on purchasing computers and modems for the Early
Start providers by combining this effort with the State’s Early Intervention Initiative. The
Early Intervention Initiative has been working for over a year to develop a data
collection and reporting system for all 88 of Ohio’s counties. In the meantime, providers
submit their Early Start data to the Department on paper and the Department then hires
temporary data processors to do data entry.

The primary users of this data will be the State. The data is not intended to be used for
compliance purposes but as an aid to determine if counties are on target in reaching
their goals and objectives and where program improvements might be made. A spin off
of this effort, is that the Early Start staff will also be collaboratively working with The
Temporary Assistance for Needy Families’ JOBS (Jobs Opportunities and Basic Skills)
program. In addition, it is believed that this data may support increased funding for the
Early Intervention Initiative.

Process for Developing the Performance Systems

The Ohio Department of Health’s Early Start initiative is actually an initiative of the
Ohio FAMILY & CHILDREN FIRST (OFCF) INITIATIVE. Fully supported by the
Governor, the OFCF initiative was formed in 1992 as a part of the National Governor’s
Association System Reform project to streamline State bureaucracy. A Cabinet Council
was formed, composed of the Directors of the Ohio Departments of: Health, Education,
Mental Health, Menta! Retardation and Developmental Disabilities, Human Services,
Drug and Alcohol, Youth, and the Office of Management and Budget. Each cabinet
director was designated lead for a specific topic area and responsible for coordinating
activities related to their topic area across all departments. (For example, the Director
of the Department of Health’s topic area is improving the public’s access to services.) A
State Action Team made up of State agency department or bureau chiefs was formed to

85


Area: Infectious Disease Control

Measures: (congenital syphilis, HIV seropositivity in childbearing women,
tuberculosis, tuberculosis in children under 15, measles in children
under 18, haemophilus influezae type B in children under 5, high
risk infants beginning immunizations)

Area: Chronic Disease Control

Measures: (metastic breast cancer at diagnosis, metastatic cervical cancer at
diagnosis, years of potential life lost under age 65)

Management Challenges

Our informants in the State office generally spoke very positively of the QI effort, but
noted several factors which make optimal use of the system more complicated. First,
local public units differ substantially from county to county in their roles and capacities.
Generally, public health units in Florida (as in many other parts of the country) saw their
role in the 1970s and early 1980s as one of ensuring overall community public health. In
the mid-1980s, many units reoriented their services towards the provision of direct
primary care to indigent or at risk populations. Today, some public health units see
themselves as needing to move away from the primary care model and back towards the
earlier model of assuring community health. Such a transition requires shifts in resources
that may not be available. In less densely populated areas with few providers, the
pressure for the local public health unit to focus on delivery of primary care to clients is
significant.

These differences in how units see their mission makes development and use of
performance indicators and data very difficult. Some units accept accountability only ‘or
their own client base, rather than the general community. Some units, more focused on
direct provision of care, may lack the analytical capacities to use and apply data they
receive. Differences in the size and population base of local health units also create
significant variation in the orientation to and use of data. Our informants at the State
level characterized local health units as still very process oriented and driven, with a
focus. on patient care rather than data.

A further complication is the absence of some types of data. There can be a significant
time lag in receiving certain data, depending on its source. Our informants described the
data from vital statistics as very timely; but noted that data from cancer registries can lag
behind as much as 3 years. The absence of good child morbidity data was mentioned.
The difficulty in obtaining data from other departments--Medicaid, highway safety and
motor vehicles, corrections--was identified as an obstacle. And, informants also noted
that the spread of managed care might well complicate further their ability to obtain
important data. Nonetheless, it was the effective exploitation and use of existing data,
rather than the need for additional data, that was identified as the most significant
challenge remaining within the QI process.

13


NEW YORK

Department of Public Health

Overview

In the past year, the New York State Department of Public Health has initiated a
process to develop results-based accountability systems for its Statewide public health
programs. As a first step, it initiated a comprehensive review process that identified 12
public health areas of high priority for future action. Each priority area has measurable
objectives to guide State and community actions for improving the health of New
Yorkers.

Nature, Extent, and Uses of Performance Measures

The Department recently completed work on a year-long effort to identify goals and
measurable objectives for high priority areas of opportunity to improve the health of New
Yorkers. This review process culminated in the publication of "Communities Working
Together for a Healthier New York" in September 1996. This report, directed to local
communities, focuses on prevention. It establishes priority areas for action over the next
10 years. This priority-setting was the first step of a long-term process to improve public
health through partnerships among State programs and communities, guided by specific,
quantifiable objectives for measuring progress along the way.

Rather than focusing on health problems directly, the process emphasized the causes of
poor health and other adverse outcomes. It also searched for effective interventions
appropriate for both communities and individuals. From this process, according to the
Tteport, emerged:

Twelve priority areas of opportunity for public health action (listed alphabetically, not by
importance): (1) access to, and delivery of, health care; (2) education; (3) healthy births;
(4) mental health; (5) nutrition; (6) physical activity; (7) safe and healthy work
environment; (8) sexual activity; (9) substance abuse, alcohol and other drugs;

(10) tobacco use; (11) unintentional injury; and, (12) violent and abusive behavior.”

The report describes at least one quantifiable objective (and, where available, baseline

data and references) to be achieved over the next 10 years for each of the 12 priority
areas.

° New York State Public Health Council, Communities Working Together for a Healthier New York: Opportunities
to Improve the Health of New Yorkers, September 1996, 1.

55


OREGON
Division of Health

Department of Human Resources

Overview

As of 1991, Oregon has used benchmarks to assess its progress toward achieving a broad
strategic goals set forth in the State’s strategic plan, "Oregon Shines." Oregon's
Department of Human Resources uses a Statewide system of outcomes and performance
measures to evaluate its success in meeting the State strategic plans goals and objectives.
The Department is composed of six divisions, among them the Division of Health.

Nature, Extent, and Uses of Performance Measures

The State has just revised its strategic plan (Oregon Shines II) to address the issues and
concerns of Oregonians today. The new central vision for the State outlined in the
strategic plan is that:

+ Oregon will have a high-wage economy and an excellent quality of life.
+ Oregon will be a place where all families and individuals can prosper.

+ Oregon will balance demands of a vital economy with demands of health
ecosystems.

Oregon’s benchmarks have also been amended. The total number of benchmarks has
been decreased from 259 down to 92 benchmarks that focus on measurable outcomes.
The health related benchmarks have decreased in number to 23 and the Department of
Human Resources and its Divisions are in the process of revising its outcomes and
performance measures to correspond to the new benchmarks. Six of the 20 Department
outcomes relate specifically to the Division of Health, which has developed 43 draft
performance measures for these 6 outcomes.

Oregon Benchmarks is administered by the Oregon Progress Board. Through citizen
opinion surveys and public review and comment, the Board scans trends to draw
attention to opportunities and challenges within State programs that may be addressed in
the Benchmarks. While this gives an overall direction to pursue in the Benchmarks
program, historical data are used to establish a baseline for various target benchmark
outcome and performance measures.

In response to each benchmark, Oregon’s institutions (public, nonprofit and private) take
periodic data measures that are then collected and compiled by the Progress Board in

97


The DMHDDSAS Quality Improvement section staff began with lists of questions
(outcome indicators) drawn up by the Quality Improvement Committee in December
1992 that evolved, over a period of about one year, into the draft client outcome
assessment surveys currently being piloted. Development of these surveys was delayed
until early 1996 for a variety of reasons. First, was the lengthy time it took to pass the
administrative rule reduction legislation. Secondly, as no extra funding was appropriated
for this effort, the DMHDDSAS had to shift funding and staff on an as needed basis to
work on this initiative. Also, staff assigned to develop the surveys worked within the
same section and had to fit this activity in with their other duties. Further complicating
this endeavor was the delay caused by section staff turnover that allowed the effort to lie
fallow until someone was available to work on the initiative. In addition, wanting the
survey instruments to be as succinct as possible, every section within the DMHDDSAS
had to justify the need for the information they wished providers to furnish. Finally, the
implementation of the Program/Performance Budgeting System, beginning in 1994,
changed the focus of the DMHDDSAS client outcome effort. Each Division of State
government was required to develop a limited set of client outcomes for inclusion in the
State budget. Both the requirements and limitations of the budgeting structure further
postponed development of the more comprehensive effort directed toward client
outcomes within the DMHDDSAS.

Sample Performance Measures

The DMHDDSAS has developed and identified program goals, objectives and qualitative
and quantitative outcome indicators specific to the program areas of mental health,
developmental disabilities and substance abuse. The following provides an example of a
goal (including one objective and related outcome indicators) for the mental heaith
program area in DMHDDSAS:

Goal #11: Maintain and improve health care access and treatment.

Objective #2: At least 80 percent of mental health clients and their families will be
satisfied with the mental health services received.

Outcome Indicators: Data from consumer satisfaction surveys, quantitative service
data. ~

Some examples of data collected on the Child Mental Health Update/Discharge Outcomes
Assessment Survey are provided in Section 4 of this instrument, which supplies data on
the services needed and received by the client in such areas as:

+ Social Services (i.e., protective services, family unification, family preservation,
shelter, adoption, foster care);

+ Mental Health (ie., outpatient services, case management, day treatment services,

residential group home, residential therapy home, emergency inpatient, non-
emergency inpatient, family support, respite);

76


Lessons Learned

Bureau staff recognize they are in the early phases of a long-term process, but think
these beginnings have been positive and augurs well for the future. They consider
several aspects of the process to have been particularly important. For example:

The process has been inclusive. Local programs have been involved from the beginning
in the initiative. They have had multiple opportunities for input along the way and have
had maximum discretion in deciding whether to participate in the pilot and which
outcomes measures to adopt. The Bureau included in the Pilot all sites that had
expressed interest in participating. Bureau staff think including input from other States
and from the research community has been particularly valuable.

The process has been a partnership. From the beginning, the Bureau has assigned high
priority to making the initiative meaningful and non-threatening to participants. The
Pilot is conceptualized and discussed largely as an effort to help improve programs. The
initiative is separate from the sites’ contracts with the Bureau. The sites are not being
held accountable for results during the Pilot.

Participation in the Pilot was encouraged, but voluntary. The Bureau pursued several
strategies for educating its Youth Programs about the initiative. It conducted mailings
and meetings, and identified and publicized the potential benefits, costs and the
responsibilities associated with participation in the Pilot. Ultimately, each program
decided for itself whether to participate.

The initiative is being planned and implemented incrementally. The chances of success

are maximized by beginning with several pilot sites, each of which adopts only one or two
outcomes measures. This allows for learning throughout the process.

For Further Information

Kathleen Herr-Zaya, PhD.

Planning and Development, Prevention Unit
Bureau of Substance Abuse

Massachusetts Department of Public Health. .
250 Washington Street, 3rd floor

Boston, MA 02108-4619

37


To develop outcomes, PAT collaborates with communities and organizations to develop
recommendations, and makes recommendations to the Partnership Council, which then
makes recommendations to the Policy Cabinet. To develop indicators, PAT works with
TA Team and community workteams, and reviews literature to develop recommendations
for indicators. It then solicits feedback from the Partnership Council for feedback;
identifies data needs, requirements, availability; and takes recommendations to the Policy
Cabinet. To develop performance measures, PAT collaborates with TA Team, providers,
program, customers, and communities (using, in part, focus groups). To support program
leaders and staff in the analysis of data, PAT relies on assistance of information systems
experts and the TA Team. The analysis includes indicator and performance
measurement data as well as consideration of external variables and feeds information

back to the system.

While this system is not yet in place, Nebraska is in part basing this system on its
experience with strategic management of public health. The Nebraska Department of
Health issued a report in May 1996 entitled, "Strategic Plan for Public Health for Fiscal
Years 1998-1999," which outlined a series of outcome measures and indicators, many of
which are based on Nebraska Year 2000 Objectives, patterned after Healthy People 2000.

Process for Developing Performance Systems

The genesis of Nebraska’s Partnership for Health and Human Services was an executive
order signed by Governor Benjamin Nelson in January 1995. Executive Order 95-2
instructed Lieutenant Governor Kim Robak to carry out a review of the State’s health
services infrastructure, processes and goals and determine ways in which to increase State

capacity.

The task force determined that current organizational configurations stood in the way of
a unified, results orien‘2d approach. Legislation was prepared for introduction in
January 1996 which would sunset five separate agencies in 1997--the Department of
Aging, the Department of Health, the Department of Public Institutions, Department of
Social Services and Office of Juvenile Services--and establish in their stead:

+ a Health and Human Services Policy Cabinet with responsibility for strategic
-planning, budgeting-and allocation, and evaluation;

+ a Health and Human Services Partnership Council, an advisory group to assess
achievement and communicate with the community; and,

+ three new agencies focused on function rather than population (services delivery,
regulation and licensure, and finance and support).

As part of the reorganization plan, communication strategies with stakeholder groups
were identified. Among other things, employee feedback was solicited and analyzed.
Citizen/community input was obtained through public forums and meetings. A newsletter
was distributed to approximately 17,000 individuals on a State mailing list, every 3 to 4

49


1. For a detailed discussion of general concepts of accountability and specific issues of accountability in
Federal block grant programs, see President’s Council on Integrity and Efficiency, Committee on Inspection
and Evaluation, Accountability for Block Grants, July 1996.

2. Fora discussion of Federal grant approaches to funding public health activities, see U.S. Department of
Health and Human Services, Office of Inspector General, Federal Approaches to Funding Public Health

Programs, OEI-01-94-00160, July 1995.

3. The President's Fiscal Year (FY) 1996 budget included a proposal for 16 Performance Partnerships in
the Public Health Service, including the consolidation of various grant programs within Centers for Disease
Control (CDC), Health Resources and Services Administration (HRSA), and the Substance Abuse and Mental
Health Services Administration (SAMHSA). In the FY 1997 budget, the Administration continued to propose
performance partnerships for CDC, HRSA and SAMHSA. In the FY 1998 budget, the Administration
emphasized performance partnerships within SAMHSA only.

Both Representative John Dingell and former Senator Nancy Kassebaum introduced bills in the 104th
Congress that would have implemented the performance partnership model for public health service programs,
and specifically for mental health and substance abuse block grants.

4. A recent report by the Institute of Medicine, summarizing a workshop on performance monitoring for
improving community health, identified a wide range of activities by Federal and non-Federal organizations.
For example:

The National Committee for Quality Assurance has developed the Health Plan Employer Data and
Information Set to collect standardized information from health plans for their own use in assessing
their performance, and to help purchasers make decisions among health plans. The Committee is also
active in assisting health plans to improve their capacities for data collection, analysis and reporting.

The Joint Commission on Accreditation of Healthcare Organizations is moving beyond standards to
develop indicators for assessing the reliability of the standards in predicting actual performance. The
Indicator Measurement System was originally developed for hospitals and is now being expanded.

The HHS is responsible for Healthy People 2000, a set of objectives regarding the health promotion
and disease prevention with goals to be accomplished by the issued model standards for translating
those national year 2000. The American Public Health Association has goals into community action
plans.

The HHS, through the Agency for Health Care Policy and Research, funds research on health care
outcomes and quality measurement.

The HHS, through the National Center for Health Statistics, assembles vital statistics and conducts
significant health related surveys.

For further information, see National Academy of Sciences, Institute of Medicine, Using Performance
Monitoring to Improve Community Health: Exploring the Issues, Workshop Summary, 1996.

5. Among the national health organizations working with HHS were: the Association of State and
Territorial Health Officers, the National Association of State Alcohol and Drug Abuse Directors, the National
Association of State Mental Health Program Directors, and the National Association of City and County
Health Officials.


NEW YORK

54


weeks, reporting on progress. A web site on the reorganization experienced about 400
visits a month.

In April 1996, a performance accountability work group was convened and identified four
tasks for itself to accomplish by November 1996 to: (1) develop a system of performance
accountability, (2) develop a starting list of outcomes desired by program interventions,
(3) develop a starting list of indicators for those outcomes, and (4) develop a focus for
performance measures. Work teams were established to accomplish these tasks.

The performance accountability work group was co-chaired by a State employee and
stakeholder (a beneficiary advocate). This work group, as well as the work teams, were
composed of strangers, from a variety of different private or public sector backgrounds,
with expertise and experience in a variety of different areas.

With such an imposing task and with no preexisting relationships to use as a foundation,
the steering committee relied heavily on facilitative leadership to ensure constructive and
productive conversations. The group spent time developing a set of principles to guide
their discussions together and the tasks they were assigned. The committee, and all three
work groups, were each assigned a team leader, facilitator, and support staff.

To develop a performance accountability system, the committee members first attempted
to obtain information from recent research literature, other States, and the agencies
which were being reorganized to form the Department of Health. Committee members
found helpful the work of several academics and writers in the field, who provided them
advice. The team found no useful models to follow within the Nebraska State agencies
they examined.

Work team members focused on outcomes went through an iterative process, using the
community and employee forums at which the reorganization of health functions was also
discussed, to generate discussion and input on outcomes. Based on a series of initial
forums, the work team tormed an initial list of 80 outcome statements. The list of 80
was shared at four new community and four employee sessions, for reaction,
consolidation and categorization, and priority setting. The team is now in the final stages
of testing 19 outcome statements, along with suggested indicators for each outcome
statement, as their final- product.

Work group members agreed that the performance accountability system they wanted to
develop would have a number of different characteristics:

+ It would not be a rule based performance accountability system. It would have to
apply throughout the organization, from the top down.

+ — It would be simple.

+ — It would be efficient.

+ It would be cost effective.

+ It would encourage accountability.

50


Collect data that is necessary, not what is interesting. Take the time to identify the data

that’s needed. Confer with stakeholders and justify data needs. The process for
developing the data elements on the client surveys has taken DMHDDSAS at least 1

year.

Concurrent processes. It took DMHDDSAS 2 1/2 years (from the December 1992
report to the Mental Health Study Commission) to get the administrative rule reductions
passed by the State General Assembly. During this time, further development and
refinement of the initiative (e.g. data needs, surveys, client outcomes) was put on hold.

In hindsight, the DMHDDSAS would have run the two processes simultaneously. Two
separate groups, one working on the rules reduction effort and the other working on
developing the pilot programs would have been formed. These groups would have met
periodically to coordinate their efforts. If this had been done, the pilot programs could
have been implemented immediately upon passage of the new administrative rules, rather

than over a year later.
For Further Informe*‘on

Mike Pedneau, Director

Division of Mental Health, Developmental! Disabilities
and Substance Abuse Services

Department of Human Resources

325 North Salisbury Street

Raleigh, N.C. 27603

79


Lessons Learned

Sooner rather than later. The Department believes providers would have benefitted
more from the experience of people at the public health level at the very beginning of
the initiative rather than later on in the process.

Consensus building. Reaching consensus from stakeholders at the beginning of the
process in regard to outcomes, definitions and what data will be used to measure these

outcomes is critical in the development and implementation of outcome-based
performance measures.

For Further Information

Ms. Cindy Oser, Chief

Bureau of Early Intervention Services
Ohio Department of Health

246 North High Street

P.O. Box 118

Columbus, OH 43266-0118

89


private insurers, academia and Regional Coordinating Boards. In 1994, Regional
Coordinating Boards were statutorily established to serve as forums for the discussion
and identification of those public health goals that are amenable to efforts by health plan
companies and medical care providers. A draft of the goals was also sent to over 700
individuals and organizations, including consumer and-advocacy groups, for comment.

Sample Performance Measures
Each of the public health goals has one or more objectives that were selected to serve as
indicators of progress toward the goals over time. In the following goal example, one

objective for each of the four subject areas within GOAL 1 is shown:

GOAL 1: Encourage and support community norms, practices, and policies that will reduce
the behavioral risks for chronic diseases and minimize the impact of chronic disease and
disability when it occurs.

OBJECTIVES (by the year 2000):

Tobacco Use

11 Reduce the percentage of adults who smoke from 22.5 percent to 15 percent.
(Information will be received from the Behavioral Risk Factor Surveillance System
[BRFSS], 1993.)

Physical Activity

1.8 Increase the percentage of children and adolescents reporting moderate daily
physical activity to at least 50 percent. (No Minnesota baseline data available.)

Alcohol Use
1.9 Reduce from 41 percent to 35 percent the proportion of high school seniors who

have used alcohol at least once a month for the past 12 months. (Information will
be received from the Minnesota Student Survey, 1992.)

Nutrition

1.12 Increase the percentage of people 18 years and older who report consuming five
or more servings of fruits and vegetables daily. (BRFSS, 1994.)

Management Challenges

Development of data systems. The State believes that: "Ultimately, limited resources
must be prioritized and allocated to those activities that have the greatest impact on
improving or maintaining the health status of the population." The development of
systems to collect reliable and current health data has been identified as a critical need in

44


APPENDIX A

Federal Public Health Block Grant Programs



« NORTH CAROLINA ....... 2... eee e eee eect t ee e eee ence eens 66

Department of Environment, Health and Natural Resources .............. 67
Division of Mental Health, Developmental Disabilities

and Substance Abuse Services... 0.6.6.0 s ee eee cece etree teens 72

Fk ©) © 1 ( © aaa aI 80

Department of Mental Health «1.2.2.0... eee e eee eee eee eens 81

Bureau of Early Intervention Services ....... 606.4 e cece eee eee 84

Five Point Plan For Strengthening Public Health in Ohio ..............-. 90

POE ©) 22, ©, @)\ Sa 96

Division of Health 2.0... cc ene eee eee ene eee 97

e WASHINGTON ......... 2c cece eee reece eet eee e ene eeeeees 102

Public Health Improvement Plan ......... 0s ee eet eee eee eee eens 103

Division of Alcohol and Substance Abuse ....... 6.62. s ee eee eee eens 108

APPENDICES
A: Federal Public Health Block Grant Programs .......-.. esses eee eeeeee A-1

B: Endnotes 2.2... 2. cece eee cece ee eee ete ete t een e ences B-1


identify strategies and the outcome, output, and efficiency measures which relate to each
strategy, as well as actual data from FY 1994 and 1995, and estimated achievements in
FY 1996 through 1999.” Because of their prior work, the Department of Health was
able to comply with this directive rather smoothly.

Lessons Learned

Officials we interviewed ascribed their success to date to leadership, stakeholder
consensus, and support from the legislature.

Because of the historical support of LHJs for the Department of Health, State and local
health officials enjoyed a good, constructive working relationship. Officials we
interviewed thought that the inherent trust in this relationship made the process work:
"The process was built on consensus. People have to brought along. When we were
finished, we had consensus; we didn’t have a product we had to market or sell."

Officials we intervieed told us they first tried another approach, and carried it out four
or five steps, before abandoning it as too cumbersome and finding the right direction in
developing the first PHIP. "Mistakes are inevitable," said one official. “That’s not failure.
Your tolerance for ambiguity has to be pretty high."

Officials who were responsible for guiding the development of the PHIP were particularly
encouraged by the legislature’s support of PHIP at a time when many other provisions in
the original Act estab!'shing the initiative were being repealed. They attribute this
support to the emphasis they placed on accountability and the focus of PHIP on
community empowerment.

For Further Information

Lucia Miltenberger

Health Policy Analyst

Office of Policy and Planning
Department of Health

1102 SE Quince Street

P.O. Box 47815

Olympia, WA 98504-7815

20 State of Washington, Office of Financial Management, Operating Budget Instructions, 1997-99 Biennium, May
1996, B-96-01.

107


NORTH CAROLINA

The Department of Environment, Health and Natural Resources

Overview

The Department of Environment, Health and Natural Resources (DEHNR) is in the
process of developing a public health accountability system for its local health
departments. This system will allow DEHNR to employ performance-based management
to measure local health department progress in applying health outcome standards and
achieving health outcome objectives established by the State’s Commission for Health
Services.

Nature, Extent and Uses of Performance Measures

The DEHNR’s proposed Accountability System consists of three levels of assessment.
The design of this system was stratified to take into consideration those areas where the
local health departments have a limited or negligible impact. While Level I and II
indicators are still in the draft stage, data for these indicators are now being collected for
the counties and local health departments. There are 86 local health departments (or
jurisdictions) in the State, some encompassing more than one county. Level III
assessment, as currently proposed, will consist of monitoring activities currently being
carried out by the health divisions.

Level I consists of a Community Wellness Index, a "snapshot" of 8 health status indicators
for each of North Carolina’s 100 counties. These indicators were selected because it was
believed they represented the most pressing public health problems in the State. Each of
these indicators also has two or more population-based health outcome measures (rathe:
than local health department-specific measures) on which counties are ranked against
each other. These data are easily accessible to DEHNR and will be updated annually.
The data will provide a "sound bite" measure of health status in the county while, at the
same time, avoid measuring local health department performance in an inappropriate
way.

The 32 Level II indicators add process measures to outcome measures in an attempt to
answer questions about the nature, quality and quantity of public health services being
provided and, in some cases, the impact of these services on health status. They
examine, in greater detail, the areas addressed in the Level I indicators. In some
instances, Level II data may be qualitative in nature, such as that derived from surveys.
A decision has not yet been reached as to whether the data for Level II indicators will be
updated yearly or biennially.

The DEHNR is just completing (January 1997) its data collection for Level I and II

outcome measures. These benchmark data will provide the State with a score card
assessment of the status of public health in individual counties and within local health

67


membership of the team; and, (5) a single sponsor. Reorganization teams without such
characteristics, these members believed, were not as successful.

For Further Information

Davie Shutzer, Administrator
Performance Accountability Division
H&HS Regulation and Licensure
P.O. Box 95007

Lincoln, NE 68509-5007

53


WASHINGTON

102


outcome based performance accountability program was needed, and thus was born the
"Community Focused Prevention Through Risk Reduction" process.

This process was derived from more than a decade of research conducted by Drs.
Catalano and Hawkins and colleagues at the University of Washington and other
universities around the country and is an adaption of work performed under DASA
contract by:

+ Prevention First, Inc.,
+ Lighthouse Institute, and
* University of Illinois Center for Prevention Research and Development.

The reasons for implementing and maintaining a performance accountability system are
even more imperative in today’s environment of Federal and State government cost
cutting. Prevention providers are under pressure to demonstrate the impact of their
programs in an environment where: (1) ATOD abuse is a central factor in crime, school
dropout and workplace under-performance; (2) the public is growing increasingly
frustrated at the continuing degradation of the quality of life in their communities;

(3) legislators and the public are focusing more on punishment versus cost of prevention
programs; and, (4) government funded programs are under intense scrutiny to
demonstrate efficiency and effectiveness.

Pilot projects testing the CPRR process began in 1991, prior to implementing the new
system Statewide between 1992 and 1993.

Prevention providers are aided in their efforts by the InTouch System, a statewide
prevention network that offers technical assistance and training to enhance prevention
efforts funded by DASA and by other sources. InTouch coordinating offices bring
together community p;2vention activities to ensure that these efforts complement one
another without unnecessary duplication.

Sample Performance Measures

The outcome measurement surveys used to assess prevention providers’ programs may
address one or-a-number of the risk factors and may be used-for. children, youth or
adults, or any combination of the three age groups. For example, the survey tool
“Attitudes and Beliefs About Alcohol," is given to adults or youth. The IRRA risk factors
this survey addresses are: (1) availability of ATOD, (2) parental drug use and pro-
ATOD family norms, and (3) community laws and norms favorable toward ATOD. The
survey uses a Likert scale of 1 to 5 inwhich clients are asked to indicate a level of
agreement or disagreement to a series of statements or indicators. Some sample
indicators from this survey tool are:

+ It is OK for anyone to drink if they are not driving.
+ Alcohol use is a way to appear more adult.

27


Lessons Learned

Involve all stakeholders from the beginning. Making this effort community based and

moving quickly to engage all stakeholders in the process was critical in the development
of the system.

Do not get caught up in the political rhetoric. Take the time to think about what you

want to accomplish and then determine how this can be achieved in a results-based
accountability framework.

For Further Information

Janet S. Bittner, Executive Director

Georgia Policy Council for
Children and Families

47 Trinity Avenue, S.W., Suite 501

Atlanta, GA 30334

23


+ LHI provides coordination, direction and leadership within the community to
improve the access to, and quality of, health services.

+ LHI has access to people technically skilled in carrying out policy development
activities.

In its 1994 PHIP report, standards were included for core function capacities. For
example, under health assessment, examples of standards included:

+ All public health jurisdictions...must have access to an integrated, centrally
managed electronic network that provides access to Federal, State, and local
information systems.

+ All public health jurisdictions...must develop, operate and assure the quality of
data management systems which meet local needs in order to systematically
collect, analyze, and monitor standardized baseline data.

+ All public health jurisdictions...must conduct and publicize epidemiologic,
sociologic, economic and other investigations which assess the health of the
community and access to health care.

Management Challenges

Obtaining consistent data is a continuing challenge, and the Department of Health is
leading an effort to assess data needs and sources for public health purposes throughout
the State. To date, no Statewide system of data exists which can be easily used by State
officials or local health jurisdictions to obtain key data on health indicators, risk factors,
or outcomes. Officials also expressed concern about managed care organizations
reluctant to share data that might give competitors a competitive advantage.

Those involved with PHIP continue to struggle with different capacities and needs of
extremely different sized counties. Of 33 local health jurisdictions, 20 serve populations
of less than 50,000. King County serves over 1.5 million people.

Key State and local officials working on PHIP believe they may be 3 years away from
tying capacity to health outcomes. In the meantime, the use of assessment data (local
communities’ health status and risk factors) will be used to guide programmatic decisions
on priorities and activities at the local level.

Independently, the State Office of Financial Management is pursuing goals that are
compatible with the core functions approach. Washington State is encouraging the
development of performance measurement systems within the State by requiring each
agency to submit 6-year strategic plans as part of their 1997-99 biennium budget
submissions. Among other things, strategic plans must include the agency’s mission
description, strategies, goals, objectives, timelines, and performance measures. A
separate form for performance measures has been developed, in which agencies must

106


Involving Local Youth Programs Directors

Staff from the community-based Youth Programs projects have been involved from the
beginning of the process. Some are members of the Unit’s Youth Programs Outcome
Committee, which spun off from a larger advisory group to the Bureau. It focused on
developing a menu of outcome measures for the prevention programs.

All Youth Programs Directors were surveyed for input into the measures development
process. In mid-summer 1996, they were invited to participate in a day-long training and
orientation session to become familiar with outcomes planning. The Bureau’s orientation
session included discussion of the potential benefits as well as costs, requirements, and
responsibilities associated with being a pilot site.

Program sites interested in participating were asked to volunteer. In so doing, the
Bureau asked each to agree to certain responsibilities, such as regular data collection and
attendance at various POP-related meetings.

Implementing the Pilot

The site application and selection processes occurred during August 1996. The Bureau
sought balance among the pilot sites in terms of geographic location, size, and type of
program. Important to the Bureau, too, was the willingness of each site to share its
experiences during the Pilot with other Youth Programs.

Eighteen sites volunteered. The Bureau divided them into two groups. Eight sites, those
assigned to the Individual Technical Assistance (ITA) group, receive individual technical
assistance from the Bureau’s research consultant. The 10 other sites, assigned to the
Group Technical Assistance (GTA) group, work with the quality improvement sub-unit of
the contractor.

During the fall of 1996, the sites began working with their respective groups to identify

and select appropriate outcome measures. The two groups are operating somewhat

differently at present. A primary difference is that each ITA site selects its own specific

outcome measure(s); the GTA sites are working to develop a common outcome measure
~ to be used.

Sample Performance Measures

The Bureau distributed to the Youth Programs a menu of options for outcomes
measurement related to alcohol, tobacco, and other drug use. It included some used by
the Illinois Department of Alcoholism and Substance Abuse and the Institute for
Prevention Research, Cornell Medical Center. These measures are derived from surveys
of children and youth that focus on topics such as substance use, coping and problem
solving skills, attitudes about substance use, and sense of community.

35


activities in the State; the need of the State for technical assistance; efforts by the State
to improve such activities; and, the extent to which availability of such activities matches
the needs.

States must provide for independent peer review to assess the quality, appropriateness,
and efficacy of treatment services provided in the State to individuals under the program.

No more than 5 percent of the total grant to a State may be used for administrative
expenses.

Preventive Health and Health Services Block Grants

The Public Health Service Act provides for block grants for preventive health and health
services. Allotments under this section are for preventive health services, comprehensive
public health services, and emergency medical services. Payments to States may be used
for activities designed to make progress towards Healthy People 2000 objectives.

States must develop their plans in consultation with State Preventive Health Advisory
Committees. They must specify strategies for making progress toward improving the
health status of the population and establish reasonable criteria to evaluate the
performance of entities receiving payments from the State. The statute also specifies
that States must report to the Federal government on progress in meeting the year 2000
health objectives and use the uniform collecting and reporting formats developed by the
Secretary for this purpose.

No more than 10 percent of the total grant to a State may be used for administrative
expenses.

Maternal and Child Health Services Block Grant

The Public Health Service Act provides for funds to improve the health of all mothers
and children consistent with the applicable health status goals and national health
objectives established by the Secretary for the year 2000. The grants are to be used to
provide and assure mothers and children access and quality health services, reduce infant
mortality and incidence of preventable disease and handicapping conditions among
children, reduce the need for inpatient and long-term care services, increase the number
of immunized children, and other objectives.

States must report on a number of measures specified within the statute or by the
Secretary, including rate of infant mortality, rate of low birthweight births, rate of
maternal mortality, rate of neonatal deaths, and other such health status indicators.


As initial steps, the Department is making available to the counties $750,000 to stimulate
follow up on areas of opportunity within the plan and to encourage efforts of local
planning groups. The Department, too, is seeking for the counties $1 million to support
a behavior risk survey to gather data on key behaviors at the county level; at present, all
the behaviors data are available only on a Statewide basis.

Sample Performance Measures

The 12 priority areas identified through New York’s review process offer the
communities in New York opportunities for improving the health of their residents.
Action in these areas by communities across the State could reduce the major causes of

death, disease, and disability.

The Department’s report, Communities Working Together for a Healthier New York,
includes, for each priority area, objectives and indicators, a rationale for the particular
area, and descriptions of the size of the problem and of possible interventions.

The Nutrition priority area, for example, has the following objective:!? By the year
2006, reduce the prevalence of overweight to no more than:

+ 20 percent among adults 18 years of age and older (baseline: 27%, Behavioral
Risk Factor Surveillance System, 1994); and,

+ 15 percent of second and fifth grade school children (baseline: 34.5% New York
City, 27.9% Rest of State; New York State Department of Health Nutrition

Survey, 1990).

The Mental Health priority area’s objective is that: By the year 2006, reduce the rate of
hospitalizations due to s.lf-inflicted (intentional) injuries among persons aged 10 years
and older to:

+ no more than 50 per 100,000 persons (baseline: 62.5 per 100,000, Statewide
Planning and Research Cooperative System, 1991-93).

- Management Challenges

The challenges facing the Department in implementing its plan for action, Communities
Working Together, include:

Encouraging community action. The emphasis is on community action for addressing

public health problems. The Department needs to articulate ways of developing and
maintaining "buyin" from local health jurisdictions and of supporting their efforts to

te Communities Working Together, 39, 36.

59


data from FY 1994 and 1995, and estimated achievements in FY 1996 through 1999.7!
The DASA’s efforts thus far help position them to respond effectively to this charge.

Lessons Learned

Both consulting firms engaged by DASA to assist in the development of the TARGET
system and in performance accountability plan produced internal reports outlining lessons
learned from their efforts, which DASA officials shared with us. The DASA officials also
shared with us the central lessons they have learned from these combined initiatives.

The key ingredients in all DASA’s initiatives which led to success were leadership,
commitment, and communication. The DASA’s director commented that the initial
difficulty in the development of the TARGET system demonstrated to him the
importance of his personal, detailed attention to the project as its sponsor, to ensure
decisions consistent with the vision of the project were being made and to help "cut
through red tape" when obstacles presented themselves. The use of a variety of different
committees, representing different constituencies, ensured involvement from key parties
and widened the basis of support for DASA’s various initiatives. Each initiative
undertaken, as part of a whole effort to increase program accountability, supported each
other initiative. Commitment occurred because of an increasing consensus of the need
and urgency of developing good systems before a less favorable one was imposed from
outside, or before the program suffered because it could not demonstrate what it
believed to be true. Communication between leadership, staff, consultants, and
stakeholders was critical to ensuring common expectations, alleviating fears, and finding
optimal solutions.

Continuing threats present themselves, but the structure of leadership, commitment and
communication helps allow responses to those threats. For example, DASA faced a
legislative challenge to reduce its research investment, but was able to demonstrate the
value of its investment and rely on support from widely different stakeholders to help
make its case.

For Further Information
- Kenneth D. Stark, Director

Division of Alcohol and Substance Abuse
Washington State Department of Social and Health Services

Olympia, WA 98504-5330

a State of Washington, Office of Financial Management, Operating Budget Instructions, 1997-99 Biennium, May
1996, B-96-01.

113


+ Health Care (ie., inpatient services, specialized, immunization, nutritional services,
primary care physician);

+ Juvenile Justice (i.e., court counseling, diversion services, multi-purpose group
home, wilderness camp, adjudication, detention, training school);

+ Community (i.¢., sports/organized recreation, church, scouting, full-time
employment); and,

+ School/Vocational (e.g. Individual Family Service Plan-birth to 5 years,
developmentally delayed, alternative school, vocational training, vocational
counseling).

Management Challenges

Managing the data, The DMHDDSAS believes it will need to contract for data entry or
provide data entry screens at the local level and specific file layouts so that the data can
be submitted locally to the DMHDDSAS, especially once all 41 programs begin
submitting their client outcome assessment data. Presently, data receipt from the three
pilot programs has a time lag of about 4 months (October to January). It will be a
challenge for the DMHDDSAS to obtain and/or redirect enough resources to process
and manage the data once all 41 programs are up and running.

Understanding the data. Analyzing the client outcome assessment data, determining
what the data means and understanding its limitations will be time consuming and a
challenge for the DMHDDSAS. In addition, while the quality improvement initiative has
the support of policymakers, sometimes the level of sophistication needed to understand
the data and, more importantly, its limitations, is not what it should be. The
DMHDDSAS plans tc develop an education campaign to ensure that the data is properly
understood.

Stakeholder buyin. Area program directors are troubled about the State’s use of the
data. They have some concerns that the data may be used for punitive purposes to
defund non-performers. The DMHDDSAS did not involve the area program directors
from the very beginning of the survey instrument design process. They believe that if
they had done so, while it would have taken more time to develop the survey and they
might have had to compromise on some of the selected data elements, they would have
achieved greater buyin from these stakeholders.

Performance-based budgeting. The DMHDDSAS is currently submitting both a line item
budget and a performance-based budget to the State legislature and Budget Office. The
performance-based budget submitted was based on current data that could be translated
into an outcome. It is the line item budget, however, that is still being used by the
Budget Office and legislature for funding appropriations. Once all area programs are
submitting the client outcome assessment data and the DMHDDSAS has analyzed the
data, performance-based budgeting will be used.

77


TREATMENT PROVIDERS
Nature, Extent, and Uses of Performance Measures

The Illinois DASA has been working on the development of an outcome-based
performance measurement survey for ATOD treatment providers. The survey will be
client focused and used to help assess the impact of the treatment provider’s program on
ATOD abusers. Scheduled to begin in July 1997, providers will administer surveys to
clients upon admission to a treatment site and when the client leaves a treatment site.

The primary user of the survey data is the State. Providers will transmit the raw survey
data to DASA who will use it, along with other quantitative and qualitative data already
being collected. This will give the State global oversight of the myriad services DASA
clients receive and will be able to better assess the impact of treatment providers’
programs.

The State is dependig on providers to administer the survey, collect and report back the
data in an unbiased manner. Providers will use a standard script or questionnaire. The
DASA plans to conduct random field audits to check the accuracy and reliability of the
data. In addition, DASA has hired a private consultant to conduct random surveys of
clients 6 months after they have been treated. As a means of facilitating these client
surveys, provider contracts include the stipulation that they must participate in this study.

Process for Developing the Performance Systems

The impetus for this new assessment program was the passage of State legislation (about
2 years ago) that called for the State’s departments to conduct a yearly assessment of the
impact of their programs. In an era of cost cutting, government downsizing and limited
budgets, legislators wanted more definitive information about the effectiveness of State
programs to help them decide where funds should be allocated.

Currently, there are about 200 corporations offering 600 ATOD treatment programs
throughout Illinois. Most have been under State contract for many years, some since the
1950s. State officials met with 400 providers, 5 focus groups and 75 organizations
(including advocacy groups) to both inform and solicit comment about the new outcome-
based performance measurement program, and the outcome measures and survey
instruments. The DASA’s performance measurements (data) were developed in-house
by DASA, though treatment providers ultimately agreed upon the script to be used for
the client interview survey.

However, treatment providers have not been as receptive to this new process as the State
would have liked. Providers have expressed concern with meeting standardized
performance measures. They believe that there will be factors beyond their control that
may negatively impact the success of their treatment program(s). And, they fear the
DASA may not take these factors into consideration when evaluating their performance.

29


will be an on-going process to be implemented and coordinated by these two committees
and is key to the successful implementation of the Five Point Plan.

Stakeholder buyin. The Five Point Plan calls upon Ohio’s 150 local health departments
to reassess their public health programs and focus on community needs rather than
programs and perceived needs. It also calls for performance standards that will hold
local health departments and providers accountable for meeting community needs.
Achieving stakeholder buyin for this philosophical change in the way of doing business
will be a challenge for the Department.

A legislative mandate will be necessary for the implementation of performance standards.
Lessons Learned

Total Quality Management. Incorporating the Total Quality Management philosophy
was critical in the development of the Five Point Plan. Steve Covey’s models of circles of

influence were used to focus the Department’s efforts (e.g. being creative, adaptive,
flexible and yet viable). The Department believes it is moving in the right direction as
local health departments are beginning to see the value in the Five Point Plan.

Conduct an environmental scan. Because there will always be opposition to change, the
first thing one must do when developing a results-based accountability system is to assess
what the future may hold. Secondly, one must determine of what value the effort or
results will have to stakeholders (e.g. executive and legislative branch, Department
directors and staff, providers, advocacy groups and clients). And finally, one must be
able to demonstrate effectiveness and efficiency.

For Further Information

Ron Elble, M.P.H., Chief
Division of Local Services

Ohio Department of Health

246 North High Street, 7th Floor
Columbus, OH 43266-0118

95


Lessons Learned

The Division identified several factors important to the progress that has been made in
implementing a performance measurement system:

Starting from "ground zero" in developing measures. The Division did not simply adopt

measures from academia or other sources and assume they would meet the expectations
of the key stakeholders.

Forming key alliances, In particular, the Division worked closely with, communicated
regularly with, and sought comment and advice from legislative stakeholders and their

agents.

Focus on target populations. The Division focused on the client populations they serve
directly, with the idea that once effectiveness is demonstrated with these populations, the

legislature might expand the Division’s responsibilities to encompass other groups.
eg) ig} P Ps iP group:

The balance and competition between client and provider interests. The full

participation of both sets of stakeholders was considered critical in forming a meaningful
set of measures.

Understanding the concept that knowledge often raises more questions than answers.

Data collected as part of the performance measurement system will create discussion,
debate, and the need for additional analysis. Instead of reducing the resources needed
for monitoring, it may well increase the resources needed as more and more questions
are raised about provider performance, client needs and experiences, and the validity of
data.

For Further Information

Contact information not available.

10


Federal Challenges

NO to national performance standards. The setting of outcome measures or standards
should remain at the State level. The Federal Government should not mandate national
performance goals or standards. They believe this would be a mistake as State statutes
differ and their local and county governments have varying degrees of autonomy.

Not all systems are equal. It has been suggested that a results-based outcome
measurement system can be selected through a review of those systems in use by the
biggest behavioral health organizations. The DMHDDSAS cautions the Federal
Government against taking this course. They believe that these big organizations "cherry
pick" their clients, selecting only those that can pay. Consequently, their client outcomes
reflect very favorably on their client care system. State public health facilities, on the
other hand, do not have the luxury of preselecting their clients relying largely on the
State for reimbursement.

Lessons Learned

Identify all stakeholders. The DMHDDSAS believes it is critical to identify all the
stakeholders and ensure that all are invited to the table to participate in the process.
The move to outcome-based monitoring was too radical a change -- a paradigm shift --
not to include all who would ultimately be affected. For this process to succeed everyone
must buyin to the philosophy and for this to occur, stakeholders must be included at the
beginning of the decisionmaking process.

Understand the terminology. Before starting the process, everyone at the table must
understand the terminology of results-based outcome measurement systems. All must
learn and come to the same understanding of the meaning of: (1) an economic outcome,
(2) a clinical outcome, and (3) a process outcome.

A facilitator/consultant should be able to explain the process and terminology in language

that a lay person (not an academic) can understand. In fact, a consultant may only be

needed for the first few days of the process once everyone has grasped the concept and
the meanings of the terminology.

Do not reinvent the wheel. Research what other States and private industry are doing in
the area of outcome measurement. Draft a proposed process based on your research
and then set this draft aside. Do not be wedded to your first effort, review and reassess.

Adequate resources. The DMHDDSAS has had to develop their quality improvement
process without extra funds. This process began in early 1992 and the DMHDDSAS is
just ending the pilot phase of its client outcome assessment survey instruments. This data
will be collected and analyzed in the near future. This process could have been much
farther along if the DMHDDSAS had been authorized funds to hire and dedicate staff to

this initiative.

78


OHIO

Early Start Initiative
Bureau of Early Intervention Services

Department of Health

Overview

The Ohio Department of Health’s Early Start Initiative "...provides funding for direct
services to infants and toddlers who are extremely vulnerable because their
environmental, family or health circumstances place them at very high risk for
developmental difficulties, abuse or neglect. [This] funding is intended to provide a
collaborative, integrated system of services and supports to help strengthen and preserve
families who may be experiencing problems before they become crises." The
Department has developed outcome-based performance measures to help as-«ss the
success of this initiative.

Nature, Extent and Uses of Performance Measures

As of January 1997, 30 of Ohio’s 88 counties are participating in the Early Start
Initiative. These participating counties received their first funding in January 1996 and
were expected to start serving children and families immediately. Participating counties
were asked to provide:

+ public awareness and ways to identify potentially eligible children;

+ assessment of children (including health and developmental status) as well as the
families concerns, priorities and resources;

* an interagency individualized family service plan;

+ family support;

«+ home visits;

+ training to staff from local service agencies;

+ public awareness and child find; and,

+ expansion, enhancement, or creation of services to meet the needs and build on
the strengths of children and families in the program.

Providers are required to submit data on the above mentioned services (e.g. type and
number of services provided, number of clients served) to the Department every 6
months. Some of the survey tools providers use to capture performance data are:

« The Nursing Child Assessment Satellite Training Feeding and Teaching Scales
(NCAST). This data collection tool provides information on parent-child
interaction during feeding and teaching interactions. Use of this tool requires 30
hours of training and certification to administer. Eleven Early Start counties use
this tool as of October 1996,


FLORIDA
Division of Alcohol, Drug Abuse and Mental Health

Department of Children and Families

Overview

The Division of Alcohol, Drug Abuse and Mental Health (ADM), Department of
Children and Families, is the first State health office to develop a performance
measurement system for budgeting purposes, according to the requirements of the
Government Performance and Accountability Act (The Act). The Act was passed by the
Florida legislature in 1994 and agencies are being phased in to the new system over a
period of 8 years. At the present time, the State legislature is reviewing the Division’s
proposed system which could result in minor or significant changes to the program.

Nature, Extent and Uses of Performance Measures

The Government Performance and Accountability Act requires agencies to submit
performance measures and standards in their budget submissions. In return, agencies
would be given flexibility for the reallocation of dollars among categories and programs.
The Act requires agencies to submit input measures, output measures, and outcome
measures.

The Act vests oversight authority in both the executive and the legislative branches.
According to the Act, evaluation of State programs would be undertaken by the Office of
Planning and Budgeting (OPB) in the Executive Office of the Governor. The Office of
Program Policy Analysis und Government Accountability (OPPAGA) is responsible for
conducting program evaluations and reporting to the OPB and legislature.

In the phase in schedule for the Act, the Department of Health and Rehabilitative
Services, Division of Alcohol, Drug Abuse and Mental Health was scheduled to begin
performance-based program budgeting i in Fiscal Year 1996-1997; that schedule slipped 1
year for reasons discussed below.’ The Division is now submitting its performance-based
budget request to the Legislative Budget Committee.

In July 1996, as the Division finalized its performance accountability system, it placed
parallel performance measurements in provider contracts and required providers to
report data on all measurements. The purpose of this approach was to ensure
accountability throughout the system for the same outcomes, and to provide a continuous
source of data for all ADM measures approved for use in performance-based budgeting.

"Since the passage of the Act, the Department of Health and Rehabilitative Services has been reorganized. The
Division of Alcohol, Drug Abuse and Mental Health now operates within the Department of Children and Families.


Outcome: —_ Improvement in health

Measures: (emergency room use, hospital inpatient admissions, hospital
inpatient days, outpatient health/clinic visits, psychiatric
hospitalization, psychiatric inpatient days--all comparisons between
incidence during treatment to the year prior to treatment)

Outcome: Reduced crime

Measures: (drug offenses, domestic violence, violent crimes, property crimes,
driving under the influence, other public order offenses--all
comparisons between incidence during treatment to the year prior to
treatment)

Outcome: — Improved Employment and Self Sufficiency

Measures: (levels of employment, earnings, levels of public assistar-e, stabilized
housing--with the exception of the last, where implementation of the
measure is in progress, all comparisons between admission and

discharge)
Management Challenges

Two unresolved issues remain on the horizon. For the moment, DASA has not
developed standards for its performance accountability system. This decision has been
very purposeful. The DASA believed that without baseline data and more experience
with the system, it was unwise to develop standards. Providers continue to be wary about
the development and use of standards.

Another factor affecting accountability measures is the use of managed care for Medicaid
recipients in the State. The increasing use of managed care arrangements has an impact
on data availability. The DASA officials also fear the impact of carveouts on
performance accountability, since it might reduce the incentive to make investments in
substance abuse treatment in order to recoup benefits outside the carveout.

Washington State is encouraging the development of performance measurement systems
within the State by requiring each agency to submit 6-year strategic plans as part of their
1997-99 biennium budget submissions. Among other things, strategic plans must include
the agency’s mission description, strategies, goals, objectives, timelines, performance
measures, and performance standards. A separate budget form for performance
measures has been developed, in which agencies must identify strategies and the
outcome, output, and efficiency measures which relate to each strategy, as well as actual

112


meeting, the Task Force determined that client outcomes will be measured in four
domains: (1) clinical status, (2) functional status, (3) life satisfaction and fulfillment, and

(4) safety and welfare.
Management Challenges

Stakeholder buyin. To date (after four meetings), some friction has been encountered.
One source of friction is providers’ concern that this process may be highly resource
intensive. Providers are unsettled about the extra costs they may have to bear for data
collection, new data systems, staff training and/or new hires. The other source of friction
has been the different perspectives of various stakeholders about the appropriate
outcomes to measure.

Consumer Quality Review Teams
Nature, Extent and Uses of Performance Measures

The Department of Mental Health is piloting three initiatives to develop consumer-
driven quality assessment data for its mental health systems. Face-to-face interviews with
randomly selected consumers, family members and providers will furnish information
about the availability, accessibility and appropriateness of mental health services; the
quality of these services; treatment results; and, consumer satisfaction. Consumer quality
review teams (CORTs) will conduct the interviews. The teams will consist of 50 percent
consumer representation, 25 percent family member representation and 25 percent
provider representation.

In June 1996, the State of Ohio’s Office of Consumer Services awarded grants in three

areas of the State. The -lliance for the Mentally Ill of Ohio and The Ohio Advocates

for Mental Health will provide training and technical assistance to the consumer quality
review teams. The pilot projects are under way in the following three areas:

+ A six-county area (including cities of Akron, Youngstown and Warren);

+ A region in western Ohio between Toledo and Cincinnati consisting of 18
predominantly rural counties; and,

+ Hamilton county-(urban Cincinnati area where efforts are focusing on services for
children).

The data collected from these interviews will be, for the most part, qualitative in nature.
It will be used by the Department of Mental Health for informational purposes and not
as a means of evaluating provider performance per se. Local mental health systems will
have primary access to the data so that they might assess their own system’s
performance.

82


WASHINGTON
Division of Alcohol and Substance Abuse

Department of Social and Health Services

Overview

The Division of Alcohol and Substance Abuse in Washington State has developed a
multi-tiered approach to performance measurement which began in the late 1980s, in
response to a variety of pressures for increased accountability. Among the critical
improvements made by the division were increases in data-base capacity to obtain
relevant data and focused studies assessing outcomes for target populations.

Nature, Extent and Uses of Performance Measures

The performance accountability system for treatment programs at Washington State’s
Division of Alcohol and Substance Abuse (DASA) is composed of several key parts.

A management information system was developed with input from a variety of
stakeholders to meet information needs of a variety of audiences. The new system,
entitled TARGET (Treatment and Assessment Report Generation Tool), is focused on
performance data. Providers are responsible in their contracts for entering data into the
TARGET system. DASA reviews data continually, identifies delinquent reporting, and
audits providers for compliance.

The DASA also developed a performance accountability plan, developed with input from
a variety of stakeholders to identify expected outcomes from treatment.

Finally, DASA supports a research agenda to explore performance, through the strategic
use and combination of TARGET data, data from other State agencies, and self reported
information from the community. This agenda includes the reporting of data from
TARGET; the matching of TARGET data to other State systems, including employment
and earnings data, Medicaid data, welfare information, corrections data, and birth
outcomes and costs; and the use of self reported data from program participants. Over
the course of the last several years, DASA has conducted a series of evaluations, many of
them focused on particular populations such as adolescents, pregnant women and
substance abusers with mental illness, to assess outcomes. Information from TARGET
and these evaluations have been used to shift dollars into program and service categories
which would be a more effective use of funds.

As part of the division’s research and performance measurement efforts, in 1993,

Washington State published its first edition of "Tobacco, Alcohol and Other Drug Abuse
Trends in Washington State," building on the health goals and objectives set out in

108


10.

11.

Determine whether existing laws/rules need to be modified to develop new
performance standards.

Department staff assigned, work schedule developed, key participants recruited
(e.g. representatives from local health departments).

Work group of State Department of Health unit staff convened.

Review existing standards and national approaches (e.g. Federal, other States,
professional organizations) to the development and implementation of
performance standards.

Initiate/oversee process of developing performance standards by:

a. developing a mechanism for incorporating core public health functions into
performance standards;

b. determining the public health practices that fall under the three over-
arching core public health functions;

cC selecting those public health practices to be incorporated into the
Department’s performance standards; and,

d. developing measurable performance standards which incorporate existing
and new public health practices.

Develop a process/procedure for determining/measuring local health department
compliance with performance standards.

Develop enforcement protocol.
Develop self-assessment standards for the Ohio Department of Health.

Determine existing authority; draft legislative changes required; draft proposed

- policies; legislation and/or rules specifying State and local health departments’

responsibilities regarding the implementation of the new performance standards.
Implement new performance standards.
Develop and implement a plan for interpreting/communicating needs and benefits

of performance standards. This is an on-going activity to be undertaken by State
and local health department leadership.

The workgroup that will be in charge of developing these performance standards is to
"\.produce a written document describing topical areas for Performance Standards, how

93


mental health and education are two priority action areas as a direct result of
communities having articulated these as areas of great concern and need.

Content-related factors. The Priorities Committee and the Department were guided by
the principles of emphasizing a community orientation and focusing on behaviors rather
than specific diseases. These principles guided the framework of the report and its
recommendations. Finally, while the report emphasizes community action in priority
areas, it also addresses the importance of maintaining the public health infrastructure
across the State. One section of the report focuses on this infrastructure and its
importance to the public’s general health and to successful implementation of specific
actions as called for in other sections of the report.

For Further Information

Michael Medvesky, Director
Public Health Information Group
State of New York

Department of Health

Corning Tower

Empire State Plaza

Albany, New York 12237

61


«The process needed to pay attention to differences in priorities that would
inevitably occur among communities and agencies.

+ The review process needed to be concluded and a plan prepared within one year,
in accordance with the Commissioner’s priorities.

Leaming from Key State and Local Government Officials

New York health officials were surveyed by mail early in the process as an initial needs
assessment. They ranked the priority of the Healthy People 2000 goals on a scale of one
to five. Criteria included the size of the problem, the seriousness of the problem, the
effectiveness of known interventions, the relevance of the goal to public health, and the
degree of community consensus about the importance of the goal. This exercise served
as a useful springboard for subsequent discussions. Most program staff identified goals
and priorities in light of their own responsibilities. For example, for individuals working
in State laboratories multiple drug-resistant organisms were a key public health priority;
for those in community health, low birth weight babies were a significant public health
issue.

Other New York State agencies, such as Education, Social Services, and Mental Health,
met with the Committee to discuss their efforts and the cross-cutting issues.

Other States were surveyed to identify those who had undertaken similar efforts.

Officials from several States briefed the Committee who was interested in learning about
others’ experiences concerning such issues as: (1) the process they had used in
developing Statewide health goals and performance measures, (2) whether they had
focused on diseases or on risky behaviors, and (3) whether the processes had been driven
largely by the State departments of health or by local communities.

Seeking Input from Communities Across the State

The Committee sponsored a series of six, one-day regional workshops across the State
during May 1996. Participants identified serious health problems in their communities,
causes of those problems, and effective interventions. An estimated 1,400 participants
attended: local health care providers, government officials, staff from: community-based
organizations, educators, advocates, and business and labor representatives. The
comments and input from these local communities largely formed the final report,
Communities Working Together for a Healthier New York.

Publicity

In organizing the workshops, which were the backbone of the review and priority-setting
process within the States, the Department publicized the workshops widely through:
direct mailings to individuals and organizations, an internet web site, newspaper and
magazine articles, press releases, and use of an existing toll free phone number. In
combing through mailing lists provided by numerous agencies and organizations, the

57


Focus on mutually agreed-upon results as the measure of success;

Implement a community-based family-focused, and prevention-oriented services
strategy;

Eliminate the policy and system barriers that interfere with achieving results;
Invest local, State and Federal funds in a cost-effective service strategy; and,

Authorize new governance structures with the authority and responsibility needed
to make improvements for children and families.

This interim Policy Council also challenged communities to develop prevention-minded
services, new approaches to systems change, creative financing, collaborative planning
and local decisionmaking.

In response to the interim Policy Council’s recommendations, the Governor ~uccessfully
sponsored a legislative bill that established new forms of State and local governance
needed to implement the new framework. This legislative bill was: The Georgia Policy
Council for Children and Families Act of 1995 (Senate Bill 256). This bill authorized the:

»

Georgia Policy Council for Children and Families: A permanent, 19-member panel,
composed of leaders in business, child and family advocacy, local government and religion
together with State agency directors from the Departments of Human Resources,
Education, Medical Assistance, Children and Youth Services and the Office of Planning
and Budget. The Policy Council serves as the State’s focal point for planning and results-
based accountability for children and families and for supporting the development of
community partnerships.

The Policy Council is responsible for: (1) defining the core results to be achieved for
children and farcilies, (2) developing a comprehensive State plan, (3) evaluating
implementation of the State plan, (4) initiating needed changes in State or Federal laws or
regulations, (5) reviewing and approving strategic plans of community partnerships and
agencies, and (6) providing leadership training and other training and technical assistance
to communities and agencies.

Local Community Partnerships: Community level partnerships designated by local
government and the Policy Council to serve as the focal point at the community level for
planning and improving results for children and families. These partnerships are
responsible for: (1) achieving a core set of results defined by the community partnership
and the Policy Council; (2) developing a comprehensive plan for public and private
agencies; (3) coordinating, evaluating and providing assistance in implementing and
carrying out the comprehensive plan; and, (4) contracting with public and private agencies
to provide programs and services for children and families.

19


T qi seoTazeg ueuMY pue YATeeH JoZ dryszoujzeg eyseaqeN 2O7TOx»

ie,
aoRlg Saxe], Seto,
“Yo on
SISATIVNV “hag
Byer,
%

ABHeNS
wiaysdg jo maraay

mom iene wiesZoid /adrazas
k HeNDe 3724S Jo E90] Aq AuMUTUIO> payipowl
UI Ae pag Pure waIshs SHH ° pue
adojaaa weId02q /adiapas/Wwwo0D 103 roi pad lojaaag
P 52. vn son payuaursjdury pue
adojaaaq sai8aje1}
aoUBULIOJ Ig — Pecojeacd] Sardaieis wba werfonro, SHNOD.LNO
“SEO *: “
waeks | ADELVULSAAGLLSAS | rawr epmosa
%
% padojaaaq

SYOLVSIGNI

Mu
Gag voNeTvaPl suowwoar
sul .

ynoysnow paynsuod aq jf1M pue wayshg Arnqeinoz.y asuewsoyzag ayy ui quiod ysea ye ajo2 ayqenjea e aaey siapjoyayeys
waisds Apiqemunossy asueuopay ay) jNoYBnoN} paraptsuod are UOISIA Pur UOISSIAY By,

~WALSAS ALITIAVLNNODOV JONVNUOAUNAd

48


A series of initial pilot programs to enter the cycle have been identified and should be
underway by early 1997.

Among the most significant future challenges:
Improved information systems. Data is currently available for about half of the

indicators identified to date. The development of improved data systems will take both
money and time.

Sufficient staff for analysis. The commitment to, and investment in, analysis is critical for
the proper interpretation of data and the use of data for program improvement and

reform.

Proper use of data on indicators. The developers of the performance accountability
system do not intend that indicators be viewed in isolation. Factors external to Health
and Human Services interventions often impact the indicators. They argue that, in order
to interpret such data, one must also review data from other sources and assess the
impact of factors outside the system upon the indicators. The system is designed to
protect against the isolated use of indicator data by including performance measures and
context analysis; but, as one representative said to us, "We know there may be political
considerations which impact this system."

Lessons Learned

The representatives we spoke to about the development of the performance
accountability system were mixed in their assessment of the short timeframe in which
they had to complete their work. “I can’t tell you what a tornado this felt like," said one
official, Nonetheless, the timeframe to which so much else was tied was never
renegotiated by any party, and no motion was made to attempt a renegotiation. The
timeframe might have helped to concentrate the attention of the work group and work
team members, but combined with the reorganization being planned at the same time, it
caused problems for team members in a number of different ways. Team members were
often in too many meetings. Plans were in flux because of parallel processes and
developments affecting performance accountability occurring in other reorganization
teams. And team-members still struggle with a persistent fear that some of the buyin to
the performance accountability system was superficial and that key stakeholders were too
distracted by other reorganization issues to pay close attention to, and reflect thoughtfully
on, their efforts.

Nonetheless, the participants in the performance accountability work group believe that
they were remarkably successful in forging consensus among their members. They point
to several factors contributing to their success: (1) a strong facilitator; (2) a shared
(among a third or so) grounding of the work group members in facilitative leadership;
(3) implementation of facilitative leadership principles, [such as “going slow to go fast",
agreeing on ground rules, balancing the building of relationships with content]; (4) stable

52


OHIO

Department of Mental Health

Overview

The Department of Mental Health is involved in two assessment efforts using outcome-
based performance measures. The first effort involves a Statewide development of
outcome measures to assess the effectiveness of its mental health programs. The second
effort, is a demonstration project to develop consumer-driven quality assessment data to
help evaluate the success of Ohio’s mental health systems.

Statewide Measures
Nature, Extent and Uses of Performance Measures

Ohio’s Department of Mental Health is in the very beginning stages of developing client
and system outcome performance measures for its mental health programs. As of
February 1997, outcome measures had not yet been determined.

Process for Developing the Performance Systems

The Health Care Financing Administration’s approval of Ohio’s Section 1115 managed
care demonstration was one impetus for this effort. Concerned about ensuring quality
care in managed care programs the Program Evaluation and Research Office in Ohio’s
Department of Mental Health recommended to its director that client and system
outcome measures be developed.

Dr. Michael F. Hogan, Director of the Ohio Department of Mental Health, formed a
Blue Ribbon Task Force charged with developing client and system outcome measures
for the Department’s mental health programs. This 42 member Task Force is composed
of: State and local government officials, consumers and family members, advocates for
mental health, mental health care providers, case managers-and-local mental health
board officials.

Formed in September 1996, the Task Force has until October 1997 to complete its
mission. Scheduled to meet monthly, the Task Force has met four times as of February
1997. The first meeting’s agenda focused on the importance of evaluating systems of
care and on the mission of the Task Force. This meeting was presided over by Dr.
Hogan. In the second meeting, members developed a set of values and rules and
regulations of conduct to which all members are to observe. The third meeting involved
a brain storming session at which all members were asked to first suggest client outcomes
and then system performance outcomes. As expected, most (about 80%) of all
suggestions related to system outputs rather than client outcomes. At its most recent

81


Lessons Learned

Among the lessons learned by Department of Health officials as developing performance
measurement systems are:

Avoid overbuilding the system. In its first attempt at an overall health-oriented strategic
plan, officials believe they made the mistake of "including everything." The initial plan
was deemed too massive for actual tracking and follow up purposes; subsequent
iterations of the plan were more streamlined. Officials now see a need to expand the

number of measures.

Process is still important. Despite the emphasis on results, officials believe that
measurement systems are not well enough developed to substitute entirely for process
measurements. "Process" is still important for accountability purposes in many
policymakers’ minds. "Performance" means different things to different people.

Indicators should be -eviewed continuously. Officials mentioned that they recently
deleted a measurement, "access to care," because appropriate data was not available at
the local level. In 1994, officials made a series of changes in indicators, combining some,
adding a dental indicator, and adding administrative outcome indicators to the list of
health status indicators (e.g. client satisfaction, employee productivity and retention).
Other changes were made in 1995 and 1996.

For Further Informaticn
Meade Grigg, Director
Office of Planning, Evaluation and Data Analysis

1317 Winewood Boulevard
Tallahassee, Florida 32399-0700

14


Process for Developing the Performance Measures

A new Commissioner was appointed to head the New York State Department of Health
in mid-1995. She immediately sought a review of the Department’s health goals and
priorities with a view to developing a comprehensive plan for action.

Building on a model she had implemented in another State, the Commissioner wanted
the review process to be public, community focused, holistic, and one that would identify
key areas of opportunity for the Department in the years ahead. The effort was to be
grounded in the Department’s ongoing emphasis on using the Healthy People 2000
objectives for its individual program areas, particularly programs supported with
Preventive Health and Health Services block grant funds, as required by the Federal
Department of Health and Human Services.

The review and priority-setting process included several major steps:
Broadening Support at the Top

Early on, the Department involved the State’s Public Health Council, a statutory body
that advises the Commissioner and performs certain regulatory functions, such as
approving certificates of need. The Council became a key partner for the Department.
It guided the overall process of establishing goals, objectives and indicators for high
priority public health needs.

The Council, in turn, created a Public Health Priorities Committee to focus exclusively on
directing the effort and producing a plan for action. Its 19 members represented State
and local government, academia, business, unions, providers, and consumers. The
Committee relied on the Department of Health to provide staffing and other services to

support the task.
Clarifying Purpose, Scope, and Ground Rules

The Committee established a series of understandings about the nature of its task. They
included:

+ The foundation of the new plan would be the national Healthy People 2000
objectives. But these would be reexamined and possibly modified through the
review process.

+ The planning and priority setting needed to cross the organizational boundaries of
State agencies.

+ The process needed to establish approaches for examining the State’s progress
towards meeting its objectives.

56


A companion report, Results-Based Systems For Public Health Programs, Volume 1:
Lessons From State Initiatives, OEI-05-96-00260, discusses common experiences of the
States we selected in developing and implementing results-based systems, and draws on
these experiences to identify challenges facing States and the Federal Government as
they proceed with these initiatives.

METHODOLOGY

This report, and its companion volume, are based on a review of results-based initiatives
in the preventive health, maternal and child health, substance abuse and mental health
programs of 11 States. We conducted onsite discussions in seven of these States:
Florida, Illinois, Massachusetts, Nebraska, New York, North Carolina and Washington.
We interviewed officials by telephone in four other States: Georgia, Minnesota, Ohio
and Oregon. Two other States included in our first round of State contacts, Colorado
and Texas, were dropped from our interviews due to resource constraints.

We initially wrote to all officials in all the States mentioned above and requested
available documents on their efforts. These documents served to confirm the presence of
reported efforts and to form the basis for additional discussions with State officials. As
we interviewed State officials onsite and by telephone, we often received or requested
additional documentation which we then reviewed as well.

The criteria we used in selecting the State initiatives described here included:

(1) suggestions from government and non-government experts, researchers, a~d analysts
identifying States with positive experiences in developing and implementing results-based
systems; (2) our own review of documents associated with individual State initiatives;

(3) geographic representation and program balance to reflect the areas of maternal and
child health, substance abuse, mental health, and preventive health; and, (4) our own
judgments about whether particular initiatives were sufficiently different from, and less
well known than, others.

We developed these summaries based on our interviews and visits with State officials.
To ensure accuracy, we shared our summaries with our key contacts in the States for
review and comment. All 11 States responded and their comments have been
incorporated. into this report.

We conducted this study in accordance with the Quality Standards for Inspections issued
by the President’s Council on Integrity and Efficiency.

Definition of Terms

As we reviewed the available literature on this subject, designed this study and conducted
these interviews, we became increasingly sensitive to the importance of consistent, clear
terms in describing efforts in these areas. Terms and definitions used for the purposes of
this report and its companion volume include the following:


child mental health services, adult mental health services, child developmental disabilities
services and adult developmental disabilities services.

These survey instruments will furnish identification and descriptive data for program
performance and client outcome measures. Some examples of the type of data to be
collected (depending on the age of the client and the type of services required) are:
client record number, Medicaid eligibility, hospital admissions and care days, current
substance abuse use, academic enrollment and GPA, prior arrest record, client housing
status, and services needed and received.

Data will be collected by the provider on a sample of their clients (selected by the
DMHDDSAS) and subsequently furnished to the DMHDDSAS. These sample clients
are given an initial assessment, status surveys (every 6 months for almost all client types)
and a final outcome survey upon discharge from the provider’s program. These survey
instruments are easy to complete and color coded to distinguish between both client type
and an initial assessment versus a status or final assessment.

The DMHDDSAS expects to have benchmark data in 1998-1999. It believes it will take
that long to collect and properly and fairly analyze the client outcome assessment data.

Continuous Improvement

The DMHDDSAS plans to create liaison positions to help coordinate working
relationships where by the 41 area programs can seek help and feedback frc.a the State
to improve their programs. By July 1997, the DMHDDSAS hopes to have about eight
division staff persons solely dedicated to this effort. Data analysis of the client outcome
assessment survey data and the consumer satisfaction survey data will emphasize the
continuous improvement of all services throughout the State regardless of the current
level of functioning. The DMHDDSAS plans to use the data collected from these survey
instruments as a tool to help them identify those program providers with any deficiencies
or problems with services or consumer satisfaction. The DMHDDSAS does not plan to
use this data in a punitive way, it will not "play gotcha," but will work with the provider to
help build a "self-correcting" quality improvement plan. Down the road (about 5 years) it
is possible that the data may be used for policy purposes; to defend policy or redirect
efforts or decrease or increase funding.

Process for Establishing Performance Measures

Phase I

The impetus for this undertaking was the Mental Health Study Commission’s quality of
services initiative, which began in 1991. The Mental Health Study Commission, a
bipartisan commission created in the 1970's, studied issues and made recommendations

to the legislature and Governor concerning mental health, developmental disabilities and
substance abuse services provided to the public.

73


+ Youth who violate an alcohol law should be treated differently from adults who
violate the same law.
+ Penalties for alcoho! law violations are lenient.

Data from this survey is used to determine such outcomes as an increase or decrease in:

+ Pro-alcohol attitudes among adults or youth.

+ Beliefs about appropriate consequences for underage alcohol use.

+ Acceptability of underage alcohol use.

« Beliefs regarding individual and community responsibility for preventing underage
alcohol use.

This information would indeed have relevance for prevention providers, community
leaders and other policymakers alike.

Management Challenges

The State faces the following primary challenges in using performance measures for its
prevention providers.

Measurement philosophy. The concept of outcome performance measurement is new for
almost all prevention providers. And, the learning curve involved in understanding the
system takes time.

Provider understaffing. Currently, most of Illinois’ prevention providers are not yet ready
to produce outcomes. Many community prevention providers are staffed by just 1 or 2
people, who usually do not have a degree in the prevention area and must learn the job
more or less on their own. Also, most have not been formally educated in evaluative
techniques or sampling methodologies. Further complicating the readiness of prevention
providers to produce results is a 25 percent staff turnover rate.

For Further Information:
DASA Prevention Division

100 ‘West Randolph, Suite 5-600-
Chicago, IL 60601

28


of the DEHNR to work with local representatives to determine if a problem really does
exist and, if so, to develop a corrective action plan.

The DEHNR further cautions that ranking performance can be misleading as one may
put greater confidence in the ability of one data to discriminate among counties then they
deserve. For example, consider a county/local health department ranked 23rd out of 100
versus one that is ranked 26th: the difference between these entities’ performance,
depending on their confounders/variables, may be negligible and the county/local health
department ranked 26th, upon further analysis, may actually be the better performer.

Lessons Learned

Do not reinvent the wheel. For those States just starting to develop a results-based
accountability system, the DEHNR advises that they first review as many models of such
systems as they can find. Also, invite another State(s) to talk about the pros and cons
they experienced in the development of their own system(s). This will help expedite the
development process, decrease the frustration that inevitably comes with this effort, and
may help in avoiding some of the pitfalls another State experienced.

Identify the players. Make sure you have the right people around the discussion table.
Know to whom they are accountable and for what they are accountable. Members who

are the decisionmakers help speed up the development process.

Appoint a mentor. When Accountability Task Force I issued its report in January 1994,
there was no one to mentor this evolving accountability system. Consequently,
development of the accountability system stagnated until it was resurrected through the
interest of the Deputy State Health Director in 1996. To help ensure the viability of this
new effort, the DEHNR intends to institutionalize it by creating a "support" position
within the State Health Director’s Office. This individual will have the day-to-day
responsibility for overseeing the accountability effort.

For Further Information

Dr. Kevin Ryan
- North Carolina Department-of Environment,
Health and Natural Resources
1330 Saint Mary’s Street
Raleigh, NC 27605

71


By 1996, much of the infrastructure necessary to support the initiative had already been
developed: a single point of entry into the system for care, common assessment tools,
cross-training programs for staff, and a central registry for data.

During 1996, county and State officials worked intensively to refine the proposal. A
State-level Department team (composed of about ten members) met weekly; State and
county partners met monthly, and intensive staff work supported the development of the
proposal throughout the year. The proposal has been under review by the Federal
Government since late summer 1996.

Sample Performance Measures

The initiative identifies four operational goals, each with multiple indicators. The
indicators focus on outcomes for the Monroe County community and for the target
population served directly by the county health department. For example, one of the
goals is "healthy births." Examples of shared indicators for both community and target
populations for “healthy births" are the following:’*

+ — low birth weight births
+ early entry into prenatal care
+ infant mortality

Examples of indicators focused on the target population, defined as: families with
pregnant women at moderate to high risk for poor pregnancy outcomes based on
influence of environmental, economic, or educational factors; psychosocial, mental health,
or developmental factors; risk behaviors (such as substance use, violence, sexual
behavior); or, health issues (access to care, compliance, nutrition, previous fetal/infant
death, premature birth) are:

* pregnant women entering WIC in the first trimester

* pregnant women who delay subsequent pregnancy for at least 1 year
* pregnant women who smoke

* women who report using family planning

* consumers satisfied with Monroe County Health Department Services

Management Challenges

The most immediate challenge facing the Department of Health is securing approval
from the Federal Government for the initiative. Subsequently, it anticipates myriad
challenges as the initiative is implemented. For example, performance monitoring, both
through review of written reports and onsite visits, will be conducted in an integrated
fashion. Department staff view the project as a pilot effort from which they hope to

5 New York State Department of Health, “Monroe County Child and Family Health Grant, Goals and

Indicators” internal document, October 30, 1996.

64


(1) Magnitude of the problem - How prevalent is the problem? How much of the
State budget is affected?

(2) Seriousness of the consequences of the problem - What are the consequences of
not correcting the problem? Would other problems be reduced by correcting the
problem?

(3) Feasibility of correcting the problem - Can the problem be solved with existing
knowledge, technology and resources?

Each Task Force member rated the 40 benchmarks using the above criteria. These
ratings were then aggregated and discussed until a consensus was reached on a
preliminary list no more than five benchmarks for each result.

Presenting the preliminary list for community review.

The preliminary list of benchmarks was reviewed by the directors of the Georgia 2000
communities and with participants at the Family Connection Summer Institute. Five
Community Partnerships conducted focus groups to review the proposed benchmarks
also using the critical factors analysis methodology.

Revising the benchmarks.

The Results Accountability Task Force met to revise the proposed benchmarks to take
into consideration community comments, data collection issues and the age group
affected. Ultimately, 26 benchmarks were selected.

Results-based budgeting

The Budget Accountability and Planning Act of 1993 (Senate Bill 335) provides for
outcome-based budgeting, which is defined "as a process by which selected agencies will
be provided more expenditure flexibility when they agree to systematically assess progress
toward outcome measures." Upon completion of its first charge, benchmarking, the
Results Accountability Task Force focused its energies on what a results-based
accountability system would look like.

The Results Accountability Task Force reviewed various models of outcome-based
budgeting, both in the private sector and that occurring in government systems across the
country. The framework they recommended links results to budgets. The budgets are
developed according to the likelihood that they will yield improved results for children
and their families. Results-based budgeting has yet to be implemented in the State.

21


MASSACHUSETTS

Department of Mental Health

Overview

The Massachusetts Department of Mental Health contracts with local provider agencies
to deliver a majority of its service programs. Its Area Offices, with their citizen advisory
boards, both negotiate with, and monitor, inpatient and community-based services
providers. The Department uses a system of results-oriented performance measures with
its local providers of services.®

Nature, Extent, and Uses of Performance Measures

Since 1992, the Department has been engaged in an intensive process to develop and
implement standards and performance indicators for the services provided by its
contractors.

The process is complete for emergency, residential, respite, day programs such as
clubhouses, rehabilitation, partial hospitalization, and some inpatient services for adults,
children, and adolescents. Standards and indicators for other inpatient and outpatient
services are currently being developed.

The overall objective is to improve providers’ performance through a process that
articulates expectations and identifies standards for each type of service that apply to all
contractors. The process also involves developing performance indicators (of both
process and outcome) and, as appropriate, clinical indicators.

These standards and their performance indicators are incorporated into the Requests-for-
Proposals made available by the Department to potential vendors. Eventually, the
standards and indicators are incorporated into the contracts the Department negotiates
with the service providers. The Area Offices work with their providers to establish
specific targets that become terms of the provider’s contract.

The Department expects that, ultimately, these data will be useful in helping the
providers monitor their own performance. The Area Offices, and their citizen advisory
boards, are beginning to use these data in monitoring contractor performance, e.g.
identifying gaps in service and areas of less than satisfactory performance and in
reviewing and awarding contracts. Although the Area Offices aggregate the data from
providers in their areas, they do not routinely provide the data to the Department’s
central office.

6 , ps oh Gi

The Department also has Department-wide objectives, measures, and outputs for which it prepares quarterly

reports to the Legislature, It also has defined objectives and indicators for its annual community mental health services block
grant plan and progress report for the U.S. Department of Health and Human Services.

38


Department identified more than 35,000 individuals who were invited to participate.
Local workshop coordinators also sent invitations. An estimated 1,400 individuals
actually participated in the one day sessions.

Professional Facilitation

The Department arranged with a local university to facilitate the regional workshops.
The basis for each workshop were small group discussions. These groups were pre-
arranged by Department staff to ensure a mix of backgrounds and interests among
participants. The discussion, led by the facilitators, focused on identifying: (1) the ten
most significant adverse health outcomes in the communities, (2) the ten most significant
risk factors contributing to these adverse health outcomes, (3) effective interventions now
being applied in their communities, and (4) new interventions that might be effective.

Synthesizing the Input

The Department analyzed and summarized the input received from the regional
workshops and from other State and local government agencies. Staff reviewed this
information in the context of the Healthy People 2000 goals and of the availability and
validity of data available to monitor progress in improving health outcomes. This process
culminated in the identification of the 12 "priority areas of opportunity for improving
community health." For each of these areas, Department staff developed specific,
quantifiable objectives to allow for monitoring progress. The rationale for the specific
target levels for the objectives in each priority area were published in a separate
document prepared by the Department and the Priorities Committee.

Decisionmaking at the Top

The Public Health Priorities Committee directed the decisionmaking about the priority
areas, and the objectives and target levels for each. It actively guided preparation of the
report, Communities Working Together, which was submitted to the Commissioner in
September 1996. The Commissioner distributed the report widely this fall, indicating that
her “personal goal is that this landmark report be used as a blueprint for public health
action in communities throughout the state.""?

Beginning the Implementation

The Department is now considering its next steps for translating the document into
community action. Many communities are seeking guidance and technical assistance
from the Department about how to use the report in their local efforts.

10 New York State Department of Health, "Rationale for the Selection of the Target Levels for the Year 2006

Objectives for New York State,” internal document.

" Barbara A. DeBuono, M.D., M.P.H., Commissioner of Health, State of New York Department of Health, letter
to colleagues, November 1996.

58


ILLINOIS

24


Department of Health and Human Services’ model of health promotion and disease
prevention objectives to local public health issues.

The Department expects the performance standards to evolve from entities capacity to
perform the core public health functions at desired levels. They are intended to "serve as
benchmarks for determining what [State and local health departments] are expected to
do or must do as part of their day-to-day operations to promote health and prevent
disease, injury and premature death." The Department believes that the performance
standard initiative will:

+ provide for a more equitable system of accreditation, certification and subsidy
determination, while assuring State and local health department accountability to
the public.

+ ensure that the State maintains a minimum public health infrastructure with
minimum quality standards.

+ enhance the ability of the State’s public health system to ensure improved health
outcomes for Ohioans.

The State will be the primary users of the data. Local health departments may be
allowed access to their own data and aggregate data.

Process for Establishing Performance Measures

Self Assessment

The Ohio Department of Health (ODH) conducted a self-evaluation that ultimately led
to the reorganization of its operations and the initiation of this planning process. The
impetus for the assessment was the changing health care environment stimulated by the

following trends:

+ The belief that the health care system is in need of reform has created a demand
for greater cost savings, coordination of services and outcome data;

+ The pressure on Federal, State and local government programs to prove their
effectiveness;

+ The formation of partnerships and collaborative efforts among agencies that
increases communication and coordinates services for greater efficiency;

+ New and anticipated challenges in environmental and health threats; and,

+ The advent of nationally accepted public health functions and models that assist
State and local health departments handling of today’s public health problems.

91


the Minnesota Public Health Goals document. Currently, the State has data for between
50 and 80 percent of the goals. The data systems needed to develop and manage this
effort will be an ongoing challenge for the State. At this point, the Department of
Health is working on strengthening the data sets they already have in place (e.g. looking
at more disease registration and specific morbidities). The Department of Health is also
reviewing what data is available and the resources needed to acquire it.

Timeliness of data, With a lag time of 1 to 2 years (data due in January 1997
represented data for January 1995), timeliness of the data is a great challenge to the
State. Providers submit their data to the State and then to the counties. Once the State
receives the data, it must then: (1) code the data, (2) put the data in a uniform
standard, (3) send the data back to the provider to verify its correctness, (4) make any
coding changes to the data based on provider comments, (5) input the data, and
ultimately (6) analyze the data.

Lessons Learned

The Department of Health identified several aspects of the process to date that have
been particularly relevant in developing results-based public health goals.

The State received less stakeholder feedback on the proposed public health goals than
expected. Stakeholder’s lack of response may have been the result of a relatively passive

process whereby stakeholder comments were solicited after the goals had been chosen by
the State. Another reason may be that some stakeholders may not have see~ the need
to comment on goals that they are not mandated to meet. In hope of achieving greater
stakeholder "ownership" of the public health goals, the State plans to be more proactive
in the future and include stakeholders at the beginning of the process when it reassesses
the public health goals in 1998-1999.

Resources (funding for staff, hardware and software) and time must be committed for
the research, development and maintenance of a management information system to
ensure an effective results-based accountability system.

For Further Information:

John Oswald, Director

Center for Health Statistics
Minnesota Department of Health
717 Delaware Street Southeast
P.O. Box 9441

Minneapolis, MN 55440-9441

45


develop, coordinate and manage projects. To ensure that all stakeholders had a voice in
these proceedings, committees (headed by State Action Team members) comprised of
State agency staff, local providers, consumers and their family members were formed.

The State Action Team’s realization of a need for improving access to services for
children from birth to age 3 was the impetus for the Early Start Initiative. From concept
(July 1995) to implementation (January 1996) the Early Start project took shape in just 6
months.

The performance goals were developed with input from the following sources and
stakeholders: local providers and parents, State agency staff, Family and Children First
State Team members, evaluators from the Healthy Start Program in Hawaii and the
Smart Start Program in North Carolina, and research and evaluation experts from the
University of Cincinnati, Ohio State University, the University of Washington (Seattle),
and the University of Arkansas, Those State legislators with an interest in early
childhood education and the Early Start Initiative were kept informed throughout its
development.

Input from the Governor and Cabinet Council members provided four long-term
performance indicators. However, these indicators could conceivably take 4 or more
years to achieve. Realizing this, the State Action Team members decided that interim
goals were needed to assess the progress being made towards achieving the long-term
indicators. A two day retreat was held with a research and evaluation group to help
facilitate in the development of the short-term goals. Ultimately, a set of three short-
term performance goals with accompanying objectives/activities were defined.

The development of the performance measures began with a research design. The
Department considers the research design as the one critical element that held the
initiative together and kept the process moving along. From the very beginning, and
throughout the development of the performance goals and objectives/activities, the
Department focused on how they were going to evaluate the effectiveness of county
programs and what data was needed to accurately assess their performance. Focus
groups with State staff and some of the participating counties were held to determine
counties’ capacity to do data collection. The Department determined, upfront, data that
currently existed and was readily available from data that needed to be created. In
addition, prior to county implementation of the Early Start programs, consensus among
stakeholders (e.g. State and county officials and participating providers) was reached as
to: how the programs would be evaluated, what data would be collected, and how that
data was to be collected and by whom.

The Department provided data collection and assessment skills training for provider staff,
all of whom had to be certified in these skills before they were allowed to gather data.
Two video teleconferences for providers, live and interactive, were held at six to seven
sites across the State for further clarification and instruction and for those people that
could not attend one of the regional meetings.

86


and function for those infants and children with special health care or developmental
needs, and effective and responsive systems to support maternal and child health."

Indicators and targets are mutually negotiated by a team of county and Department staff.
For the first grant year, these are based on 1995 baseline data to the extent possible.
The team also defined "mutually acceptable ranges [emphasis added} of success" for
progress in meeting each target to avoid inappropriate focus on a precise figure. Results
outside the variance will trigger discussion, review, and technical assistance.

Health officials will monitor the program’s progress in reaching the targets. "Should
Monroe County’s progress in achieving the Child and Family Health grant’s stated goals
be deemed unsatisfactory, State and County staff will review the County’s strategies and
efforts to determine if programmatic changes should be made... . If the County were
found to be ineffective in achieving substantial progress toward one or more goals, the
allocation of funding in future grant years associated with those goals could be
affected.""*

The State will continue to monitor the program in ways that go beyond review of data
collected for performance indicators. The Department will conduct onsite visits and
provide technical assistance. It also plans to hire an outside evaluator to assess the
program.

Process for Developing the Performance Systems

County health department officials have spearheaded the effort to consolidate funding.
They have long had an interest in trying out such a consolidated approach, having been
frustrated over the years with meeting the different contract requirements of multiple
categorical funding sources that support their child and family health programs. And, the
county had gained some experience with consolidation under an earlier service
integration grant from the Robert Wood Johnson Foundation, funded at 10 sites across
the country in 1993, to integrate children’s health services.

The county had unsuccessfully sought support for the initiative from the Department in
earlier years. In 1996, however, the environment for pursuing such an approach was

~more-favorable. The Department, for various reasons, was more.receptive to trying
consolidation. And it, together with the county partner, were encouraged by the Vice-
President’s National Performance Review, which supported efforts to consolidate grants
and to develop results-based systems for accountability. The initiative is similar, in many
respects, with the performance partnership grants envisioned by the Federal
Government.

13 New York State Department of Health, "New York State/Monroe County Child and Family Health Grant:

Executive Summary,” internal document, 1-2.

14 New York State/Monroe County Child and Family Health Grant: Executive Summary," 13.

63


Sample Performance Measures

The four long-term performance indicators identified by the Governor and The Cabinet
Council are:

1. To decrease the number of subsequent pregnancies for teen parents.

2. To decrease the number of substantiated/indicated cases of abuse and neglect
among children age birth to three.

3. To decrease the number of children birth to three placed into out-of-home care.

4. To increase the percentage of substance abusing pregnant women and parents of
children birth to three receiving substance abuse treatment.

The three short-term goals and one example of an objective/activity for each goal are:

Goal #1: To help create a family environment conducive to the growth and development
of children.

Qbjective/Activity: To increase parental knowledge and skills related to parent-child
interaction and developmental expectations. [Data sources are: NCAST, HOME and the
nutrition checklist information from the Data Collection Tool.]

Goal #2: To ensure children birth to three have proper medical care.

Objective/Activity: To increase the percentage of children participating in Early Start
with a primary health care provider. [Data source is: Data Collection Tool.]

Goal #3: To bring well coordinated services to children and families who need help.

Qbjective/Activity: To increase identification and referral of eligible children and
families. [Data sources are: BabyNet, intake forms, list of referral sources.]

An example-of one of the data sources for goal #1 is-the Home Observation for
Measurement of the Environment Scale which is used for children from birth to age
three. The original form was developed by the University of Washington, School of
Nursing. The version used by the Bureau of Early Intervention Services has the following
six sections: (1) Emotional and verbal responsitivity of mother, (2) Avoidance of
restriction and punishment, (3) Organization of environment, (4) Provision of appropriate
play material, (5) Maternal involvement with child, and (6) Opportunities for variety in
daily stimulation.

Each section requires the observing party to indicate a yes/no response for a variety of
possible caregiver-child interactions. Two examples of these interactions are:

87


MASSACHUSETTS

32


emotional disturbances, children at risk of service emotional disturbances, people with
forensic orders and adults with a mental health disability.

In July 1996, the State placed performance measurements in provider contracts to
provide a continuous source of data for all its ADM measures approved for use in
performance-based budgeting. All of the measures included in the Division’s submission
to the legislature as reflective of its performance are also included in provider contracts.

In attaching the same set of performance measures to its contracts, the Division wrestled
with a number of questions, including: (1) whether data would be collected on all clients,
(2) whether providers would be responsible for all data required, and (3) how the results
would be used in assessing provider performance. Ultimately, the Division decided to
require reporting on all clients, not just a sample. They decided to require providers to
report on all data (not just data related to the particular service they delivered).
Providers preferred that the Division use collateral data sources, such as other State
databases (from the Department of Corrections, for example), but the Division rejected
this option because they feared such data would not be timely, and because they wanted
providers to become responsible for knowing key data on their clients. Finally, the
Division set fairly loose performance standards--provider performance targets are based
on obtaining results within two standard deviations of a performance goal for all
providers, and a failure to meet that standard does not automatically trigger contract
termination. Failure would trigger further inquiry, and initiate a series of questions about
service delivery, population demographics, and management and administration.

Sample Performance Measures

As discussed above, the Division identified target groups and developed outcome
statements for the group. It then identified subpopulations within the target group and
performance measures for each group.

Example: Substance Abuse
Target Group: Adults who abuse drugs and/or alcohol.

Outcome: ...Will be drug free upon completion of treatment; will reduce substance
abuse.

Subpopulation: Adults abusing substances involved in the criminal justice system
community.

Measures: (number served, substance abuse, arrests, individual satisfaction,
employment)


NEBRASKA

Nebraska Partnership for Health and Human Services

Overview

The State of Nebraska has initiated an ambitious project in which, within 1 year’s time, a
complete reorganization of State health functions will have been designed and
implemented. As part of this effort, a performance accountability system has been
developed to identify outcomes, outcome indicators and performance measures for
programs supported by the new Health and Human Services organization.

Nature, Extent and Uses of Performance Measures
The mission of the Nebraska Partnership for Health and Human Services is:

To create and sustain a unified, accessible, caring and competent health and
human services system for each Nebraskan that maximizes local determination to
achieve measurable outcomes [emphasis added]. To this end, the state will work in
partnership with communities and their public and private sector entities?

Exhibit 1, on the next page, outlines the relationships between outcomes, indicators,
performance measures, analysis, and systems or strategies, for performance accountability
proposed for the Nebraska Partnership. Nineteen outcomes have been developed thus
far. A series of possible indicators have also been developed, which are expected to be
refined by agency professional staff based on data availability (meaning that data is now
available, or could be made available). Performance measures, which focus on efficiency,
quality and effectiveness and more directly relate to system or strategy under scrutiny,
will be customized for each program or strategy. In order to apply the data to improve
programs and interventions, analysis of data from the "outer loop," or systems level,
together with data from the “inner loop," or program level, would occur. This allows for
evaluation, interpretation and understanding of the system and program interactions and
dynamics within a larger context.

According to the implementation plan, the Performance Accountability Team (PAT) will
consist of members from all three HHS agencies, reporting to the Director of Regulation
and Licensure. Other members from programs, the community and technical side are
brought in as necessary. The Community/State Support and Technical Assistance Team
(TA Team) works closely with communities to develop capacities, as well as providing
leadership for the feedback, analysis and evaluation of the data obtained. The PAT
drives much of the development and recommendation of measures, as well as analysis of
the data obtained.

8 Nebraska Partnership Project, "Unified Health and Human Services for Nebraska’s Future: A Blueprint for
Action," 1995.

47


Process for Developing the Performance Systems

The Act arose from the combined efforts of the Governor and key House and Senate
committees responsible for oversight of government operations. The impetus to the Act
included: (1) low public confidence in State programs and cperations; (2) performance-
based management approaches gaining acceptance in private sector management theory
and practice; (3) national support for performance-based public administration, including
the National Academy of Public Administration/American Society for Public
Administration 1991 statements in support of performance measurement; and, (4) the
Government Performance and Results Act of 1993.

The Act built on prior efforts in Florida as well, including the Frederick Commission,
which in 1991 recommended community-oriented, customer driven, results focused
management in public programs.

To implement performance-based budgeting according to the Act’s provisions, the
Division developed target populations, established an overall goal, and worked with
stakeholders to identify measures. In focusing on target populations, the Division
ensured that measurement would be oriented towards people rather than services,
believing this to be a key to effective and innovative program operations as well as to
creating opportunities for services integration. The Division also established a “core” goal
for its target populations: self-sufficiency. The consideration of factors which contribute
to self-sufficiency drove the development of performance indicators. The Division used
workgroups with stakeholders--providers, clients, legislative staff, OPAGA staff--to
develop measures. These workgroups were arranged around target populations,
consisted of 15 to 30 people, and met according to their own needs and schedules (some
for a full day, some for two full days).

Within the workgroups, providers and client or client representatives often had different
perspectives on appropriate measures and outcomes. Providers generally wanted more
controllable, limited measurements connected to outputs and the results of parucular
services (e.g. day treatment). Clients and their representatives were often forceful in
agitating for broader outcomes and measurements to reflect what they considered their
true needs and requirements (e.g. for obtaining sustained work and income levels). Even
State employees-brought their own perspectives: -our informants characterized employees
with backgrounds in mental health as more accepting of the work on outcomes
measurement than their colleagues who had worked in the substance abuse arena.

In June 1995, the final set of performance measurements were approved. However, the
Division lacked baseline data for measures relating to persons with mental illness. As a
result, the schedule for performance budgeting was delayed one year while these data
were collected. The State contracted with the Florida Mental Health Institute (FMHI)
to develop survey instruments and train district staff and volunteers on data collection.
Through this mechanism, in May 1996, the State was able to successfully survey a sample
of individuals in provider agencies representing people in crisis, children with severe


+ developing partnerships and relationships with a wide range of stakeholders both
inside and outside government; and,

+ developing, for each of its programs, standards of care as well as performance
indicators.

The Department recognizes that achieving consensus from a broad spectrum of
stakeholders on the standards and indicators has been a time-consuming process. It
recommends that other States may benefit from establishing definite timelines for
decisionmaking and clearly defining the steps and responsibilities in the process.

For Further Information

Gary Pastva
Assistant Commissioner for

Clinical and Professional Services
Massachusetts Department of Mental Health
25 Staniford Street
Boston, MA 02114-2575

Joan Kerzner

Director of Policy Development
Massachusetts Department of Mental Health
25 Staniford Street

Boston, MA 02114-2575

41


The Results Accountability Task Force:

In May 1995, the Governor’s Policy Council commissioned a Results Accountability Task
Force to: (1) establish clear benchmarks of progress that will assist communities and the
State in tracking the extent to which results are being achieved, and (2) develop a
framework and recommendations for how to use results and benchmarks for policy and
budget decisionmaking. This 15-member Task Force was comprised of leaders from
business, education, health care, human services, philanthropy, government and the
community.

Benchmarking

It took the Results Accountability Task Force about 6 months to develop 26 benchmarks
for the Policy Council’s 5 results areas. The following process was used to develop the
benchmarks:

Setting criteria for selecting benchmarks.
In the first meeting, the Results Accountability Task Force decided upon four criteria for
selecting benchmarks (measures). Staff were instructed to compose a list of possible

benchmarks for the Results Accountability Task Force to review based on the following
criteria:

(1) Measures indicating where early intervention can prevent later problems took
precedence.

(2) Measures should demonstrate a link with desired results.

(3)  County-level data, updated regularly, should already exist for the measures.

(4) Measures should build on existing work with Kids Count, Council for School
Performance, Savannah Youth Futures Authority and the Governor’s Council for
Economic Development.

-Reviewing-the literature-and meeting with experts.

Prior to presenting the list of proposed benchmarks to the Results Accountability Task

Force, staff refined the list through an extensive literature review and discussions with

subject area experts. A draft of 40 benchmarks were proposed for the S results.

Prioritizing the Benchmarks.
The Results Accountability Task Force determined that each result would have no more

than five benchmarks. They used the critical factors analysis methodology to prioritize
the proposed 40 benchmarks according to the following 3 standards:

20


GEORGIA

15


Results-based System: An initiative focused on using measurement systems to gauge
program outcomes and effectiveness, with accountability attributes which might include

public reporting, goal setting, and standards or requirements for meeting goals applied to
program officials, providers, contractors or grantees.

Outcomes Measures: Measures reflecting ultimate programmatic results, including health
status and risk behaviors.

Performance Measures: Measures reflecting a program’s more immediate effectiveness
or efficiency, including activity levels and direct accomplishments as a result of services
rendered.

Standard: A measure set as a requirement and expected threshold for States counties,
providers, or other partners in the results-based system.

Goal: A measure representing the end to which efforts by States, counties, providers or
other partners in the results-based system are aimed.


Research Agenda

The DASA hired a full time researcher to help guide its research agenda. The DASA is
assisted in this research agenda by two coordinating councils. The first, the Washington
Interagency Network, is composed of representatives from 13 different State agencies
having a role in or affected by alcohol and drug abuse, including corrections, education,
law enforcement, prevention and treatment. Originally formed to develop consensus on
responses to legislative proposals, the relationships formed over time and regular
meetings have allowed joint projects to be pursued and data sharing facilitated. The
second, a research committee composed of State and academic researchers, including
representatives from the Veterans Administration, Washington State University and the
University of Washington, meets three to four times a year to discuss research initiatives.
Discussion within this group helps ensure research is conducted and supported which is
relevant to program operations.

Sample Performance Measures

As indicated above, the Division uses various "tiers" of measures to report on
performance.

For example, in its last report on tobacco, alcohol, and other drug abuse trends in
Washington State (1995), the division reported on such outcomes as the percent of
Washington State’s high school seniors who had never smoked a cigarette, the percent of
high school seniors who reported never having had a drink, deaths per 100,000 persons
due to alcohol related motor vehicle accidents, rates of deaths due to liver cirrhosis and
lung cancer, and arrests for drug abuse violations, and compared those results to the
nation as a whole. The report also discusses policy questions and data needs arising from
the Division’s examination of these data.

The Division’s TARGET data supports outcomes measurements more directly related to
programmatic interventions. For example, the Division has developed outcomes
measures for four programs: ADATSA ("a state funded program that provides a
continuum of care to persons who are indigent and deemed unemployable as a result of
alcoholism and/or drug addition"), detoxication, youth assessment and treatment, and
treatment of pregnant and-parenting women. A series of outcomes and measures have
been developed for each program. The following provides some examples of ADATSA
outcomes and measures:

Outcome: Reduction in alcohol and other drug use
Measures: _(re-admissions to detox and inpatient treatment, abstinence or

reduced use of alcohol and/or other drugs--tracked 12 months after
discharge)

11


2. Child Health
Measure: Child (ages 1-9) fatality rate

3. Heart Disease and Stroke
Measure: Heart disease rate

4. Cancer
Measure: Breast cancer death rate

3. Diabetes and Other Chronic Disease
Measure: Diabetes-related death rate

6. Injury
Measure: Motor vehicle injury rate

7. Communicable Disease
Measure: AIDS case rate

& Adolescent (ages 10-17) Health Status
Measure: Adolescent fatality rate

Data for Level II indicators are provided on a local health department basis. Some
examples of the 32 Level II indicators include:

+ Percent of live births with adequate/adequate plus prenatal care.

+ Percent of mothers who smoked during pregnancy.

+ Percent of Medicaid children utilizing health check preventative care.

+ Percent of inspections completed on food, lodging and institutions.

+ Public schools are smoke-free.

+ Rate of unsatisfactory pap smears submitted to State laboratory.
Management Challenges

Understanding the data. The DEHNR believes it will be a challenge to accurately assess
the impact local health departments have on their communities.

Level I and II outcome measures may not always be accurate indicators of poor health
status in a community or of deficient local health department performance. These
measures will probably not definitively identify a problem. Rather, they are intended to
"red flag" an area where some evidence of a problem exists. It is then the responsibility

70


ILLINOIS

Department of Alcoholism and Substance Abuse

Overview

The Illinois Department of Alcoholism and Substance Abuse (DASA) has two efforts
underway to evaluate the impact of its treatment and prevention programs. Prevention
providers have been using an assessment process to also target their efforts more
efficiently. An outcome-based performance measurement client survey is currently being
developed for use by treatment providers.

PREVENTION PROVIDERS
Nature, Extent and Uses of Performance Measures

"Community Focused Prevention Through Risk Reduction" (CPRR) is a five-step process
used by prevention providers to aid in targeting their efforts more efficiently and to help
document the impact of prevention programs.

The CPRR five steps are:

Assess Community Risk Factors
Target Desired Outcomes
Develop and Implement Programs
Review Program Implementation
Measure Outcomes

VRYNE

To assess their community’s risk factors, step 1, prevention providers use the Illinois Risk
and Resiliency Assessment (IRRA) tool. The IRRA tool is a database of archival data
from many Illinois agencies and organizations that has been collected for the 104
counties of Illinois to identify or rate what the highest risk factors are for each county.
The IRRA risk factors are:

Availability of ATOD (alcohol, tobacco and other drugs)
Stresses associated with transitions and mobility

Unsafe neighborhoods and community disorganization

Parental drug use and pro-ATOD family norms

Early initiation of ATOD use

Latch-key status and unsupervised time among youth

Lack of family resources or supports

Family management problems and stressful home environments
Pro-ATOD media messages

Community laws and norms favorable toward ATOD

SCSI AAR WHE

25


FLORIDA
Department of Health
Overview

The Department of Health, reorganized as such effective January 1, 1997, consists of a
number of agencies that have been involved in performance measurement efforts over a
period of years. While part of the Florida Department of Health and Rehabilitative
Services (HRS), officials now in the Department of Health (as well as the Department of
Children and Families) developed strategic plans for Floridians’ health. A quality
improvement process was also used to assess the performance of local health units
responsible for preventive health, maternal and child health, environmental health, and
disease control.

Nature, Extent and Uses of Performance Measures

The Department issued its latest strategic plan in January 1997. The plan consists of
priority service areas, one priority indicator for each area, a set of key indicators for each
area (no more than 10, composed largely of outcome measures or measures for reducing
risk), and intervention strategies. Baseline data for 1995 are included for a number of
measures.

The Department also released an evaluation of their success in meeting objectives
outlined in a 1994 strategic plan. Based on this experience, it also modified some of the
indicators.

The State reviews District and County level performance based on a series of indicators.
It conducts a quality improvement peer review process in which indicators, struc ture and
process are reviewed for county public health units. The Public Health Indicators Data
System (PHIDS) contains data available for use in constructing indicators and measuring
results, based on data from vital statistics, trauma, census, epidemiology programs, and
other sources.

Process for Developing Performance Systems

The Department’s work on strategic planning began in 1976, when staff codified local
health department activities and developed a comprehensive reporting system and initial
plan. In 1988, the Department initiated a series of regional meetings to develop health
status and outcome indicators and process measurements. That effort was largely based
on the national Healthy People 2000 effort.

The effort in developing a quality improvement (QI) process began in 1990. State and
local health officials worked together to develop a program structure, system, and
indicators, through a Public Health Services Council. The Council used Healthy People
2000 and Healthy Communities 2000, in addition to other documents such as the HRS

11


Sample Performance Measures

The scripted client interview survey will be used by the DASA to help assess the impact
of treatment providers’ programs. These surveys may also be client self-administered.
The surveys will provide outcome baseline data from all clients upon admission (except
those under age 18 or undergoing detox) and outcome follow-up data from all clients
when they are discharged from the treatment program.

The scripted survey, still in draft, currently consists of about 23 questions or indicators.
The indicators ask for personal information about the client’s: alcohol and drug use;
family, education, health and mental health, financial and employment status; health care
utilization; living arrangements; and, interaction within the criminal justice system. Some
examples of the indicators are:

+ How old were you the first time you tried alcohol or drugs?

+ | Which item from the following list best describes your current status with the legal
system?

- Not currently involved with the criminal justice system
- In jail or prison

- On probation

- On parole

Treatment providers already collect client demographic data (e.g. name, address, age,
gender, race, referral source) and payment and eligibility data (e.g. family income and
size, insurance coverage) from all clients upon admission into their treatment programs.
In addition, a clinician or counselor must conduct a clinical assessment of the client upon
admission.

Management Challenges

Although this system will not be implemented until July 1997, officials have found several
aspects of the process, to date, to be particularly important.

Provider acceptance of new process. The DASA has found that the concept of
evaluation through results versus outputs is so foreign to many providers and even some
State officials, that informational meetings early on with stakeholders, about the new
process and outcome measures, were indispensable in the development of the survey
interview script. However, today (1 1/2 years into the development of the performance
system) DASA still does not have full provider buyin. In hindsight, the DASA believes
they should have developed the performance system in house but then spent more time
soliciting feedback on how the process should work.

Management information system. The DASA has come to believe that the management
information system is the most critical element in a results-based performance

30


In April 1996, Governor John Kitzhaber, unwilling to abort the program outright,
appointed a 45-member Governor’s Oregon Shines Task Force to work with the Oregon
Progress Board to reassess the State strategic plan and its benchmarks in relation to
today’s realities. Task Force members included legislators; county and local officials; and,
representatives from business, labor, academia and the media. In addition, meetings
were held across the State during 1996 to obtain input from Oregonians about the State
strategic plan.

Based on the benchmark data gathered since 1991, the Task Force determined that the
State strategic plan needed to focus more on: "...the well-being of families and
communities and the condition of our surroundings." They found that Oregon’s
",.mproved economy has not reduced some of [its] social problems as much as had been
expected and that our economic expansion may be threatening the very quality of life
that makes Oregon such a special place to live." New goals, objectives and key
benchmarks were developed aimed at ensuring Oregon’s economic success, improving the
health of communities and citizens, and protecting the environment. The revised Oregon
Shines II strategic goals are:

Quality jobs for all Oregonians;
Safe, caring and engaged communities; and,
Healthy, sustainable surroundings.

Each goal’s objectives identified by the Task Force are projected out to the years 2000
and/or 2010. The Task Force also worked on decreasing and refining the 259
benchmarks into "statements of measurable, realistic outcomes" with more specific
performance measures. Ultimately, all but 92 benchmarks were eliminated.

Prior to the establishment of the Task Force, the Department of Human Resources had
initiated two efforts, working simultaneously, to reassess its progress toward meeting the
strategic goals. These efforts are:

+ The Department Director and the six division heads (The Cabinet) and the
assistant administrators (The Department Policy Council) held focus groups to
review the Department’s status in meeting its objectives and the relevance of
outcomes and key benchmarks.

+ The Performance Measurement Committee (composed of performance measures
coordinators for each of the six Divisions within the Department) has been
meeting monthly for about 2 years to develop broader outcome measures and
more specific performance measures.

The Performance Measurement Committee has just completed (February 1997) a draft
of revised outcomes and performance measures, tailored to the 92 rew benchmarks, for
each Division in the Department. The next step in the process calls for a review of these
revised outcomes and performance measures by the Cabinet for approval.

99


Phase I of the Five Point Plan

In February 1995, Dr. Peter Somani (Director of Ohio’s Department of Health) created
a core/steering committee composed of ODH staff and representatives from public
health to re-evaluate the role of the State’s public health system - a system that was
service driven and fragmented. Believing that the public health system was not flexible
enough to deal with the changing health care and political environment (e.g. the advent
of managed care, a call for privatization, the maxim of doing more with less) Dr. Somani
charged the core committee with developing a public health strategic plan that would
focus on community needs and accountability rather than programs and perceived needs.

The development of the Five Point Plan was, and still is, a collaboration between the
ODH, local health departments, local Boards of Health, public health associations, health
care providers, academia and experts in the public health arena. A Steering Committee,
consisting of ODH senior staff, hired a consultant (a research and planning organization)
to facilitate the planning process. A Technical Advisory Committee consisting of
representatives from public health associations was convened to assist in drafting the Five
Point Plan. And, to ensure input from local health departments, health commissioners
from each of Ohio’s local health departments were asked to complete a survey designed
to elicit their views and opinions on public health.

The Steering Committee and the Technical Advisory Committee met over a period of 9
months to develop the: Five Point Plan mission and vision statements; purpose and
scope of the Plan; and, definition, goals, objectives, and strategies for implementation
(action steps) for each of the five initiatives.

Phase Il of the Five Point Plan

In February 1996, five workgroups were created (one each for the five strategic ureas) to
flesh out the elements to be included in each strategic area. Participants were asked to:
(1) develop and shape the initiatives; (2) determine resources required to develop and
implement the initiatives; and, (3) develop fiscal impact analysis for the respective
initiatives, including funding options. This stage of the process has just been completed.

Each of the workgroups was comprised of 25 to 30 people, experts from Ohio’s public
health system. Members included ODH and local health department staff,
representatives from the four major public health associations, and from universities and
the private sector. The workgroups were further divided into four to five subworkgroups
of four to five people. Because most of the workgroup members were strangers to each
other, the first thing each workgroup did was to create a vision and values statement for
its program area. This process allowed all involved to get to know and trust each other.

As of January 1997, performance standards have not yet been developed. This will be
accomplished in the next phase of the planning process. However, during Phase I,
"Essential Action Steps" for the development of new performance standards were
outlined. These Action Steps are:

92


strategic plan and State Health Plan, to develop its indicators. The criteria developed for
indicators was that they would be: understandable, acceptable to a wide range of
stakeholders, measurable using data that was available or could be made available, and

influenced by the activities of public health units.

In the past, local health units, district offices, and the State had discrete roles in the
system.” Local health units establish their targets for each indicator and conduct
ongoing analysis of results. Districts provide technical assistance. The State establishes
Statewide goals. Both the district and State staff have been involved in onsite peer
reviews of local units.

Sample Performance Measures
Departmental Strategic Plan
Priority Service Area: Preventing Infectious Disease

Priority Indicators:

+ Reduce rubeola (measles) case rate per 10,000 children under age 18 from 0.038
in Calendar Year (CY) 1994 to zero by CY 2000.

+ Reduce HIV seropositivity per 1000 women giving birth from 4.26 in CY 1994 to
3.69 by CY 2000.

+ Reduce incidence of tuberculosis cases per 100,000 persons from 12.65 in CY 1994
to 11.05 by CY 2000. :

District Performance Measures
Area: Maternal and Child Health

Measures: (resident live births, infant mortality, neonatal mortality, low and
very low birthweight, births to mothers 15-19, repeat births to
mothers 15-19, births to children 10-14)

As part of the departmental reorganization, the new Department of Health no longer has district offices, which
were assigned to the Department of Children and Families. There will be no comparable replacement for the districts. To
manage 67 CHDS, the central office is developing a comprehensive performance measurement system encompassing
outcomes, the quality of service delivery, and outputs. The performance reports will be generated on a routine schedule and
the measures, or a subset, wil! be used for performance reporting. This includes Agency Strategic Indicators and Performance

_ Based Budgeting indicators.

12


biennial reports to the State legislature. One example of such a survey is the Youth Risk
Behavior Survey which is administered every 2 years to a sample of students in grades
9-12. These surveys are used to monitor progress toward the Oregon Benchmarks and to
track the health status of youth, in general, and in relation to such health risks as
tobacco, alcohol and other drugs, for example.

Information collected in the form of the Oregon Benchmarks report is used in several
capacities. Not only does it contribute to setting the legislative agenda in establishing
budget priorities but it also helps focus management attention on high priority matters,
useful in the community planning process. Overall, the Department of Human
Resources and its Divisions use the data, not for compliance purposes, but as an aid in
determining how well the State and county health department programs are performing
so that the State may focus its attention on those areas that need improvement.

Process for Developing the Performance System

The Oregon Benchmarks program has been heralded as a pioneer in the move toward
legislative enactment of measurement indicators to guide and monitor a State’s progress.
This model for a union between public, private, and nonprofit organizations in a shared
vision garnered the State widespread recognition for its award-winning idea (e.g. 1994
Innovations in Government Award presented by the Ford Foundation and the Kennedy
School of Government at Harvard University; included as a section in the 1993 report on
reinventing government by the National Performance Review; featured in the National
Governors’ Association policy paper on the redesign of State government).

The idea for the Benchmarks program was hatched in the mid-80’s by a group of
corporate leaders as a response to Governor Neil Goldschmidt’s request for an economic
growth strategy. Oregon Shines, the State strategic plan, was issued in 1989. Later in
that same year, the Oregon Progress Board was created and charged with: (1) taking the
State’s strategic plan to the public for review, (2) identifying key steps necessary to
achieve the strategic vision, and (3) translating the strategies in Oregon Shines into
measurable goals for the State.

The strategic plan’s goals and strategies were the cornerstone for creating the Oregon
Benchmarks. The actual beginning of the program is marked by the passage of the first
160 benchmarks into law during the 1991 legislative session. The program swelled to 272
benchmarks in 1993 when then Governor Barbara Roberts offered funding incentives to
agencies that adopted benchmarks. In addition, all State agencies were directed to give
priority to critical benchmarks in developing their budget proposals and to develop
performance measures consistent with these benchmarks.

However, by 1995, with 259 benchmarks, many within and outside State government felt

that the benchmarks had become unmanageable, many being unmeasurable or
impractical. In fact, the 1995 State legislature declined to renew the program.

98


WASHINGTON

Public Health Improvement Plan

Overview

Washington State’s Public Health Improvement Plan (PHIP) is a multi-year plan. To
date, the efforts under the PHIP have focused on developing an understanding of the
core functions and capacities of the State’s public health system, including State and local
jurisdictions. They have also included the development of outcome standards for
improved health of the community, building on Healthy People 2000.

Nature, Extent and Uses of Performance Measures

The Department published its first Public Health Improvement Plan (PHIP) in 1994. It
outlined core function capacity standards in “community health assessment, development
of public health policy, assuring community access to quality health care within the
community, and providing the leadership financial and organization administration
required to integrate these functions into a coordinated, adaptive and effective public
health system."© A total of 88 core function capacity standards were included in the
1994 plan.

The plan also included "long term objectives, or outcome standards for the improved
health status of the people who live in Washington State, at the same time recognizing
that the public health system is not, and should not be, solely responsible for
achievement of these objectives. The outcome measures are...generally for the year 2000.
They define optimal, measurable future levels of health status, maximum acceptable
levels of disease, injury, or dysfunction, and in some cases the degree to which a
particular service or program is operational. Many of the outcome standards...are
identical in subject matter to the national year 2000 health objectives contained in
Healthy People 2000. The actual quantitative objectives for Washington will most often
be different from those for the nation, since the baselines are usually different."””

In September 1996, the Department of Health published, “The Health of Washington
State," containing Statewide data on health status, risk factors, and year 2000 goals. All

33 local health jurisdictions have been working on reports on health status and risk
indicators for their communities, though with varying results.

A 1996 PHIP report was finalized in December 1996. This second report focuses more
on the development of measurement systems. Performance Measures Technical Advisory

6 ‘Washington State Department of Health, Public Health Improvement Plan, November 29, 1994, p. 29.

Tid, p84.

103


6. National Academy of Sciences, Committee on National Statistics, Assessment of Performance Measures
in Public Health, Phase 1 Report, Draft: for Comment, September 1996. The report proposed performance
measures in the areas of chronic disease, HIV/STD/TB, mental health, immunization, substance abuse, sexual
assault prevention, disability prevention, and emergen~; medical services.


NORTH CAROLINA

66


NORTH CAROLINA

Division of Mental Health, Developmental Disabilities
and Substance Abuse Services

Department of Human Resources

Overview

The Division of Mental Health, Developmental Disabilities and Substance Abuse Services
(DMHDDSAS) is developing a proactive "continuous quality improvement" approach to
assess the efficacy of its public mental health, developmental disabilities and substance
abuse service system. Consumer satisfaction surveys are conducted in all area programs
and institutions. Client outcome assessment instruments are now being tested by a few
area programs.

Nature, Extent and Uses of Performance Measures
Consumer Satisfaction Surveys

The DMHDDSAS uses consumer satisfaction surveys as part of its quality improvement
process, to help determine that basic consumer needs are met and to identify possible
deficiencies in service. These surveys are administered twice a year in all 41 Area
Programs and 16 Institutions, with the exception of Developmental Disability Residential
Centers where they are conducted annually. The survey consists of ten questions that
focus on consumers’ satisfaction in the areas of quality of services, meeting of service
needs, staffing and facilities. The DMHDDSAS operated facilities, area programs and
contracted providers .dminister consumer satisfaction surveys to all their active clients
during a one week period specified by the DMHDDSAS.

The first time these surveys were administered was in November 1995 when 29,896
surveys were collected. To date, the consumer satisfaction surveys have been
administered three times, with a fourth administration scheduled for May 1997.
Approximately 75,000 such surveys have been collected thus far. Analysis of this data is
ongoing. Currently, the survey results are distributed to the Area Program Directors,
Institutional Directors, Executive Staff and the Advocacy Advisory Committee for
informational purposes.

Client Outcome Assessment Surveys
The DMHDDSAS is piloting mental health, developmental disabilities and substance
abuse client survey instruments in 3 of its 41 area programs prior to Statewide

implementation which will begin in 1997. Separate survey instruments were created for:
child and adult substance abuse services, perinatal/maternal substance abuse services,

72


performance measures and the programs they describe have to do with "means." For
example:

A Result is a condition of well-being for families, children or communities, such as
children succeeding in school.

A Benchmark is a measure for which data are available that help quantify the
achievement of results, such as the percentage of students graduating from high school
on time.

A Performance Measure is a measure of effectiveness, such as the percentage of students
enrolled in an alternative school program who are promoted to the next grade.

Georgia has selected 26 benchmarks as indicators of the broader results. The benchmark
data are designed to be used as a planning and evaluation tool by the Community
Partnerships and the State. These benchmark data are expected to: provide a general
sense of the well-being of children and families over time, signal whether strategies are
making a difference, and serve as a catalyst for change.

However, these benchmarks are not all inclusive. As a result, the State has decided not
to set Statewide goals because of the possibility that some of the benchmarks may not be
relevant in some communities. When this situation occurs, the Community
Partnerships/Family Connection Collaboratives are encouraged to develop other
benchmarks as measures of progress.

Results-Based Budgeting

Results-based budgeting is another component of Georgia’s results-based accountability
system for children and families. Program results will be linked to budgets, thereby
creating a system of accountability that explicitly works towards achieving agreed upon
goals. Both the starting and the ending points of the budget process will become the
results. Thus, budgeting begins with the desired results, systematically derives spending
plans based on the results expected, and then measures success by demonstrated progress
toward achieving the results.

The State is not yet using a results-based budgeting process. They are just beginning to
implement this effort. The State plans, over the next 2 to 3 years, to move all its
programs from line item budgeting to one that is linked to results. The 1995-1996 budget
represents the first time State agencies have budgeted on a "program" basis rather than
on a line item basis. Agencies’ budgets are expected to be tied to performance beginning
next year.

Barriers Busting Process
The Community Partnerships and Policy Council have also developed a process they

hope will effect policy change. This mechanism is called the “Barriers Busting Process."

17


Sample Performance Measures

Each of the Division of Health’s outcomes has three or more performance measures. An
example of one performance measure for each of the Division of Health’s six outcomes is
provided below. The number in parenthesis represents the total number of Health
Division performance measures for this outcome. These 6 outcomes and 43 performance
measures are still in draft.

Outcome: Decrease the percentage of people using tobacco.

Performance Measures (4): Increase the percentage of adults who do not
currently smoke tobacco.

Outcome: Decrease the number of people at risk of alcohol and drug abuse.

Performance Measures (3): Decrease the percentage of 8th grade students who
used alcohol in the previous month.

Outcome: _ Increase the percentage of healthy babies.
Performance Measures (5): Decrease the infant mortality rate.

Outcome: Decrease the number of deaths, illness, and injuries due to preventable
environmental and personal causes.

Performance Measures (21): Decrease the number of deaths due to AIDS.
Outcome: Decrease the teen pregnancy rate.

Performance Measures (3): Decrease the rate of teenage pregnancy. (Pregnancy
rate per 1,000 females age 10-17.)

Outcome: Decrease the rate of preventable deaths due to suicide and injuries.
Performance Measures (7): Decrease the rate‘of'deaths due to unintentional
injuries.

Lessons Learned

Stakeholder buyin. In the beginning of the State benchmarking initiative, the effort was

discussed and promoted in terms that led State officials and staff to feel that failure to

achieve the benchmark(s) could result in the loss of their jobs. Consequently, many
people were apprehensive about moving to an outcome-based performance system and

development of such a system was slow to be achieved. It took about 2 to 3 years for
people to feel comfortable with the new process and to work cooperatively with each

100


Sample Performance Measures
The following provides examples of two benchmarks for each of the five results:
Result 1. Healthy Children

+ Reduce the percentage of children who have untreated vision, hearing or health
problems at school entry.
+ Reduce the pregnancy rate among school-age girls.

Result 2. Children Ready for School

+ — Increase the percentage of low-income students in Head Start or pre-kindergarten
programs.

+ Increase the percentage of students passing the Georgia Kindergarten Assessment
Program.

Result 3. Children Succeeding in School

+ Reduce the percentage of students who are absent 10 or more days from school
annually.
+ Increase the percentage of students who graduate from high school on time.

Result 4. Strong Families

- Increase the percentage of stable new families (with the first birth to a mother
who has completed high school and is age 20 or older, and with the father’s name
recorded on the birth certificate).

+ Reduce the incide.ce of confirmed child abuse or neglect.

Result 5. Self-sufficient Families

+ Increase the percentage of welfare recipients leaving public assistance because of
employment or higher incomes.
+ Increase affordable,-accessible, quality child care.

Management Challenges

Limited resources. The State is ever being asked to do more with less resources (e.g. to
increase county participation in Community Partnerships, to increase the number of
clients served, to train community partnership staff). The Policy Council does not have
the resources to be there for the Community Partnerships 100 percent of the time. As a
result, the Community Partnerships have begun to formally establish peer-to-peer
networks and are now assisting each other. For example, staff training is one area where
the networks are working together to help each other.

22


+ It would address the two key issues of lack of stakeholder confidence in programs,
and limited or declining resources available for programs, by providing
measurable, objective feedback on the results achieved and the use of those
resources.

+ It would renew itself as a dynamic system, rather than a static model.

Sample Measures

As discussed above, new tiers of measures will be put in place by the new system:
outcome level goals and indicators, and performance level goals and indicators. For
example, a series of outcome level goals and indicators have been drafted for various
“cluster areas." A partial listing of outcomes and suggested indicators in the cluster area,
Health and Well Being are:

Outcome: Nebraska residents are free from preventable disease.

Indicators:  (incide~ce of sexually transmitted disease, HIV infection, cardiovascular
disease, cancer, type II diabetes, vaccine preventable disease, tobacco
related deaths)

Outcome: Nebraska residents are free from substance abuse.

Indicators: (percent of teens who report binge drinking, percent of adults who report
binge dr:aking, percent of teens who report substance abuse, percent of
adults who report substance abuse, percent of teens using tobacco, percent
of adults using tobacco, number of deaths and injuries related to alcohol
use, number of deaths and injuries related to substance abuse)

Outcome: __ Nebraska residents do not experience death, disease or injury related to lack of
appropriate treatment.

Indicators: (percent of residents who reside in medically underserved areas, percent of
deaths due to lack of emergency treatment)

Performance measures will be developed on-a-program-by- program basis.

Management Challenges

Once implemented, officials we interviewed believe the success of the system will rely
heavily on several key factors: (1) the constitution and competence of the performance
accountability team (including its support from departmental leadership), which would
facilitate efforts on performance reporting; (2) the availability of needed data; and, (3)
the investment in analysis and evaluation capability.

Team participants believe that it will take another 2 1/2 years for everyone within the

Department to be in the performance accountability system, at some point in the cycle.

51


OREGON

96


The Bureau is allowing the sites to develop and/or choose a measure of their own; they
need not select from the Bureau’s menu. It also is allowing sites considerable latitude to
determine approaches to measure their performance in achieving the outcome. Among
the approaches are: standardized surveys and questionnaires, problem-solving scenarios
for youth that are presented both pre- and post-intervention, focus groups, and interviews
with youths’ parents.

The Bureau is encouraging the pilot sites to use this outcome process to improve their
programs. In its communications with the Youth Programs, it has emphasized the
importance of the process for identifying the most effective program activities and for
demonstrating the value of prevention efforts from a broad perspective. While not
holding the sites accountable for results during the pilot phase, the Bureau has
communicated its intention to use the outcomes data to monitor program performance
and that “eventually programs’ demonstrated ability to achieve outcomes may strengthen
their funding requests."

Process for Developing the Performance Systems

The impetus for the specific POP initiative derived largely from the Director and staff of
the Bureau’s Prevention Unit. Work began in early 1996. By year’s end, the 18
participating sites had been selected and the process of identifying outcome measures for
each was well underway.

The primary actors in the process have included the Bureau’s Prevention Unit staff, a
private research firm that contracts with the Bureau, and providers from the local
prevention projects. The process involved several major steps:

Seeking Input from Other States

Early on, the Bureau staff participated in a regional meeting of States, convened by the
Federal Center for Substance Abuse Prevention, to share their experiences in developing
outcome measurement systems and information on research tools for measuring
outcomes in prevention settings.

‘Collaborating Closely with the Research Community
The Bureau’s research contractor has played a central role in the process of developing

measures, defining the strategy for implementation of the system, and working with both
the Unit staff and the providers during this pilot phase.

* Herr-Zaya and Rahn-Lopez, May 30, 1996.

34


It was designed to: (1) focus on results, (2) promote top down and bottom up problem
solving, (3) use common sense, and (4) be flexible and accountable. For each barrier, a
lead State partner (e.g. the Georgia Child Care Council or the Departments of Human
Resources or Medical Assistance) is assigned the responsibility for working with
communities and other State partners to resolve the problem. Such actions may involve
something complex and time consuming (such as new State and/or Federal legislation), or
something more simple (such as a change in procedure that would eliminate one or move
levels of management review).

This process is currently being tested. In May 1996, the Policy Council adopted Starting
Points, a model system for prevention and early care designed to support families from
birth to pre-kindergarten enrollment. In September 1996, participating communities
were asked to identify any policies hindering their development of this model system.
The Policy Council determined the top six priority barriers and decided to test the
barriers busting process and its effectiveness in resolving these obstacles. An example of
one of the barriers identified is: multiple intake and eligibility requirements whereby
families have to complete different forms for every service.

The Department of Human Resources has the lead responsibility for mitigating this
barrier. Resolution of this problem will involve streamlining intake and eligibility through
its Common Intake and Eligibility and Integrated Computer System Initiatives, which will
be no easy task as this effort could involve dealing with as many as 40 different funding
sources.

Process for Establishing Performance Measures

The impetus for this effort began in 1991, when Governor Zell Miller challenged the
State Departments of Human Resources, Education and Medical Assistance to work
together to address the needs of Georgia’s children. During that same year, The Joseph
B. Whitehead Foundation provided funding to help 95 selected Georgia communities
work in partnership with the State to test new and innovative approaches to the way
services were reaching children and families. These efforts have evolved over the years
into 86 partnerships/Family Connection Communities engaged in collaborative planning
and integrated service strategies that revolve around: community-based decisionmaking,
a focus on families, an.orientation toward results, intense planning, creative strategies,
and an emphasis on prevention and early intervention.

Policy Council for Children and Families

In 1994, by executive order, the Governor established the interim Policy Council for
Children and Families, a 21-member panel composed of: elected and appointed officials;
various State agency department heads; and, community, advocacy and business leaders.
This interim Policy Council issued a report in December 1994 titled: "On Behalf of Our
Children: A Framework for Improving Results." The basic premise of this report calls for
the State to:

18


NEW YORK

Monroe County Department of Health

Overview

New York’s State Department of Health is working with Monroe County health officials
on an initiative to: (1) support child and family health services through a single funding
stream of combined State and Federal funding sources, and (2) rely on results-oriented

objectives and indicators for ensuring accountability for performance. The Department

submitted the proposal to the Federal Government for review in late summer 1996.

Nature, Extent, and Uses of Performance Measures

The proposal calls for consolidating the contracts supporting eight child and family grant
programs into a sing'e, integrated grant. The services would be supported by a budget of
$3.4 million, resulting from combining State and Federal funds for several categorical
programs administered by several State and Federal agencies. These include, for
example, the United States Departments of Agriculture, Education and Health and
Human Services (the Centers for Disease Control and Prevention and the Health
Resources and Services Administration), and New York’s Medicaid program and various
programs of the Department of Health.

The proposal calls for a single work plan and budget, streamlined administrative
processes, and maintaining current reporting requirements of all funding agencies. The
proposal also calls for relying heavily on outcomes measures to assure accountability for
the program’s performance.

Fiscal Accountability

Issues of fiscal accountability require the State to seek approval from the Federal
Government for the proposal. The Department is asking that Circular A-87, issued by
the President’s Office of Management and Budget, be interpreted similarly by the
Federal Departments and that it emphasize results-based accounting rather than
program-specific accounting.

Performance Accountability
The outcomes-based system for accountability includes several program goals with
indicators and targets for each. The proposal states that the program goals are "based

on federal and state grant objectives and the Healthy People 2000 goals. The goals
include: healthy births, optimal healthy development of children, enhanced development

62


Among the current block grants administered by the United States Department of Health
and Human Services (HHS) for public health purposes are the Community Mental
Health Services Block Grants, Substance Abuse Block Grant, Preventive Health and
Health Services Block Grants, and Maternal and Child Health Services Block Grant.

Consistent with the character of block grants, the legislation establishing these grants
specifies purposes, populations to be served, specific activities or processes to be carried
out, reporting requirements, and limits or minimums on the uses of funds for particular
purposes. A brief description of each of these block grants follows below.

Community Mental Health Services Block Grants

The Public Health Service Act provides for block grants for community mental health
services. The Secretary of HHS, through the Director of the Center for Mental Health
Services, makes grants to the States each fiscal year for the purposes of providing
comprehensive community health services under a State plan, evaluating programs and
services carried out under the plan, and planning, administration and educational services
under the plan. State plans provide for the establishment and implementation of an
organized community-based system of care for adults with serious mental illness or
children with serious emotional disturbance. The plan must contain quantitative targets
to be achieved, although the Act only specifies that such targets include the numbers of
such individuals residing in the areas to be served under the system. Among other
things, specific attention must be given to: (1) reducing the rate of hospitalization for
eligible individuals, (2) providing outreach and services to eligible homeless individuals,
and (3) estimating incidence and prevalence in the State of the conditions of eligibility.

States must establish mental health planning councils to review State plans, serve as an
advocate for individuals with mental illnesses or emotional problems, and monitor, review
and evaluate the allocation and adequacy of mental health services within the State.

No more than 5 percent of the total grant to a State may be used for administrative
expenses.

Substance Abuse Block Grants

The Public Health Service Act provides for block grants for the prevention and treatment
of substance abuse. The HHS Secretary, through the Director of the Center for
Substance Abuse, makes grants to the States each fiscal year. States must allocate a
specific minimum percentage of funds for alcoho! prevention and treatment and for
prevention and treatment activities regarding other drugs. States must also spend a
specific portion of their funds to educate and counsel individuals not requiring treatment
about abuse and provide for activities to reduce their risk of such abuse.

States must submit an assessment of needs to the Secretary in order to receive a grant.

This assessment must include information by localities within the State on the incidence
and prevalence of drug abuse and alcohol abuse; current prevention and treatment

A-2


APPENDIX B

Endnotes



FLORIDA



the Performance Standards will incorporate essential public health practices, how
compliance will be managed, including enforcement protocol.

Sample Performance Measures

The Five Point Plan defines performance standards "...as statutorily-based, standardized
criteria for documenting levels of compliance with public health laws, rules and
regulations, and professionally accepted public health practices." Though performance
standards have not yet been developed, the Department has decided that Ohio’s existing
minimum standards will be updated and modified to focus on outcome-based
performance measures for the three core public health functions. In addition,
performance standards will be developed for each of the following 10 public health
practices:

Assessment Standards

1. Capacity for assessing and monitoring health needs.
2. Capacity to investigate and to remedy health hazards.
3. Ability to analyze health determinants, barriers and needs.

Policy Developmemt Standards

4. Capacity to mobilize constituents, communicate and advocate.
5. Ability to establish priorities.
6. Capacity to develop and articulate policies.

Assurance Standards

7. Organizational and resource management capacities.
8. Ability to plan and implement programs.

9. Evaluation and quality assurance capacities.

10. Capacity to inform/educate the public.

Management Challenges

Data development and management is seen as the most critical area in the strategic plan.
The Department plans a total systems improvement that will be funded over a 5-year

period. Because of the possible changes in agency department directors and in the
legislative and executive level of government that can occur during this period of time, a
continuing education process to promote this effort will be implemented.

Keeping the effort alive. Recognizing the importance of providing accurate and
consistent information about the Five Point Plan to the public health community, the
Governor, legislators and other public officials, a short-term marketing plan (designed to
interpret the purpose, progress, rational and expected benefits of the Plan) was adopted
by the Steering Committee and Technical Advisory Committee. This marketing effort

94


Process for Developing the Performance Systems

The impetus for developing program standards and performance indicators to be used
Statewide for all the Department’s programs originated largely with the appointment of a
new Commissioner in 1991. She and her top staff were committed to this effort during
her almost 5-year tenure, during which time the process became institutionalized. The
commitment to monitoring provider performance continues under the current
Commissioner, who was appointed in February 1996.

Developing the standards and indicators for each type of service has been a time-
consuming, resource intensive, broadly-based effort. Because all contracts for a given
type of program (e.g. respite or residential services) are bid during a particular year, the
Department anticipates each program cycle and engages the process before the contracts
for each type of service are re-bid for their 5-year contract periods.

Developing standards and indicators for residential services, the largest undertaking so
far, involved participation of Department staff, providers, consumers, and family
members. The Department used focus groups to solicit feedback from these
stakeholders on draft proposals it had initially prepared. The Department then
considered the recommendations of the stakeholders as it modified the drafts. Final
drafts were ultimately reviewed by a senior Department policy group before being
approved by the Commissioner for implementation.

Sample Performance Measures

The Department has published its program standards and indicators for emergency
services in a 28-page document that includes: (1) an overview defining psychiatric crises
and describing emergency programs; (2) the clinical and programmatic standards for
emergency programs and crisis stabilization; (3) the performance indicators; and,

(4) sections on definitions of, and qualifications for, professional staff and guidelines for
working with deaf and hard-of-hearing clients. Examples of performance indicators for
the emergency programs include:’

STANDARD: Emergency Program services are provided in the least restrictive, clinically
“appropriate setting using the least intrusive interventions. The Program has the ability to
triage, assess, and make appropriate clinical disposition decisions, including the capacity to
access inpatient services or less restrictive alternatives.

Indicator 1: Percent of individuals evaluated and hospitalized.

Massachusetts Department of Mental Health, “Emergency Program Standards," approved 8/6/96.

39


other and among different Divisions and Departments. Today, benchmarking is viewed
as a partnership towards achieving the State’s strategic plan goals and objectives.

For Further Information

Claudia Bingham

Performance Measure Coordinator
Oregon Health Division
Department of Human Resources
Office of Administration, STE 925
800 NE Oregon Street, #21
Portland, OR 97232

101


MINNESOTA

42