Evaluating

Your Program

Family 

Psychoeducation

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES

Substance Abuse and Mental Health Services Administration

Center for Mental Health Services

www.samhsa.gov

Family 

Psychoeducation

Evaluating

Your Program

U.S. Department of Health and Human Services

Substance Abuse and Mental Health Services Administration

Center for Mental Health Services
Acknowledgments

This document was prepared for the Substance Abuse and Mental Health Services 

Administration (SAMHSA) by the New Hampshire-Dartmouth Psychiatric Research Center 

under contract number 280-00-8049 and Westat under contract number 270-03-6005, 

with SAMHSA, U.S. Department of Health and Human Services (HHS). Neal Brown, M.P.A., 

and Crystal Blyler, Ph.D., served as SAMHSA Government Project Officers.

Disclaimer

The views, opinions, and content of this publication are those of the authors and contributors 

and do not necessarily reflect the views, opinions, or policies of the Center for Mental Health 

Services (CMHS), SAMHSA, or HHS.

Public Domain Notice

All material appearing in this document is in the public domain and may be reproduced 

or copied without permission from SAMHSA. Citation of the source is appreciated. However, 

this publication may not be reproduced or distributed for a fee without the specific, written 

authorization from the Office of Communications, SAMHSA, HHS.

Electronic Access and Copies of Publication

This publication may be downloaded or ordered at http://www.samhsa.gov/shin. Or, please 

call SAMHSA’s Health Information Network at 1-877-SAMHSA-7 (1-877-726-4727) (English 

and Español).

Recommended Citation

Substance Abuse and Mental Health Services Administration. Family Psychoeducation: Evaluating 

Your Program. HHS Pub. No. SMA-09-4422, Rockville, MD: Center for Mental Health Services, 

Substance Abuse and Mental Health Services Administration, U.S. Department of Health and 

Human Services, 2009.

Originating Office

Center for Mental Health Services

Substance Abuse and Mental Health Services Administration

1 Choke Cherry Road

Rockville, MD 20857

HHS Publication No. SMA-09-4422

Printed 2009
Family 

Psychoeducation

Evaluating Your Program

Evaluating Your Program shows quality assurance team members 

how to evaluate the effectiveness of your Family Psychoeducation 

program. It includes the following:

n A readiness assessment;

n The Family Psychoeducation Fidelity Scale;

n The General Organizational Index; and

n Outcome measures that are specific to your program.

You will also find instructions for conducting assessments and tips 

on how to use the data to improve your program.

For references see the booklet, The Evidence.
This KIT is part of a series of Evidence-Based Practices KITs created 

by the Center for Mental Health Services, Substance Abuse and 

Mental Health Services Administration, U.S. Department of Health 

and Human Services.

This booklet is part of the Family Psychoeducation KIT that includes 

a DVD, CD-ROM, and seven booklets:

How to Use the Evidence-Based Practices KITs

Getting Started with Evidence-Based Practices

Building Your Program

Training Frontline Staff

Evaluating Your Program

The Evidence

Using Multimedia to Introduce Your EBP
What’s in Evaluating Your Program

Why Evaluate Your Family Psychoeducation Program?. . . . . . 1

Conduct a Readiness Assessment. . . . . . . . . . . . . . . . . . . . . 3

Conduct Process Assessments. . . . . . . . . . . . . . . . . . . . . . . . 5

Monitor Outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Use Data to Improve Your Program. . . . . . . . . . . . . . . . . . . 17

Appendix A: Cover Sheet—Family Psychoeducation Fidelity 

Scale and General Organizational Index. . . . . . 23

Appendix B: Checklist—Observation of Multifamily 

Group Sessions. . . . . . . . . . . . . . . . . . . . . . . . 27

Appendix C: Family Psychoeducation Fidelity 

Scale and Scoresheet. . . . . . . . . . . . . . . . . . . 31

Appendix D: Family Psychoeducation Fidelity 

Scale Protocol. . . . . . . . . . . . . . . . . . . . . . . . 37

Appendix E: General Organizational 

Index and Scoresheet. . . . . . . . . . . . . . . . . . . 49

Appendix F: General Organizational Index Protocol. . . . . . . 55

Appendix G: Outcomes Report Form. . . . . . . . . . . . . . . . . . 69

Appendix H: Instructions for the Outcomes Report Form. . . 73

Appendix I: Assessor Training and Work 

Performance Checklist. . . . . . . . . . . . . . . . . . 79

Family 

Psychoeducation

Evaluating Your Program 1 Why Evaluate Your Family Psychoeducation Program

Evaluating Your Program

Why Evaluate Your Family Psychoeducation Program?

Key stakeholders who are implementing 

Family Psychoeducation (FPE) programs 

may find themselves asking two questions:

n	Has the program been implemented 

as planned?

n	Has the program resulted in the 

expected outcomes?

Asking these two questions and using the 

answers to help improve your program are 

critical for ensuring the success of your 

FPE program.

To answer the first question, collect 

process measures (by using the FPE 

Fidelity Scale and General Organizational 

Index). Process measures capture how 

services are provided. To answer the second 

question, collect outcome measures. 

Outcome measures capture the results 

or achievements of your program.

As you prepare to implement your 

program, we strongly recommend that you 

develop a quality assurance system using 

both process and outcome measures to 

monitor and improve the quality of the 

program from the startup phase and 

continuing through the life of the program.
Why Evaluate Your Family Psychoeducation Program 2 Evaluating Your Program

Why you should collect 

process measures

Process measures give you an objective, structured 

way to determine if you are delivering services in 

the way that research has shown will result in 

desired outcomes. Process measures allow agencies 

to understand whether they are providing services 

that are faithful to the evidence-based model. 

Programs that adhere closely to the FPE model 

are more effective than those that do not follow 

the model. Adhering to the model is called fidelity.

Collecting process measures is an excellent method 

to diagnose program weaknesses while helping 

to clarify program strengths. Once FPE programs 

reach high fidelity, ongoing monitoring allows 

you to test local innovations while ensuring that 

programs do not drift from the core principles 

of the evidence-based practice.

Process measures also give mental health 

authorities a comparative framework to evaluate 

the quality of FPE programs across the state. 

They allow mental health authorities to identify 

statewide trends and exceptions to those trends.

Why you should collect 

outcome measures

While process measures capture how services are 

provided, outcome measures capture the program’s 

results. Every service intervention has both 

immediate and long-term consumer goals. In 

addition, consumers have goals for themselves, 

which they hope to attain with the help of mental 

health services. These goals translate into outcomes 

and the outcomes translate into specific measures.

Consumer outcomes are the bottom line for 

mental health agencies, like profit is in business. 

No successful businessperson would assume that 

the business was profitable just because employees 

work hard.

Why develop a quality 

assurance system

In your mental health system, you should develop 

a quality assurance system that collects not only 

process measures such as those on the FPE Fidelity 

Scale and General Organizational Index, but also 

outcome measures such as those specified above 

to show the effect of FPE. Developing a quality 

assurance system will help you do the following:

n	Diagnose your program’s strengths and 

weaknesses;

n	Formulate action plans for improving your 

program;

n	Help consumers achieve their goals for recovery; 

and

n	Deliver mental health services both efficiently 

and effectively.

Research Has Shown That You Can 

Expect These Outcomes from Your 

FPE Program

n Reduced relapse and hospitalization

n Improved family well-being

n Increased participation in vocational 

rehabilitation

n Higher rates of employment, when 

combined with Supported Employment

n Decreased costs of care
Evaluating Your Program 3 Conduct a Readiness Assessment

Evaluating Your Program

Conduct a Readiness Assessment

Let’s assume that administrators and 

family intervention coordinators have read 

Building Your Program. Your new FPE 

practitioners have completed Training 

Frontline Staff. How do you know if you 

are ready to begin providing FPE services 

to consumers?

The Readiness Assessment on the next 

page will help quality assurance team 

members, advisory group leaders, and 

family intervention coordinators track 

the processes and administrative tasks 

required to develop an FPE program.

Answering these questions will help 

you generate an ongoing to-do list (or 

implementation plan) to guide your steps 

in implementing your FPE program. Your 

answers will also help you understand the 

components of the FPE model that are 

already in place in your agency and the 

work that still remains.
Conduct a Readiness Assessment 4 Evaluating Your Program

Readiness Assessment

Check any areas that you feel you do NOT completely understand.

q Which practitioners will be designated as staff for your 

FPE program?

q Who will supervise and direct the FPE program 

(who will be the family intervention coordinator)?

q What are the roles of the family intervention 

coordinator and practitioners?

q What is the size of the FPE practitioners’ caseloads?

q What is the supervisory structure (how often does 

the family intervention coordinator meet with FPE 

practitioners and the agency director)?

q How will you identify and refer consumers to your 

FPE program?

q How will you inform consumers, families, and others 

of your FPE program?

q What are your assessment procedures for consumers 

in your FPE program?

q How will you document the provision of FPE services?

q How often will FPE sessions be offered?

q How long will consumers and families receive FPE?

q What is your planning process for arranging the 

FPE one-day educational workshop?

q How will FPE consumers and families have access 

to multimedia educational materials?

q When will you offer FPE multifamily groups?

q How will you measure your program’s fidelity to 

the evidence-based model and use this information 

to improve your program?

q How will you collect and use consumer outcomes 

data?

q How does your FPE staff relate to advisory groups?

Note areas where you still are unclear or have questions. Arrange to speak to an expert consultant or experienced family 

intervention coordinator.
Evaluating Your Program 5 Conduct Process Assessments

Evaluating Your Program

Conduct Process Assessments

In addition to the Readiness Assessment, 

you should conduct your first process 

assessment before you begin providing 

any FPE services. By doing so you will 

determine whether your agency has core 

components of the evidence-based practice 

in place. During the first 2 years of 

implementing your FPE program, plan 

to assess your program every 6 months. 

After your program has matured and 

achieved high fidelity, you may choose to 

conduct assessments once a year. Agencies 

that have successfully implemented FPE 

programs indicate that you must continue 

to evaluate the process to ensure that you 

do not revert to previous practice patterns.

Once your program has achieved high 

fidelity to the evidence-based model, 

FPE practitioners may tailor the program 

to meet individual needs of the community. 

If you continue to use process assessments 

along with outcomes monitoring, you 

will be able to understand the extent 

to which your changes result in your 

program’s departure from model fidelity 

and whether the changes positively or 

negatively affect consumers. 
Conduct Process Assessments 6 Evaluating Your Program

How to use process measures

Two tools have been developed to monitor how 

FPE is provided:

n	The FPE Fidelity Scale; and 

n	General Organizational Index. 

You may administer both tools at the same time. 

The FPE Fidelity Scale has 14 program-specific 

items. Each item is rated on a 5-point scale, ranging 

from 1 (not implemented) to 5 (fully implemented). 

The items assess whether the program is provided 

as the evidence-based model prescribes.

The General Organizational Index is a second set 

of process measures that has been developed. 

In contrast to fidelity scales, which are practicespecific, this assessment can be used when 

implementing any evidence-based practice. 

It measures agency-wide operating procedures 

that have been found to affect agencies’ overall 

capacity to implement and sustain any evidencebased practice. 

For the FPE Fidelity Scale and General 

Organizational Index, see Appendices C and E. 

You can also print these forms from the CD-ROM 

in the KIT.

About the Process Measures that Are Included in the KIT

Quality assurance measures have been developed 

and are included in all Evidence-Based Practices KITs. 

The FPE Fidelity Scale was developed by a 

group of researchers at Indiana University-Purdue 

University, Indianapolis, and the developers of 

the KIT. The standards used for establishing the 

anchors for the “fully implemented” ratings were 

determined through a variety of expert sources 

as well as through empirical research. The scale 

has undergone numerous drafts and review by 

many groups. Revisions were also made based 

on feedback from a variety of sources during 

the 3-year pilot testing of the KIT materials. 

The General Organizational Index, developed 

by Robert Drake and Charlie Rapp, is a newly 

developed scale. This scale has undergone multiple 

revisions based on feedback gathered during the 

3-year pilot testing of the KIT materials. 
Evaluating Your Program 7 Conduct Process Assessments

Who can conduct process assessments?

We recommend enlisting two assessors to conduct 

your process assessment. Data collected by two 

assessors simultaneously increase the likelihood 

that information will be reliable and valid. 

Agencies that have successfully implemented FPE 

programs have taken different approaches to 

identify assessors. Some agencies train FPE 

Advisory Committee members as assessors and 

rotate the responsibility of completing assessments. 

Others have pre-existing quality assurance teams 

and simply designate members of the team to 

complete assessments of their FPE program. In 

other cases, the mental health authorities have 

designated staff to conduct assessments. 

Assessments can be conducted either internally by 

your agency or program or externally by a review 

group. External review groups have a distinct 

advantage because they use assessors who are 

familiar with FPE but, at the same time, are 

independent. The goal is to select objective and 

competent assessors. 

Although we recommend using external assessors, 

agencies can also use internal staff to rate their 

own programs. The validity of these ratings (or any 

ratings, for that matter) depends on the following:

n	The knowledge of the person making the ratings; 

n	Access to accurate information pertaining to the 

ratings; and 

n	The objectivity of the ratings. 

If you do conduct your assessments using internal 

staff, beware of potential biases of raters who are 

invested in seeing the program look good or who 

do not fully understand FPE. It is important for 

ratings to be made objectively and that they be 

based on hard evidence. 

Circumstances will dictate decisions in this area, 

but we encourage agencies to choose a review 

process that fosters objectivity in ratings, for 

example, by involving a practitioner who is not 

centrally involved in providing FPE. Only people 

who have experience and training in interviewing 

and data collection procedures (including chart 

reviews) should conduct assessments. Additionally, 

assessors need to understand the nature and 

critical ingredients of the evidence-based model. 

If your agency chooses to use a consultant or 

trainer to help implement your FPE program, 

involving that person in the assessment process 

will enhance the technical assistance you receive. 

Whichever approach you choose, we encourage 

you to make these decisions early in the planning 

process. For a checklist to help evaluate assessors’ 

training and work performance, see Appendix I.

How to conduct process assessments

A number of activities take place before, during, 

and after a process assessment. In general, 

assessments include the following: 

n	Interviewing administrators, the family 

intervention coordinator, FPE practitioners, 

consumers, and families;

n	Interviewing other agency staff (psychiatrists, 

therapists, or case managers); 

n	Observing one or more group or individual 

sessions;

n	Observing a planning and supervisory meeting; 

and 

n	Conducting a chart review.

Collecting information from multiples sources 

helps assessors more accurately capture how 

services are provided. A day-long site visit 

is the best way to learn this information. 
Conduct Process Assessments 8 Evaluating Your Program

To save time, you may interview FPE practitioners 

in a group. If the FPE program has three or fewer 

FPE practitioners, you should interview all of 

them. If the program has more than three FPE 

practitioners, you should try to interview at least 

three of them. 

For the items that require interviews with 

consumers and family members, we suggest that 

you interview at least three (from unique families). 

Try to interview families who are at different stages 

of the educational process. Contact the family 

intervention coordinator to help identify and set up 

these interviews. The following suggestions outline 

steps in the assessment process.

Before the process assessment

n n n Prepare your assessment questions

A detailed protocol has been developed to 

help you understand each item on the FPE 

Fidelity Scale and General Organizational 

Index, the rationale for why it was included, 

guidelines for the types of information to 

collect, and instructions for completing 

your ratings. Use the protocols to help 

prepare the questions that you will ask 

during your assessment visit. For the FPE 

Fidelity Scale and General Organizational 

Index protocols, see Appendices D and F. 

While we expect that quality assurance 

teams will select which outcome measures 

meet your agency’s needs, you should use 

the FPE Fidelity Scale and General 

Organizational Index in full. Collecting data 

for all the items on these scales will allow 

your agency to gain a comprehensive 

understanding of how closely your program 

resembles the evidence-based model. 

n n n Create a timeline for the assessment 

List all the necessary activities leading 

up to and during the visit and create a 

timeline for completing each task. Carefully 

coordinating efforts, particularly if you have 

multiple assessors, will help you complete 

your assessment in a timely fashion. 

n n n Establish a contact person

Have one key person in the FPE program 

arrange your visit and communicate 

beforehand the purpose and scope of your 

assessment to people who will participate 

in interviews. Typically, this contact person 

will be the family intervention coordinator. 

Exercise common courtesy and show 

respect for competing time demands by 

scheduling well in advance and making 

reminder calls to confirm interview dates 

and times.

n n n Establish a shared understanding 

with the staff of the FPE program

The most successful assessments are those 

in which assessors and the FPE staff share 

the goal of understanding how the program 

is progressing according to evidence-based 

principles. If administrators or FPE 

practitioners fear that they will lose funding 

or look bad if they don’t score well, then the 

accuracy of the data may be compromised. 

The best assessment is one in which all 

parties are interested in learning the truth. 

n n n Indicate what you will need from 

respondents during your visit

In addition to the purpose of the 

assessment, briefly describe what 

information you need, with whom you 

must speak, and how long each interview 

will take to complete. 
Evaluating Your Program 9 Conduct Process Assessments

The visit will be most efficient if the family 

intervention coordinator gathers 

beforehand as much of the following 

information as possible:

n	 Roster of FPE staff (roles and full-time 

equivalents [FTEs]);

n	 Number of consumers the agency serves;

n	 Number of consumers actively receiving 

FPE;

n	 Number of consumers and families 

who have attended:

Three or more joining sessions;

Educational workshop; and

Each multifamily group;

n	 Number of consumers served through 

the FPE program in the past 6 months;

n	 Number of consumers who have 

dropped out of the FPE program in the 

past 6 months;

n	 A copy of the agency’s brochure or 

mission statement for the FPE program; 

n	 A copy of the policies, procedures, and 

forms used to identify consumers for 

FPE;

n	 A copy of the policies, procedures, and 

forms used with consumers in the FPE 

program for assessment and treatment 

planning;

n	 A copy of the curriculum used in the 

educational workshop; 

n	 A copy of the curriculum used to train 

agency staff on the evidence-based 

model; and

n	 A copy of the agency’s quality assurance 

procedures, specifically a list of process 

and outcome measures used to evaluate 

the FPE program.

Reassure the family intervention 

coordinator that you will be able to 

conduct the assessment, even if all of the 

requested information is unavailable. 

Indicate that some information is more 

critical (for example, number of FPE 

practitioners and number of consumers in 

the FPE program) than other information.

Tell the contact person that you must 

observe a planning meeting, a group 

supervision meeting, and a multifamily 

group session during your visit. These are 

important factors in determining when you 

should schedule your visit. 

Observing an FPE multifamily group 

is integral to the assessment process. 

If observing a multifamily group session 

is impossible during your visit, arrange to 

have the sessions videotaped before your 

site visit.

n n n Alert your contact person that you will 

need to sample 10 charts

From an efficiency standpoint, it is 

preferable that the charts be drawn 

beforehand, using a random selection 

procedure. There may be a concern that 

the evaluation may be invalidated if FPE 

practitioners handpick charts or update 

them before the visit. If you both 

understand that the goal is to learn how 

the program is implementing services, 

this is less likely to occur. 

Additionally, you can further ensure 

random selection by asking for 20 charts 

and randomly selecting 10 to review. 

Other options include asking for a deidentified list (i.e., with names removed) 

of consumers who receive FPE and using 

the list to choose 10 charts to review. 
Conduct Process Assessments 10 Evaluating Your Program

If the program only has one FPE 

practitioner with fewer than 10 consumers 

on its caseload, then review the charts for 

all consumers in the program.

n n n Clarify reporting procedures

With the appropriate people (agency 

administrators, the mental health authority, 

or the family intervention coordinator), 

clarify who should receive a report of the 

assessment results. Recipients may include 

the following:

n	 Agency administrators; 

n	 Members of the agency’s quality 

assurance team;

n	 Members of the FPE Advisory 

Committee; 

n	 The family intervention coordinator;

n	 FPE practitioners; and

n	 Consumers and families. 

Assessors should also clarify how the 

agency would like the report to be 

distributed. For example, assessors may 

mail or fax the report and follow up to 

discuss the results in a meeting or by 

conference call. 

n n n Organize your assessment materials

Four forms have been created to help you 

conduct your assessment: 

n	 The first form is a cover sheet for 

the FPE Fidelity Scale and General 

Organizational Index, which is 

intended to help you organize your 

process assessment. It captures general 

descriptive information about the 

agency, data collection, and community 

characteristics. 

n	 The second form is a designed to help 

you collect data on two FPE fidelity 

items (Items 11 and 12). Complete this 

form to record information collected 

during your observation of FPE 

multifamily group sessions.

n	 The third and fourth forms are 

scoresheets for the two scales. They 

help you compare assessment ratings 

from one time period to the next. They 

may also be useful if you are interested 

in graphing results to examine your 

progress over time.

For the FPE Fidelity Scale and General 

Organizational Index instruments, cover 

sheet, checklist, and scoresheets, see 

Appendices A, B, C and E. You can also 

print these forms from the CD-ROM in 

the KIT. 

During your assessment visit

n n n Tailor your terminology

To avoid confusion during your interviews, 

tailor your terminology. For example, an 

FPE program may use client instead of 

consumer or it may use clinician instead of 

practitioner. Every agency has specific job 

titles for particular staff roles. By adopting 

the local terminology, you will 

improve communication.

n n n Conduct your chart review

It is important that you conduct your chart 

review from a representative sample of 

charts. When you begin your chart review, 

note whether your sample reflects families 

of consumers in different stages of the 

educational process. You should also note 

whether your sample includes consumer 

charts from each FPE practitioners’ 

caseload. Selecting charts of consumers 

who have received at least five FPE 

sessions is preferred. If your random 

sample is not representative in this 

manner, consider supplementing your 

sample with selected charts that will 

increase its representativeness. 
Evaluating Your Program 11 Conduct Process Assessments

Within each chart, examine the screening, 

referral, assessment, and treatment 

planning forms. Review recent Progress 

Notes to understand the amount and type 

of contact that FPE practitioners have with 

the consumers on their caseloads and with 

their treatment team members. If Progress 

Notes are not integrated into consumer 

charts, then ask if FPE practitioners have 

any additional files that you may review. 

In some cases, a lag may exist between 

when a service is given and when it is 

documented in the consumer’s chart. 

To get the most accurate representation 

of services rendered when you sample 

chart data, try to gather data from the 

most recent time period in which 

documentation is completed in full. 

To ascertain the most up-to-date time 

period, ask the family intervention 

coordinator, FPE practitioners, or 

administrative staff. Avoid getting an 

inaccurate sampling of data where officebased services might be charted more 

quickly than services given in the field.

n n n If discrepancies between sources occur, 

query the family intervention coordinator 

The general strategy in conducting fidelity 

assessments is to obtain data from as many 

sources as possible. When all these data 

sources converge, you can be more 

confident in the validity of the ratings. 

However, sometimes sources disagree. 

The most common discrepancy is likely 

to occur when the family intervention 

coordinator’s interview gives a more 

idealistic picture of the team’s functioning 

than the chart and observational data do. 

For example, on the FPE Fidelity Scale, 

Assertive engagement and outreach (Item 

14) assesses whether FPE practitioners 

assertively engage all potential consumers 

and family members in the FPE program. 

The chart review may show that consumers 

who drop out of the program are not 

contacted, while the family intervention 

coordinator may indicate that FPE 

practitioners expend considerable time 

reaching out to consumers who have 

disengaged from the program. 

To understand and resolve this 

discrepancy, the assessor should ask 

the family intervention coordinator 

the following:

Our chart review shows 10 percent of 

consumers who disengage are contacted, 

but your estimate is much higher. Would 

you help us understand the difference? 

Often the family intervention coordinator 

can provide information that will resolve 

the discrepancy. 

n n n Before you leave, check for missing data

Fidelity scales should be completed in full, 

with no missing data on any items. Check 

in with the family intervention coordinator 

at the end of the visit to collect any 

additional information you may need. 
Conduct Process Assessments 12 Evaluating Your Program

After your assessment visit

n n n Followup

It is important to collect any missing data 

before completing your rating. If necessary, 

follow up on any missing data (for example, 

by calling or sending an e-mail). This 

would include discussing with the family 

intervention coordinator any discrepancies 

between data sources that you notice after 

you’ve completed the visit. 

n n n Score the scales

The purpose of the scale is to assess fidelity 

to the evidence-based practice at the 

program level, rather than at the level of 

a specific practitioner. Ratings are based 

on current behavior and activities, not 

on planned or intended behavior. For 

example, to get full credit (to code the 

item as “5”) for Family intervention 

coordinator (Item 1), the program must 

have a designated staff member fulfilling 

the tasks of this position. If the agency 

plans to hire personnel to fill the position, 

it would not receive credit. If you assess 

an agency for the first time to determine 

which components of the evidence-based 

model the agency already has in place, 

some items may not apply. 

Many agencies that are developing a new 

FPE program will receive low fidelity 

ratings on items for which the agency 

has not yet formulated its policies and 

procedures. For example, several items 

are based on evaluating services that are 

provided by designated trained FPE 

practitioners. Agencies that have not yet 

hired or assigned and trained FPE 

practitioners, identified consumers and 

families, offered an FPE 1-day educational 

workshop, or started an FPE multifamily 

group cannot be rated for these items. 

If an item cannot be rated, code the item 

as “1.” 

To receive full credit, many items require 

that the family intervention coordinator 

and practitioners both understand and 

apply the evidence-based practice principle. 

If FPE practitioners generally do not 

understand the concepts, then code that 

item as “1.” If they understand parts of 

the concept and apply the understanding 

consistently, code the item as “3.” To 

receive full credit, there must be evidence 

that the concepts are applied consistently. 

For a complete explanation of how to rate 

each item, see the FPE Fidelity Scale 

Protocol and General Organizational Index 

Protocols in Appendices D and F. 

n n n Complete scales independently

If you have two assessors, both should 

independently review the data collected and 

rate the scales. They should then compare 

their ratings, resolve any disagreements, 

and devise a consensus rating.

n n n Complete the scoresheets 

Tally the item scores and determine 

the level of implementation achieved.
Evaluating Your Program 13 Monitor Outcomes

Evaluating Your Program

Monitor Outcomes 

Unlike process measures, which must be 

used in full to comprehensively understand 

how services are provided, you must 

decide which outcome measures will be 

most informative for your program. 

Initially, your outcomes monitoring system 

should be simple to use and maintain. 

Complexity has doomed many wellintended attempts to collect and use 

outcomes data. 

One way to simplify is to limit the number 

of outcome measures. Select your outcome 

measures based on the 

type of information that will be most useful 

to your agency. Based on the research 

literature, we suggest that you monitor a 

core set of outcomes such as the following: 

n	Relapse and hospitalization; 

n	Family well-being; 

n	Participation in Supported Employment 

or vocational rehabilitation; 

n	Employment rates; and 

n	Cost of care. 
Monitor Outcomes 14 Evaluating Your Program

These few outcomes reflect the primary goals 

of FPE. Specifically, the goal of FPE is to help 

consumers move forward in the process of 

recovering from mental illnesses and pursuing 

meaningful life goals. For this reason, it is 

important for you to capture outcomes for recovery 

in a way that is most useful for your program. 

For data to be useful, they must be valid. That is, 

the data must measure what they are supposed 

to measure. Thus, the outcomes must be few and 

concrete for FPE practitioners to focus on key 

outcomes, to understand them in a similar way, 

and to make their ratings in a consistent and errorfree fashion. 

To enhance validity, we recommend using simple 

ratings initially. Limiting your outcome measures 

to concrete measures will also allow you to collect 

data from FPE practitioners. 

Develop procedures 

Agencies may choose either to develop the outcomes 

portion of their quality assurance system from 

scratch or to use existing outcomes monitoring 

systems. A number of electronic evaluation 

programs are available to help you develop 

comprehensive, integrated, user-friendly outcome 

monitoring systems. Examples include the following: 

n	Publicly available tools such as the Consumer 

Outcomes Monitoring Package (see box 

below), the Decision Support 2000+ Online 

(http://www.ds2kplus.org); or 

n	Various commercially available products. 

What Is the Consumer Outcomes Monitoring Package? 

Sponsored in part by the Substance Abuse and 

Mental Health Services Administration (SAMHSA), 

the Consumer Outcomes Monitoring Package 

(COMP) was designed by a team at the School of 

Social Welfare, University of Kansas. This computer 

application allows agencies to choose from a preestablished list of outcomes developed for each 

evidence-based practice. Data may be entered 

for the chosen outcomes, and reports can be 

generated quarterly or monthly. The COMP also 

allows agencies to view their outcomes data using 

a variety of tables and graphs. 

The designers of COMP tried to make the 

computer application as easy and as flexible 

to use as possible. You may access COMP through 

the Web. Agencies can download the computer 

application and print out Installation Instructions 

and a User Manual, which provides definitions 

and forms.

To download COMP 

n Go to http://research.socwel.ku.edu/ebp.

n Click on the link to the download page. 

n Click on the links to download the Installation 

Instructions and User Manual. 

n Follow the instructions to install the application. 
Evaluating Your Program 15 Monitor Outcomes

When deciding whether to use an existing 

outcomes monitoring package or to design your 

own, it is important to keep your agency’s 

capabilities in mind. The system must not create 

undue burden for FPE practitioners, and it must 

give them information that is useful in their jobs. 

The system should fit into the workflow of the 

agency, whether that means making ratings on 

paper, using the COMP computer application, or 

developing your own outcomes monitoring package. 

Start with whatever means are available and expand 

the system from there. In the beginning, you may 

collect data with a simple report form and you can 

report hand-tallied summaries to FPE practitioners. 

Computer software that allows for data entry 

and manipulation (for example, Microsoft Access, 

Excel, and Lotus) makes tabulating data and 

graphing easier than doing them by hand. A 

computerized system for data entry and report 

generation presents a clear advantage and it may 

be the goal, but do not wait for it. Feedback does 

not have to come from a sophisticated computer 

system to be useful. It is more important that it is 

meaningful and frequent. For a sample Outcomes 

Report Form, see Appendix G, which is an 

example of a simple, paper-based way to collect 

participation and outcomes data regularly. For 

instructions for using the Outcomes Report Form, 

see Appendix H. 

Expanding Your Outcome Measures 

Once you have established your core outcomes 

monitoring system, have learned how to routinely 

collect data, and are accustomed to using it to 

improve your FPE program, you will be ready 

to expand your outcomes measures.

Consider asking consumers and families for input 

about how to improve your FPE program, both 

practically and clinically. Consumers and families are 

important informants for agencies that are seeking 

to improve outcomes. Agencies may want to know 

the following:

n If consumers and families are satisfied with their 

services;

n How services have affected their quality of life; 

and

n Whether consumers believe the services are 

helping them achieve their recovery goals.

While collecting data from consumers and families 

requires more staff time than the information that 

may be reported quickly by FPE practitioners, 

consumers and families can give valuable feedback.

We recommend the following surveys for collecting 

information from consumers and families:

n The Mental Health Statistics Improvement 

Program (MHSIP) Consumer Satisfaction Survey 

at http://www.mhsip.org

n Recovery measurement instruments such as 

those described in Measuring the Promise: A 

Compendium of Recovery Measures, Volume II, 

available through http://www.tecathsri.org

It is difficult to get a representative sample 

of consumer and family respondents since 

mailed surveys are often not returned and 

interviews may only be done with people who 

are cooperative and easy to reach. Samples that 

are not representative may be biased.

Avoid bias in your consumer and family data by 

using a variety of mechanisms to conduct your 

assessments. For example, consider combining 

feedback collected through surveys with that 

obtained from focus groups. Another option is to 

hire a consultant to conduct qualitative interviews 

with a small group of consumers or families.
Monitor Outcomes 16 Evaluating Your Program

How often should you collect 

outcomes data? 

Plan to monitor the outcomes for consumers in 

your FPE program every 3 months and share the 

data with program staff. Collecting data at regular 

and short intervals will enhance the reliability of 

your outcomes data. 

While we recommend that you design a system for 

collecting outcomes early in the implementation 

process, FPE programs should not expect to see 

the desired results until the program is fully 

operational. Depending on resources available to 

your program, this may take anywhere from 6 to 18 

months to accomplish. 

How should you identify data collectors? 

Agency administrators or mental health authorities 

may assign the responsibility for collecting 

outcomes data to the following: 

n	The family intervention coordinator; 

n	Members of the FPE Advisory Committee; 

n	The quality assurance team; 

n	Independent consultants, including consumers 

and family members; and 

n	Other staff. 

Unlike collecting process measures, collecting 

outcome measures does not require a day-long 

assessment process. Many standard outcome 

measures will be information that FPE 

practitioners can report from their daily work 

with consumers. 

It is important to develop a quick, easy, standardized 

approach to collect outcomes data. For example, 

create a simple form or computer database that 

FPE practitioners can routinely update.
Evaluating Your Program 17 Use Data to Improve Your Program

Evaluating Your Program

Use Data to Improve Your Program 

As you develop a quality assurance system, 

family intervention coordinators and FPE 

practitioners will weave it into the fabric 

of their daily routines. Process assessments 

will give you a window into the demanding 

work done every day. Outcome reports will 

give you tangible evidence of the use and 

value of services, and they will become a 

basis for decisionmaking and supervision. 

At some point, your program staff may 

wonder how they did their jobs without 

an information system. They will come to 

view it as an essential ingredient of wellimplemented evidence-based practices. 
Use Data to Improve Your Program 18 Evaluating Your Program

n n n Create reports from your assessments

For your process data, in addition to 

completing the FPE Fidelity Scale, 

General Organizational Index, and 

scoresheets, assessors should write a report 

explaining their scores. The report should 

include the following:

n	 An interpretation of the results of the 

assessment; 

n	 Strengths and weaknesses of the FPE 

program; and 

n	 Clear recommendations to help the 

program improve. 

The report should be informative, factual, 

and constructive. Since some process 

measures assess adherence to the 

evidence-based model at both the agency

and program staff levels, remember 

to target recommendations to 

administrators, the family intervention 

coordinator, and FPE practitioners.

When summarizing outcomes data, start 

with simple, easy-to-read reports. Then let 

experience determine what additional 

reports you need. You can design your 

reports to give information about individual 

consumers, a single FPE practitioner’s 

caseload, or the program as a whole. For 

example, reports generated for individual 

consumers may track the consumer’s 

participation in specific stages of treatment 

and outcomes over time. You could enter 

these reports in consumers’ charts, and they 

could be the basis for discussions about 

consumers’ progress.

n n n Use tables and graphs to understand 

your outcomes data

After the first process and outcomes 

assessments, it is often useful to provide 

a visual representation of a program’s 

progress over time. We recommend that 

you use tables and graphs to help 

understand and report the results. 

By graphing your fidelity score, you have a 

visual representation of how your program 

has changed over time. For an example, see 

Figure 1. For your process data, you may 

simply graph the results using a spreadsheet 

and include this in your report. 

When your program shows greater fidelity 

over time, the graph will display it and 

reinforce your efforts. Additionally, as you 

can see in Figure 1, the graph allows you 

to quickly compare one team to another. 

In this example, Team A struggled in the 

first 6 months. Understanding Team A’s 

progress compared to Team B’s allowed 

the teams to partner and share strategies. 

Consequently, Team A improved 

dramatically over the next 6-month period. 

Another feature of graphing assessment 

scores is to examine the cut-off scores for 

fair (52) or good (62) implementation. Your 

program can use these scores as targets.
Evaluating Your Program 19 Use Data to Improve Your Program

Figure 1. Fidelity Over Time

70

80

90

100

110

120

130

140

▲

▲

▲

▲

▲

■

■

■

■

■

▲■

Team A

Team B

Date of Assessment

Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Oct

 2006 2007

Note: 62 – 70 = good implementation 

52 – 61 = fair implementation 

51 and below = not evidence-based practice

Here are three examples of tables and graphs 

that can help you understand and use your 

outcomes data.

Example 1: Periodic summary tables

Periodic summary tables summarize your 

outcomes data each quarter and address these 

kinds of questions: 

n	How many consumers participated in our FPE 

program during the last quarter? 

n	What proportion of consumers in our FPE 

program were hospitalized last quarter? 

n	How did the hospitalization rate for those 

participating in FPE compare to the rate for 

consumers in standard treatment?

Agencies often use this type of table to understand 

consumer participation or to compare actual 

results with agency targets or goals. These tables 

are also frequently used to describe agencies’ 

services in annual reports or for external 

community presentations.

Table 1: Sample Periodic Summary Table of Enrollment 

in Evidence-Based Practices

Not 

eligible

Eligible but 

NOT in EBP 

service

Enrolled Percent 

of eligible 

consumers 

enrolled

Family 

Psychoeducation

0 30 60 67%

Assertive 

Community 

Treatment

30 25 90 78%

This agency provided both Family Psychoeducation 

(FPE) and Assertive Community Treatment (ACT). 

The FPE staff identified 90 consumers for the 

program. Of those, 60 received FPE, while 30 

consumers were eligible but received another 

service. Consequently, 67 percent of consumers 

who were eligible for the FPE program participated 

in the program. 
Use Data to Improve Your Program 20 Evaluating Your Program

Example 2: Movement tables

Tables that track changes in consumer 

characteristics (called movement tables) can give 

you a quick reference for determining service 

effectiveness. For example, Table 2 compares 

consumers’ residential status between two quarters. 

Table 2: Sample Movement Table

To FY ’06 Qtr 3

From: 

FY ‘06 

Qtr: 2

Institutional Substantial 

care

SemiindependantIndependant Total

Institutional 2 1 1 3 7

Substantial care 3 8 1 3 15

Semiindependant 1 0 2 4 7

Independant 1 3 2 100 106

Total 7 12 6 110 135

To create this table, the data were collapsed into 

the four broad categories. The vertical data cells 

reflect the residential status for consumers for 

the beginning quarter. The horizontal data cells 

reflect the most recent quarterly information. 

The residential status categories are then ordered 

from the most restrictive setting (institutional) 

to the least restrictive (independent). 

The data in this table are presented in three colors. 

The purple cells are those above the diagonal, the 

blue cells are those below the diagonal, and the 

white cells are those within the diagonal. The data 

cells above the diagonal represent consumers who 

moved into a less restrictive environment between 

quarters. As you can see, one consumer moved 

from institutional to substantial care, one to semiindependent care, and three to independent living. 

Furthermore, one consumer moved from 

Above the diagonal

Below the diagonal 

Within the diagonal

substantial care to semi-independent care, three 

consumers moved from substantial care to 

independent care and four consumers moved 

from semi-independent care to independent care. 

These 13 consumers (10 percent of the 135 

consumers in the program) moved to a more 

desirable stage of treatment between quarters. 

The data reported in the diagonal cells ranging 

from the upper left quadrant to the lower right 

reflect consumers who remained in the same 

residential status between quarters. Two 

consumers were in an institution for both quarters 

of this report; eight remained in substantial care, 

two in semi-independent and 100 in independent 

living. These 112 consumers (83 percent of the 135 

consumers in the program) remained stable 

between quarters. 
Evaluating Your Program 21 Use Data to Improve Your Program

The cells below the diagonal line represent 

consumers who moved into a more restrictive 

setting between quarters. Three consumers moved 

from substantial to institutional care, one consumer 

moved from semi-independent care to institutional 

care, one consumer moved from independent 

living to institutional care, three moved from 

independent living to substantial care, and 

two moved from independent living to semiindependent care. These 10 consumers (7 percent 

of the 135 consumers in the program) experienced 

some setbacks between quarters. The column 

totals show the number of consumers in a given 

residential status for the current quarter, and the 

row totals show the prior quarter. 

You can use movement tables to portray changes 

in outcomes that are important to consumers, 

supervisors, and policymakers. The data may 

stimulate discussion about the progress that 

consumers are making or the challenges with 

which they are presented.

Example 3: Longitudinal plots

A longitudinal plot is an efficient and informative 

way to display participation or outcomes data for 

more than two successive periods. The goal is to 

view performance in the long term. You can use 

a longitudinal plot for a consumer, a caseload, 

a specific evidence-based practice, or an entire 

program. A single plot can also contain longitudinal 

data for multiple consumers, caseloads, or programs 

for comparison. Figure 2 presents an example of a 

longitudinal plot comparing critical incidents for one 

FPE program over an 11-month period.

This plot reveals that with the exception of private 

psychiatric hospitalizations, all other critical 

incidents appear to be going in a positive direction 

(that is, there is a reduction in incidence). 

Longitudinal plots are powerful feedback tools 

because they permit a longer range perspective 

on participation and outcome, whether for a single 

consumer or a group of consumers. They enable a 

meaningful evaluation of the success of a program, 

and they provide a basis for setting goals for future 

performance.

n n n Share your results

The single factor that will most likely 

determine the success of a quality 

assurance system is its ability to give useful 

and timely feedback to key stakeholders. 

It is fine to worry about what to enter into 

a system, but ultimately its worth is in 

converting data into meaningful 

information. For example, data may show 

that 20 consumers were homeless during 

the past quarter, but it is more informative 

to know that this represents 10 percent 

of the consumers in the FPE program. 

For information to influence practice, it 

must be understandable and meaningful, 

and it must be delivered in a timely way. 

Figure 2. Sample Longitudinal Plot for Monthly Frequency

 of Negative Incidents for Consumers

◆

✖

✖

✖

✖

✖

✖

✖

✖

✖

✖

✖

✖

▲ ▲▲

▲

▲

▲

▲

▲

▲

▲

▲

◆

◆

◆

◆ ◆

◆

◆

◆

■

■

■ ■

■

■

■

■

■

■

◆■ ◆

◆

Homeless

Incarcerated

In-Patient substance abuse

Priv. Psych. Hosp.

■

▲

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

 2007

16

14

12

10

8

6

4

2

0
Use Data to Improve Your Program 22 Evaluating Your Program

In addition, the quality assurance system 

must tailor the information to suit the 

needs of various users and to answer 

their questions.

 Sharing results with FPE practitioners 

After each assessment, dedicate time 

during a supervisory meeting to discuss 

the results. Numbers that reflect above 

average or exceptional performance should 

trigger recognition, compliments, or other 

rewards. Data that reflect below average 

performance should evoke a search for 

underlying reasons and should generate 

strategies that offer the promise of 

improvement. By doing this regularly, the 

family intervention coordinator will create 

a learning organization characterized by 

adaptive responses to information that aim 

to improve consumer outcomes.

 Sharing results with your FPE Advisory 

Committee or quality assurance team 

You may also use this information to keep 

external stakeholders engaged. Sharing 

information with vested members of the 

community, staff from your mental health 

authority, and consumers and family 

advocates can be valuable. Through these 

channels, you may develop support for 

the FPE program, increase consumer 

participation, and raise private funds 

for your agency.

 Sharing results internally

Agencies may distribute reports during 

all staff and manager-level meetings to 

keep staff across the agency informed and 

engaged in the process of implementing 

your FPE program. Agencies with 

successful FPE programs highlight the 

importance of developing an 

understanding and support for the 

evidence-based model across the agency. 

Additionally, integrating consumer-specific 

reports into clinical charts may help you 

monitor consumers’ progress over time. 

Reporting consumer-specific outcomes 

information at the treatment team meetings 

also helps keep the team focused on 

consumers’ goals.

 Sharing results with consumers 

and families

Agencies may highlight assessment results 

in consumer and family meetings. 

Increasing consumers’ and families’ 

understanding of the FPE program may 

motivate them to participate in the 

treatment process and build trust in the 

consumer-provider relationship. 

Also, sharing results may create hope 

and enthusiasm for your FPE program. 

Sharing information motivates people and 

stimulates changes in behavior. Sharing the 

results of your assessments with a variety 

of stakeholders is the key to improving 

your program. 
Evaluating Your Program 23 Appendix A: Cover Sheet

Evaluating Your Program

Appendix A: Cover Sheet—

Family Psychoeducation Fidelity Scale 

and General Organizational Index 

Evaluating Your Program 25 Appendix A: Cover Sheet

Cover Sheet: Family Psychoeducation Fidelity Scale 

and General Organizational Index 

 Today’s date

Assessors’ names

Program name (or Program code)

Agency name

Agency address

Street

City State ZIP code

Family intervention coordinator or contact person 

Names of FPE practitioners 

Telephone E-mail

Sources used for assessments:	 q Chart review: Number reviewed

	 q FPE multifamily group observation

q Planning and supervisory meeting observation

q Family intervention coordinator interview

q FPE practitioner interviews Number interviewed

q Consumer interviews Number interviewed

q Family member interviews Number interviewed

q Other staff interviews Number interviewed

q Brochure review

q Other 

Number of FPE practitioners

Number of consumers/families in the program

Number of consumers/families who left the program in the past 6 months

Number of consumers/families served in the past 6 months

Funding source

Agency location: q Urban

q Rural

Date program was started

( ) -

/ /

/ /

Evaluating Your Program 27 Appendix B: Checklist

Evaluating Your Program

Appendix B: Checklist—

Observation of Multifamily Group Sessions

Evaluating Your Program 29 Appendix B: Checklist

Checklist—Observation of Multifamily Group Sessions

 Today’s date

Assessors’ names

Program name (or Program code)

Agency name

Agency address

Street

City State ZIP code

Names of FPE practitioners 

Number of consumer participants

Number of family participants

Frequency of sessions

Item 11. Structured Group Sessions 

Yes No

1. Beginning socialization q	 q

2. Review progress from last session’s action plan q	 q

3. Go-round q	 q

4. Selection of a single problem q	 q

5. Structured problem-solving q	 q

6. End with socialization q	 q Rating

Item 12. Structured Problem-Solving Technique 

Yes No

1. Define the problem 

2. Generate solutions 

q	 q

q q

3. Discuss advantages and disadvantages of each solution 

4. Choose the best solution 

q	 q

q q

5. Form an action plan

6. Review the action plan

q	 q

q q Rating

/ /

Evaluating Your Program 31 Appendix C: Fidelity Scale and Score Sheet

Evaluating Your Program

Appendix C: Family Psychoeducation Fidelity Scale 

and Score Sheet

Evaluating Your Program 33 Appendix C: Fidelity Scale and Score Sheet

Family Psychoeducation Fidelity Scale 

Criteria

Ratings / Anchors

1 2 3 4 5

1. Family intervention coordinator: 

Designated clinical administrator 

who performs the following tasks:

n Establishes, monitors, and 

automates family intake and 

engagement procedures

n Assigns potential FPE 

consumers to FPE practitioners

n Monitors and adjusts FPE 

practitioner caseloads

n Arranges for training new FPE 

practitioners and continuing 

education of existing FPE staff

n Supervises FPE staff

Agency does not 

have a designated 

staff member 

OR

Cannot rate due 

to no fit.

Agency has a 

designated staff 

member who 

performs 1 or 2 

of the tasks.

Agency has a 

designated staff 

member who 

performs 3 of 

the tasks.

Agency has a 

designated staff 

member who 

performs 4 of 

the tasks.

Agency has a 

designated staff 

member who 

performs all tasks.

2. Session frequency:

Families and consumers 

participate biweekly in FPE 

sessions.

< Every 3 months 

OR

Cannot rate due 

to no fit.

Every 3 months Every 2 months Monthly At least twice 

a month

3. Long-term FPE:

Families and consumers are 

provided with long-term 

FPE; specifically, at least one 

family member per consumer 

participates in FPE sessions for 

at least 9 months.

Most families and 

consumers receive 

less than 6 months 

of FPE sessions 

OR

Cannot rate due 

to no fit.

Most families and 

consumers receive 

6–7 months of FPE 

sessions.

Most families and 

consumers receive 

7–8 months of FPE 

sessions.

Most families and 

consumers receive 

8–9 months of FPE 

sessions. 

More than 90% 

of families and 

consumers receive 

at least 9 months 

of FPE sessions.

4. Quality of practitionerconsumer-family alliance

FPE practitioners engage family 

members and consumers with 

warmth, empathy, acceptance, 

and attention to each individual’s 

needs and desires.

High dropout rate

OR

Cannot rate due 

to no fit. 

Sources indicate 

that alliance is often 

poor, leading to high 

dropout rate.

MSources indicate 

alliance is 

inconsistent or barely 

adequate, leading to 

moderate dropout 

rate, 

OR

Information is 

inconsistent

Sources indicate a 

fairly strong alliance.

Sources consistently 

indicate a strong 

alliance.

5. Detailed family reaction:

FPE practitioners identify and 

specify the family’s reaction to 

their relative’s mental illnesses.

There is consistent 

evidence for less 

than 33% of 

involved families. 

There is consistent 

evidence for 

33–49% of involved 

families.

There is consistent 

evidence for 

50–64% of involved 

families.

There is consistent 

evidence for 

65–79% of involved 

families.

There is consistent 

evidence for 80% 

or more of involved 

families.

6. Precipitating factors:

FPE practitioners, consumers, 

and families identify and specify 

precipitating factors for the 

consumers’ mental illnesses.

There is consistent 

evidence for less 

than 33% of 

involved families 

and consumers.

There is consistent 

evidence for 33-49% 

of involved families 

and consumers.

There is consistent 

evidence for 

50–64% of involved 

families and 

consumers.

There is consistent 

evidence for 

65–79% of involved 

families and 

consumers.

There is consistent 

evidence for 80% 

or more of involved 

families and 

consumers.

7. Prodromal signs 

and symptoms: 

FPE practitioners, consumers, 

and families identify and specify 

prodromal signs and symptoms 

of the consumer’s mental illnesses.

There is consistent 

evidence for less 

than 33% of 

involved families and 

consumers. 

There is consistent 

evidence for 

33–49% of involved 

families and 

consumers. 

There is consistent 

evidence for 

50–64% of involved 

families and 

consumers.

There is consistent 

evidence for 

65–79% of involved 

families and 

consumers.

There is consistent 

evidence for 80% 

or more of involved 

families and 

consumers.
Appendix C: Fidelity Scale and Score Sheet 34 Evaluating Your Program

Family Psychoeducation Fidelity Scale 

Criteria

Ratings / Anchors

1 2 3 4 5

8. Coping strategies: 

FPE practitioners identify, 

describe, clarify, and teach coping 

strategies.

There is consistent 

evidence for less 

than 33% of 

involved families and 

consumers. 

There is consistent 

evidence for 

33–49% of involved 

families and 

consumers.

There is consistent 

evidence for 

50–64% of involved 

families and 

consumers.

There is consistent 

evidence for 

65–79% of involved 

families and 

consumers.

There is consistent 

evidence for 80% 

or more of involved 

families and 

consumers.

9. Educational curriculum:

FPE practitioners use a 

standardized curriculum to teach 

families about mental illnesses. 

The curriculum covers six topics: 

n Psychobiology of the specific 

mental illness;

n Diagnosis;

n Treatment and rehabilitation;

n Impact of mental illness 

on the family;

n Relapse prevention; and

n Family guidelines.

Less than 33% 

of involved 

families receive 

a standardized 

educational 

curriculum, no 

standardized 

educational 

curriculum exists, 

OR

Only 1–2 topics 

are covered

33–49% of involved 

families receive 

a standardized 

educational 

curriculum covering 

all 6 topics 

OR

Only 3 topics 

are covered.

50–64% of involved 

families receive 

a standardized 

educational 

curriculum covering 

all 6 topics 

OR

Only 4–5 topics 

are covered.

65–79% of involved 

families receive 

a standardized 

educational 

curriculum covering 

all 6 topics.

80% or more 

of involved 

families receive 

a standardized 

educational 

curriculum covering 

all 6 topics.

10. Multimedia education:

Consumers and family members 

are given educational materials 

about mental illnesses in several 

formats (for example, paper, 

video, and Web sites).

Less than 33% 

of families and 

consumers receive 

educational materials 

OR

Cannot rate due 

to no fit.

33–49% of families 

and consumers 

receive educational 

materials 

OR

Materials are given 

in only 1 format.

50–64% of families 

and consumers 

receive educational 

materials 

OR

Materials are given 

in only 2 formats.

65–79% of families 

and consumers 

receive educational 

materials in all 3 

formats.

80% or more 

of families and 

consumers receive 

educational materials 

in all 3 formats.

11 Structured group sessions: 

FPE practitioners follow a 

structured procedure that includes 

the following:

n Beginning socialization;

n Review progress from last 

session’s action plan;

n Go-round;

n Selection of a single problem;

n Structured problem solving; 

and 

n Ending with socialization.

Groups include 2 or 

fewer components. 

Groups include 3 of 

the 6 components. 

Groups include 4 of 

the 6 components.

Groups include 5 of 

the 6 components. 

Groups include all 

6 components.
Evaluating Your Program 35 Appendix C: Fidelity Scale and Score Sheet

Family Psychoeducation Fidelity Scale 

Criteria

Ratings / Anchors

1 2 3 4 5

12. Structured problem–solving: 

FPE practitioners use a 

standardized approach to help 

consumers and families with 

problem solving, which includes 

the following:

n Define the problem;

n Generate solutions;

n Discuss the advantages and 

disadvantages of each solution;

n Choose the best solution;

n Form an action plan; and

n Review the action plan.

No more than 2 of 

6 components of the 

structured problemsolving are used. 

3 of 6 components 

of the structured 

problem-solving 

are used.

4 of 6 components 

of the structured 

problem-solving 

are used.

5 of 6 components 

of the structured 

problem-solving 

are used.

All 6 components 

of the structured 

problem-solving 

are used.

13. Stage-wise provision 

of services:

FPE services are provided 

in the following:

n	Engagement;

n 3 or more joining sessions; 

n Educational workshop; and

n Multifamily group.

Families and 

consumers begin 

multifamily groups 

with minimal or 

no engagement, 

no joining sessions, 

or no education.

Engagement is 

minimal and only 

1 joining session is 

completed before 

entry into the 

multifamily group. 

Education is delayed 

or absent.

Engagement and 

2 joining sessions 

are completed 

before entry into the 

multifamily group. 

Education is delayed 

or absent.

Most steps are 

done in order; 

however, families 

enter multifamily 

groups before 3 

joining sessions 

are completed 

or education is 

provided.

Engagement, all 

3 joining sessions, 

and education are 

completed before 

entry into the 

multifamily group. 

14. Assertive engagement 

and outreach: 

FPE practitioners assertively 

engage all potential consumers 

and family members by phone, 

by mail, or in person (in the 

agency or in the community) 

on an ongoing basis.

FPE practitioners do 

not engage potential 

consumers and 

family members.

FPE practitioners 

engage potential 

consumers and 

family members 

only once as part of 

initial engagement.

FPE practitioners 

engage potential 

consumers and 

family members 

2 times as part of 

initial engagement.

FPE practitioners 

assertively engage 

some potential 

consumers and 

family members 

using all necessary 

means on a timelimited basis.

FPE practitioners 

assertively engage all 

potential consumers 

and family members 

using all necessary 

contact means on an 

ongoing basis.

FPE practitioners 

demonstrate 

tolerance of different 

levels of readiness 

using gentle 

encouragement.
Appendix C: Fidelity Scale and Score Sheet 36 Evaluating Your Program

Score Sheet: Family Psychoeducation Fidelity Scale 

 Date of visit

Agency name

Assessors’ names

Assessor 1 Assessor 2 Consensus

1 Family intervention coordinator

2 Session frequency

3 Long-term FPE

4 Quality of practitioner-consumer-family alliance

5 Detailed family reaction

6 Precipitating factors

7 Prodromal signs and symptoms

8 Coping strategies

9 Educational curriculum

10 Multimedia education

11 Structured group sessions

12 Structured problem-solving

13 Stage-wise provision of services

14 Assertive engagement and outreach

Total score

Items not rated

62-70 = Good implementation

52-61 = Fair implementation

51 and below = Not evidence-based practice 

/ /
Evaluating Your Program 37 Appendix D: Fidelity Scale Protocol

Evaluating Your Program

Appendix D: Family Psychoeducation 

Fidelity Scale Protocol

Evaluating Your Program 39 Appendix D: Fidelity Scale Protocol

Family Psychoeducation Fidelity 

Scale Protocol 

This protocol explains how to rate each item on 

the FPE Fidelity Scale. In particular, it provides 

the following:

n	A definition and rationale for each fidelity 

item. These items have been derived from 

comprehensive, evidence-based literature. 

n	A list of data sources most appropriate for 

each fidelity item (for example, chart review, 

family intervention coordinator interview, FPE 

practitioners, consumers, or families). When 

appropriate, a set of probe questions is provided 

to help you elicit the critical information 

needed to code the item. These questions 

were specifically generated to help you collect 

information from respondents that is free from 

bias such as social desirability.

n	Decision rules that will help score each item 

correctly. As you collect information from various 

sources, these rules will help you determine the 

specific rating to give for each item. 

1. Family intervention coordinator

Definition: One clinical administrator is designated to 

oversee the FPE program for a substantial 

portion of the job (time depends on size of 

program). This person’s role includes 

activities such as the following:

n	 Establishing, monitoring, and automating 

family intake and engagement 

procedures;

n	 Assigning potential FPE consumers 

to FPE practitioners; 

n	 Monitoring and adjusting FPE 

practitioners’ caseloads;

n	 Arranging for training of new staff 

and continuing education of existing 

FPE staff;

n	 Supervising FPE practitioners.

Rationale: Delivery of services to families must be 

subject to accountability and tracking. One 

effective way for agencies to monitor the 

delivery of family services is to create a 

position of family intervention coordinator, 

who would also serve as the contact person 

for FPE services, facilitate communication 

between staff and families, and supervise 

FPE practitioners. 

Sources of information: Before the site visit, determine 

whether the organization has someone who 

has a title of family intervention coordinator 

or its equivalent. During the fidelity visit, 

interview the agency director, family 

intervention coordinator, practitioners, 

consumers, and family members.

Item response coding: The agency director and family 

intervention coordinator are the primary sources of 

information for this item. If other sources do not report 

these responsibilities performed by the coordinator, then 

fidelity assessors should follow up with the agency 

director and family intervention coordinator with 

clarifying questions and documentation (at end of the 

fidelity visit day or in follow-up call). If the program does 

not have a designated position of family intervention 

coordinator (or an equivalent), code the item as “1.” 

If the program has a designated staff member who 

performs all five tasks, code the item as “5.” 

Probe questions

For family intervention coordinators: 

n	 “What is your role in the FPE program? How 

much time do you devote to this? What kinds 

of responsibilities do you have?” [Check who 

performs the tasks specified above.] 

n	 “Can you explain intake procedures, monitoring, 

training schedule, and supervision schedule?”

For FPE practitioners:

n	 “What functions does the family intervention 

coordinator perform?” 

n	 [Read list of five tasks listed above.] “Is anyone 

responsible for these tasks?” 

For consumers and family members: “What 

functions does [family intervention coordinator’s 

name] perform?”
Appendix D: Fidelity Scale Protocol 40 Evaluating Your Program

2. Session frequency

Definition: Families and consumers participate at least 

in biweekly FPE sessions. 

Rationale: It is presumed that families benefit more if 

sessions are offered regularly and predictably. 

Sources of information: Chart review, roster of sessions, 

and interviews with family intervention 

coordinator, FPE practitioners, consumers, 

and family members. 

Item response coding: The primary evidence for coding 

this item would be attendance rosters or a calendar of 

scheduled events, if such documents exist. The 

program should have some way of documenting the 

frequency of FPE sessions. If the documentation 

suggests that the organization provides at least 

biweekly FPE sessions, code the item as “5.” 

Probe questions

For family intervention coordinators: 

n	 “How often are FPE sessions held for family 

members?” 

n	 “Do you have attendance rosters, a calendar of 

events, or other documentation to verify this?”

For FPE practitioners:

n	 “How often are FPE sessions held for family 

members?” 

n	 “Do you have attendance rosters, a calendar of 

events, or other documentation to verify this?”

For consumers and family members: “How often 

are FPE sessions held for family members?” 

3. Long-term FPE

Definition: Families and consumers are provided with 

long-term FPE; specifically, at least one 

family member per consumer participates 

in FPE sessions for at least 9 months.

Rationale: In general, 9 months of biweekly equivalent 

FPE sessions are required for families 

and consumers to learn the necessary 

information and problem-solving skills. 

After completing the program, families and 

consumers may also benefit from booster 

sessions or support groups.

Sources of information: Chart review, roster of sessions, 

and interviews with the family intervention 

coordinator, FPE practitioners, consumers, 

and family members. 

Item response coding: The primary evidence for coding 

this item would be a report containing the number of 

families and consumers completing FPE and how long 

they attended, records of duration of FPE groups, or 

attendance sheets. In the absence of written records, 

the assessment will depend on interviews. Excluding 

dropouts, if there is evidence that 90 percent or less of 

families receive at least 9 months of FPE sessions, 

code the item as “5.” 

Probe questions

For family intervention coordinators or FPE practitioners: 

n	 “How long do family members attend FPE 

before they graduate?” 

n	 “Do you have attendance rosters, a calendar of 

events, or other documentation to verify this?”

For consumers and family members:

n	 “How long have you attended FPE sessions?” 

n	 “How long do you intend to attend?”
Evaluating Your Program 41 Appendix D: Fidelity Scale Protocol

4. Quality of Practitioner-ConsumerFamily Alliance

Definition: FPE practitioners engage family members 

and consumers with warmth, empathy, 

acceptance, and attention to each 

individual’s needs and desires.

Rationale: When the alliance between practitioners, 

consumers, and families is poor, family 

members and consumers are less likely to 

participate fully or at all in FPE programs 

and, as a result, are less likely to benefit 

from FPE services.

Sources of information: Interviews with FPE 

practitioners, family members, and 

consumers. Observations of FPE sessions.

Item response coding: The primary source for rating this 

item is direct observation. This item requires clinical 

judgment and is based on the fidelity assessor’s 

experience. Negative indicators would include 

comments in interviews, FPE sessions, or charts 

expressing judgmental or blaming attitudes. If sources 

consistently indicate a strong alliance for all FPE 

practitioners, code the item as “5.” 

Probe questions

For FPE practitioners:

n	 “How do you establish rapport or develop an 

alliance with family members and consumers?”

n	 “How would you rate or describe your alliance 

with [family and consumer’s name]?” [Select one 

family and consumer with whom the practitioner 

works.]

n	 “Are there any family members or consumers 

with whom you feel your relationship is 

counterproductive or poor?”

For family members and consumers:

n	 “How would you describe your relationship with 

[FPE practitioner’s name]?”

n	 “Do you feel that [FPE practitioner’s name] has 

worked to establish a good relationship with you? 

What has he or she done to connect with you? 

What has he or she done that makes it more 

difficult for you to work with him or her?”

n	 “What would you change about your working 

relationship with [FPE practitioner’s name] to 

make it better?”
Appendix D: Fidelity Scale Protocol 42 Evaluating Your Program

5. Detailed Family Reaction 

Definition: FPE practitioners identify and specify the 

family’s reaction to their relative’s mental 

illnesses. Reactions are emotional and 

behavioral responses (note the distinction 

from coping strategies in Item 8).

Rationale: A core principle of FPE is to help family 

members achieve a basic understanding of 

serious mental illnesses as well as to resolve 

family conflict by listening and responding 

sensitively to each family’s emotional 

distress related to having a relative with 

serious mental illnesses. 

Sources of information: Chart review (especially 

treatment plan) and interviews with FPE 

practitioners, consumers, and families.

Item response coding: If documentation in the treatment 

plan and reports by FPE practitioners, consumers, and 

families corroborate that family reactions are identified 

and specified in joining sessions for 80 percent or more 

of involved families, code the item as “5.” 

Probe questions

For practitioners:

n	 “What sorts of issues do you discuss in joining 

sessions?” 

n	 “Do you address how families react emotionally 

or behaviorally to their relatives’ mental illnesses?” 

n	 “What sorts of activities do you engage in to help 

them deal with their reactions?” 

n	 Using a chart for a family member seen by the 

practitioner, ask the practitioner to explain the 

specifics.

For consumers and family members:

n	 “What sorts of issues did you discuss during the 

first couple of FPE sessions?” 

n	 “Earlier in the FPE sessions, did you spend time 

discussing how you felt and reacted about the 

illness?” 

n	 “Did the practitioner lead you in activities to 

help you deal with your feelings and reactions?”

6. Precipitating Factors 

Definition: FPE practitioners, consumers, and families 

identify and specify precipitating factors for 

consumers’ mental illnesses. 

 Rationale: Exploring factors that have precipitated 

relapse in the past is a crucial step to 

developing individualized relapse 

prevention and illness management 

strategies. Involving consumers and families 

as equal partners in planning and delivering 

treatment is a core principle of FPE. 

Sources of information: Chart review (especially 

treatment plan) and interviews with FPE 

practitioners, consumers, and families.

Item response coding: If documentation in the treatment 

plan and reports by FPE practitioners, consumers, 

and families corroborate that precipitating factors 

are identified and specified in joining sessions for 80 

percent or more of involved families and consumers, 

code the item as “5.” 

Probe questions

For FPE practitioners: 

n	 “In joining sessions, do you discuss the 

precipitating factors of the illness with families 

and consumers?” [If yes, “Can you describe the 

process you use to discuss them? Can you show 

me examples?”] 

n	 Using a chart, ask the FPE practitioner 

to explain the specifics.

For consumers and family members:

n	 “Earlier in the FPE sessions, did the FPE 

practitioner identify precipitating factors for 

[your or your relative’s] illness?” [If yes, “Please 

give examples.”]

n	 “Did you discuss how to respond to them once 

you notice these factors? Have you reviewed 

these strategies in later sessions?” 
Evaluating Your Program 43 Appendix D: Fidelity Scale Protocol

7. Prodromal Signs and Symptoms

Definition: FPE practitioners, consumers, and families 

identify and specify prodromal signs and 

symptoms of consumers’ mental illnesses.

Rationale: Exploring consumers’ prodromal signs 

and symptoms is another crucial step 

to developing individualized relapse 

prevention and illness management 

strategies. Involving consumers and families 

as equal partners in planning and delivering 

treatment is a core principle of FPE. 

Sources of information: Chart review (especially 

treatment plan) and interviews with FPE 

practitioners, consumers, and families.

Item response coding: If documentation in the treatment 

plan and reports by FPE practitioners, consumers, 

and families corroborate that prodromal signs and 

symptoms are identified and specified in joining 

sessions for 80 percent or more of involved families, 

code the item as “5.” 

Probe questions

For FPE practitioners: 

n	 “In joining sessions, do you identify prodromal 

symptoms with consumers and families?” 

[If yes, “Can you describe the process you use 

to identify them? Can you give an example?”] 

n	 Using a chart, ask the practitioner to explain 

the specifics.

For consumers and family members:

n	 “Earlier in the FPE sessions, did the FPE 

practitioner discuss the signs that you (or your 

family member) may be becoming symptomatic?”

n	 “What sorts of things were suggested in your 

sessions for recognizing the early signs and 

symptoms of the illness? Please give examples. 

Have you reviewed these suggestions in later 

sessions?”

8. Coping Strategies

Definition: FPE practitioners identify, describe, clarify, 

and teach coping strategies. Coping 

strategies are intentional and thoughtful 

attempts to change behavior or symptoms 

related to mental illnesses (note the 

distinction from family reactions in Item 5).

Rationale: Exploring coping strategies that have 

and have not worked is a crucial step 

to developing individualized relapse 

prevention and illness management 

strategies. Insight into patterns of 

ineffective interactions and behaviors is 

likely to motivate consumers and families 

toward desired change. 

Sources of information: Chart review (especially 

treatment plan) and interviews with FPE 

practitioners, consumers, and families.

Item response coding: If documentation in the treatment 

plan and reports by FPE practitioners, consumers, and 

families corroborate that practitioners help 80 percent 

or more of involved families and consumers to identify, 

describe, clarify, and learn coping strategies in joining 

sessions, code the item as “5.” 

Probe questions

For FPE practitioners:

n	 “Do you identify coping strategies with 

consumers and families?” [If yes, “Can 

you describe the process you use?”] 

n	 Using a chart, ask the FPE practitioner 

to explain the specifics.

For consumers and family members:

n	 “Have you discussed coping strategies? 

What sorts of things did you talk about?” 

n	 “Did you discuss alternative ways of coping 

with [your or your relative’s] illness?” 
Appendix D: Fidelity Scale Protocol 44 Evaluating Your Program

9. Educational Curriculum

Definition: FPE practitioners use a standardized 

curriculum to teach families about mental 

illnesses. The curriculum covers six topics: 

n Psychobiology of the specific 

mental illness;

n Diagnosis; 

n Treatment and rehabilitation;

n Impact of mental illness on the family;

n Relapse prevention; and

n Family guidelines.

Rationale: Effectively teaching families new 

information and skills requires structure 

and systematically using specific evidencebased techniques and strategies. Therefore, 

it is critical that an FPE program has a 

standardized educational curriculum that 

specifies what is taught and how it is taught. 

Sources of information: Curriculum review, schedule 

of completed session, and interviews with 

family intervention coordinator, FPE 

practitioners, and families. 

Item response coding: If 80 percent or more of involved 

families receive a standardized educational curriculum 

covering all six topics, code the item as “5.” 

Probe questions

For family intervention coordinators:

n	 “Does your program have a standardized 

educational curriculum?” [If yes, “May I have 

a copy for review? How was it developed?”] 

n	 “How do you ensure that the curriculum is 

followed? Do you periodically evaluate and 

update the curriculum? Do you have a schedule 

of completed sessions and their content?”

n	 Ask about each area listed above and whether 

they are included.

For FPE practitioners:

n	 “Do you use a standardized educational 

curriculum?” [If yes, “Are there any areas you 

teach differently from the curriculum?”] 

n	 “Do you have a schedule of completed sessions 

and their content?”

n	 Ask about each area listed above and whether 

they are included.

For family members: 

n	 Have you attended a 1-day educational 

workshop? [If yes, “What topics were covered?”] 

n	 Ask about each area listed above. 

n	 “Did the FPE practitioners review these 

educational topic areas with you individually or 

in a group session?”
Evaluating Your Program 45 Appendix D: Fidelity Scale Protocol

10. Multimedia Education 

Definition: Consumers and family members are given 

educational materials about mental illnesses 

in several formats (for example, paper, 

video, and Web sites).

Rationale: Consumers and families benefit from 

receiving educational materials in a variety 

of formats. Some people may be more likely 

to watch a video or search a website than to 

read the same information in a document.

Sources of information: Review of educational materials 

and interviews with the family intervention 

coordinator, FPE practitioners, and families.

Item response coding: If educational materials are 

provided to families and consumers in all three 

formats, code the item as “5.” 

Probe questions

For family intervention coordinators and FPE 

practitioners:

n	 Ask to see the materials. 

n	 “Do you provide educational materials to 

families and consumers? How many families 

and consumers on your caseload or in your FPE 

program have received educational materials?”

n	 “Can you give or show me examples or the 

types of materials that you give to families 

and consumers?” 

For family members and consumers:

n	 What types of educational materials have 

you received through the FPE program?” 

[If they suggest only written materials have 

been provided, “Have you ever been offered 

or given videos, Web site addresses, or material 

in other formats?”

11. Structured Group Sessions

Definition: FPE practitioners adhere to a structured 

procedure that includes:

n	 Beginning socialization;

n	 Review the last session’s action plan;

n	 Go-round;

n	 Selection of a single problem; 

n	 Structured problem-solving; and

n	 Ending with socialization.

Rationale: Families and consumers benefit from 

structured sessions that follow a predictable 

pattern. FPE practitioners should establish 

a clear agenda, goals, and expectations for 

each FPE session.

Sources of information: Observation of FPE multifamily 

group sessions and interviews with family 

intervention coordinator, FPE practitioners, 

consumers, and families.

Item response coding: If FPE multifamily group sessions 

include all six components listed above, code the item 

as “5.” 

Probe questions

For family intervention coordinators 

and FPE practitioners:

n	 Can you describe the typical FPE multifamily 

group session?” 

For consumers and family members:

n	 “Can you describe what you do at the beginning 

of each multifamily group session? In the 

middle? At the end?”

n	 “Does the FPE practitioner seem to have 

a structured approach to each session?”

n	 “Is it clear to you what will be accomplished 

in each session?”
Appendix D: Fidelity Scale Protocol 46 Evaluating Your Program

12. Structured Problem-Solving

Definition: FPE practitioners use a standardized 

approach to help consumers and families 

with problem-solving, which includes:

n	 Define the problem; 

n	 Generate solutions;

n	 Discuss the advantages and disadvantages 

of each solution;

n	 Choose the best solutions; 

n	 Form an action plan; and 

n	 Review the action plan.

Rationale: Studies show that collaborative and 

structured problem-solving techniques 

involving setting realistic goals and priorities 

and breaking goals into small behavioral 

steps are effective in improving consumers’ 

functioning and families’ coping. 

Sources of information: Observation of FPE multifamily 

group sessions and interviews with family 

intervention coordinator, FPE practitioners, 

consumers, and families.

Item response coding: If all six components of structured 

problem-solving were used, code the item as “5.” 

Probe questions

For family intervention coordinators and FPE 

practitioners:

n	 “Do you focus on problem-solving in multifamily 

groups?” [If yes, “What strategies do you use? 

Do you follow the same process during every 

session?”]

n	 Listen for the list of six components given above. 

If a component is omitted, probe for whether it 

is included.

For the family members and consumers: 

n	 In the multifamily groups, do you discuss how to 

address problems that may arise?” [If yes, “What 

sorts of activities do you do in the sessions to 

work on problems you may be having? Do you 

ever generate plans of action? Is it a step-by-step 

procedure? Can you describe the steps?”]
Evaluating Your Program 47 Appendix D: Fidelity Scale Protocol

13. Stage-Wise Provision of Services

Definition: FPE services are provided in the following 

order: 

1. Engagement;

2. Three or more joining sessions;

3. Educational workshop; and

4. Multifamily group. 

Rationale: FPE is most effective if all components of 

the evidence-based model are followed in 

order. Effective FPE programs ensure that 

consumers and families are well informed 

about the practice, establish a strong 

working alliance, receive a standardized 

educational curriculum, and develop clear 

treatment goals before entering into the 

multifamily group. 

Sources of information: Chart review and interviews 

with family intervention coordinator, FPE 

practitioners, consumers, and families.

Item response coding: If sources corroborate that 

engagement, joining sessions, and the educational 

workshop are completed in a step-wise manner before 

entering into the multifamily group, code the item 

as “5.” 

Probe questions

For family intervention coordinators and FPE 

practitioners:

n	 “How do you engage consumers and families 

who would benefit from FPE?”

n	 “Do you provide joining sessions for consumers 

and families?” [If yes, “How many joining 

sessions has each consumer and family on your 

caseload had? What kind of topics do you cover 

in your joining sessions?”] 

n	 “Did you offer a 1-day educational workshop? 

When was it offered? How many consumers 

and families attended? Did all the attendees 

complete three or more joining sessions before 

participating in the workshop?” 

n	 “When did the multifamily group begin? Did 

all group participants complete three or more 

joining sessions and participate in the workshop 

before the group began?” 

For consumers and family members:

n	 Ask if he or she has received each of the four 

services. Probe further about the timeframe 

and content of each service.

n	 “Did you feel that you had a good understanding 

of FPE before the multifamily group began?” 
Appendix D: Fidelity Scale Protocol 48 Evaluating Your Program

14. Assertive Engagement and Outreach

Definition: FPE practitioners assertively engage all 

potential consumers and family members by 

phone, by mail, or in person (in the agency 

or in the community) on an ongoing basis. 

Rationale: All consumers and families who may benefit 

from FPE should be educated about the 

practice so that they can make informed 

decisions about participation. Effective FPE 

programs are flexible in meeting the needs 

of individual families and consumers and 

use a variety of means for reaching out to 

them. Assertive engagement and outreach 

is also crucial in overcoming barriers to 

participation such as stigma and 

hopelessness. 

Sources of information: Chart review and interviews 

with family intervention coordinator, FPE 

practitioners, consumers, and families.

Item response coding: If FPE practitioners actively 

engage all potential consumers and family members 

through all necessary means on an ongoing basis, code 

the item as “5.” 

Probe questions

For family intervention coordinators and FPE 

practitioners:

n	 “How do you engage consumers and families 

who would benefit from FPE?” 

n	 “How do you engage hard-to-reach consumers 

and family members? For example, some 

consumers may not have a phone number to 

contact. Or, you may not be able to reach some 

family members during your office hours 

because they work.” 

n	 “What would you do if a consumer or a family 

member told you he or she was not ready 

for FPE?” 

n	 “What do you do with families who don’t show 

up for treatment? What about families who 

drop out of treatment? How do you engage 

or re-engage these families?”

For consumers and family members:

n	 “How did you come to participate in this FPE 

program? Did the program do a good job in 

helping you understand FPE, explore your 

expectations about the program, and make 

an informed decision about participating?” 

n	 “Have you ever felt discouraged or ambivalent 

about participating in FPE or stopped showing 

up for sessions?” [If yes, “What did the FPE 

practitioner do to re-engage you in FPE?”]

n	 “How do you feel about the availability of 

your FPE practitioner? Do you feel that your 

practitioner actively reaches out to you?” 

[If yes, “How does he or she do so?”]
Evaluating Your Program 49 Appendix E: General Organizational Index

Evaluating Your Program

Appendix E: General Organizational Index 

and Score Sheet

Evaluating Your Program 51 Appendix E: General Organizational Index

General Organizational Index

1 2 3 4 5

G1. Program philosophy

Committed to clearly articulated 

philosophy consistent with specific 

evidence-based practice (EBP) model, 

based on these five sources:

n Program leader

n Senior staff (for example, executive 

director, psychiatrist)

n Practitioners providing the EBP

n Consumers and families receiving EBP

n Written materials (for example, 

brochures)

No more than 

1 of 5 sources 

shows clear 

understanding 

of program 

philosophy.

OR

All sources 

have numerous 

major areas of 

discrepancy.

2 of 5 sources 

show clear 

understanding 

of program 

philosophy.

OR

All sources 

have several 

major areas of 

discrepancy.

3 of 5 sources 

show clear 

understanding 

of program 

philosophy.

OR

Sources mostly 

aligned to 

program 

philosophy, but 

have 1 major area 

of discrepancy.

4 of 5 sources 

show clear 

understanding 

of program 

philosophy.

OR

Sources mostly 

aligned to 

program 

philosophy, 

but have 1 or 2 

minor areas of 

discrepancy.

All 5 sources 

show clear 

understanding 

and commitment 

to program 

philosophy for 

specific EBP. 

*G2. Eligibility or consumer 

identification

All consumers with serious mental 

illnesses in the community support 

program, crisis consumers, and 

institutionalized consumers are screened 

to determine if they qualify for EBP 

using standardized tools or admission 

criteria consistent with EBP. Also, agency 

systematically tracks number of eligible 

consumers.

20% of 

consumers receive 

standardized 

screening and/

or agency 

DOES NOT 

systematically 

track eligibility.

21–40% of 

consumers receive 

standardized 

screening 

and agency 

systematically 

tracks eligibility.

41–60% of 

consumers receive 

standardized 

screening 

and agency 

systematically 

tracks eligibility.

61–80% of 

consumers receive 

standardized 

screening 

and agency 

systematically 

tracks eligibility.

>80% of 

consumers receive 

standardized 

screening 

and agency 

systematically 

tracks eligibility.

*G3. Penetration

Maximum number of eligible consumers 

served by EBP, as defined by the ratio:

Number of consumers receiving EBP

Number of consumers eligible for EBP

Ratio .20 Ratio .21 – .40 Ratio .41 – .60 Ratio .61 – .80 Ratio > .80

* These two items coded based on all consumers with serious mental illnesses at the site or sites where EBP is being implemented; all other 

items refer specifically to those receiving the EBP.

Total number of consumers in target population

Total number of consumers eligible for EBP % % eligible:

Total number of consumers receiving EBP Penetration rate
Appendix E: General Organizational Index 52 Evaluating Your Program

1 2 3 4 5

G4. Assessment Assessments are 

completely absent 

or completely 

non-standardized.

Pervasive 

deficiencies in 2 

of the following:

n	Standardization;

n	Quality of 

assessments;

n	Timeliness; and

n	Comprehensiveness.

Pervasive 

deficiencies in 1 

of the following: 

n	Standardization;

n	Quality of 

assessments;

n	Timeliness; and

n	Comprehensiveness.

61%-80% of 

consumers receive 

standardized, 

high-quality 

assessments at 

least annually.

OR

Information is 

deficient for 1 

or 2 assessment 

domains.

More than 80% 

of consumers 

receive 

standardized, 

high-quality 

assessments, the 

information is 

comprehensive 

across all 

assessment 

domains, and it 

is updated at least 

annually.

Full standardized assessment of all 

consumers who receive EBP services. 

Assessment includes the following:

n	History and treatment of medical, 

psychiatric, substance use disorders 

n	Current stages of all existing disorders 

n	Vocational history 

n	Any existing support network 

n	Evaluation of biopsychosocial 

risk factors

G5. Individualized treatment plan 20% of 

consumers 

EBP serves 

have explicit 

individualized 

treatment plan, 

related to EBP, 

updated every 

3 months.

21–40% 

of consumers 

EBP serves 

have explicit 

individualized 

treatment plan, 

related to EBP, 

updated every 

3 months.

41–60% of 

consumers 

EBP serves 

have explicit 

individualized 

treatment plan, 

related to EBP, 

updated every 

3 months.

OR

Individualized 

treatment plan 

updated every 

6 months for all 

consumers.

61–80% of 

consumers 

EBP serves 

have explicit 

individualized 

treatment plan, 

related to EBP, 

updated every 

3 months.

More than 80% 

of consumers 

EBP serves 

have explicit 

individualized 

treatment plan 

related to EBP, 

updated every 

3 months.

For all EBP consumers, an explicit, 

individualized treatment plan exists 

related to the EBP that is consistent with 

assessment and updated every 3 months

G6. Individualized treatment 20% of 

consumers EBP 

serves receive 

individualized 

services meeting 

goals of EBP.

21–40% of 

consumers EBP 

serves receive 

individualized 

services meeting 

goals of EBP.

41–60% of 

consumers EBP 

serves receive 

individualized 

services meeting 

goals of EBP.

61–80% of 

consumers EBP 

serves receive 

individualized 

services meeting 

goals of EBP.

More than 80% 

of consumers EBP 

serves receive 

individualized 

services meeting 

goals of EBP.

All EBP consumers receive individualized 

treatment meeting goals of EBP

G7. Training 20% of program 

staff receive 

standardized 

training annually.

21–40% 

of program 

staff receive 

standardized 

training annually.

41–60% 

of program 

staff receive 

standardized 

training annually.

61–80% 

of program 

staff receive 

standardized 

training annually.

More than 80% 

of program 

staff receive 

standardized 

training annually.

All new program staff receive 

standardized training in EBP (at least 

a 2-day workshop or equivalent) within 

2 months after hiring. Existing program 

staff receive annual refresher training 

(at least 1-day workshop or equivalent).

G8. Supervision 20% of EBP 

practitioners 

receive 

supervision.

21–40% of EBP 

practitioners 

receive weekly 

structured, 

consumercentered 

supervision.

OR

All EBP 

practitioners 

receive informal 

supervision. 

41–60% of EBP 

practitioners 

receive weekly 

structured, 

consumercentered 

supervision.

OR

All EBP 

practitioners 

receive monthly 

supervision.

61–80% of EBP 

practitioners 

receive weekly 

structured, 

consumercentered 

supervision. 

OR

All EBP 

practitioners 

receive 

supervision 2 

times a month.

More than 

80% of EBP 

practitioners 

receive 

structured weekly 

supervision, 

focusing 

on specific 

consumers, 

in sessions that 

explicitly address 

EBP model and 

its application.

EBP practitioners receive structured, 

weekly supervision (group or individual 

format) from a supervisor experienced 

in particular EBP. Supervision should 

be consumer-centered and explicitly 

address EBP model and its application 

to specific consumer situations.
Evaluating Your Program 53 Appendix E: General Organizational Index

1 2 3 4 5

G9. Process monitoring No attempt 

at process 

monitoring 

is made.

Informal process 

monitoring is 

used at least 

annually.

Process monitoring is deficient 

on 2 of these 3 

criteria:

n	Comprehensive and 

standardized;

n	Completed 

every 6 

months; and 

n	Used to guide 

program 

improvements.

OR

Standardized 

monitoring done 

annually only.

Process 

monitoring is 

deficient on 1 of 

these 3 criteria: 

n	Comprehensive and 

standardized;

n	Completed 

every 6 

months; and 

n	Used to guide 

program 

improvements.

Standardized 

comprehensive 

process 

monitoring 

occurs at least 

every 6 months 

and is used to 

guide program 

improvements.

Program leaders and administrators 

monitor process of implementing EBP 

every 6 months and use the data to 

improve the program. Monitoring 

involves a standardized approach, for 

example, using fidelity scale or other 

comprehensive set of process indicators.

G10. Outcome monitoring No outcome 

monitoring 

occurs.

Outcome 

monitoring 

occurs at least 1 

time a year, but 

results are not 

shared with EBP 

practitioners.

Standardized 

outcome monitoring occurs at 

least 1 time a 

year. Results are 

shared with EBP 

practitioners.

Standardized 

outcome 

monitoring occurs 

at least 2 times a 

year. Results are 

shared with EBP 

practitioners. 

Standardized 

outcome 

monitoring 

occurs quarterly. 

Results are 

shared with EBP 

practitioners.

Program leaders and administrators 

monitor outcomes for EBP consumers 

every 3 months and share data with 

EBP practitioners. Monitoring involves 

standardized approach to assessing a key 

outcome related to EBP, for example, 

psychiatric admissions, substance abuse 

treatment scale, or employment rate.

G11. Quality Assurance (QA) No review or 

no committee.

QA committee has 

been formed, but 

no reviews have 

been completed.

Explicit QA review 

occurs less than 

annually. 

OR

QA review 

is superficial.

Explicit QA review 

occurs annually

Explicit review 

occurs every 6 

months by QA 

group or steering 

committee for 

EBP

Agency has QA committee or 

implementation steering committee 

with an explicit plan to review EBP 

or components of the program every 

6 months.

G12. Consumer choice about

service provision

Consumercentered services 

are absent 

(or practitioners 

make all EBP 

decisions).

Few sources agree 

that type and 

frequency of EBP 

services reflect 

consumer choice

Half of the 

sources agree 

that type and 

frequency of EBP 

services reflect 

consumer choice.

Most sources 

agree that type 

and frequency 

of EBP services 

reflect consumer 

choice.

OR

Agency fully 

embraces 

consumer choice 

with 1 exception.

All sources agree 

that type and 

frequency of EBP 

services reflect 

consumer choice.

All consumers receiving EBP services 

are offered choices; EBP practitioners 

consider and abide by consumer 

preferences for treatment when offering 

and providing services. Score Sheet: 

General Organizational Index 
Appendix E: General Organizational Index 54 Evaluating Your Program

Score Sheet: General Organizational Index 

 Date of visit

Agency name

Assessors’ names 

Assessor 1 Assessor 2 Consensus

G1 Program philosophy

 G2 Eligibility or consumer identification

 G3 Penetration

 G4 Assessment 

 G5 Individualized treatment plan

 G6 Individualized treatment

 G7 Training

 G8 Supervision

 G9 Process monitoring

 G10 Outcome monitoring 

 G11 Quality Assurance (QA)

 G12 Consumer choice regarding service provision

Total mean score

/ /
Evaluating Your Program 55 Appendix F: GOI Protocol

Evaluating Your Program

Appendix F: General Organizational Index Protocol

Evaluating Your Program 57 Appendix F: GOI Protocol

General Organizational 

Index Protocol

The General Organizational Index Protocol 

explains how to rate each item of the index. 

In particular, it provides the following:

n	A definition and rationale for each item; and 

n	A list of data sources most appropriate for each 

fidelity item (for example, chart review, program 

leader, practitioners, consumers, and family 

interviews). 

When appropriate, a set of probe questions is 

provided to help you elicit the critical information 

needed to code the fidelity item. These probe 

questions were specifically generated to help you 

collect information from respondents that is 

relatively free from bias, such as social desirability.

Decision rules will help you code each item 

correctly. As you collect information from various 

sources, these rules will help you determine the 

specific rating to give for each item. 

G1. Program Philosophy 

Definition: The program is committed to a clearly 

articulated philosophy consistent with the 

specific evidence-based practice (EBP), 

based on the following five sources:

n	 Family intervention coordinator;

n	 Senior staff (for example, executive 

director, psychiatrists);

n	 FPE practitioners;

n	 Consumers and family members; and

n	 Written materials (for example, 

brochures).

Rationale: In agencies that truly endorse EBPs, staff 

members at all levels embrace the program 

philosophy and practice it in their daily work. 

Sources of information:

Overview: During the site visit, be alert to indicators of 

program philosophy consistent or inconsistent with the 

EBP, including observations from casual conversations, 

staff and consumer activities, etc. Statements that 

suggest misconceptions or reservations about the 

practice are negative indicators, while statements that 

show enthusiasm for and understanding of the practice 

are positive indicators. 

The intent of this item is to gauge the understanding of 

and commitment toward the practice. It is not 

necessary that every element of the practice is 

currently in place (this is gauged by the EBP-specific 

fidelity scale), but rather whether all those involved are 

committed to implementing a high-fidelity EBP.

The practitioners rated for this item are limited 

to those implementing this practice. Similarly, the 

consumers rated are those receiving the practice. 

1. Family intervention coordinator, senior staff, 

and practitioner interviews

At the beginning of the interview, have 

practitioners briefly describe the program. 

n	 “What are the critical ingredients or principles of 

your services?” 

n	 “What is the goal of your program?”

n	 “How do you define [EBP area]?”

2. Consumer interview

n	 “What kind of services do you receive from this 

program?”

n	 Using a layperson’s language, describe to the 

consumer or family the principles of the specific 

EBP area. [Probe if the program offers services 

that reflect each principle.]

n	 “Do you feel the practitioners of this program 

are competent and help you address your 

problems?”

3. Written material review (for example, brochure)

n	 Does the site have written materials on the 

EBP? If not, then rate item down one scale point 

(i.e., lower fidelity).

n	 Does the written material articulate a program 

philosophy that is consistent with the EBP? 
Appendix F: GOI Protocol 58 Evaluating Your Program

Item response coding: The goal of this item is not to quiz 

every practitioner to determine if each can recite every 

critical ingredient. Rather, the goal is to gauge whether 

the understanding is generally accurate and not contrary 

to the EBP. For example, if a senior staff member says, 

“We are having trouble identifying consumers for our 

FPE program since most families are unsupportive,” 

then that would be a red flag for the practice of FPE. 

If all sources show evidence that they clearly understand 

the program philosophy, code the item as “5.” For a 

source type that is based on more than one person (for 

example, practitioner interviews) determine the majority 

opinion when rating whether that source endorses a 

clear program philosophy. Note: If no written material 

exists, then count that source as unsatisfactory.

G2. Eligibility/Consumer Identification

Definition: For EBPs implemented in a mental health 

center: All consumers in the community 

support program, consumers in crisis, and 

those in the hospital are screened using 

standardized tools or admission criteria that 

are consistent with the EBP. 

For EBPs implemented in a service area: 

All consumers within the jurisdiction of the 

service area are screened using standardized 

tools or admission criteria that are consistent 

with the EBP. For example, in New York, 

county mental health administrations are 

responsible for identifying consumers who 

will be served by Assertive Community 

Treatment (ACT) programs. 

The target population refers to all adults 

with serious mental illness (SMI) served by 

the provider agency or service area. If the 

agency serves consumers at multiple sites, 

then assessment is limited to the site or sites 

that are targeted for the EBP. If the target 

population is served in discrete programs 

(for example, case management, residential, 

day treatment), then ordinarily all adults 

with serious mental illnesses are included 

in this definition.

Screening will vary according to the EBP. 

The intent is to identify all who could 

benefit from the EBP. In every case, the 

program should have an explicit, systematic 

method for identifying the eligibility of 

every consumer. Screening typically occurs 

at program admission; programs that are 

newly adopting an EBP should have a plan 

for systematically reviewing consumers who 

are already active in the agency. 

Rationale: Accurately identifying consumers who 

would benefit most from the EBP requires 

routinely reviewing eligibility, based on 

criteria that are consistent with the EBP. 

Sources of information: 

1. Family intervention coordinator, senior staff, 

and practitioner interviews

n	 “Describe the eligibility criteria for your 

program.”

n	 “How are consumers referred to your program? 

How does the agency identify consumers who 

would benefit from your program? Do all new 

consumers receive screening for substance abuse 

or severe mental illness (SMI) diagnosis?”

n	 “What about consumers who are in crisis 

(or institutionalized)?”

n	 Ask for a copy of the screening instrument that 

the agency uses.

2. Chart review

Review documentation of the screening process 

and results.

3. County mental health administrators (where applicable)

If eligibility is determined at the service-area level 

(such as the New York example), then interview the 

people who are responsible for this screening.

Item response coding: This item refers to all consumers 

with SMI in the community support program or its 

equivalent at the sites where the EBP is being 

implemented; it is not limited to consumers who 

receive EBP services only. Calculate this percentage 

and record it on the fidelity scale in the space 

provided. If 80 percent or more of these consumers 

receive standardized screening, code the item as “5.”
Evaluating Your Program 59 Appendix F: GOI Protocol

G3. Penetration

Definition: Penetration is defined as the percentage 

of consumers who have access to an EBP 

as measured against the total number of 

consumers who could benefit from the EBP. 

Numerically, this proportion is defined by––

Number of consumers receiving an EBP

Number of consumers eligible for the EBP

As in the preceding item, the numbers used 

in this calculation are specific to the site or 

sites where the EBP is being implemented. 

Rationale: Surveys have repeatedly shown that people 

with SMI often have a limited access to 

EBPs. The goal of EBP dissemination is not 

simply to create small exclusive programs, 

but to make these practices easily accessible 

within the public mental health system.

Sources of information:

The calculation of the penetration rate depends on the 

availability of the two statistics defining this rate. 

Numerator: The number receiving the service is based 

on a roster of names that the family intervention 

coordinator maintains. Ideally, this total should be 

corroborated with service contact sheets and other 

supporting evidence that the identified consumers are 

actively receiving treatment. As a practical matter, 

agencies have many conventions for defining active 

consumers and dropouts, so that it may be difficult to 

standardize the definition for this item. Use the best 

estimate of the number actively receiving treatment.

Denominator: If the agency systematically tracks 

eligibility, then use this number in the denominator. 

(See the rules listed in G2 to determine the target 

population before using estimates below.) If the agency 

doesn’t track eligibility, then estimate the denominator 

by multiplying the total target population by the 

corresponding percentage based on the literature for 

each EBP. 

According to the literature, the estimates for EBP 

KITs available at this writing should be as follows:

n	 Integrated Treatment for Co-Occurring 

Disorders––40 percent

n	 Supported Employment––60 percent

n	 Illness Management and Recovery––100 percent

n	 Family Psychoeducation––100 percent (some 

kind of significant other)

n	 Assertive Community Treatment––20 percent

Item response coding: Calculate this ratio and record it 

on the fidelity scale. If the program serves more than 

80 percent of eligible consumers, code the item as “5.”
Appendix F: GOI Protocol 60 Evaluating Your Program

G4. Assessment

Definition: All EBP consumers receive standardized, 

high-quality, comprehensive, and timely 

assessments. 

Standardization refers to a reporting format 

that is easily interpreted and consistent 

across consumers. 

High quality refers to assessments that 

provide concrete, specific information that 

differentiates among consumers. If most 

consumers are assessed using identical 

words or if the assessment consists of broad, 

noninformative checklists, then consider 

this to be low quality. 

Comprehensive assessments include the 

following:

n	 History and treatment of medical, 

psychiatric, and substance use disorders; 

n	 Current stages of all existing disorders;

n	 Vocational history;

n	 Any existing support network; and 

n	 Evaluation of biopsychosocial risk factors. 

Timely assessments are those updated 

at least annually.

Rationale: Comprehensive assessment or reassessment is indispensable in identifying 

target domains of functioning that may need 

intervention, in addition to consumers’ 

progress toward recovery. 

Sources of information: 

1. Family intervention coordinator, senior staff, and 

practitioner interviews

n	 “Do you give a comprehensive assessment 

to new consumers? What are the components 

thatxyou assess?”

n	 Ask for a copy of the standardized assessment 

form, if available, and have practitioners go 

through the form. 

n	 “How often do you re-assess consumers?” 

2. Chart review

n	 Look for comprehensiveness of assessment 

by looking at multiple completed assessments 

to see if they address each component of the 

comprehensive assessment every time an 

assessment is performed.

n	 “Is the assessment updated at least yearly?”

Item response coding: If more than 80 percent 

of consumers receive standardized, high-quality, 

comprehensive, and timely assessments, code the 

item as “5.”
Evaluating Your Program 61 Appendix F: GOI Protocol

G5. Individualized Treatment Plan

Definition: For all EBP consumers, an explicit, 

individualized treatment plan exists (even if 

it is not called this) related to the EBP that 

is consistent with assessment and updated 

every 3 months. 

Individualized means that goals, steps 

to reaching the goals, services and 

interventions, and intensity of involvement 

are unique to this consumer. Plans that 

are the same or similar across consumers 

are not individualized. One test is to place 

a treatment plan without identifying 

information in front of supervisors to 

see if they can identify the consumer.

Rationale: Core values of EBP include individualizing 

services and supporting consumers’ pursuit 

of their goals and progress in their recovery 

at their own pace. Therefore, treatment 

plans need ongoing evaluation 

and modification.

Sources of information: 

Note: Assess this item and the next together; that is, 

ask questions about specific treatment plans along 

with questions about the treatment.

1. Chart review (treatment plan)

Using the same charts as examined during 

the EBP-specific fidelity assessment, look for 

documentation of specific goals and consumerbased, goal-setting process.

n	 “Are the treatment recommendations consistent 

with assessment?”

n	 “What evidence is used for a quarterly review?”

2. Family intervention coordinator interview

“Describe the process of developing a treatment 

plan. What are the critical components of a typical 

treatment plan and how are they documented?” 

3. Practitioner interview

When feasible, use the specific charts selected 

above. Ask practitioners to go over a sample 

treatment plan. 

n	 “How do you come up with consumer goals?” 

[Listen for consumer involvement and 

individualization of goals.]

n	 “How often do you review (or follow up on) 

the treatment plan?”

4. Consumer interview

n	 “What are your goals in this program? How did 

you set these goals?” 

n	 “Do you and your practitioners together review 

your progress toward achieving your goals?” 

[If yes, “How often? Please describe the 

review process.”]

5. Team meeting and supervision observation, if available

Observe how the treatment plan is developed. 

Listen especially for discussion of assessment, 

consumer preferences, and individualization 

of treatment. Do they review treatment plans? 

Item response coding: If more than 80 percent of EBP 

consumers have an explicit, individualized treatment 

plan that is updated every 3 months, code the item 

as “5.” 

If the treatment plan is individualized but updated 

only every 6 months, code the item as “3.”
Appendix F: GOI Protocol 62 Evaluating Your Program

G6. Individualized Treatment 

Definition: All EBP consumers receive individualized 

treatment meeting the goals of the EBP. 

Individualized treatment means that steps, 

strategies, services, interventions, and 

intensity of involvement are focused on 

specific consumer goals and are unique 

for each consumer. Progress Notes are 

often a good source of what really goes on. 

Treatment could be highly individualized, 

despite the presence of generic 

treatment plans. 

An example of a low score on this item 

for Integrated Treatment of Co-Occurring 

Disorders is the following: 

 If consumers in the engagement phase of recovery 

are assigned to a relapse prevention group and are 

constantly told they need to quit using, rather than 

using motivational interventions.

Rationale: The key to the success of an EBP is 

implementing a plan that is individualized 

and meets the goals for the EBP for 

each consumer.

Sources of information: 

1. Chart review (treatment plan)

Using the same charts as examined during the 

EBP-specific fidelity assessment, examine the 

treatment provided. Limit the focus to a recent 

treatment plan related to the EBP. Judge whether 

an appropriate treatment occurred during the time 

frame indicated by the treatment plan.

2. Practitioner interview

When feasible, use the specific charts selected 

above. Ask practitioners to go over a sample 

treatment plan and treatment. 

3. Consumer interview

“Tell me about how this program is helping 

you meet your goals.” 

Item response coding: If more than 80 percent of EBP 

consumers receive treatment that is consistent with 

the goals of the EBP, code the item as “5.”

G7. Training

Definition: All new practitioners receive standardized 

training in the EBP (at least a 2-day 

workshop or its equivalent) within 2 months 

after they are hired. Existing practitioners 

receive annual refresher training (at least 

a 1-day workshop or its equivalent).

Rationale: Practitioner training and retraining are 

warranted to ensure that evidence-based 

services are provided in a standardized 

manner, across practitioners and over time. 

Sources of information: 

1. Family intervention coordinator, senior staff, 

and practitioner interviews

n	 “Do you provide new practitioners with 

systematic training for [EBP area]?” [If yes, 

probe for specifics: Mandatory or optional? 

Length? Frequency? Content? Group or 

individual format? Who trains? In-house 

or outside training?]

n	 “Do practitioners receive refresher trainings?” 

[If yes, probe for specifics.]

2. Review training curriculum and schedule, if available

Does the curriculum appropriately cover 

the critical ingredients for [EBP area]?

3. Practitioners interview

n	 “When you first started in this program, did 

you receive a systematic and formal training 

for [EBP area]?” [If yes, probe for specifics: 

Mandatory or optional? Length? Frequency? 

Content? Group or individual format? 

Who trains? In-house or outside training?] 

n	 “Do you receive refresher trainings?” [If yes, 

probe for specifics.]

Item response coding: If more than 80 percent of 

practitioners receive at least yearly, standardized 

training for [EBP area], code the item as “5.”
Evaluating Your Program 63 Appendix F: GOI Protocol

G8. Supervision

Definition: FPE practitioners receive structured, 

weekly supervision from a supervisor 

experienced in the particular EBP. 

The supervision can be either group or 

individual, but CANNOT be peers-only 

supervision without a supervisor. The 

supervision should be consumer-centered 

and explicitly address the EBP model and 

how it applies to specific consumer 

situations. Administrative meetings and 

meetings that are not specifically devoted to 

the EBP do not fit the criteria for this item. 

The consumer-specific EBP supervision 

should be at least 1 hour each week.

Rationale: Regular supervision is critical not only 

for individualizing treatment, but also 

for ensuring the standardized provision 

of evidence-based services.

Sources of information: 

1. Family intervention coordinator, senior staff, 

and practitioner interviews

Probe for logistics of supervision: length, frequency, 

group size, etc. 

n	 “Describe what a typical supervision session 

looks like.”

n	 “How does the supervision help your work?”

2. Team meeting and supervision observation, if available

Listen for discussion of [EBP area] in each 

case reviewed. 

3. Supervision logs documenting frequency of meetings 

Item response coding: If more than 80 percent of FPE 

practitioners receive weekly supervision, code the item 

as “5.”

G9. Process Monitoring

Definition: Family intervention coordinators and 

administrators monitor the process of 

implementing the EBP every 6 months and 

use the data to improve the program. Process 

monitoring involves a standardized approach, 

for example, using a fidelity scale or other 

comprehensive set of process indicators. 

An example of a process indicator would 

be a systematic measurement of how much 

time case managers spend in the community 

instead of in the office. Process indicators 

could include items related to training or 

supervision. The underlying principle is that 

whatever is being measured is related to 

implementing the EBP and is not being 

measured to track billing or productivity.

Rationale: Systematically and regularly collecting 

process data is imperative in evaluating 

program fidelity to EBP. 

Sources of information: 

1. Family intervention coordinator, senior staff, 

and practitioners interviews

n	 “Does your program collect process data 

regularly?” [If yes, probe for specifics. 

Frequency? Who? How (using [EBP area] 

fidelity scale vs. other scales)? etc.]

n	 “Does your program collect data on consumer 

service use and treatment attendance?” 

n	 “Have the process data affected how your 

services are provided?

2. Review of internal reports and documentation, 

if available

Item response coding: If evidence exists that 

standardized process monitoring occurs at least 

every 6 months, code the item as “5.”
Appendix F: GOI Protocol 64 Evaluating Your Program

G10. Outcome Monitoring

Definition: Family intervention coordinators and 

administrators monitor the outcomes of 

EBP consumers every 3 months and share 

the data with FPE practitioners in an effort 

to improve services. Outcome monitoring 

involves a standardized approach to 

assessing consumers. 

Rationale: Systematically and regularly collecting 

outcomes data is imperative in evaluating 

program effectiveness. Effective programs 

also analyze such data to ascertain what is 

working and what is not working and use 

the results to improve the quality of services 

they provide.

Key outcome indicators for each EBP are 

discussed in the EBP KITs. A provisional 

list is as follows:

n	 Integrated Treatment for Co-Occurring 

Disorders—substance use (such as 

the Stages of Treatment Scale);

n	 Supported Employment—competitive 

employment rate;

n	 Illness Management and Recovery— 

hospitalization rates, relapse prevention 

plans, medication compliance rates;

n	 Family Psychoeducation—hospitalization 

and family well-being; and

n	 Assertive Community Treatment— 

hospitalization and housing.

Sources of information: 

1. Family intervention coordinator, senior staff, 

and practitioner interviews

n	 “Does your program have a systematic method 

for tracking outcomes data?” [If yes, probe for 

specifics: How (computerized vs. chart only)? 

How often? Type of outcome variables? 

Who collects data?]

n	 “Do you use any checklist or scale to monitor 

consumer outcome (for example, Substance 

Abuse Treatment Scale)?” 

n	 “What do you do with the outcomes data? 

Do your practitioners review the data regularly?” 

[If yes, “How is the review done (for example, 

cumulative graph)?”]

n	 “Have the outcomes data affected how your 

services are provided?” [If yes, “How?”]

2. Review of internal reports and documentation, 

if available

Item response coding: If standardized outcome 

monitoring occurs quarterly and results are shared 

with FPE practitioners, code the item as “5.”
Evaluating Your Program 65 Appendix F: GOI Protocol

G11. Quality Assurance

Definition: The agency’s quality assurance (QA) 

committee has an explicit plan to review the 

EBP or components of the program every 6 

months. The steering committee for the 

EBP can serve this function. 

Good QA committees help the agency in 

important decisions, such as penetration 

goals, placement of the EBP within the 

agency, and hiring and staffing needs. QA 

committees also help guide and sustain the 

implementation by doing the following:

n	 Reviewing fidelity to the EBP model; 

n	 Making recommendations for 

improvement; 

n	 Advocating and promoting the EBP 

within the agency and in the community; 

and 

n	 Deciding on and keeping track of key 

outcomes relevant to the EBP.

Rationale: Research has shown that programs that 

most successfully implement EBPs have 

better outcomes. Again, systematically and 

regularly collecting process and outcomes 

data is imperative in evaluating program 

effectiveness.

Sources of information: 

1. Family intervention coordinator interview

“Does your agency have an established team or 

committee that is in charge of reviewing the 

components of your [EBP area] program?” [If yes, 

probe for specifics. “Who? How? When?”] 

2. QA committee member interview

n	 “Please describe the tasks and responsibilities 

of the QA committee.” [Probe for specifics. 

“What is the purpose? Who? How? When?”] 

n	 “How do you use your reviews to improve the 

program’s services?”

Item response coding: If the agency has an established 

QA or steering committee that reviews the EBP or 

components of the program every 6 months, code the 

item as “5.”
Appendix F: GOI Protocol 66 Evaluating Your Program

G12. Consumer Choice About Service 

Provision

Definition: All consumers who receive EBP services are 

offered a reasonable range of choices 

consistent with the EBP; practitioners 

consider and abide by consumer 

preferences for treatment when they offer 

and provide services.

Choice is defined narrowly in this item to 

refer to services provided. This item does 

not address broader issues of consumer 

choice such as choosing to engage in selfdestructive behaviors.

To score high on this item, it is not sufficient 

that a program offers choices. The choices 

must be consonant with the EBP. So, for 

example, an agency implementing 

Integrated Treatment for Co-Occurring 

Disorders would score low if it only worked 

with consumers who were abstinent. 

A reasonable range of choices means that 

FPE practitioners offer realistic options to 

consumers rather than prescribing only one 

or a couple of choices or dictating a fixed 

sequence or prescribing conditions that 

consumers must complete before becoming 

eligible for a service.

Examples of Relevant Choices 

by EBPs 

Current at this writing 

Integrated Treatment for Co-Occurring 

Disorders 

n Group or individual counseling sessions

n Frequency of treatment

n Specific self-management goals

n Selection of other supporters to be involved

Supported Employment

n Type of occupation

n Type of work setting

n Schedules of work and number of hours

n Whether to disclose

n Nature of accommodations

n Type and frequency of followup supports

Family Psychoeducation

n Consumer readiness for involving family

n Who to involve 

n Choice of problems and issues to work on

Illness Management and Recovery 

n Selection of other supporters to be involved

n Specific self-management goals

n Nature of behavioral tailoring

n Skills to be taught

Assertive Community Treatment 

n Type and location of housing

n Nature of health promotion

n Nature of assistance with financial 

management

n Specific goals

n Daily living skills to be taught

n Nature of medication support

n Nature of substance abuse treatment
Evaluating Your Program 67 Appendix F: GOI Protocol

Rationale: A major premise of EBP is that consumers 

are capable of playing a vital role in 

managing their illnesses and in making 

progress towards achieving their goals. 

Providers accept the responsibility for getting 

information to consumers so that they can 

more effectively participate in treatment.

Sources of information:

1. Family intervention coordinator interview

n	 “Tell us what your program philosophy 

is about consumer choice. How do you 

incorporate consumers’ preferences in 

the services you provide?” 

n	 “What options exist for your services? 

Give examples.” 

2. Practitioner interview

n	 “What do you do when a disagreement occurs 

between what you think is the best treatment 

for consumers and what they want?”

n	 “Describe a time when you were unable to abide 

by a consumer’s preferences.” 

3. Consumer interview: 

n	 “Does the program give you options 

for the services you receive?” 

n	 Are you receiving the services you want?”

4. Team meeting and supervision observation

Look for discussion of service options 

and consumer preferences. 

5. Chart review (especially treatment plan)

Look for documentation of consumer preferences 

and choices. 

Item response coding: If all sources support that type 

and frequency of EBP services always reflect consumer 

choice, code the item as “5.” 

If the agency embraces consumer choice fully except 

in one area (for example, requiring the agency to 

assume representative payeeships for all consumers), 

then code the item as “4.”

Note: Ratings for both scales are based on current 

behavior and activities, not planned or intended 

behavior. 

The standards used for establishing the anchors for the 

fully implemented ratings were determined through a 

variety of expert sources as well as empirical research.

Evaluating Your Program 69 Appendix G: Outcomes Report Form

Evaluating Your Program

Appendix G: Outcomes Report Form

Evaluating Your Program 71 Appendix G: Outcomes Report Form

Outcomes Report Form

Quarter q January, February, March Year

q April, May, June

q July, August, September

q October, November, December

Reported by

Agency Team

About the consumer

Consumer ID Discharge date Date of birth 

 q Male Ethnicity

 q Female Primary diagnosis

What was the consumer’s evidence-based service status on the last day of the quarter?

Unknown Not Eligible Eligible Enrolled

Integrated Treatment for Co-Occurring Disorders q q q q

Supported Employment q q q q

Assertive Community Treatment q q q q

Illness Management and Recovery q q q q

Family Psychoeducation q q q q

In the past 3 months, how often has the consumer... Number of days Number of 

incidents

Been homeless?

Been incarcerated?

Been in a State psychiatric hospital?

Been in a private psychiatric hospital?

Been hospitalized for substance abuse reasons?

/ / / /
Appendix G: Outcomes Report Form 72 Evaluating Your Program

In the past 3 months, how many days was the consumer 

competitively employed? (Use 0 if the consumer has not been 

competitively employed.)

_____________ Days

Was the consumer competitively employed on the last day 

of the reporting period?

q Yes

q No

What was the consumer’s stage of substance abuse treatment 

on the last day of the quarter? Check one.

q Not applicable

q Pre-engagement

q Engagement

q Early persuasion

q Late persuasion

q Early active treatment

q Late active treatment

q Relapse prevention

q In remission or recovery

What was the consumer’s living arrangement on the last day 

of the quarter? Check one.

q Not applicable or unknown 

q Psychiatric hospital 

q Substance abuse hospitalization 

q General hospital psychiatric ward 

q Nursing home 

q Family care home

q Living with relatives (heavily dependent for personal care) 

q Group home

q Boarding house 

q Supervised apartment program 

q Living with relatives (but is largely independent)

q Living independently

q Homeless

q Emergency shelter

q Other (specify): ____________________________ 

What was the consumer’s educational status on the last day 

of the quarter? Check one.

q Not applicable or unknown 

q No educational participation 

q Avocational/Educational involvement 

q Pre-educational explorations 

q Working on General Educational Development (GED) diploma 

q Working on English as Second Language

q Basic educational skills

q Attending vocational school, vocational program, 

apprenticeship, or high school

q Attending college: 1 to 6 hours

q Attending college: 7 or more hours

q Other (specify): ____________________________

What is the consumer’s highest level of education? Check one.

q No high school 

q High school diploma or General Educational Development 

(GED) diploma 

q Some college 

q Associate degree 

q Vocational training certificate 

q Bachelor of Arts or Bachelor of Science

q Master’s degree or Ph.D.
Evaluating Your Program 73 Appendix H: Instructions for the Outcomes Report Form

Evaluating Your Program

Appendix H: Instructions for the Outcomes Report Form

Evaluating Your Program 75 Appendix H: Instructions for the Outcomes Report Form

Instructions for the Outcomes Report Form

Before you fill out the Outcomes Report Form, become familiar with the definitions of the data elements to 

provide consistency among reporters.

Evidence-based service status

What was the consumer’s evidence-based service 

status on the last day of the quarter? Check the 

appropriate boxes according to these definitions:

Eligible: Does the consumer meet the participation 

criteria for a specific EBP? Each EBP has 

criteria for program participation that 

should be used to determine eligibility.

Enrolled: Is the consumer participating in a particular 

EBP service or has the consumer 

participated in the EBP in the past period? 

Note: Aggregate data about eligibility and 

enrollment can be used to determine the 

percent of eligible consumers who 

received services.

Incident reporting

For the following outcomes, record the number of 

days and number of incidents that the consumer spent 

in each category during the reporting period.

Categories:

n	 Been homeless: Number of days that the 

consumer was homeless and how many times 

the consumer was homeless during the reporting 

period. Homeless refers to consumers who lack a 

fixed, regular, and adequate nighttime residence.

n	 Been incarcerated: Number of days and incidents 

that the consumer spent incarcerated in jails or 

in other criminal justice lockups.

n	 Been in a state psychiatric hospital: Number of 

days and incidents that the consumer spent 

hospitalized primarily for treatment of psychiatric 

disorders in a state psychiatric hospital.

General data

Quarter: Check the time frame for the reporting 

period.

Year: Fill in the current year.

Reported by: Fill in the name and title of the person 

who completed the form.

Agency: Identify the agency name.

Team: Write the team name or number.

About the consumer

Consumer ID: Write the consumer ID that is used 

at your agency, usually a name or an 

identifying number. This information will 

be accessible only to the agency providing 

the service.

Discharge date: If the consumer has been discharged 

during this report period, fill in the 

discharge date.

Date of birth: Fill in the consumer’s date of birth 

(Example: 09/22/1950).

Gender: Check the appropriate box.

Ethnicity: Fill in the consumer’s ethnicity.

Primary diagnosis: Write the DSM diagnosis.
Appendix H: Instructions for the Outcomes Report Form 76 Evaluating Your Program

n	 Been in a private psychiatric hospital: Number 

of days and incidents that the consumer spent 

hospitalized primarily for treatment of 

psychiatric disorders in a private 

psychiatric hospital.

n	 Been hospitalized for substance abuse reasons: 

Number of days and incidents that the consumer 

spent hospitalized primarily for treatment of 

substance use disorders, including both public 

and private hospitals whose primary function 

is treating substance use disorders.

Competitive employment

In the past 3 months, how many days was the consumer 

competitively employed? Competitive employment

means working in a paid position (almost always 

outside the mental health center) that would be open 

to all community members to apply. Competitive 

employment excludes consumers working in sheltered 

workshops, transitional employment positions, or 

volunteering. It may include consumers who are selfemployed but only if the consumer works regularly and 

is paid for the work.

Stage of substance abuse treatment

What was the consumer’s stage of substance abuse 

treatment on the last day of the quarter? Record the 

consumer’s stage of substance abuse recovery, 

according to the following nine categories:

n	 Not applicable: No history of substance abuse 

disorder.

n	 Pre-engagement: No contacts with a case 

manager, mental health counselor, or integrated 

treatment specialist.

n	 Engagement: Contact with an assigned case 

manager or counselor, but does not have regular 

contacts. The lack of regular contact implies lack 

of a working alliance.

n	 Early persuasion: Regular contacts with a case 

manager or counselor, but has not reduced 

substance use for more than a month. Regular 

contacts imply having a working alliance and a 

relationship in which substance abuse can be 

discussed.

n	 Late persuasion: Engaged in a relationship with a 

case manager or counselor, is discussing 

substance use or attending a group, and shows 

evidence of reducing use for at least 1 month 

(fewer drugs, smaller quantities, or both). 

External controls (for example, Antabuse) may 

be involved in reduction.

n	 Early active treatment: Engaged in treatment, is 

discussing substance use or attending a group, 

has reduced use for at least 1 month, and is 

working toward abstinence (or controlled use 

without associated problems) as a goal, even 

though consumer may still be abusing.

n	 Late active treatment: Engaged in treatment, has 

acknowledged that substance abuse is a problem, 

and has achieved abstinence (or controlled use 

without associated problems) but for less than 6 

months.

n	 Relapse prevention: Engaged in treatment, has 

acknowledged that substance abuse is a problem, 

and has achieved abstinence (or controlled use 

without associated problems) for at least 6 

months. Occasional lapses, not days of 

problematic use, are allowed.

n	 In remission or recovery: No problems related to 

substance use for more than 1 year and is no 

longer in any type of substance abuse treatment.

Living arrangement

What was the consumer’s living arrangement on the last 

day of the quarter? These data give your agency an 

ongoing record of the consumer’s residential status.

n	 Not applicable or unknown

n	 Psychiatric hospital: Those hospitals, both public 

and private, whose primary function is treating 

mental disorders. This includes state hospitals 

and other freestanding psychiatric hospitals.

n	 Substance use hospitalization: Those hospitals, 

both public and private, whose primary function 

is treating substance use disorders.

n	 General hospital psychiatric ward: Psychiatric wards 

located in general medical centers that provide 

short-term, acute crisis care.
Evaluating Your Program 77 Appendix H: Instructions for the Outcomes Report Form

n	 Nursing home: Facilities that are responsible for 

the medical and physical care of consumers and 

have been licensed as such by the state.

n	 Family care home: Consumers live in single-family 

dwellings with non-relatives who provide 

substantial care. Substantial care is determined 

by the degree to which non-relatives are 

responsible for the daily care of consumers. Such 

things as medication management, 

transportation, cooking, cleaning, restrictions on 

leaving the home, and money management are 

considered. Non-relatives may have guardianship 

responsibilities. If consumers are unable to do 

most daily living tasks without the aid of 

caretakers, consider caretakers to be providing 

substantial care.

n	 Lives with relatives (heavily dependent for personal 

care): Consult consumers and relatives about how 

much family members are responsible for 

consumers’ daily care. An important distinction 

between this status and supervised apartment 

program is to ask, “If the family were not 

involved, would the consumer be living in a 

more restrictive setting?” In assessing the extent 

to which family members provide substantial 

care, consider such things as taking medication, 

using transportation, cooking, cleaning, having 

control of leaving the home, and managing 

money. If consumers are unable to 

independently perform most daily living 

functions, consider family members to be 

providing substantial care.

n	 Group home: A residence that is run by staff who 

provide many functions (shopping, meal 

preparation, laundry, etc.) that are essential to 

living independently.

n	 Boarding house: A facility that provides a place to 

sleep and meals but it is not seen as an extension 

of a mental health agency nor is it staffed with 

mental health personnel. These facilities are 

largely privately run and consumers have a high 

degree of autonomy.

n	 Supervised apartment program: Consumers live 

(fairly independently) in an apartment sponsored 

by a mental health agency. In determining 

whether someone fits this category, look at the 

extent to which mental health staff have control 

over key aspects of the living arrangements. 

Example characteristics of control include 

the following:

The mental health agency signs the lease.

The mental health agency has keys to the 

house or apartment.

Mental health agency staff provides onsite day 

or evening coverage.

The mental health agency mandates that 

consumers participate in certain mental health 

services—medication clinic, day program, 

etc., to live in the house or apartment.

Note: Consumers who receive only case 

management support or financial aid are NOT 

included in this category; they are considered to 

be living independently.

n	 Lives with relatives (but is largely independent): An 

assignment to this category requires having 

information from consumers and families. The 

key consideration relates to the degree to which 

consumers can perform most tasks essential to 

daily living without being supervised by family 

members.

n	 Living independently: Consumers who live 

independently and are capable of self-care, 

including those who live independently with case 

management support. This category also 

includes consumers who are largely independent 

and choose to live with others for reasons 

unrelated to mental illness. They may live with 

friends, a spouse, or other family members. The 

reasons for shared housing could include 

personal choice related to culture or financial 

considerations.

n	 Homeless: Consumers who lack a fixed, regular, 

and adequate nighttime residence.

n	 Emergency shelter: Temporary arrangements due 

to a crisis or misfortune that are not specifically 

related to a recurrence of the consumer’s illness. 

While many emergency shelters provide 

emotional support, the need for emergency 

shelter is due to an immediate crisis unrelated to 

the consumer’s mental illness.

n	 Other: Those who complete the form should 

clearly define this status in the space provided.
Appendix H: Instructions for the Outcomes Report Form 78 Evaluating Your Program

Educational status

What was the consumer’s educational status on the last day 

of the quarter? These data give your agency an ongoing 

record of the consumer’s educational status.

n	 Not applicable or unknown

n	 No educational participation: Consumer is not 

participating in educational activities.

n	 Avocational/educational involvement: Organized 

classes in which consumers enroll consistently 

and expect to take part for the purpose of life 

enrichment, hobbies, recreation, etc. These 

classes must be community-based, not run by 

the mental health center. Classes are those in 

which anyone could participate, not just 

consumers. If any of these activities involve 

college enrollment, use the categories below.

n	 Pre-educational explorations: Consumers in this 

status are engaged in educational activities with 

the specific purpose of working toward an 

educational goal. This includes consumers who 

attend a college orientation class with the goal of 

enrolling, meet with the financial aid office to 

apply for scholarships, or apply for admission to 

enroll. This status also includes consumers who 

attend a mental health center-sponsored activity 

focusing on an educational goal (for example, 

campus visits with a case manager to survey the 

location of classrooms; meetings with the case 

manager and college staff to secure 

entitlements).

n	 Working on General Educational Development (GED) 

diploma: Consumers who are taking classes to 

earn their GED diploma.

n	 Working on English as Second Language:

Consumers who are taking classes in English as a 

Second Language in a community setting.

n	 Basic educational skills: Consumers who are taking 

adult educational classes focused on basic skills, 

such as math and reading.

n	 Attending vocational school or apprenticeship, 

vocational program, or attending high school: 

Consumers who are––

Participating in community-based vocational 

schools;

Learning skills through an apprenticeship, 

internship, or in a practicum setting;

Involved in on-the-job training to acquire 

more advanced skills;

Participating in correspondence courses 

which lead to job certification; and

Young adults attending high school.

n	 Attending college: 1 to 6 hours: Consumers who 

attend college for 6 hours or fewer per term. 

This status continues over breaks, etc., if 

consumers plan to continue enrollment. This 

status suggests that consumers regularly attend 

college and includes correspondence, TV, or 

video courses for college credit.

n	 Attending college: 7 or more hours: Consumers 

attend college for 7 or more hours per term. 

This status continues over breaks, etc., if 

consumers plan to continue enrollment.

Regular attendance with expectations of 

completing course work is essential for 

assignment to this status.

n	 Other: Those who complete the form should 

clearly define this status in the space provided.
Evaluating Your Program 79 Appendix I: Assessor Training

Evaluating Your Program

Appendix I: Assessor Training

and Work Performance Checklist

Evaluating Your Program 81 Appendix I: Assessor Training

Assessor Training and Work Performance Checklist

 Assessment date 

Assessor’s name ____________________________________________________________________ _________________________________

First Middle Initial Last Title

Agency visited _______________________________________________________________________________________________________

Agency address _______________________________________________________________________________________________________

Street

 _______________________________________________________________________________________________________

City State ZIP code

EBP assessed _______________________________________________________________________________________________________

Assessor qualifications

Yes

q 1a. Data collection and skills: Assessor’s skills are evidenced by his or her prior work experience, credentials, 

or supervisor’s observations.

q 1b. EBP knowledge: Assessor’s knowledge is evidenced by his or her prior work experience, credentials, 

or passing a knowledge test on a specific EBP.

q 1c. Training: Assessors receive at least 8 hours of systematic training on chart review, interviewing techniques, 

and process assessment.

q 1d. Shadowing: Assessors complete at least 1 assessment with an experienced assessor before the first official 

process assessment.

q 1e. Practice rating: Assessors co-rate as practice before being official assessors and agree exactly 

with an experienced assessor on ratings for at least 80% of items.

/5 Subtotal

Data Collection

/ /
Appendix I: Assessor Training 82 Evaluating Your Program

q 2a. Contact and scheduling: With contact person, assessors identify a date convenient to site, explain 

purpose of the assessment, identify information to be assembled ahead of time, and develop specific 

schedule of interviews and assessment activities. 

q 2b. Number of assessors: Two or more assessors are present during the assessment visit and independently 

rate all items. If agency is working with a consultant, assessor may join with consultant to conduct 

assessments.

q 2c. Time management: Sufficient time is allotted and all necessary materials reviewed (2 days for 2 

assessors).

q 2d. Interviewing: Interview all the sources stipulated in the protocol (e.g., for IMR, interviews with the 

program director, 3 ACT team members, and 3 consumers).

q 2e. Completion of documents: Complete score sheet, cover sheet, and any other supplemental documents 

relating to the agency.

q 2f. Documentation supporting rating: Each assessor provides written documentation for evidence 

supporting the rating for each item (e.g., marginal notes). 

q 2g. Chart selection and documentation: Chart selection follows guidelines provided in the protocol 

(e.g., randomized, appropriate type and number of charts). Assessors note discrepancies (e.g., chart 

unavailability).

q 2h. Chart review: Both assessors review all charts and rate them independently.

q 2i. Resolution of discrepancies: When a discrepancy exists between sources (e.g., charts and ACT team 

members), assessors make followup probes with an appropriate informant (typically the ACT leader 

or relevant staff members).

q 2j. Independent ratings: No later than 1 day after the assessment, assessors independently complete scales 

before discussing ratings.

/10 Subtotal

Post-assessment visit

q 3a. Timely consensus: Within 5 working days after the assessment, assessors discuss their ratings 

to determine consensus ratings, identifying any followup information needed. A third assessor 

(e.g., supervisor) may be consulted to resolve difficult ratings. 

q 3b. Inter-rater reliability: Raters agree exactly on ratings for at least 80% of the items. Sources of 

unreliability are discussed with supervisor and strategies developed to reduce future unreliability. 

q 3c. Follow up on missing data: If followup calls are needed to complete an item, information obtained 

within 3 working days.

/3 Subtotal

Comprehensive report writing
Evaluating Your Program 83 Appendix I: Assessor Training

q 4a. Documentation of background information:

n List recipients of report in the header (usually the agency director and ACT leader; 

add others by mutual agreement).

n Summarize time, place, and method.

n Provide background about scale.

q 4b. Site and normative fidelity data: Provide a table with item-level (consensus) scores, along with 

normative data (if available). Normative data include both national and State norms. In this table, provide 

comparative site data from prior assessments. On second and later assessments, provide a graph of global 

fidelity ratings over time for the site (trend line).

q 4c. Quantitative summary: Provide narrative summary of quantitative data. List strengths and weaknesses.

q 4d. Score interpretations:

n Interpret overall score.

n Include other pertinent observations. 

n Provide overall summary.

n Provide opportunity for site to comment and clarify.

q 4e. Report editing: If agency is working with a consultant, consultant may write report. Assessor and 

supervisor review draft of the report before it is submitted to the agency.

/5 Subtotal

Report submission and followup

q 5a. Timely report: Report sent to agency director within 2 weeks of visit.

q 5b. Follow up on report: If agency is working with a consultant, consultant discusses report with designated 

agency staff within 1 month of assessment.

/2 Subtotal

Quality control

q 6. Quality control: Supervisor reviews assessments and gives feedback, as necessary, to assessors. 

Depending on skill level of assessors, supervisor periodically accompanies assessors on assessment 

for quality assurance purposes. 

/1 Subtotal

/27 Total — Add the subtotals.


HHS Publication No. SMA-09-4422

Printed 2009

26173.0709.7765020404