Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

Health Care Anti-Fraud Volunteer Project 

Performance Measures 

JUNE GIBBS BROWN 

Inspector General 

APRIL1998 

OEI-02-97-00520 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through 

a nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI’s New York regional office prepared this report under the direction of John I. Molnar, 

Regional Inspector General. Principal OEI staff included: 

REGION HEADQUARTERS 

Miriam Gareau Susan Burbach, Program Specialist 

To obtain copies of this report, please call the New York Regional Office at 2 12-264-1998. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SU’MMARY 

PURPOSE 

To assist the Administration on Aging in the development of performance measures for the 

Health Care Anti-Fraud, Waste, and Abuse Community Volunteer Demonstration Program. 

BACKGROUND 

Congress authorized the Health Care Anti-Fraud, Waste, and Abuse Community Volunteer 

Demonstration Program with the Omnibus Consolidated Appropriation Act of 1997 (P.L. 104-

208) in order to bring more resources to bear on the effort to curb losses to Medicare. To fund 

this program the Senate Report (104-368) directed that $2,000,000 be transferred to the 

Administration on Aging (AoA) from the Health Care Financing Administration’s research and 

demonstration budget. These funds would support different approaches to recruit and train 

retired professionals as local, volunteer resources and educators. The Senate Report directed 

AoA to consult with the Office of Inspector General (OIG) and to develop outcome measures to 

test the effectiveness of different approaches. In May of 1997, AoA awarded funds to 12 

organizations to implement these projects. The funds went to two area agencies on aging, six 

state units on aging, and four private aging organizations. 

DEVELOPING PERFORMANCE MEASURES 

In the first step in developing the performance measures, we reviewed the funded grant 

applications in order to identify commonly proposed performance measures and activities 

(Appendix A). We were able to do this because grantees had been asked to discuss outcomes in 

their grant applications. Based on that analysis, we developed a logic model (Appendix B) for 

the projects that described the inputs, outputs and outcomes. Based on the proposed outcome 

measures and logic model, we developed a set of draft performance measures (Appendix C). 

In the next step, we shared the logic model and draft performance measures with the projects for 

review. Then we called each project to discuss their experiences in implementing the grant and to 

get their assessment of the appropriateness and feasibility of collecting data for each of the draft 

performance measures. Based on their input we revised the performance measures (Page 6). 

The next step is to implement these performance measures. OIG and AOA staff will develop a 

reporting format; OIG will then pre-test it by asking projects to supply performance data. The 

OIG will issue further reports, as appropriate, tracking implementation and reporting on 

outcomes. 

i 
TABLE OF CONTENTS 

PAGE 

EXECUTIVE SUMMARY ..................................................... i 

INTRODUCTION ............................................................ . 

FINDINGS .................................................................. . 

Review of Approved Applications . . . . . . . O. . O. . . . . . . . . . . . . . . . . . . o. . a. D. . . . . . .3 

Early Implementation . . . . . . . . . . . . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . .4 

Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 

CONCLUSION AND NEXT STEPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 

APPENDICES 

A: Summary of Proposed Project Activities ................................ A- 1 

B:LogicModel ..................................................... ..B- 1 

C: Discussion Draft Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-l 
INTRODUCTION 

PURPOSE 

To assist the Administration on Aging in the development of performance measures for the 

Health Care Anti-Fraud, Waste, and Abuse Community Volunteer Demonstration Program. 

BACKGROUND 

Health Care Anti-Fraud Volunteer Projkcts: Congress authorized the Health Care AntiFraud, Waste, and Abuse Community Volunteer Demonstration Program in the Omnibus 

Consolidated Appropriation Act of 1997 (P.L. 104-208) in order to bring more resources to bear 

on the effort to curb losses to Medicare. The Senate Committee believed that thousands of retired 

accountants, health professionals, investigators, teachers, and others could serve as community 

volunteers in this effort. More specifically, these retired professionals, with appropriate training, 

would be asked to assist other Medicare beneficiaries in detecting and reporting fraud, waste, and 

abuse. Because the language for this program was introduced by Senator Tom Harkin of Iowa, 

these grants are commonly referred to as “Harkin Projects.” 

To fund this program the Senate Report (104-368) directed that $2,000,000 be transferred to the 

Administration on Aging (AoA) from the Health Care Financing Administration’s research and 

demonstration budget. These funds would support different approaches to recruit and train 

retired professionals as local, volunteer resources and educators. Additionally, the Senate Report 

directed AoA to consult with the Office of Inspector General (OIG) and the General Accounting 

Office in implementing the program, and to develop outcome measures to test the effectiveness 

of different approaches. 

In May of 1997, AoA awarded funds to 12 organizations to implement these projects. The funds 

went to two area agencies on aging, six state units on aging, and four private aging organizations. 

Different types of grantees were funded to support Congress’ desire to test different approaches 

for training volunteers. The application kit for these projects asked prospective projects to define 

a specific service area, develop a collaborative approach and coordinate with other agencies, 

develop outreach strategies, expand current activities to enlist volunteers, and collect information 

on participants and track referrals (suspected cases). 

Outreach: A larger initiative of the Department of Health and Human Services also seeks to 

educate Medicare beneficiaries on Medicare fraud. The OIG, in partnership with AoA, the 

Health Care Financing Administration, and the American Association of Retired Persons, is 

planning a nationwide Medicare fraud outreach campaign. The campaign will use mass media to 

1 
encourage Medicare beneficiaries to review their “Explanation of Medicare Benefits” to identify 

possible instances of fraud or abuse and to report them to the OIG Hotline. The projects have 

been invited to participate in the campaign. 

METHODOLOGY 

This inspection was conducted in two phases. In the first phase we reviewed the approved 

project grant applications and reviewed relevant literature on outcome measurement. These 

application reviews identified the proposed activities and proposed outcome measures for each of 

the projects. 

In the second phase of the inspection, we developed a draft logic model and draft performance 

measures. The model and measures were shared with each of the projects for review. We then 

called each of the projects to discuss their progress in implementing their project and their 

reaction to the draft performance measures. Based on their comments we modified the 

performance measures. 

2 
REVIEW OF APPROVED APPLICATIONS 

All of the proposals submitted by the projects met the requirements listed in the application kit. 

However, most projects proposed to go further than the requirements by training individuals 

other than retired professionals and by doing community education activities. Specifically, they 

proposed to train other members of the aging network such as ombudsman staff, health insurance 

counselors, and other AAA staff. See Appendix A. 

Applicants were also asked to discuss outcome measures to be used in evaluating their approach. 

An analysis of the proposed outcome measures from’the funded projects’ applications shows a 

number of common measures. Not surprisingly these include: 

the number of retired professionals who become trainers, 

the number of Medicare beneficiaries trained by the trainers, 

the number of training sessions led by the trainers, 

the results of pre- & post-tests at the training sessions, and 

the numbers of referrals, cases, and recoveries attributable to the project. 

Additionally, since most of the projects planned to do more activities than train retired 

professionals, most of the applications contained additional measures. These performance 

measures include: 

the number of media or public education products developed, 

the number of media spots and public service announcements (PSAs), 

the number of hits to the project’s web site, and 

feedback to hotline callers. 

3 
EARLY IMPLEMENTATION 

As part of our efforts to develop performance measures for the Health Care Anti-Fraud Volunteer 

Projects, we discussed the implementation of each project director. These discussions revealed 

the following issues. 

Varying Stages of Implementation: The twelve projects were in various stages of 

implementation in January 1998. Five projects had begun training volunteers, and trainers for 

two have begun to train Medicare beneficiaries. Among the other projects, four have begun to 

recruit volunteers, while the others are just getting started. All are optimistic about their 

potential to identify fraud and abuse. 

Wrestling with the Term “Professional”:’ The projects understand the intent of the program is 

“to recruit and train retired professionals,” but find it difficult to strictly implement. Projects 

report that they have approached selected organizations for retired professionals in their 

recruitment efforts, but they find it difficult to limit their recruiting to only retired professional 

volunteers. The nature of the aging network at the State and local levels, makes it difficult for 

the projects to turn away volunteers simply because they had not been professionals. Examples 

of farmers, union workers, or professional volunteers were commonly sited. 

Recruiting Within and Outside the Network: Projects are training individuals who are already 

in the aging network, as well as recruiting new volunteers. Current members of the network who 

are likely to be trained are ombudsmen, health insurance counselors, and senior center staff. 

People from outside the network are being recruited through outreach efforts aimed at 

associations for retired individual and at senior centers, as well as through mass media. 

Creating One-on-One and Group Trainers: Health Care Anti-Fraud Volunteer Projects are 

creating two types of trainers. One trainer speaks to groups of seniors about how to review 

Medicare Summary Notices or Explanation of Medicare Benefits, identify instances of fraud, 

and report suspicious claims. The other trainer is similar to the health insurance counselor. This 

trainer works one-on-one, usually in senior center, providing in depth guidance to individuals on 

their Medicare claims. This type of training is totally consistent with the program mandate, but 

will make the measuring of outcomes more difficult. The activities of each type of trainer and 

related outcomes will have to be counted and measured separately. 

Relying on Hotlines for Tracking: In general, projects are relying on their own or the 

HHS/OIG hotline as a way to identify referrals. The referrals are then tracked to identify if any 

cases or any financial settlements arise. Projects that use one-on-one trainers do not require 

hotlines for those referrals, because they can track referrals from the one-on-one meetings. 
PERFORMANCE MEASURES 

The first step in addressing Congress’ directive to measure outcomes is to develop a system of 

performance measures. This is because outcomes cannot be measures in isolation and are a 

function of inputs and outputs. The first step in developing performance measures is to develop 

a logic model for the program (See Appendix B). We did this based on the language in the 

committee report, the application kit, and the approved grant applications. 

The next step was to develop a set of draft performance measures (See Appendix C). The 

measures were based on the logic model and the outcome measures proposed by the projects in 

their applications. We then shared the logic model and draft performance measures with each of 

the projects for review. In our discussion with the projects we asked about the appropriateness of 

each measure and the feasibility of collecting the relevant data. 

Based on our discussions with the projects, we developed the following set of performance 

measure (See Table 1, page 6). We believe that they will meet AoA’s need to monitor 

performance and report on outcomes, while not placing a significant reporting burden on the 

projects. They do not include “pre- and post testing” because many of the projects that had 

proposed such testing found it either difficult to implement or potentially intimidating to 

volunteers and beneficiaries who would have to take the tests. 

These performance measures require some common definitions in order to assure consistent 

implementation among the projects. They are: 

Retired Professional retired professionals recruited (new to the network) and trained 

specifically for the project 

Other any other volunteer recruited and trained by the project 

Community Education all education activities other than group or one-on-one sessions led 

by project trainers 

Media Hit any media hit electronic or print, including PSAs 

Referral suspected case of fraud or abuse 

Case a referral that results in an indictment or settlement 

Projects will be asked to report these performance data in a format to be developed by AoA and 

OIG . The data will be reported on the 12, 18,24, and 30 month anniversary of the initial grant. 

The OIG will prepare a report summarizing and analyzing the data for each of the reporting 

periods. 

5 
...................... ................ 

.............. .............. :...:. ..: 

.............. ............................................................... .............. 

.................. .............. 

.............................................................................. .... 

Table 1 

PERFORMANCE MEASURES 

..:..::............ ...... ................................. ...... ............:t.:.>:.-.’ :.>>:.;.>, ,.)j:.):.:.:.):.:.:.:.:.:.:.:.:.:.:.::,::::::::::.~ :. ............... ........:::.:...:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.~.:.:.:.:.:.:.:.:.:.::.: ....I,,, ...................... ............................ ~ii:~~~~~~~:::liil: ...................... .............. .................::.:.>:.:.:. ......................... ........................... ...::::.:::........... ...... ....................... ... ... ............................................ .......................................... ........................ ................. ~:.~;:..::!j:..j::..:. ................................................. .: ............... ....Y...............:~... :.:::,:~::.::.::::,‘:::::::::::;;::::‘:::::::.:.:.:.:.:.:.:.:. I 

..!... ...................................... ::::::~:.:.~:.):.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.: . ...,.,.,.: ................... .............. 

1 1$ spent on recruiting retired professional I 

~2 / $ spent on recruiting others 

,3 / $ spent on training retired professionals 

4 ! $ spent on training others 

5 I $ spent to support retired professionals 

6 1 $ spent to support others 

I 

7 ~$ spent on community education activities II 

8 / $ spent on tracking system .‘............................................. ........ .............. :::::::.::.::...:.:::::::::::::::::::::::::::.::::::::::::::::::::::.:::.: .........:.:.:.:.:.,:.....................::.:.:.:.:.:.:.:.:.:.:.~.~. :.:.:.:.::..;:::::::: :.,,:,,..~.:.:.:.:.:.:.:.:.:...:.:.:.:.:.:.:.:::.:.::::: ............................................................................................................................................................................

.................. ... ,,,,,,,,~.~~~............................................................................................................................................ ........ ............................................................................................... ::,:,:,:,i:i:i:i:i:i:~‘~~~.~~~~~~~~~~~~~~~~~i:ira::.::::iiiiiiiiii::_ .>:.:?: ................................. 

.............. i:iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii::i:iii:j:i:ii.:iiliri~~:~~~:~ .................... :/::::::::::::. ..... ::.>:.>:.:.:. ....:.::...::::. ...................................................................................... .:.:.:.:.:.:.:.:.:.:.....................................................................................................

........................................................................... ........ .............. ................................. I

............................................................::.:.:.:.:.:.:.~:.):.:.:.:.:.:.:.:.:.:.:.:.:.:::.::.:.:.:.:.:.:.:.:.:.:.:.:.: .. :......................................................................................................................................................................... ,:.:.:.,.::.:.:.:.:. ..............

9 # of retired professional trained 

10 1# of other people trained 

11 !

i # of public service announcements II 

12 ~# of community education activities II 

13 I # of group sessions for beneficiaries led by retired professionals 

14 / # of group session for beneficiaries led by others 

15 / #ofb eneficiaries who attended the sessions led by retired professionals II 

16 ~# of beneficiaries who attended the sessions led by others 

17 j # of one-on-one encounters led by retired professionals 

18 I # of one-on-one encounters led by others II 

19 1 estimated # of people reached by media hits I

1 

20 / estimated # of people reached by community education 

21 i # of referrals attributable to the project 

I 

22 j # of cases attributable to the project II 

23 1 $ recouped attributable to the project II 

6 
CONCLUSION AND NEXT STEPS 

Based on discussions with the projects and a review of the applications and relevant literature, 

we believe this set of performance measures will meet AoA’s need to monitor the Health Care 

Anti-Fraud Volunteer Projects’ performance and report on outcomes, while not placing a 

significant reporting burden on the individual projects. 

Two steps must be taken in order to implement these performance measures: 

1. OIG and AOA staff will develop a reporting format for the performance measures; and 

2. OIG will pre-test the reporting format by asking projects to supply performance data. 

7 
SUMMARY OF PROPOSED PROJECT ACTIVITIES 

I 

PROJECT ~Total $ i Federal $ i Train 1PSAs 1 800 Web 

1 (x000) ’ (x000) / Trainers I # Site 

I 

1 

MD SUA 

Baltimore, MD 

Suburban AAA 

CWAG ~ 173 ~ 127 ~ X ~ ~ x ~ x 

Madison, WI I 1 ! I

/ / 

MN SUA 239 ~ 178 1 X 

Ix! ~ St. Paul, MN I j (multi) i 1 

1 I 

IA SUA 349 i 187 / x 

Ix ‘x; 

Des Moines, IA 1 

I 

Dist 3 AAA 158 / 117 j x /x /x / 

farrensburg, MO 1 I (multi) I I 

I 

CHA i 289 188 I x / x 1 x 

San Mateo, CA / j (multi) j I 

1 -I 

HI SUA 141 ~ 100 ~ x ix 1x 

Honolulu, HI i (multi) i 

r TOTAL ; 2,755 i 1,935 / 12 / 8 9 j 4 

multi = project will train multiple types of trainers 

A-l 
APPENDIX B 

LOGIC MODEL 

The logic model of the Health Care Anti-Fraud Volunteer Projects flows from the mandate from 

Congress, and includes the resources, the activities, the accomplishments, and the effect of the 

projects. “Program” is used to refer to the entire effort, while “project” refers to individual 

grantees. Activities not included in the mandate are presented in italics. 

MANDATE 

To recruit and train retired professionals to serve as volunteer expert resources 

and educators for other Medicare beneficiaries 

To test different training models 

INPUT- What resources support the program? 

AoA funding, grantee funds, support from the aging network in the 

project’s service area (probably unmeasurable) 

PROCESS- What do the projects do? 

recruit retired professionals and others 

train and support trainers 

count and track referrals 

design community education program 

OUTPUT- What do the projects accomplish? 

retired professionals trained as trainers 

others trained as trainers 

system developed to identify and track referrals 

PSAs and community education programs 

B-l 
OUTCOME- What is the effect of the accomplishments? 

Initial sessions led by trained trainers (by retired professionals and by others) 

Medicare beneficiaries trained or aided (by retired professionals and by 

others) 

increased understanding of examples of Medicare fraud as demonstrated 

by pre- and post-testing 

Long-term calls, referrals, cases, recouped funds (by retired professionals and by 

others) 

individuals reached by public education programs (probably 

unmeasurable) 

B-2 
APPENDIX C 

DISCUSSION DRAFT PERFORMANCE MEASURES 

SHARED WITH THE PROJECTS 

INPUTS 

$ expended to recruit, train, and support retired professionals and $ for other 

trainers 

$ expended on community education activities 

$ expended on tracking system 

OUTPUTS 

# of retired professionals and # of others trained 

# of PSAs and community education activities 

system to identify and track referrals 

OUTCOMES 

Initial # of sessions led by retired professionals and # by other trainers 

# of Medicare beneficiaries trained by the retired professionals and # by other 

trainers 

average percent increase in post-tests scores over pre-tests score 

Long-term # of referrals and cases attributable to both types of trainers 

recouped $ attributable to both types of trainers 

# of individuals reached by PSAs and community education programs 

C-l