Department  of Health  and Human  Services 
OFFICE  OF 
INSPECTOR  GENERAL 
Health  Care  Anti-Fraud  Volunteer  Project 
Performance  Measures 
JUNE  GIBBS  BROWN 
Inspector  General 
APRIL1998 
OEI-02-97-00520  OFFICE  OF INSPECTOR  GENERAL 
The mission  of the Office  of Inspector  General (OIG),  as mandated by Public  Law  95-452,  is to 
protect the integrity  of the Department  of Health  and Human  Services programs  as well  as the 
health and welfare  of beneficiaries  served by them.  This statutory  mission  is carried out through 
a nationwide  program  of audits, investigations,  inspections,  sanctions, and fraud alerts.  The 
Inspector  General informs  the Secretary of program  and management problems  and recommends 
legislative,  regulatory,  and operational  approaches to correct them. 
Office  of Evaluation  and Inspections 
The Office  of Evaluation  and Inspections  (OEI)  is one of several components  of the Office  of 
Inspector  General.  It conducts short-term  management and program  evaluations  (called 
inspections)  that focus on issues of concern to the Department,  the Congress, and the public.  The 
inspection  reports provide  findings  and recommendations  on the efficiency,  vulnerability,  and 
effectiveness  of departmental  programs. 
OEIs New  York  regional  office  prepared this report under the direction  of John I. Molnar, 
Regional  Inspector  General.  Principal  OEI staff included: 
REGION  HEADQUARTERS 
Miriam  Gareau  Susan Burbach,  Program  Specialist 
To obtain copies of this report, please call the New York Regional Office at 2 12-264-1998. 
Reports are also available  on the World  Wide Web at our home page address: 
http://www.dhhs.gov/progorg/oei  EXECUTIVE  SUMMARY 
PURPOSE 
To assist the Administration  on Aging  in the development  of  performance  measures for the 
Health Care Anti-Fraud,  Waste, and Abuse Community  Volunteer  Demonstration  Program. 
BACKGROUND 
Congress authorized  the Health  Care Anti-Fraud,  Waste, and Abuse Community  Volunteer 
Demonstration  Program with  the Omnibus  Consolidated  Appropriation  Act  of 1997 (P.L.  104-
208) in order to bring  more resources to bear on the effort  to curb losses to Medicare.  To fund 
this program  the Senate Report  (104-368)  directed that $2,000,000  be transferred  to the 
Administration  on Aging  (AoA)  from the Health  Care Financing  Administrations  research and 
demonstration  budget.  These funds would  support different  approaches to recruit  and train 
retired professionals  as local, volunteer  resources and educators.  The Senate Report directed 
AoA  to consult with  the Office  of Inspector  General (OIG)  and to develop outcome measures to 
test the effectiveness  of different  approaches.  In May  of 1997, AoA  awarded funds to 12 
organizations  to implement  these projects.  The funds went to two  area agencies on aging, six 
state units on aging, and four private  aging organizations. 
DEVELOPING  PERFORMANCE  MEASURES 
In the first step in developing  the performance  measures, we reviewed  the funded grant 
applications  in order to identify  commonly  proposed performance  measures and activities 
(Appendix  A).  We were able to do this because grantees had been asked to discuss outcomes in 
their grant applications.  Based on that analysis, we developed  a logic  model (Appendix  B) for 
the projects that described the inputs, outputs and outcomes.  Based on the proposed outcome 
measures and logic  model, we developed  a set of draft performance  measures (Appendix  C). 
In the next step, we shared the logic  model and draft performance  measures with  the projects  for 
review.  Then we called each project  to discuss their experiences in implementing  the grant and to 
get their assessment of the appropriateness  and feasibility  of collecting  data for each of the draft 
performance  measures.  Based on their input we revised the performance  measures (Page 6). 
The next step is to implement  these performance  measures. OIG and AOA  staff will  develop a 
reporting  format;  OIG will  then pre-test it by asking projects  to supply performance  data.  The 
OIG will  issue further  reports, as appropriate,  tracking  implementation  and reporting  on 
outcomes. 
i  TABLE  OF  CONTENTS 

PAGE 
EXECUTIVE  SUMMARY  .....................................................  i 
INTRODUCTION  ............................................................  . 
FINDINGS  ..................................................................  . 
Review  of Approved  Applications  . . . . . . . O. . O. . . . . . . . . . . . . . . . . . . o. . a. D. . . . . . .3 
Early  Implementation  . . . . . . . . . . . . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . .4 
Performance  Measures  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 
CONCLUSION  AND  NEXT  STEPS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 
APPENDICES 
A:  Summary  of Proposed Project Activities  ................................  A- 1 
B:LogicModel  .....................................................  ..B- 1 
C: Discussion  Draft  Performance  Measures  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-l  INTRODUCTION 

PURPOSE 
To assist the Administration  on Aging  in the development  of  performance  measures for the 
Health  Care Anti-Fraud,  Waste, and Abuse Community  Volunteer  Demonstration  Program. 
BACKGROUND 
Health  Care  Anti-Fraud  Volunteer  Projkcts:  Congress authorized  the Health  Care Anti-
Fraud, Waste, and Abuse Community  Volunteer  Demonstration  Program  in the Omnibus 
Consolidated  Appropriation  Act of 1997 (P.L.  104-208)  in order to bring  more resources to bear 
on the effort  to curb losses to Medicare.  The Senate Committee  believed that thousands of retired 
accountants,  health professionals,  investigators,  teachers, and others could  serve as community 
volunteers  in this effort.  More  specifically,  these retired professionals,  with  appropriate  training, 
would  be asked to assist other Medicare  beneficiaries  in detecting  and reporting  fraud, waste, and 
abuse.  Because the language for this program  was introduced  by Senator Tom Harkin  of Iowa, 
these grants are commonly  referred to as Harkin  Projects. 
To fund this program  the Senate Report (104-368)  directed that $2,000,000  be transferred  to the 
Administration  on Aging  (AoA)  from the Health  Care Financing  Administrations  research and 
demonstration  budget.  These funds would  support different  approaches to recruit  and train 
retired professionals  as local, volunteer  resources and educators.  Additionally,  the Senate Report 
directed  AoA  to consult with  the Office  of Inspector  General (OIG)  and the General Accounting 
Office  in implementing  the program,  and to develop outcome measures to test the effectiveness 
of different  approaches. 
In May  of 1997, AoA  awarded funds to 12 organizations  to implement  these projects.  The funds 
went to two  area agencies on aging, six state units on aging, and four private  aging organizations. 
Different  types of grantees were funded to support Congress  desire to test different  approaches 
for training  volunteers.  The application  kit for these projects  asked prospective  projects  to define 
a specific  service area, develop a collaborative  approach and coordinate  with  other agencies, 
develop outreach strategies, expand current activities  to enlist volunteers,  and collect  information 
on participants  and track referrals (suspected cases). 
Outreach:  A larger initiative  of the Department  of Health  and Human  Services also seeks to 
educate Medicare  beneficiaries  on Medicare  fraud.  The OIG, in partnership  with  AoA,  the 
Health  Care Financing  Administration,  and the American  Association  of Retired  Persons, is 
planning  a nationwide  Medicare  fraud outreach campaign.  The campaign  will  use mass media to 
1 
 encourage Medicare  beneficiaries  to review  their  Explanation  of Medicare  Benefits  to identify 
possible instances of fraud or abuse and to report them to the OIG Hotline.  The projects  have 
been invited  to participate  in the campaign. 
METHODOLOGY 
This inspection  was conducted  in two phases.  In the first phase we reviewed  the approved 
project  grant applications  and reviewed  relevant  literature  on outcome measurement.  These 
application  reviews  identified  the proposed  activities  and proposed outcome  measures for each of 
the projects. 
In the second phase of the inspection,  we developed  a draft logic  model and draft performance 
measures.  The model  and measures were shared with  each of the projects  for review.  We then 
called each of the projects to discuss their progress in implementing  their project  and their 
reaction  to the draft performance  measures.  Based on their comments we modified  the 
performance  measures. 
2 
 REVIEW  OF APPROVED  APPLICATIONS 
All  of the proposals  submitted  by the projects  met the requirements  listed in the application  kit. 
However,  most projects proposed  to go further  than the requirements  by training  individuals 
other than retired  professionals  and by doing  community  education  activities.  Specifically,  they 
proposed to train  other members of the aging network  such as ombudsman  staff, health insurance 
counselors,  and other AAA  staff.  See Appendix  A. 
Applicants  were also asked to discuss outcome  measures to be used in evaluating  their approach. 
An analysis of the proposed  outcome  measures fromthe  funded projects  applications  shows a 
number of common  measures.  Not  surprisingly  these include: 
the number  of retired professionals  who become trainers, 
the number  of Medicare  beneficiaries  trained by the trainers, 
the number  of training  sessions led by the trainers, 
the results of pre- & post-tests at the training  sessions, and 
the numbers of referrals,  cases, and recoveries attributable  to the project. 
Additionally,  since most of the projects  planned to do more activities  than train retired 
professionals,  most of the applications  contained  additional  measures.  These performance 
measures include: 
the number  of media or public  education  products  developed, 
the number  of media spots and public  service announcements  (PSAs), 
the number  of hits to the projects  web site, and 
feedback to hotline  callers. 
3 
 EARLY  IMPLEMENTATION 
As part of our efforts to develop performance  measures for the Health  Care Anti-Fraud  Volunteer 
Projects, we discussed the implementation  of each project  director.  These discussions revealed 
the following  issues. 
Varying  Stages of Implementation:  The twelve  projects  were in various  stages of 
implementation  in January  1998.  Five projects  had begun training  volunteers,  and trainers for 
two have begun to train Medicare  beneficiaries.  Among  the other projects,  four have begun to 
recruit  volunteers,  while  the others are just  getting  started.  All  are optimistic  about their 
potential  to identify  fraud and abuse. 
Wrestling  with  the Term  Professional:  The projects  understand  the intent  of the program  is 
to recruit  and train retired professionals,  but find it difficult  to strictly  implement.  Projects 
report that they have approached  selected organizations  for retired  professionals  in their 
recruitment  efforts, but they find it difficult  to limit  their recruiting  to only  retired professional 
volunteers.  The nature of the aging network  at the State and local levels, makes it difficult  for 
the projects  to turn away volunteers  simply  because they had not been professionals.  Examples 
of farmers, union workers,  or professional  volunteers  were commonly  sited. 
Recruiting  Within  and Outside  the Network:  Projects are training  individuals  who are already 
in the aging network,  as well  as recruiting  new volunteers.  Current  members of the network  who 
are likely  to be trained are ombudsmen,  health insurance counselors,  and senior center staff. 
People from  outside the network  are being recruited  through  outreach efforts  aimed at 
associations  for retired  individual  and at senior centers, as well  as through  mass media. 
Creating  One-on-One  and  Group  Trainers:  Health  Care Anti-Fraud  Volunteer  Projects are 
creating two  types of trainers.  One trainer  speaks to groups of seniors about how to review 
Medicare  Summary Notices  or Explanation  of  Medicare  Benefits,  identify  instances of fraud, 
and  report  suspicious claims.  The other trainer  is similar  to the health insurance counselor.  This 
trainer works  one-on-one,  usually  in senior center, providing  in depth guidance to individuals  on 
their Medicare  claims.  This type of training  is totally  consistent  with  the program  mandate, but 
will  make the measuring  of outcomes more difficult.  The activities  of each type of trainer  and 
related outcomes will  have to be counted and measured separately. 
Relying  on Hotlines  for  Tracking:  In general, projects  are relying  on their  own or the 
HHS/OIG  hotline  as a way to identify  referrals.  The referrals  are then tracked to identify  if any 
cases or any financial  settlements arise.  Projects that use one-on-one  trainers  do not require 
hotlines  for those referrals, because they can track referrals  from the one-on-one  meetings.  PERFORMANCE  MEASURES 
The first  step in addressing Congress  directive  to measure outcomes  is to develop a system of 
performance  measures.  This is because outcomes cannot be measures in isolation  and are a 
function  of inputs and outputs.  The first step in developing  performance  measures is to develop 
a logic  model  for the program  (See Appendix  B).  We did this based on the language in the 
committee  report, the application  kit, and the approved grant applications. 
The next step was to develop a set of draft performance  measures (See Appendix  C).  The 
measures were based on the logic  model and the outcome measures proposed  by the projects  in 
their applications.  We then shared the logic  model and draft performance  measures with  each of 
the projects  for review.  In our discussion  with  the projects  we asked about the appropriateness  of 
each measure and the feasibility  of collecting  the relevant  data. 
Based on our discussions with  the projects,  we developed the following  set of performance 
measure (See Table  1, page 6).  We believe that they will  meet AoAs  need to monitor 
performance  and report on outcomes, while  not placing  a significant  reporting  burden on the 
projects.  They do not include  pre- and post testing  because many of the projects  that had 
proposed  such testing found it either difficult  to implement  or potentially  intimidating  to 
volunteers  and beneficiaries  who would  have to take the tests. 
These performance  measures require  some common  definitions  in order to assure consistent 
implementation  among the projects.  They are: 
Retired  Professional 	 retired professionals  recruited  (new to the network)  and trained 
specifically  for the project 
Other  any other volunteer  recruited  and trained  by the project 
Community  Education 	 all education  activities  other than group or one-on-one  sessions led 
by project  trainers 
Media  Hit  any media hit electronic  or print,  including  PSAs 
Referral  suspected case of fraud or abuse 
Case  a referral  that results in an indictment  or settlement 
Projects will  be asked to report these performance  data in a format  to be developed  by AoA  and 
OIG . The data will  be reported  on the 12, 18,24,  and 30 month  anniversary  of the initial  grant. 
The OIG will  prepare a report  summarizing  and analyzing  the data for each of the reporting 
periods. 
5 
 Table  1 

PERFORMANCE  MEASURES 

:.>>,:..);j.:>.),: .:.:.):.:.:..:......:...:....:.......:.:...:.........:.............:.............:..............:...............:..:.....:........,...
...:..: .:..t.:....:. :.:...:.>.:..:...::...:.. -..~...  .................................:..:..:.....:.... ....:...:...: .:.:.:.:..:...:.....:....I :,.,:,.:.:.:.~.:.:.:.:.:.:.:.:.:.::.:
~ii:~~~~~~~:::liil: 
.............................................................................................  .. ...........................................................................................!.........:.......:.....:.:..:.:.:..:.:.....:.:..::.:...:.:..>~..... ..:.:.......:.:.......:.~ .... .:.....). :...:...:.....:.......:.........: .......:~...... ..::,..........:~,....;...::,.......:.:...: ..::...! .:j..:.:.......:j...:..::Y... :...:.:... .:....:.......:.......:.........:.............:...~.................. ...........:.....:..:.::...,~... ..:..:.... :.:...:.:.:.:.,..:.:.:.:.:..:...:...:...:...:...:...;...;.: .:...:....:.....:.:.::... :. .: .:
.:.:.:.:...:..:.:.:.:.:.:.:.
I 
1  1$ spent on recruiting  retired professional 

I 

~2  / $ spent on recruiting  others 
,3  / $ spent on training  retired professionals 
4  ! $ spent on training  others 
5  I $ spent to support retired  professionals 
6  1 $ spent to support others 

I 

7  ~$ spent on community  education  activities  II 

8 	 / $ spent on tracking  system
.........:...........:.........., .........:.........,........,.:........,.,........,.:.. .....,.,........,.i.......,.:..........,.i.........:,...........i~.... .:.    ......i....:....~..i.....:.. ..~.i....:.....~~................~............~.............~...................~.............~.... .......~...........~..........~...........~..........~..........~..........~...........~............~............~...... .....~..........~...........:.~..........:...~........:...~........:::...i..:...:..:::..i...r:.....,::..a.i.::...:...::..:i.....i..:.. i::.....i::..:.i.....:.i.:..i:...i:;.:...i.:.i...ii.:...i.i.:.i...i::i...i.i.:.i...i::i...i.i::...i.i::.i..i.: i...:.i:...:.i:..._i.:...i.:.i.. :.:i:....i:...i::...i.:...i::i....i:...:i:...:.:...:::...:i.:....::,:...,i.::...i.:,:.i....,.::...~:.j:....:.:....:i.:..::..:...i:.:..i:.......:.:..::..:..i.~.:.i.:.l:....i.:..:r~.:....i:..:~..:....:..:~.:....:..:::...~:..::...~:...:...:.~.::...:.:.::.~..:..::......::...:..::......::.....:.....:..:..:.....:.....:..:..:..:..:.............................................. 
:.................................................................................................. .... :.........:....:...............:....:....::......... ...............................................................................................................:...........:...........:............:.......,.....::............:::....../:......:::......:......::,.......:......:.:.::.....:...::......:.:......:.::.......:.....:. ::.......:....:...:...... ....:..:.. ......:.. .......:.........~.........:.........)....: .......:.......:.......:.........:.........:.........:.........:.........:.........>.:........:.:...........:..:....?..........:. ...:....:....:.:....:......:.>........:.:...:.......>...:......:..:......:..:......:..:......:.....:.....:.....:.....:.....:.....:.....:.....:.............................................. ...
I
9  # of retired  professional  trained 
10  1# of other people trained 
11  !
i # of public  service announcements 
II 

12  ~# of community  education  activities 
II 

13  I # of group  sessions for beneficiaries  led by retired professionals 
14  / # of group  session for beneficiaries  led by others 
15  / #ofb  eneficiaries  who  attended the sessions led by retired professionals 
II 

16  ~# of beneficiaries  who  attended the sessions led by others 
17  j # of one-on-one  encounters  led by retired professionals 
18  I # of one-on-one  encounters  led by others  II 

19  1 estimated # of people reached by media hits

I
1 

20  / estimated # of people reached by community  education 

21  i # of referrals  attributable  to the project 
I 

22  j # of cases attributable  to the project 
II 

23  1 $ recouped  attributable  to the project  II 

6 
 CONCLUSION  AND  NEXT  STEPS 

Based on discussions  with  the projects  and a review  of the applications  and relevant literature, 
we believe this set of performance  measures will  meet AoAs  need to monitor  the Health  Care 
Anti-Fraud  Volunteer  Projects  performance  and report on outcomes, while  not placing  a 
significant  reporting  burden  on the individual  projects. 
Two  steps must be taken in order to implement  these performance  measures: 
1. OIG and AOA  staff will  develop  a reporting  format for the performance  measures; and 
2.  OIG will  pre-test the reporting  format  by asking projects to supply performance  data. 
7 
 SUMMARY  OF PROPOSED  PROJECT  ACTIVITIES 

I 

PROJECT  ~Total $  i Federal $  i  Train  1PSAs  1 800  Web 

1 (x000)    (x000)  / Trainers  I  #  Site 

I 

1 

MD  SUA 
Baltimore,  MD 
Suburban AAA 
CWAG  ~  173  ~  127  ~  X  ~  ~ x  ~  x 

Madison,  WI  I  1  /  / 

!  I
MN  SUA  239  ~  178  1  X 

St. Paul, MN  I  j (multi)  Ii x!  ~

1 

I 

1 
IA  SUA  349  i  187  Ix  x; 

/  x 
Des Moines,  IA  1 
I 

Dist 3 AAA  158  /  117  j  x  /x  /x  / 
farrensburg,  MO  1  I  (multi)  I  I 

I 

CHA  i  289  188  I  x  /  x  1  x 

San Mateo, CA  /  j (multi)  j  I 

1 -I 

HI  SUA  141  ~  100  ~  x  ix  1x 
Honolulu,  HI  i (multi)  i 

r 
TOTAL  ;  2,755  i  1,935  /  12  /  8  9  j  4 

multi  = project  will  train multiple  types of trainers 
A-l  APPENDIX  B 
LOGIC  MODEL 
The logic  model  of the Health  Care Anti-Fraud  Volunteer  Projects flows  from the mandate from 
Congress, and includes the resources, the activities,  the accomplishments,  and the effect of the 
projects.  Program  is used to refer to the entire effort,  while  project  refers to individual 
grantees.  Activities  not included  in the mandate are presented  in italics. 
MANDATE 
To recruit  and train retired professionals  to serve as volunteer  expert resources 
and educators for other Medicare  beneficiaries 
To test different  training  models 
INPUT- What  resources  support  the program? 
AoA  funding,  grantee funds, support from the aging network  in the 
projects  service area (probably  unmeasurable) 
PROCESS- What  do the projects  do? 
recruit  retired professionals  and others 
train and support trainers 
count and track referrals 
design community  education program 
OUTPUT- What  do the projects  accomplish? 
retired professionals  trained  as trainers 
others trained  as trainers 
system developed to identify  and track referrals 
PSAs and community  education programs 
B-l  OUTCOME- What  is the effect of the accomplishments? 
Initial  sessions led by trained trainers (by retired professionals  and by others) 
Medicare  beneficiaries  trained or aided (by retired professionals  and by 
others) 
increased understanding  of examples of Medicare  fraud as demonstrated 
by pre- and post-testing 
Long-term 	 calls, referrals,  cases, recouped funds (by retired professionals  and by 
others) 
individuals  reached  by public  education programs  (probably 
unmeasurable) 
B-2 
 APPENDIX  C 
DISCUSSION  DRAFT  PERFORMANCE  MEASURES 
SHARED  WITH  THE  PROJECTS 
INPUTS 
$ expended to recruit,  train,  and support retired professionals  and $ for other 
trainers 
$ expended on community  education  activities 
$ expended on tracking  system 
OUTPUTS 
# of retired professionals  and # of others trained 
# of PSAs and community  education  activities 
system to identify  and track referrals 
OUTCOMES 
Initial  # of sessions led by retired professionals  and # by other trainers 
# of Medicare  beneficiaries  trained by the retired professionals  and # by other 
trainers 
average percent increase in post-tests scores over pre-tests score 
Long-term  # of referrals  and cases attributable  to both types of trainers 
recouped  $ attributable  to both types of trainers 
# of individuals  reached by PSAs and community  education  programs 
C-l