. .
Department of Health and Human Services

OFFICE OF

INSPECTOR 
GENERAL

OMBUDSMAN OUTPUT MEASURES

MANAGEMENT ADVISORY REPORT

f,RVICES' 
Richard P. Kusserow 
INSPECTOR GENERAL 
'0 
JUNE 1991 
lt  "d:m  OFFICE OF INSPECTOR GENERAL

The mission of the Office of Inspector Genera (OIG), as madated by Public Law 95-452, as 
amended, is to protect the integrty of the Deparent of Health and Human Services ' (HS)
progrs as well as the health and welfare of beneficiares served by those progrs. This 
statutory mission is cared out through a nationwide network of audits, investigations, and 
inspections conducted by thr OIG operating components: the Offce of Audit Services, the 
Office of Investigations, and the Office of Evaluation and Inspections.. The OIG also informs the 
progr 
Secretar of HHS of  and management problems, and reommends courses to correct 
them. 
OFFICE OF AUDIT SERVICES 
The OIG' s Offce of Audit Services (OAS) provides all auditing services for HHS, either by 
conducting audits with its own audit resources or by overseeing audit work done by others. 
Audits examne the performance of HHS programs and/or its grantees and contractors in 
caring out their respective responsibilities, and are intended to provide independent 
assessments of HHS programs and operations in order to reduce waste, abuse and 
mismanagement and to promote economy and effciency throughout the Deparment. 
OFFICE OF INVESTIGATIONS 
The OIG' s Office of Investigations (01) conducts crimial, civil, and admistrative 
investigations of allegations of wrongdoing in HHS programs or to HHS beneficiares and of 
unjust enrchment by providers. The investigative efforts of 01 lead to criminal convictions 
administrative sanctions, or civil money penalties. The 01 also oversees State Medicaid fraud 
control units which investigate and prosecute fraud and patient abuse in the Medicaid program. 
OFFICE OF EVALUATION AND INSPECTIONS 
The OIG' s Offce of Evaluation and Inspections (OEI) conducts short-term management and 
program evaluations (called inspections) that focus on issues of concern to the Deparment
, the 
Congress, and the public. The fmdings and recommendations contained in the inspections 
reports generate rapid, accurate, and up-to-date information on the effciency, vulnerability, and 
effectiveness of deparmental programs. 
This report was prepard in the New York Regional Office under the direction of Regional 
Inspector General Thomas F. Tully. Project 
staf included: 
Renee Schlesinger Project Leader  Headquarters
Alan S. Levine, 

Demetr Arapakos 
Harson 
N aney 
Alan S. Meyer  Deparment of Health and Human Services

. OFFCE 
INSPECTOR GENERA

OMBUDSMAN OUTPUT 
MEASURES

MAAGE!vNT 
ADVISORY REPORT

uw'C'r$' 
CfJ' 
Richard P. Kusserow 
INSPECTOR GENERAL 
'0 
OEI-02-90-02121 
lt ;(It''ofia  EXECUTIVE SUMMARY

PURPOSE 
The purose is to identify methods for measurig the success of ombudsman programs. 
BACKGROUND 
We recently conducted an inspection of State Long Term Car Ombudsman Programs. See our 
report OEI-02-90-02120 entitled "Successful Ombudsman Progrs. " In that inspection, we 
interviewed all State ombudsmen to determine the characteristics of successful programs and to 
help identify highly successful programs. 
In analyzing this information, we created a multi-dimensional index which served as a 
conceptual basis for ratig the activities of all ombudsman programs. We believe that this index 
can serve as an example of an output measurement system for periodcally monitoring and 
progrs. 
appraising the effectiveness of ombudsmen 
The criteria used to evaluate these programs fall into two dimensions: visibility and complaint 
resolution. A third dimension--peer recommendations--was also incorporated into the index. . 
Four levels of success were then identified by reviewing the frequency distrbution of each of 
these criteria. 
RECOMMENDATION 
The index is meant to demonstrte that, despite certai limtations, such a measurement system is 
possible. The development of a measurement or rating system, using either our methodology or 
an alternative, would provide a comparative basis for analyzing State programs, measuring 
progress, and targeting technical assistance. We recommend that you implement such a system.  .... ....-.. ................ ........................ ..................................................... ........................................................................ ........ ..................:.......................................................... ................
TABLE OF CONTENTS

INTRODUCTION..................................................................................................... .

Purose.... .... 

Background.............. .........................................................................

OMBUDSMAN OUTPUT MEASURES.............. 

Criteria .................................................................................................... .............

Scoring........ .........................................................................................................

Results.......................... ........................................................................................ 3 
Limtations...... . 
Recommendation.......... .......................................................................................

A ...... .................. .................... .......................................................... .. A
APPENDIX 
B .................... ............................ ................................................ ........ . 
APPENDIX 
C ......................................................................................................... C
APPENDIX  INTRODUCTION

PURPOSE 
The purose is to identiy methods for measurg the success of ombudsman programs. 
BACKGROUND 
We recently conducted an inspection of State Long Term Car Ombudsman Programs. See our 
draft report OEI-02-90-02120 entitled "Successful Ombudsman Programs" issued in April 1991. 
In that inspection, ombudsmen from all States, the Distrct of Columbia (DC) and Puerto Rico 
(PR) were interviewed by telephone to discuss what makes an ombudsman program successful. 
They were asked to (1) suggest criteria and standards for judging ombudsman program activities; 
(2) report their experiences in these activities; and (3) identify States they feel have successful 
progrs. The purose of these interviews was to determne the characteristics of successful 
progrs 
and to help identify highly successful programs which we would look at in greater 
depth. 
In analyzing this information, we created a multi-diensional index which served as a 
conceptual basis for ratig the activities of all ombudsman programs. We believe that this index 
might serve as an example of an output measurment system that could be used to periodically 
monitor and apprase the effectiveness of ombudsmen progrs. We are providing this as one 
methodological tool in support of the inspection s recommendation that the Admnistration on 
Aging (AoA) should work with States to develop model operational guidelines in areas such as 
frquency of visits; staf-to-bed ratios; vo1unteer-to-bed ratio; complaint response time; 
complait resolution percentages; recruitment, trning and retention of staff and volunteers; and 
progr 
publicity.  OMBUDSMAN OUTPUT MEASURES

CRITERIA 
Telephone interviews revealed performance criteria which ombudsmen f el should be used to 
evaluate their program. These criteria  two dimensions: visibiliiy and complaint 
fal into 
progrs 
resolution. The 51 responding State ombudsman  were rated on their reported activities 
A third diension of success, peer recommendations, was also incorporated into 
in these aras. 
the index. We asked each State to identiy other States that they feel have successful programs; 
the recommendations each State program received from fellow ombudsmen were scored. Table 
each dimension. 
1 lists the crteria which fal under 
TABLE 1 
Criteria for Measuring Success 
of Ombudsman Programs 
VISIBILITY 
Frequency of visits to facilities annually 
Ratio of professional staff to facility beds 
Ratio of volunteers to facilty beds 
II.  COMPLAINT RESOLUTION 
Response time for life-threatening complaints 
Response time for other complaints. 
percentage of complaints resolved per year 
II. 
PEER RECOMMENDATIONS 
Number of times fellow ombudsmen recommended State as having a 
model program 
SCORING 
Four levels of success were identified by reviewing the frequency distrbution of each of these 
criteria. Every State s perfonnance was scored from high (4 points for the best reported 
experiences) to low (1 point). For instance, a State progrm that makes regular weekly visits to 
long tenn car facilties eared four points in that activity; a State that only visits facilities to 
follow up on complaints but never makes routine visits eared one point. The scoring system for 
each criterion is outlined in Table 2 as follows:  TABLE 2 
Scoring System 
VISIBILITY: 
Points Frequency  Ratio of  Ratio of 
regular visits  staff to beds  volunteers to beds 
weekly  1499 beds  169 beds 
1 :1- 1 :1-
monthly  1 :1500-2999  1 :170-999 
4 times/yr  1 :3000-4499  1 :1000 plus 
only follow-up  1 :4500 plus  no volunteers 
COMPLAINT RESOLUTION: 
Percent of 
Points  Response time to: complaints 
lie-threatening 
non lie-threatening  resolved per year 
24 hours  24 hours  over 90% 
24-48 hours  3 days  75-89% 
5 days  5 days  65-74% 
1 week  over 5 days  less than 65% 
PEER RECOMMENDATIONS: 
Points  Number of votes as model program 
20 votes 
1 4 -
7 - 1 3 votes 
2 - 6 votes 
1 vote 
RESULTS 
The points eared for visibilty, complaint resolution, and for being a model were totalled to 
obtain a final score for each State. As shown in Table 3 the final scores range from a low of 11 
points to a high of 27 (out of a possible 28 points).  TABLE 3 
Results of Scores 
COMPLAINT  PEER 
RECOM
VISIBiliTY  RESOLUTION 
MENDA-
Response Time .  TlONS 
STATE  requency  Ratio of  Ratio of  Ufe- Non Ufe %of  Peer 
votes for  Total 
staff to  voluntee Threat- threat cotTplalnt 
regular  bes  to bes enlng  ening  resolved  model  score 
visits  states 
Alabama 
Alaska 
Arizona 
Arkansas 
California 
Colorado 
Connecticut 
Delaware 
District of Columbia 
Florida 
GeorQia 
Hawaii 
Idaho 
Iowa 
Ilinois 
Indiana 
Kansas 
Kentuckv 
Louisiana 
Maine 
Marvland 
Massachusetts 
Michiaan 
Minnesota 
Mississippi 
Missouri 
Montana 
Nebraska 
Nevada 
New Hampshire 
New Jersev 
New Mexico 
New York 
North Carolina  TABLE 3 (cant.) 
COMPLAINT  PEER 
RECOM
VISIBILITY  RESOLUTION 
MENDA-
Response Time  TIONS 
STATE  requency  Ratio of  Ratio of  Ufe- Non Ufe %of  Peer 
votes for  Total 
staff to  volunteer  Threat.  threat.  complaint 
regular  model  score 
beds  to bes  enlng  enlng  resolved 
visits  states 
North Dakota 
Ohio 
Oklahoma 
Oreaon 
Pennsvlvania 
Rhode Island 
South Carolina 
South Dakota 
Tennessee  1 . 
Texas 
Utah 
Vermont 
Virainia 
Washinaton 
West Virainia 
Wisconsin 
Wvomina 
Based on these scores we were able to group the State programs into three levels of success: 
most, moderate and least. 
LIMITATIONS 
Limtations exist in this index in that the information is basically self-reported. Also, many 
somewhat intangible factors, which contrbute to the success of an ombudsman program, are not 
captured. These include: program independence, the ability of a program to influence legislation 
or change policy, the relationship of the ombudsman program with other State agencies and 
providers and the impact of the personality and leadership style of the State ombudsman. To 
some degre, however, the recommendations of model programs incorporate these intangibles. 
RECOMMENDATION 
The above index is meant to demonstrate that, despite certain limitations, such a measurement 
rag 
system is possible. The development of a measurment or  system, using either our methodology 
or an alterative, would prvide a compartive basis for analyzing State progrs, measuring progress 
and targeting technical assistance. We recommend that AoA implement such a system.  APPENDIX A

TABLE 4

Output Measures for Ombudsman Programs:

Visibilty Criteria. Reported Activity

STATE  Frequency of regular  Ratio of staff  Ratio of volunteers to 
visits  to beds  beds 
Alabama  3 x per year  1 cer 385 
Alaska  yearly  1 per 1123  1 per 79 
Arizona  weekly  1 per3018  1 per 246 
Arkansas  4 x cer year  1 cer 1267  1 cer 1900 
California  weekly  1 cer 2000  1 cer 800 
Colorado  monthly  1 per 2558  1 per 1023 
Connecticut  2 - 3 x cer month  1 cer 4020  1 cer 670 
Delaware  weekly  1 cer 1980  1 cer 135 
District of Columbia  2 x cer year  1 cer 872  1 cer 74 
Florida  yearlv  1 cer 9474  1 cer 632 
GeorQia  4 x cer year  1 cer 1259  1 cer 871 
Hawaii  no reQ. visit  1 per 3262 
Idaho  4 x cer year  1 cer 849 
Iowa  no rea. visit  1 cer 43000  1 cer 18 
Illnois  vearlv  1 cer 3640  1 cer 557 
Indiana  vearlv  1 cer4127 
Kansas  2 x cer year  1 cer 9694 
Kentuckv  1 cer 1748  1 cer 157 
month Iv 

Louisiana  weeklv  1 cer 1892  1 cer 80 
Maine  no reQ. visit  1 per 4290  1 per 757 
Marvland  monthly  1 cer 2094  1 cer 481 
Massachusetts  weekly  1 cer 590  1 cer 235 
Michiaan  weeklv  1 cer 4255  1 cer 1337 
Minnesota  2 - 5 x cer year  1 cer 3329  1 cer 370 
Mississippi  4 x per year  1 per 1248  1 per 999 
Missouri  weekly  1 cer 5329  1 cer 213 
Montana  monthly  1 cer 3250  1 cer 144 
Nebraska  yearly  1 cer 1 0558  1 cer 1920 
Nevada  4 x cer year  1 cer 1591 
New Hamcshire  monthlY  1 cer 3113  1 per 1556 
New Jersev  no rea. visit  1 cer 7143 
New Mexico  weeklv  1 cer 1272  1 cer 191 
New York  weekly  1 per 2660  1 cer 346 
North Carolina  4 x cer year  1 cer 4427  1 cer 48 
North Dakota  2 x cer year  1 cer 4045  1 cer 202 
Ohio  weekly  1 cer 2060  1 cer 412  TABLE  4 (cont. 
Ratio of staff  Ratio of volunteers to 
STATE 
to beds  beds 
Oklahoma  er 2174

er 3215

Ore on

Penns Ivania  er 3596

Rhode Island  er 10331

South Carolina  er 5665

South Dakota  er 1409

Tennessee  1 er3912

Texas  er 3441

Utah  er 1062

Vermont  er 854

Vir inia  er 3596

Washin ton  er 6678

West Vir inia  er 1500

Wisconsin  er 6685

W omin  er 214
 APPENDIX 
TABLE 5 
Output Measures for Ombudsman Programs: 
Reported Activitf
Complaint Resolution Criteria 
Complaint Response Time 
STATE  % of complaints 
Ufe threatening  Non Ufe threatening 
resolved Der vear

Alabama  3 - 5 days  3 - 5 days  83% 
Alaska  24 hours  24 hours
 93% 
Arizona  5 days max.  5 days max.  74% 
Arkansas  48 hours
 2 - 3 days  87% 
California  24 hours
 24 hours
 75% 
Colorado  24 hours
 24 hours
 83% 
Connecticut  24 hours  3 days  96% 
Delaware  24 hours  24 hours
 67% 
District of Columbia  24 hours
 24 hours
 90% 
Florida  24 hours
 5 days  50% 
Georaia  24 hours
 1 week  75% 
Hawaii  24 - 48 hours  2 days  72% 
Idaho  24 hours
 5 days  72% 
Iowa  24 hours
 priority  35% 
Illinois  48 hours
 2 days  81% 
Indiana  24 hours
 2 weeks  69% 
Kansas  24 - 48 hours
 2 days  25% 
Kentucky  24 hours
 24 hours  65% 
Louisiana  48 hours
 2 days  80% 
Maine  24 hours
 1 week  77% 
Marvland  24 - 48 hours
 2 days  68% 
Massachusetts  24 hours
 24 hours
 95% 
Michiaan  24 hours
 24 hours
 77% 
Minnesota  5 days
 5 days  77% 
Mississippi  24 hours
 24 hours
 60% 
Missouri  24 - 48 hours
 2 days  90% 
Montana  24 hours
 1 week  100% 
Nebraska  24 hours
 1 0 days  85% 
Nevada  24 hours
 3 days  80% 
New Hampshire  24 hours
 months  43% 
New Jersey  24 hours
 3 days  D/K* 
New Mexico  24 hours
 24 hours
 75% 
New York  48 hours
 2 days  92% 
North Carolina  24 hours
 1 week  60% 
North Dakota  24 hours
 4 days  80%  TABLE 5 (cont. 
Complaint Response Time 
STATE  % of complaints 
Ufe threatening  Non Ufe threatening 
resolved per year 
Ohio  24 hours  1 week  67% 
Oklahoma  3 - 5 davs  3 - 5 davs  80% 
Oreaon  24 hours  24 hours  75% 
Pennsylvania  24 hours  5 davs  40% 
Rhode Island  24 hours  24 hours  85% 
South Carolina  24 hours  24 hours  60% 
South Dakota  24 hours  3 davs  100% 
Tennessee  3 davs  3 davs  70% 
Texas  24 hours  3 davs  92% 
Utah  3 davs  3 weeks  98% 
Vermont  24 hours  3 davs  70% 
Virainia  48 hours  48 hours  84% 
Washinaton  1 week  1 week  80% 
West Virainia  24 hours  24 hours  79% 
Wisconsin  24 hours  4 davs  76% 
Wyoming  24 hours  3 davs  72% 
* New Jersey does not keep statistics on the percent of complaints resolved in a year. In order 
to include this statistic in the index we projected an estimated value based on their other reported 
activities.  APPENDIX 

Number of Votes Received by Ombudsmen Programs 
as Model States 
STATE  OF VOTES
NUMBER 
Ohio 
Michigan 
Massachusetts 
California 
Colorado 
Minnesota 
Distrct of Columbia 
Florida 
Georgia 
Oklahoma 
New Jersey 
Marland 
Louisiana 
Wisconsin 
Maine 
Idaho 
Kentucky 
Oregon 
Texas 
Virginia 
Delawar 
Ilinois 
New Hampshire 
New Mexico 
New York 
Nort Dakota

South Carolina 
Tennessee 
Vermont 
Washington 
State programs did not receive any votes 
The remaining  22 
from their fellow ombudsmen. 
r. - 1