. .

Department of Health and Human Services


OFFICE OF


INSPECTOR GENERAL


OMBUDSMAN OUTPUT MEASURES


MANAGEMENT ADVISORY REPORT


f,RVICES' 

Richard P. Kusserow 

INSPECTOR GENERAL '0 

lt "d:m 

JUNE 1991 
OFFICE OF INSPECTOR GENERAL


The mission of the Office of Inspector Genera (OIG), as madated by Public Law 95-452, as 

amended, is to protect the integrty of the Deparent of Health and Human Services ' (HS) progrs as well as the health and welfare of beneficiares served by those progrs. This 

statutory mission is cared out through a nationwide network of audits, investigations, and 

inspections conducted by thr OIG operating components: the Offce of Audit Services, the 

Office of Investigations, and the Office of Evaluation and Inspections.. The OIG also informs the 

Secretar of HHS of progr and management problems, and reommends courses to correct 

them. 

OFFICE OF AUDIT SERVICES 

The OIG' s Offce of Audit Services (OAS) provides all auditing services for HHS, either by 

conducting audits with its own audit resources or by overseeing audit work done by others. 

Audits examne the performance of HHS programs and/or its grantees and contractors in 

caring out their respective responsibilities, and are intended to provide independent 

assessments of HHS programs and operations in order to reduce waste, abuse and 

mismanagement and to promote economy and effciency throughout the Deparment. 

OFFICE OF INVESTIGATIONS 

The OIG' s Office of Investigations (01) conducts crimial, civil, and admistrative 

investigations of allegations of wrongdoing in HHS programs or to HHS beneficiares and of 

unjust enrchment by providers. The investigative efforts of 01 lead to criminal convictions 

administrative sanctions , or civil money penalties. The 01 also oversees State Medicaid fraud 

control units which investigate and prosecute fraud and patient abuse in the Medicaid program. 

OFFICE OF EVALUATION AND INSPECTIONS 

The OIG' s Offce of Evaluation and Inspections (OEI) conducts short-term management and 

program evaluations (called inspections) that focus on issues of concern to the Deparment, the 

Congress, and the public. The fmdings and recommendations contained in the inspections 

reports generate rapid, accurate, and up-to-date information on the effciency, vulnerability, and 

effectiveness of deparmental programs. 

This report was prepard in the New York Regional Office under the direction of Regional 

Inspector General Thomas F. Tully. Project staf included: 

Renee Schlesinger Project Leader Alan S. Levine, 
 Headquarters

Demetr Arapakos 

N aney Harson 

Alan S. Meyer 
Deparment of Health and Human Services


. OFFCE 

INSPECTOR GENERA


OMBUDSMAN OUTPUT MEASURES


MAAGE!vNT ADVISORY REPORT


uw'C'r$' CfJ' 

'0 

lt ;(It''ofia 

Richard P. Kusserow 

INSPECTOR GENERAL 

OEI-02-90-02121 
EXECUTIVE SUMMARY


PURPOSE 

The purose is to identify methods for measurig the success of ombudsman programs. 

BACKGROUND 

We recently conducted an inspection of State Long Term Car Ombudsman Programs. See our 

report OEI-02-90-02120 entitled "Successful Ombudsman Progrs. " In that inspection, we 

interviewed all State ombudsmen to determine the characteristics of successful programs and to 

help identify highly successful programs. 

In analyzing this information, we created a multi-dimensional index which served as a 

conceptual basis for ratig the activities of all ombudsman programs. We believe that this index 

can serve as an example of an output measurement system for periodcally monitoring and 

appraising the effectiveness of ombudsmen progrs. 

The criteria used to evaluate these programs fall into two dimensions: visibility and complaint 

resolution. A third dimension--peer recommendations--was also incorporated into the index. . 

Four levels of success were then identified by reviewing the frequency distrbution of each of 

these criteria. 

RECOMMENDATION 

The index is meant to demonstrte that, despite certai limtations, such a measurement system is 

possible. The development of a measurement or rating system, using either our methodology or 

an alternative, would provide a comparative basis for analyzing State programs, measuring 

progress, and targeting technical assistance. We recommend that you implement such a system. 
............ .................................................. . ................................................ .... ........................ :..................

TABLE OF CONTENTS


INTRODUCTION..................................................................................................... .


Purose.... .... 


Background.............. .........................................................................


OMBUDSMAN OUTPUT MEASURES.............. 


Criteria .................................................................................................... .............


Scoring........ .........................................................................................................


Results.......................... ........................................................................................ 3 

Limtations...... . 

Recommendation.......... .......................................................................................


APPENDIX A ...... .................. .................... .......................................................... .. AAPPENDIX B .................... ............................ ................................................ ........ . 

APPENDIX C ......................................................................................................... C­
INTRODUCTION


PURPOSE 

The purose is to identiy methods for measurg the success of ombudsman programs. 

BACKGROUND 

We recently conducted an inspection of State Long Term Car Ombudsman Programs. See our 

draft report OEI-02-90-02120 entitled "Successful Ombudsman Programs" issued in April 1991. 

In that inspection, ombudsmen from all States, the Distrct of Columbia (DC) and Puerto Rico 

(PR) were interviewed by telephone to discuss what makes an ombudsman program successful. 

They were asked to (1) suggest criteria and standards for judging ombudsman program activities; 

(2) report their experiences in these activities; and (3) identify States they feel have successful 

progrs. The purose of these interviews was to determne the characteristics of successful 

progrs and to help identify highly successful programs which we would look at in greater 

depth. 

In analyzing this information, we created a multi-diensional index which served as a 

conceptual basis for ratig the activities of all ombudsman programs. We believe that this index 

might serve as an example of an output measurment system that could be used to periodically 

monitor and apprase the effectiveness of ombudsmen progrs. We are providing this as one 

methodological tool in support of the inspection s recommendation that the Admnistration on 

Aging (AoA) should work with States to develop model operational guidelines in areas such as 

frquency of visits; staf-to-bed ratios; vo1unteer-to-bed ratio; complaint response time; 

complait resolution percentages; recruitment, trning and retention of staff and volunteers; and 

progr publicity. 
OMBUDSMAN OUTPUT MEASURES


CRITERIA 

Telephone interviews revealed performance criteria which ombudsmen f el should be used to 

evaluate their program. These criteria fal into two dimensions: visibiliiy and complaint 

resolution. The 51 responding State ombudsman progrs were rated on their reported activities 

in these aras. A third diension of success, peer recommendations, was also incorporated into 

the index. We asked each State to identiy other States that they feel have successful programs; 

the recommendations each State program received from fellow ombudsmen were scored. Table 

1 lists the crteria which fal under each dimension. 

TABLE 1 

Criteria for Measuring Success 

of Ombudsman Programs 

VISIBILITY 

Frequency of visits to facilities annually 

Ratio of professional staff to facility beds 

Ratio of volunteers to facilty beds 

II. COMPLAINT RESOLUTION 

Response time for life-threatening complaints 

Response time for other complaints. 

percentage of complaints resolved per year 

II. PEER RECOMMENDATIONS 

Number of times fellow ombudsmen recommended State as having a 

model program 

SCORING 

Four levels of success were identified by reviewing the frequency distrbution of each of these 

criteria. Every State s perfonnance was scored from high (4 points for the best reported 

experiences) to low (1 point). For instance, a State progrm that makes regular weekly visits to 

long tenn car facilties eared four points in that activity; a State that only visits facilities to 

follow up on complaints but never makes routine visits eared one point. The scoring system for 

each criterion is outlined in Table 2 as follows: 
TABLE 2 

Scoring System 

VISIBILITY: 

Points Frequency 

regular visits 

Ratio of 

staff to beds 

Ratio of 

volunteers to beds 

weekly 1 :1- 1499 beds 1 :1- 169 beds 

monthly 1 :1500-2999 1 :170-999 

4 times/yr 1 :3000-4499 1 :1000 plus 

only follow-up 1 :4500 plus no volunteers 

COMPLAINT RESOLUTION: 

Percent of 

Points Response time to: lie-threatening non lie-threatening 

complaints 

resolved per year 

24 hours 24 hours over 90% 

24-48 hours 3 days 75-89% 

5 days 5 days 65-74% 

1 week over 5 days less than 65% 

PEER RECOMMENDATIONS: 

Points Number of votes as model program 

1 4 - 20 votes 

7 - 1 3 votes 

2 - 6 votes 

1 vote 

RESULTS 

The points eared for visibilty, complaint resolution, and for being a model were totalled to 

obtain a final score for each State. As shown in Table 3 the final scores range from a low of 11 

points to a high of 27 (out of a possible 28 points). 
TABLE 3 

Results of Scores 

VISIBiliTY 

requency Ratio of Ratio of 

staff to voluntee regular bes to bes visits 

COMPLAINT 

RESOLUTION 

Response Time . 

Ufe- Non Ufe­ %of 

Threat- threat­ cotTplalnt 

enlng ening resolved 

PEER 

RECOMMENDATlONS 

Peer 

votes for 

model 

states 

Total 

score 

STATE 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District of Columbia 

Florida 

GeorQia 

Hawaii 

Idaho 

Iowa 

Ilinois 

Indiana 

Kansas 

Kentuckv 

Louisiana 

Maine 

Marvland 

Massachusetts 

Michiaan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 

Nevada 

New Hampshire 

New Jersev 

New Mexico 

New York 

North Carolina 
TABLE 3 (cant.) 

COMPLAINT PEER 

VISIBILITY RESOLUTION RECOMMENDAResponse Time TIONS 

STATE requency 

regular 

visits 

Ratio of 

staff to 

beds 

Ratio of 

volunteer 

to bes 

UfeThreat. 

enlng 

Non Ufethreat. 

enlng 

%of 

complaint 

resolved 

Peer 

votes for 

model 

states 

Total 

score 

North Dakota 

Ohio 

Oklahoma 

Oreaon 

Pennsvlvania 

Rhode Island 

South Carolina 

South Dakota 

Tennessee 1 . 

Texas 

Utah 

Vermont 

Virainia 

Washinaton 

West Virainia 

Wisconsin 

Wvomina 

Based on these scores we were able to group the State programs into three levels of success: 

most, moderate and least. 

LIMITATIONS 

Limtations exist in this index in that the information is basically self-reported. Also, many 

somewhat intangible factors, which contrbute to the success of an ombudsman program, are not 

captured. These include: program independence, the ability of a program to influence legislation 

or change policy, the relationship of the ombudsman program with other State agencies and 

providers and the impact of the personality and leadership style of the State ombudsman. To 

some degre, however, the recommendations of model programs incorporate these intangibles. 

RECOMMENDATION 

The above index is meant to demonstrate that, despite certain limitations, such a measurement 

system is possible. The development of a measurment or rag system, using either our methodology 

or an alterative, would prvide a compartive basis for analyzing State progrs, measuring progress 

and targeting technical assistance. We recommend that AoA implement such a system. 
APPENDIX A


TABLE 4


Output Measures for Ombudsman Programs:


Visibilty Criteria. Reported Activity


STATE 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District of Columbia 

Florida 

GeorQia 

Hawaii 

Idaho 

Iowa 

Illnois 

Indiana 

Kansas 

Kentuckv 

Louisiana 

Maine 

Marvland 

Massachusetts 

Michiaan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 

Nevada 

New Hamcshire 

New Jersev 

New Mexico 

New York 

North Carolina 

North Dakota 

Ohio 

Frequency of regular 

visits 

3 x per year 

yearly 

weekly 

4 x cer year 

weekly 

monthly 

2 - 3 x cer month 

weekly 

2 x cer year 

yearlv 

4 x cer year 

no reQ. visit 

4 x cer year 

no rea. visit 

vearlv 

vearlv 

2 x cer year 

month Iv 


weeklv 

no reQ. visit 

monthly 

weekly 

weeklv 

2 - 5 x cer year 

4 x per year 

weekly 

monthly 

yearly 

4 x cer year 

monthlY 

no rea. visit 

weeklv 

weekly 

4 x cer year 

2 x cer year 

weekly 

Ratio of staff Ratio of volunteers to 

to beds beds 

1 cer 385 

1 per 1123 1 per 79 

1 per3018 1 per 246 

1 cer 1267 1 cer 1900 

1 cer 2000 1 cer 800 

1 per 2558 1 per 1023 

1 cer 4020 1 cer 670 

1 cer 1980 1 cer 135 

1 cer 872 1 cer 74 

1 cer 9474 1 cer 632 

1 cer 1259 1 cer 871 

1 per 3262 

1 cer 849 

1 cer 43000 1 cer 18 

1 cer 3640 1 cer 557 

1 cer4127 

1 cer 9694 

1 cer 1748 1 cer 157 

1 cer 1892 1 cer 80 

1 per 4290 1 per 757 

1 cer 2094 1 cer 481 

1 cer 590 1 cer 235 

1 cer 4255 1 cer 1337 

1 cer 3329 1 cer 370 

1 per 1248 1 per 999 

1 cer 5329 1 cer 213 

1 cer 3250 1 cer 144 

1 cer 1 0558 1 cer 1920 

1 cer 1591 

1 cer 3113 1 per 1556 

1 cer 7143 

1 cer 1272 1 cer 191 

1 per 2660 1 cer 346 

1 cer 4427 1 cer 48 

1 cer 4045 1 cer 202 

1 cer 2060 1 cer 412 
TABLE 4 (cont. 

STATE 

Oklahoma 

Ore on


Penns Ivania 

Rhode Island 

South Carolina 

South Dakota 

Tennessee 

Texas 

Utah 

Vermont 

Vir inia 

Washin ton 

West Vir inia 

Wisconsin 

W omin 

Ratio of staff 

to beds 

er 2174


er 3215


er 3596


er 10331


er 5665


er 1409


1 er3912


er 3441


er 1062


er 854


er 3596


er 6678


er 1500


er 6685


er 214


Ratio of volunteers to 

beds 
APPENDIX 

TABLE 5 

Output Measures for Ombudsman Programs: 

Complaint Resolution Criteria Reported Activitf

Complaint Response Time 

STATE 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District of Columbia 

Florida 

Georaia 

Hawaii 

Idaho 

Iowa 

Illinois 

Indiana 

Kansas 

Kentucky 

Louisiana 

Maine 

Marvland 

Massachusetts 

Michiaan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 

Nevada 

New Hampshire 

New Jersey 

New Mexico 

New York 

North Carolina 

North Dakota 

Ufe threatening 

3 - 5 days 

24 hours 

5 days max. 

48 hours


24 hours


24 hours


24 hours 

24 hours 

24 hours


24 hours


24 hours


24 - 48 hours 

24 hours


24 hours


48 hours


24 hours


24 - 48 hours


24 hours


48 hours


24 hours


24 - 48 hours


24 hours


24 hours


5 days


24 hours


24 - 48 hours


24 hours


24 hours


24 hours


24 hours


24 hours


24 hours


48 hours


24 hours


24 hours


Non Ufe threatening 

3 - 5 days 

24 hours


5 days max. 

2 - 3 days 

24 hours


24 hours


3 days 

24 hours


24 hours


5 days 

1 week 

2 days 

5 days 

priority 

2 days 

2 weeks 

2 days 

24 hours 

2 days 

1 week 

2 days 

24 hours


24 hours


5 days 

24 hours


2 days 

1 week 

1 0 days 

3 days 

months 

3 days 

24 hours


2 days 

1 week 

4 days 

% of complaints 

resolved Der vear


83% 

93% 

74% 

87% 

75% 

83% 

96% 

67% 

90% 

50% 

75% 

72% 

72% 

35% 

81% 

69% 

25% 

65% 

80% 

77% 

68% 

95% 

77% 

77% 

60% 

90% 

100% 

85% 

80% 

43% 

D/K* 

75% 

92% 

60% 

80% 
TABLE 5 (cont. 

Complaint Response Time 

STATE Ufe threatening Non Ufe threatening % of complaints 

resolved per year 

Ohio 24 hours 1 week 67% 

Oklahoma 3 - 5 davs 3 - 5 davs 80% 

Oreaon 24 hours 24 hours 75% 

Pennsylvania 24 hours 5 davs 40% 

Rhode Island 24 hours 24 hours 85% 

South Carolina 24 hours 24 hours 60% 

South Dakota 24 hours 3 davs 100% 

Tennessee 3 davs 3 davs 70% 

Texas 24 hours 3 davs 92% 

Utah 3 davs 3 weeks 98% 

Vermont 24 hours 3 davs 70% 

Virainia 48 hours 48 hours 84% 

Washinaton 1 week 1 week 80% 

West Virainia 24 hours 24 hours 79% 

Wisconsin 24 hours 4 davs 76% 

Wyoming 24 hours 3 davs 72% 

* New Jersey does not keep statistics on the percent of complaints resolved in a year. In order 

to include this statistic in the index we projected an estimated value based on their other reported 

activities. 
APPENDIX 


Number of Votes Received by Ombudsmen Programs 

as Model States 

STATE NUMBER OF VOTES

Ohio 

Michigan 

Massachusetts 

California 

Colorado 

Minnesota 

Distrct of Columbia 

Florida 

Georgia 

Oklahoma 

New Jersey 

Marland 

Louisiana 

Wisconsin 

Maine 

Idaho 

Kentucky 

Oregon 

Texas 

Virginia 

Delawar 

Ilinois 

New Hampshire 

New Mexico 

New York 

Nort Dakota


South Carolina 

Tennessee 

Vermont 

Washington 

The remaining 22 State programs did not receive any votes 

from their fellow ombudsmen. 

r. - 1