THE UTILIZATION AND QUALITY CONTROL

PEER REVIEW ORGANIZATION (PRO) PROGRA: 
AN EXPLORATION OF PROGRA EFFECTIVENESS 
RICHAD P. KUSSEROW 
INSPECTOR GENERAL  JANARY 1989 
CONTROL # OAI-Ol-88-00572
 TABLE OF CONTENTS

EXECUTIVE SUMRY 
INTRODUCTION. . 

PRO DESCRIPTIVE DATA. 
LIST OF ABBREVIATIONS

FINDINGS. . 
PRO-RELATED FINDINGS. 

Most respondents regarded the PROs as at least

moderately effective in carrying out their mission

Many respondents noted that the PROs' effectiveness

is hampered by both internal and external constraints. 

Respondents reflected considerable consensus regarding

activities. 
the PROs' most and least useful HCFA-defined 
The PROs reported somewhat more effective relationships

wi th their Medicare fiscal intermediaries than with

their Medicare carriers. 

There has been considerable variation among the PROs in

the degree to which their review decisions have con

curred with those of Super PRO. . 

HCFA-RELATED FINDINGS 
There has been significant improvement in HCFA' s 
management and administration of the PRO program

period. 
during the second contract 
s abil i ty 
The HCFA'  to assess PRO performance has been
limi ted 
by the lack of refinement and integration of 
its evaluation tools

The HCFA appears to have given the PROs insufficient

information on its criteria for making contract

renewal decisions. 

The use of 2-year , fixed-price contracts appears to 
have created inefficiencies in the PRO program

The HCFA' s second scope of work has been highly

prescriptive and appears to have provided insufficient

incentives for PRO initiative  . 23

The dynamism and broad scope of the PRO program have

impaired HCFA' s ability to manage the program. 
 RECOMMENDA TI ON S 
APPENDIX I 
ENDNOTES 
APPENDIX II 
BACKGROUND OF THE PRO INSPECTION 
APPENDIX III 
BACKGROUND OF THE PRO PROGRA.  . 37 
APPENDIX IV 
CHART OF PRO ACTIVITIES.  . 47 
APPENDIX V 
COMPARISON OF 1984 SCOPE OF WORK TO 
1986 SCOPE OF WORK  . 48 
APPENDIX VI 
SUMRY OF RECENT PRO-RELATED LEGISLATIVE PROVISIONS  . 50 
APPENDIX VII 
SUMMRY OF PRO-RELATED STUDIES 
APPENDIX VIII 
THE PROS WITH MORE THAN . ONE CONTRACT 
APPENDIX IX 
MEMORABLE  " PRO- ISMS" 
APPENDIX X 
TABLES 1-4  0 . 
APPENDIX XI 
METHODOLOGICAL NOTES  . 64 
APPENDIX XII 
COMMENTS ON THE DRAFT REPORT AND OIG RESPONSES TO THE 
COMMENTS  . 69  SUMY 
EXECUTIVE 
PURPOSE AND OBJECTIVES: 
This is the third and final inspection report in a series

assessing the performance of the utilization and Quality Control

Peer Review Organization (PRO) program. The purpose of this PRO

inspection is to promote a better understanding of the PROs' 
mission and activities. This final report offers an exploration

of the overall effectiveness of the PRO program, focusing on an

assessment of both the PROs themselves and on the Health Care

Financing Administration'  (HCFA) oversight of them.

METHODOLOGY: 
The inspection grew out of the Inspector General' s desire to

obtain a broad perspective .
on the PROs' performance during their 
(1986-88). To that. end 
second contract period  , we pursued three 
inquiry:  als 
primary lines of  (1) interviews with 211 individ 
associated with the PRO program, including all the PRO Chief

Executive Officers (CEOs) and representatives of other

Government, provider, and consumer groups associated with the

PROs  (2) site visits to 12 PROs selected for case study, and 
(3) review of pertinent literature and data bases.
MAOR FINDINGS: 
PRO-RELATED FINDINGS 
MOST RESPONDENTS REGARDED THE PROs AS AT LEAST MODERATELY

EFFECTIVE IN CARYING OUT THEIR MISSION. The vast maj ori ty 
(83 percent) of respondents reported favorably on the PROs' 
effectiveness. The highest ratings came from the PRO CEOs

congressional 
staff, and hospital association representatives. 
Three of the 12 case study sites received consistently high

effectiveness ratings. They differed in sizes and geographic

locations but shared several characteristics, including high

staff morale, relatively strong sanction referral records

relatively visible board chairmen and CEOs and strong

entities. 
communication links with external 
THE PROs REPORTED SOMEWHT MORE EFFECTIVE RELATIONSHIPS WITH 
THEIR MEDICARE FISCAL INTERMEDIARIES (FIs) THA WITH THEIR 
MEDICARE CARRIERS. Ninety-one percent of the PROs reported at

least moderately effective relationships with their fiscal

intermediaries and 70 percent reported similar levels of

carriers. 
effectiveness with their  Despite generally close 
working relationships between PROs and their FIs, some PRO and

HCFA staff noted concerns about FIs failing to make timely

paYment adj ustments based on the PROs' DRG de"terminations. Such

concerns have been supported by a recent Office of Inspector

General 
(OIG) audit which found that as of July 1987  approximately 54 000 claims adjustmenFtIs st.o taling about $51

million remained unprocessed by the  In addition, - although 
the PROs have had rather limited interactions with their

carriers 
, the need for stronger PRO, FI, and carrier coordination 
will increase as the PROs begin to review in non-hospital

settings. 
THE PROs VARY IN THE DEGREE TO WHICH THEIR REVIEW DECISIONS

CONCUR WITH THOSE OF SUPERPRO. Because no acceptable range of 
disagreement between the PROs and SuperPRO has been established

and no national trend analysis of variation among quality

problems identified has been undertaken, it is impossible to

data. 
assess a particular PRO' s performance based on SuperPRO 
However, our analysis of PRO/SuperPRO variations wi thin the PROs

second contract period reflected strong differences between PRO

and SuperPRO determinations, with the. exception of premature 
discharges, where both groups identified relatively few problems. 
Furthermore, comparison of PRO/SuperPRO differences wi thin the

first and second contract periods reflected that the PROs ' use of

generic quality screens appears to have had no effect in

narrowing the margin of difference between PRO and SuperPRO

determinations related to the identification of quality problems

reviewers. 
or the appropriate referral of cases to physician 
HCFA-RELATED FINDINGS 
THERE HAS BEEN SIGNIFICANT IMPROVEMENT IN HCFA' S MAAGEMENT AND

ADMINISTRATION OF THE PRO PROGRA DURING THE SECOND CONTRACT 
PERIOD. Despite the problems enumerated in other findings of 
this report, HCFA has made important strides in improving the PRO

program, as reflected in part by the fact that a majority of all

respondents considered 
HCFA' s management of the program at least 
moderately effective" and noted an improvement in that oversight

since the start of the program. 
Our interviews and review of 
HCFA materials confirm that HCFA has taken important steps to

improve the program, particularly by broadening its communication

ties. 
efforts with both the PRO and external entity communi 
THE HCFA' S ABILITY TO ASSESS PRO PERFORMCE HAS BEEN LIMITED BY 
THE LACK OF REFINEMENT AND INTEGRATION OF ITS EVALUATION TOOLS. 
Although we had hoped to assess PRO effectiveness by

supplementing our interview-generated information with HCFA

evaluation data, the inherent limitations of HCFA' s evaluation

tools made such analysis impossible. As currently structured

these tools have not been sufficiently integrated into a

comprehensive, coordinated system for evaluating the PROs' 
performance. Our interviews and review of HCFA documents

revealed both individual and collective problems with HCFA' 

three evaluation tools--the PRO Monitoring Protocol and Tracking

System 
(PROMPTS-2), SuperPRO, and the PROs' ongoing data-
reporting requirements.
 The PROMPTS-2 has focused much more on process than outcome

evaluation of the PROs and its format has failed to capture

gradations in PRO performance. 
The SuperPRO process appears duplicative of the medical

review component of PROMPTS-2 , and most respondents

including a number of HCFA regional office stdafaf,t an.o ted

uncertainty about how HCFA has used SuperPRO 
addition, many PRO respondents complained about their lack

of contact with SuperPRO reviewers and SuperPRO' s limited

use of local reviewers. 
The PROs' ongoing reporting requirements have been

cumbersome and complex, involving 23 separate reports, 90

percent of which must be submitted on a monthly or quarterly

basis. 
Furthermore, in-depth data analysis has been
limited. 
However, HCFA has recently taken steps to 
strengthen its data analysis capability and is planning to

contract cycle. 

reduce the PROs' reporting requirements in the third

THE HCFA APPEARS TO HAVE GIVEN THE PROS INSUFFICIENT INFORMTION 
ON ITS CRITERIA FOR MAKING CONTRACT RENEWAL DECISIONS. 
Respondents cited a wide range of activities as most important to

HCFA' s evaluation of the PROs. Al though the PROs' second scope 
of work requirements reflected greater emphasis on activities

related to quality review, only 30 percent of all respondents

cited such activities when asked to identify what HCFA has

considered most important in evaluating PRO performance.

Furthermore, wi thin HCFA' s scoring system for the PROs' second 
PROMPTS-2 evaluation, the numerical weights assigned to specific

activities do not appear to be consistent with the relative

period. 
importance of those activities within the second contract 
THE USE OF 2-YEAR, FIXED-PRICE CONTRACTS APPEARS TO HAVE CREATED

INEFFICIENCIES IN THE PRO PROGRA. Seventy-eight percent of all 
respondents thought that the 2-year length of the PROs' contracts

has been too short, especially given the ever-increasing

PROs. 
responsibilities of the  The short contract period has 
given the PROs little time for operational stability between

contract renewal cycles and has hampered HCFA' s planning and

evaluation efforts. Therefore, we support the recent Onmibus

Budget Reconciliation Act of 1987 (OBRA 1987) provision that

lengthens the PRO contract period to 3 years and allows greater

spacing between termination dates for the 54 PRO contracts.

Although fewer respondents directly noted the inefficiencies of

fixed-
price contracts, many complained about the delays inherent 
in HCFA' s processing the large volume of PRO contract

modifications  product of using fixed-price 
, which have been a by-
contracts for a rapidly changing program. HCFA has taken

important steps to improve the timely processing of contract

modifications and is intending to experiment with alternative

contracting mechanisms for the third contract cycle. 
iii  THE HCFA' S SECOND SCOPE OF WORK HAS BEEN HIGHLY PRESCRIPTIVE AND 
APPEARS TO HAVE PROVIDED INSUFFICIENT INCENTIVES FOR PRO

INITIATIVE. Many respondents criticized the highly detailed menu

of PRO activities prescribed by HCFA, r commending that several

of those activities should be de-emphasized or eliminated from

' contract requirements. 
the PROs  In determining the PROs ' second 
scope of work, HCFA clearly had to balance competing forces for

flexibility and uniformity. The HCFA appears to have developed

the second and third scopes of work without adequate pilot

testing of activities prior to full-scale implementation by all

PROs. . Several  oncern about the PROs' 
respondents also voiced 
current requirement to perform comparable levels of review at all

hospitals regardless of their levels of performance.

THE DYNAMISM AND BROAD SCOPE OF THE PRO PROGRA HAVE IMPAIRED 
HCFA'S ABILITY TO MAAGE IT. HCFA has had to implement a

complex, ever-changing program which is of strong interest to

Congress and the professional medical community. 
The HCFA' 
programmatic work load has included implementation of planning

and evaluation systems , negotiation of 2 basic contracts and

300 contract modifications for 54 PRO areas, and coordination

entities. 
with other HCFA and outside  HCFA' s large work load has 
been accompanied by frequent staff turnover at both the agency

levels. 
administration and program management 
Such pressures may have inhibited short-term planning, as

reflected by HCFA' s delay in implementing legislative mandates

and by its failure to incorporate policy changes into the PRO

manual in a timely manner. Long-term planning may also have been

compromised as evidenced by the fact that in our interviews and

review of HCFA materials, we saw no indication of HCFA having a

long-range perspective on future directions for the PRO program. 
Al though the PROs' responsibilities may have strengthened their 
role in protecting the integrity of the Medicare program, severa 1

responden  s raised substantial and legitimate concerns that ever-
increasing PRO mandates could precipitate the program' s implosior.

rather than facilitate its improvement. 
RECOMMENDATIONS 
Al though HCFA has made important strides in the communications, 
contracting, and data areas, our findings suggest the need for

HCFA to take the following actions in the areas of evaluation and

planning: 
Improve coordination and integration of its evaluation tools

(PROMPTS-2 , SuperPRO and PRO data reports) to ensure 
comprehensive assessments of PRO performance. This might

incl ude 
 synchronizing the PROMPTS-2 review with the due dates

of SuperPRO reports; and

incorporating SuperPRO results into the PROMPTS-2

document along with data from PRO reports to maximize

the comprehensiveness of the evaluation.

Encourage consistency and objectivity in PRO renewal

decisions 
by: 
establishing rating standards for PROMPTS-2 and for

acceptable ranges of variance for SuperPRO;

formally informing the PROs, regional office staff and

review panel members of the weights to be assigned

SuperPRO , HMO/CMP, cost/savings ratio factors; and 
establishing a standing evaluation panel of HCFA

regional and central office staff to review regional

recommendations. 
Participation by PRO staff before 
closed session action would permit discussion of mutual

questions and concerns.

Re-examine the purpose for and validity of the SuperPRO

review process. This assessment should address such issues

as: 
whether SuperPRO uses appropriate reviewers anq

criteria to render valid judgments about local PRO

decisions; 
whether SuperPRO is intended to complement PROMPTS-2

it; 
medical review or to duplicate 
whether SuperPRO is intended to validate PROMPTS-2

size, 
review decisions. If so, whether its sample 
criteria and record selection process permit valid

comparison; and

whether there are better ways to use SuperPRO

expertise, such as having SuperPRO perform on-site

review and provide technical assistance to those PROs

most consistently differing from SuperPRO in their

review decisions or by having SuperPRO assess what

types of quality problems that are being identified by

PROs. 
Create forums for SuperPRO, HCFA and PRO discussions.

Examine ways to streamline the SuperPRO process to minimize

the administrative burden on PROs and to maximize its

utili ty to 
HCFA, such as:  . ..  ...
reducing the sample size for small PROs;

revising the schedule for reporting results to reduce

time lag between PROMPTS-2 and SuperPRO reviews; and

facili tating 
closer interactions between PRO and 
SuperPRO reviewers in resolving PRO/SuperPRO

differences. 
Improve the PROMPTS-2 process by:

developing standards for judging the manner of PRO

performance, i. e. 
, "Are PRO/provider relationships
effective 
, "Is the PRO successfully tracking
problems 
; and 
yes/no" 
establishing standards for rating "
acceptability of overall activity category based on

number of " 
yes/no" ratings of subcategories; 
developing more outcome measures of PRO effectiveness

through research and demonstrations.

Develop and release comparative PRO performance data to the

PROs and other interested parties.

Develop and distribute requirements, instructions , and

policies in a more timely manner to allow 'sufficient lead-

time for PRO implementation. 
Improve the consistency in policy interpretation among PROs

and HCFA staff by updating the PRO Manual on an on-going 
basis and by providing training and orientation sessions for

both PROs and regional staff as needed. 
Review the current roles of carriers , fiscal intermediaries

and PROs to assess the appropriateness of the current

distribution of responsibilities and assess the coordination

of these entities, such as the timeliness of. F.

adj ustments . 
Strengthen long-range planning of future directions and

appropriate roles for the PRO program by: 
increasing the emphasis on research and demonstrations

in such areas as review methodologies, patient outcome

and severity measures and data system requirements;

exploring mechanisms to encourage PRO innovations and

reduce the prescriptive nature of the current review

requirements;  exploring better long-term ways of structuring PROs 

activities so they are cost-effective and complement

the efforts of other review entities such as state

medical licensure boards and hospital qual i ty assurance
committees. 
For instance, HCFA might explore the 
possibility of having SuperPRO responsible for making

DRG determinations and have the PROs focus on quality

review; and

establishing an advisory group of PRO and other

relevant enti ty representatives to provide input to 
issues. 
HCFA on long-range planning 
COMMENTS ON DRAFT REPORT

We received written comments from the Health Care Financing

Administration 
(HCFA), the Assistant Secretary for Planning and
Evaluation 
(ASPE), and the American Hospital Association (AHA)" 
The ASPE and AHA raised some questions but were basically

report. 
positive about the  The HCFA raised significant concerns 
about the basis for the findings and about many of the

recommendations. See 
appendix XII for a complete presentation of 
the comments and of our response to them. 
vii
 INTRODUCTION

The Office of Inspector General (OIG) recently completed an

inspection of the Utilization and Quality Control Peer Review

Organization 
(PRO) program. The primary purpose of this broad-
based study was to assess PRO performance and to promote a better

understanding of the PROs' mission and activities. To that end, 
factors: 
the study focused on the following 
the implications of the changes in the PROs' scope of

work from the first to second contract period;

the maj or differences in perception among the PROs and

other entities (e.g., health providers, consumers

Government officials, public interest advocates) 
regarding the PROs' mission and performance;

the significant variations that exist among PROs in

carrying out their scope of work responsibilities;

the PRO practices that appeared to be exemplary; and

the potential weaknesses or vulnerabilities of the

program. 
(See appendix II for a more detailed explanation of the back

ground for th s inspection.

In the course of this OIG inspection of PRO performance, we

conducted in-depth interviews with a wide range of individuals

associated with the PRO program, including all PRO chief execu

tive officers (CEOs), and a sample of other PRO staff, as well as

national and local external entities. We visited 12 of the 44

PtiRoOsn wsh:o  are conducting reviews in 16 of the 54 PRO jurisdic

California, Colorado, Florida, Georgia, Indiana , Iowa, 
Massachusetts, New York, Oregon, Rhode Island, Texas, and West

Virginia (for the 
Delaware PRO area). In addition to this 
primary data, we collected and analyzed PRO-related performance

entities. 
data from HCFA and other  (See appendix XI for a more 
detailed description of our methodology.

This is the third and final report of a series summarizing the

study. 
findings of our  The first report focused on the PROs 
qualityt ireesvi.e w activities, and the second, on the PROs' sanction

acti vi  This final report offers an exploration of the 
overall effectiveness of the PRO program, focusing on an

oversight of
assessment of both the PROs themselves and HCFA' s 
them. It is important to note that our assessment of the PRO

program is based primarily on interviews with a wide variety of

well-informed individuals closely associated with the PROs and

HCFA. Wherever possible, we have supplemented our analysis of 
data. 
this qualitative data with available quantitative  Owing to our time constraints , the limitations of the study

design  ' activities 
, and the breadth of the PROs  , it is impossible 
to draw definitive conclusions about the program' s overall

effectiveness. Nevertheless , if, as, ,lo nteh ius. s. Senator noted,

reputation is a proxy for quality, 
report should offer 
significant indications of the PROs' and HCFA' s performance

during the second contract period. 
Our two previous reports included detailed explanations of the

PRO program. To minimize repetition for readers , we have placed

the detailed summary of the PRO program in appendix III of this

report. 
Appendix III also includes a more detailed discussion of 
HCFA and its tools for evaluating the PROs, as well as an

s fiscal 
explanation of the responsibilities of HCFA' 
intermediaries and carriers and their interaction with the PROs. 
A chart of PRO activities during the second scope of work is

included in appendix IV.  PRO DESCRIPTIVE DATA

In the course of our study, we learned that the PROs vary

significantly in their structures and functions. The 44 or
ganizations that have contracted with HCFA to carry out the

requirements of the second scope of work in the 54 areas reflect

the following patterns:

86 percent conduct all reviews themselves , whereas 

percent of them subcontract w  h other organizations;

68 percent were previously Professional Standards

Review Organizations (PSROs); 
84 percent are physician-sponsored and 16 percent are

physician-access organizations; 
68 percent derive the large majority of their work fro

Medicare , whereas 5 percent of them spend less than 50 
percent of their time on Medicare-related activities;

73 percent reported conducting review activities for

business entities , and 68 percent of them reported

contracting with state Medicaid programs;

approximately 50 percent subcontract for their data

processing, whereas the others maintain their own

systems; 
61 percent reported "
usually" performing on-site
review; 20 percent reported "always" performing ' on-site 
rather than off-site review.

In addition, our 12 case study sites reflected the following

variations in board composition and staff morale: 
75 percent had a clear separation between board and

staff functions, whereas 25 percent of them had one

person simultaneously serving as board chairman and

medical director; and

66 percent reported at least average staff morale,

whereas 33 percent of them had at least some staff who

reported low staff morale: board membersst atefnfd.e d to

rate staff morale more highly than PRO  Most 
respondents reported that the key factors inhibiting

stronger staff morale included the lack of competitive

salaries 
, the high turnover in nurse reviewers, and the
frustrations inherent in the shifting priori 
ties of a 
quickly changing program.
 The case sites also showed great variation in their quality

review systems and relationships with external entities, which

were highli hted in our first report on the PROs' quality review

activities. 
LIST OF ABBREVIATIONS

AAP:  American Association of Retired Persons 
AM:  American Medical Association 
BPO:  Bureau of Program Operations , Health Care 
Financing Administration

Carrier: 
Medicare Carrier 
CEO:  Chief Executive Officer 
COBRA:  Consolidated Omnibus Budget Reconciliation 
Act of 1985

DRG:  Diagnosis-Related Groups 
EOMB:  Executive Office of Management and Budget
FI: 
Medicare fiscal intermediary 
HCFA:  Heal th Care Financing Administration 
HSQB:  Heal th standards and Qual i ty Bureau 
GAO:  u. S. General Accounting Office 
OBRA  1986:  Omnibus Budget Reconciliation Act of 1986 
o BRA  1987 :  Omnibus Budget Reconciliation Act of 1987 
OMB:  Office of Management and Budget 
PPS:  prospective Payment System 
PRO:  Utilization and Quality Control Peer Review
Organization 
PROMPTS - 2 :  Peer Review organization Monitoring Protocol 
I Second Scope

and Tracking System for the PROs 
of Work

PSRO:  Professional Standards Review Organization
SuperPRO:  SysteMetrics , Inc.  :);,
:/ / . ~/~/' ~/~" ~';~. ' ';' ';' ';' '; " .
- -  )-  /."  / /.  -=;".-'
FINDINGS 
PRO-RELATED FINDINGS 
Most resDondents reqarded the PROs as at least moderate 1 y 
effective in carryinq out their mission

The vast majority (83 percent) of respondents reported favorably

on the PROs' effectiveness (see figure I). Among the several

groups interviewed, the strongest proponents of PRO effectiveness

were the PRO CEOs, congressional staff, and hospital association

representatives. The harshest criticism of PRO effectiveness

came from some medical societies, national governmental review

staff. 
entities, and HCFA regional 
FIGURE I 
PERCEPTIONS OF THE PROS' EFFECTIVENESS E\ 
CARRYIl\G OUT THEIR MISSION 
'JV\! EFFECT,V' 
PRO Chief Exe(:. ;"e 
Off;cers (CEO 
VERY  I/  I'  I'  lf L/1  Ccs  Study P=?c- S::.
C:'IC 50arc ME: 
00 
HCF,c Rea;0  ' or. 
O:'ERATELY  CentrOi 6ff;': 
Nctional Exte n'J! 
Entities 
MINIMALLY  P' Locoi Ex-terne 
Entities 
iota; for all 
NC:i  Respondents 
r;:-:  ,OW 
6:- 100 
PERCEt'-IFGE  ()r CF.TEGOPY  RESPONDlhJG 
Source: OIG Inspection Interv;ews-
Note: N= 208 respondents. 
Of those respondents who viewed the PROs as effective, the major

reasons cited for their assessments were, in descending order of

frequency, as follows: 
the PROs had a positive impact on the quality of

patient care;

the PROs provided good educational information to the

medical community;

the PROs had a positive influence on changing utiliza

tion patterns;

the PROs had a strong sentinel effect on physician and

hospital practices.
 The maj or reasons cited for low ratings of PRO effectiveness

were , in descending order of frequency, as follows: 
I educational efforts to change physician

the PROs 
practices were inadequate; 
the PROs operated with too many constraints;

the PROs had inadequate support from the medical

communi ty . 
It is worth noting that a sizable number of both HCFA and

national external entities mentioned that the effectiveness of

individual PROs varied so widely that it was difficult or

impossible to develop a composite picture of the PROs' overall

effectiveness. 
Our analysis of response patterns among our 12 case study sites

revealed the following (see table 1, appendix X): 
The PRO CEOs and other staff tended to rate their own

PRO as at least moderately effective. Only two PROs

(sites E and K on table 1) had staff members who rated

effective. In 
their PRO as minimally or not  one case 
the reason offered for that rating was the excessive

difficulty the PRO had in recruiting physicians to

review cases and the resultant difficulty in meeting

s time frames. In 
HCFA'  the qther case, the staff 
person noted that the PRO' s ineffectiveness was due to

the overly prescribed nature of its work and the lack

of funding necessary to keep 'up with its requirements.

The hospital associations tended to give the PROs

higher effectiveness ratings than did the other

external entities. This is consistent with the fact

that the case study sites usually had closer working

relationships with the hospital associations than with

the medical societies, or state medical licensure

boards. 3 In fact, a sizable number (45 percent) of the

state medical licensure boards in our case study sites

noted that 
they "did not know" the effectiveness of 
their PROs. This was consistent with our finding in

our previous report on the PROs 
' quality review

activities that most PROs have little interaction with

their state medical licensure boards.

In one quarter of our case study sites (sites A, G,

and H), all respondents considered the PRO at least

moderately effective. In all the other sites, at least

one respondent rated the PRO less than moderately

effective or "did not know" the PRO' s effectiveness.

The three PROs that were consistently rated as at least

moderately effective by all entities were of different sizes and
 geographic locations. However, the three organizations shared

the following characteristics: 
high staff morale;

relatively strong sanction referral records;

contracts with State Medicaid and business entities in

addition to the one with HCFA;

no subcontractors for review activities;

relatively visible board chairmen and CEOs who worked

closely together;

communication links in place with most significant

external entities , through either designated positions

on their boards , active task forces , or advisory groups

to the board.

The most striking indication of the PRO program'  growth in
stature was the fact that although many people we interviewed 
voiced particular concerns about the PROs, no one questioned the
overall need for the program.

Manv respondents noted that the PROs' effectiveness has been

hampered bv both internal and external constraints

The most significant constraints mentioned in descending order of

(see figure II): 
frequency included the following 
resources: insufficient 
Limi ted  money, staff, and time 
' myriad 
to carry out the PROs  responsibilities. Most 
PRO and external entities believed that the PROs had

inadequate funding for their acti vi ties 
, whereas most 
HCFA staff thought that the PROs had adequate funding. 
As noted by one U. S. Senator: "We recognize that the

PROs have done their best to move beyond their original

mandate for utilization review and to put greater

emphasis on quality... (but) their efforts have been

severely hampered by limited funds. ,,4 
Many PRO staff mentioned their difficulty in attracting

and retaining good physician and nurse reviewers, the

former because of medical community resistance and the

latter because of an inability to provide competitive

salaries and the lack of professional stimulation

inherent in repeti ti ve patient 
record review. 
HCFA' s management: inadequate time frames, inconsis
tencies in interpretation among HCFA regional offices

and too prescriptive a process for carrying out the

PROs' responsibilities. Many PROs voiced concern about

I S 
the serious delays in HCFA  contract modification  ::-:: '!= =,: .'=: " '::' :' ; , . ' '::.: ' . ;' , , , , - ) , ) , ) , )'  '"" 
' " ' " ' " ~~~~ ::::;..
process and the corresponding need for PROs to under

take new responsibilities without the neces ary
funding. 
The American Medical Peer Review Association 
(AMPRA) conducted a PRO survey last year, which found

that contract modification delays were posing serious

cash-flow problems for several PROs. Since that time 
HCFA has implemented a computerized tracking system for

contract modifications, and Congress has mandated that

HCFA must give notice to the PROs 30 days befor

requiring new activities.

In addition, concerns were raised by PROs and some HCFA

staff themselves about both the lack of consistency in

interpretation of PRO requirements and the extensive

requirements for uniformity that HCFA places on the

PROs for selecting cases for review. 
I lack of acceptance by the medical community:

The PROs 
the PROs ' difficulty in effecting change in practice

patterns given their often adversarial relationships

wi th the medical community.

Program  s dynamism: the rapid pace of change in the 
program dictated by congressional action. 
Inconsistent expectations: the 
PROs' mandate to be 
all things to all people,,5 and the sometimes conflict

ing messages the PROs have received from government

oversight entities.

Other constraints mentioned included the PROs' lack of adequate

interaction with external entities , and their extensive oversight

entities. 
by multiple government 
FIGURE II 
PERCEPTIOl\S OF THE MAJOR CONSTRAINTS LIMITI 
THE PRO'S EFFECTIVENESS 
STRA! (;S 
PRO Chief E)t  cut... 
Officers (CECS' 
LACV. OF RESOURCES 
k: ?: 5:  Case Stuo:; PRC' S'c 
L. and Boor  m=.,: 
h : A MANAGEME NT  b? HCFA Reo.ono: on: 
l2 Centro' Of:, :,: S 
f' 
~ationc  E It t,= 
P""\' S:.:iA"  RES:STAN:E  C' ' C""  Lt..o. ctlo:' . E.\,,' 
(L E tltte 
Pr=OG::A' t DVNAM:SM  Total fo' 01: 
R"!spono 
G")'J'- 1I,:'Y6 STE ,:Y 
10
PERCE'  Tt.:;E MEI,tTIOhJING EACH C()  STRA! 
s: , :: t"s:r  ':'' H-;te.. ..i  ..s.. 
es;:'  :I.::t:..  Respondents reflected considerable consensus reqardinq the PROs' 
activities. 
most and least useful HCFA-defined 
Of the five types of review that the PROs are required to perform

on a given record (see categories A-E in appendix IV), generic

quality screens and admission reviews were seen as the most

likely tools for identifying quality problems. 
This was 
consistent with our previous report on the PROs ' quality review

activities, 6 which noted that respondents saw those quality 
activities as the most important part of the PROs ' mission.

On the other hand, no one found coverage review the most useful

(82 
type of PRO review and a large majority of the PRO CEOs 
useful. 
percent) saw it as least  Many respondents mentioned that 
the PROs were less equipped to make coverage determinations than

fiscal intermediaries, which are more versed in the intricacies

of Medicare coverage requirements.

Among the 18 types of retrospective and prospective review

mandated by HCFA (see activities 1-18, appendix IV), respondents

found the following four most useful:

3 percent random sample from each PPS hospital;

all readmissions within 15 days from a PPS acute care

bed to any acute hospital (PPS or non-PPS) for inap

propriate admissions;

performance of five objectives to eliminate adverse

outcomes and reduce unnecessary admissions and proced

ures ; 
intensified review of hospitals that reach six cases or

rate. 
5 percent trigger error 
Every respondent group rated the 3 percent random sample as most

useful for identifying utilization problems and readmission

However, 
review as most useful for identifying quality problems. 
it is worth noting that one third of the national external

entities and one half of the local external entities had no

opinions about the most and least useful activities.

included: 
Those types of review found least useful by respondents 
all percutaneous lithotripsy claims in hospitals with

extracorporeal shockwave lithotripters;

editor; 
all cases identified by the Medicare code 
all cases referred by the fiscal intermediary for

coverage issues;  50 percent random sample of day and cost outliers;

preadm1ss1on review of pacemaker and four procedures

5 percent of 
proposed by a PRO that add up to 2. 
statewide discharges;

all cases involving assistants at cataract surgery; and

all submitted claims adjustments resulting in a higher-

weighted DRG.

Percutaneous lithotripsy review was ranked as the least useful

activity by a considerable margin. 
However, the other types of 
review listed as least useful were more equally ranked by

respondents. 
Respondents' rankings of the least useful review activities

closely paralleled their rankings of those activities they

thought should be de-emphasized or eliminated from their

contracts. In 
addition to all the least useful activities 
previously outlined, a number of respondents also thought that

the PROs' requirement to perform five objectives should be de-

emphasized or eliminated, although many respondents noted it as

being a useful review. 
This indicates that there is greater 
others, 
discrepancy in perceptions about this activity than most 
which may be explained by the fact that since such objectives are

PRO-defined, there is greater variation among the PROs in their

implementation of that requirement.

Furthermore, a significant majority of PRO CEOs and staff noted

that the five objectives and the day and cost outlier activities

consumed more time than they warranted. On the other hand, they

thought that the readmissions wi thin 15 days and the 3 percent

random sample review activities were worthwhile in spite of their

being time-consuming. 
Nevertheless, several PRO CEOs and staff 
questioned their upcoming mandate to extend readmission review to

31 days. 
It is important to note that HCFA has eliminated percutaneous

lithotripsy review from the third scope of work and has decreased

outliers. 
the percentage samples of day and cost 
The PROs reported somewhat more effective relationships with

their Medicare fiscal intermediaries than with their Medicare

carriers. 
A vast majority of PROs (91 percent) reported having at least

moderately effective relationships with their local fiscal

intermediaries (FIs). A smaller number (79 percent) reported

carriers. 
similar levels of effectiveness with their  Our case study analysis revealed that most PROs had more

interaction with their fiscal intermediaries than with their

carriers. The FIs forward the Unibill tape to the PROs from

which the PROs select cases for review. The FIs also make

payment adj ustments in response to cases in which the PRO makes 
admission. In 
DRG changes or denies an  discussing their FI-
related concerns, several PRO and a few HCFA staff mentioned

concerns about the FIs' lack of timely payment adjustments and

the lack of close coordination between the two respective HCFA

components (Health Standards and Quality Bureau and the Bureau of

Program Operations) responsible for the PROs and FIs. A few of 
the PRO CEOs from the case study sites noted that their

relationships with the FIs were enhanced by regular meetings with

the FIs and appropriate HCFA personnel.

A recently conducted and as yet unreleased OIG audit provides

ack of timely adjustments. 
supporting evidence of the 
s OfficFeIs'  of Audit 
that audit, the OIG'  (OA) notes that as of July
1987 , approximately 54 000 claims adFjuIsstm.e nts totaling about $51 
million remained unprocessed by the  More than 40 percent of 
those unprocessed adjustments were pending at the FIs for over 90

days. 
The OA audit provides recommendations to HCFA for ways of 
improving the FIs' timeliness in making such adjustments.

Most PROs reported relatively little ongoing interaction with

existed. 
their carriers but found few problems with what  They 
anticipated having more interaction with them as they begin to

settings. 
conduct reviews in nonhospital  Accordingly, in our 
first inspection report, we noted the need for HCFA to facilitate

a stronger link between the Medicare Part A and Part B data

bases. 
In its recent draft report, "Medicare Improving Quality of Care

Assessment and Assurance " the General Accounting Office (GAO) 
highlighted the inadequate coordination among the PROs, FIs and

carriers on issues related to quality . 
Just as we argued in our 
first inspection report that Medicare beneficiaries would be

better served by closer coordination between the PROs and other

entities such as licensure boards and hospital quality assurance

committees, GAO recommended that HCFA "develop formal guidelines

to coordinate the systematic and timely reporting by carriers and

intermediaries to PROs of possible problems with the quality of

care provided in ambulatory and post-hospital care settings

identified in medical reviews. ,,9 
There has been 
considerable variation amonq the PROs in the 
deqree to which their review decisions have concurred with those

of SuperPRO

HCFA has contracted with SysteMetrics (more commonly referred to

as SuperPRO) to biannually review a sample of records previously

reviewed by each PRO (see appendix III for a fuller discussion) 

During the PROs ' second contract period, the first 6 months of
 which corresponded to the fourth SuperPRO cycle, SuperPRO

validated the determinations made by the PROs regarding admission

review , discharge review, DRG validations , quality review, and 
admission referrals (i. e. , whether the PROs' nurse reviewers are 
referring appropriate cases to their physician reviewers).

Because no acceptable range of disagreement between the PROs and

SuperPRO has been established and no national trend analysis of

variation among identified quality problems has been undertaken,

it is impossible to judge a particular PRO' s performance based on

SuperPRO data. However, an analysis of PRO/SuperPRO variation

within the second contract period (cycle 4) reflected the
following patterns 
(see table 2, appendix X, for a summary of the
ranges and means of these differences) : 
the PRO and SuperPRO findings corresponded most closely

in the area of premature discharge, where both groups

found relatively few problems;

the largest average difference between PRO and SuperPRO

findings was in the area of DRG changes and the widest

range of differences was in the area of admission

referrals; 
SuperPRO generally found a greater frequency of

problems in each area of review than did the PRO, with

the following exceptions:

one PRO found more quality problems;

three PROs identified more cases for review DY

physicians (i. 
, admission referrals); and 
one PRO found more cases of premature discharge.

These patterns were generally consistent across the 4 cycles of

SuperPRO review and among our 12 case sites. Two additional

considerations are worth noting. First, although HCFA required

the PROs to use generic quality screens in the second contract

period, their use appears to have had no effect in narrowing the

margin between PRO and SuperPRO differences. in either identifying 
quality problems or in determining when it is appropriate for a

nurse reviewer to refer a case to a physician reviewer

(i. e.
, physician referrals). A comparison of the PRO/SuperPRO 
differences across the four cycles (the first three of which

correspond to the PROs' first contract period) reflected that the

overall differences between the PRO and SuperPRO seem to have

widened rather than narrowed in these two areas (see table 2.
More specifically, between the third and fourth SuperPRO cycles

the range of difference between PRO and SuperPRO findings related

to quality problems and admission referrals has widened, although

constant. 
the average difference has remained relatively  This 
' variation in 
may be explained by the PROs  implementation of the 
generic quality screens noted in our first inspection report.
 ...  .-- =.. ., --  . . ~~:~:,""~  ~~.').~',  ... ""' \ . q ""  ":' ;..:  --.,.  .,:\-=
Second , in our case study analysis, we found no consistent 
association between perceptions of PRO effectiveness and

levels. 
PRO/SuperPRO agreement  Although one of the three PROs 
perceived by all respondent groups to be at least moderately

effective also had consistently high agreement with SuperPRO, the

(see 
other two reflected average differences with SuperPRO 
appendix X). 
table 3 

In response to the generally high discrepancy rates between PRO

and SuperPRO, the PROs have complained that SuperPRO fails to

employ- local physicians to review PRO cases and that they have

little if any opportunity to discuss differences with SuperPRO

reviewers. A further 
discussion of SuperPRO issues will be 
report. 
included in the HCFA-related findings section of this 
HCFA-RELATED FINDINGS 
There has been siqnificant improvement in HCFA' s manaqement and 
administration of the PRO proqram durinq the second contract

period 
Despite the problems that are enumerated in other findings of

this section, HCFA staff have made important strides in address

ing PRO-related concerns as reflected by the fact that a maj ori ty

of all respondents rated HCFA' s management of the PRO program as

at least "
moderately effective.  (see figure III). It is worth 
noting that HCFA received the highest ratings from those respon

dsetnatfs fm.o st intimately involved with the PRO program--PRO and HCFA

Respondents from entities less involved in the implemen
favorably. 
tation of the program rated HCFA' s management less 
FIGCRE III 
PERCEPTIO:'S OF THE EFFECTIYE\"ESS OF

HCFA' S MANAGEMENT 
\' E  FECTIVE 
Dn " 
tZP 
Officers (CEC:=:
"A:r, 
VERY  Case Stuay D:o':. 
and Boanj Mecnt. E-" 
I? 
HCFA Reaionc  'Jr,:
1.")2ERATE:. y  Ce"tra  Offj(  :f; 
r7 
Nationai ExterrJ 
Ent:  -=s 
Mlt.ml.A:L.Y 
L.oco: Ext",'...: 
Entities 
()tal for 0:'

NOT  Respondents 
e.: 
Y;- f NOW 
1 
IV' 
=:=r  C.:- GC'R'  .t-\  CI".  C.:.;:. ,-, Jr,  -.'.  v--:  : t. ' -
O:.  ir,s:.e:::  ;ev. 
res:):) :C-E"'  ,,,,,,,,,,,,
;: :  :,::-  \\' .-,.   ---   -- --  -:,, - :.  ." -. :. .: - '. -: : --"' -  ' :.  -, :.,;:- .;::  -:=  :::-:;'' 
Furthermore, a majority of respondents also thought that HCFA'

oversight of the PRO program had been improving since the start

(see figure IV). 
of the PRO program  Again, those respondents 
closest to the PRO program tended to give HCFA the highest marks

for improvement , but a sizable number (60 percent) of the

national external entities also saw HCFA' s management as
improv ing . 
FIGURE IV 
PERCEPTIONS OF CHANGES IN HCFA' S OVERSIGHT 
VrL7  PRO Chief ::ie':u:i'''E
Offic ers (CE Cis) 
Case Study PRe: :::,: 
GE'TING BETTER 
ena Bocrc I' 
c..: 
;. F'2': 
':e  :rc. 
::TT!f  O;'S  VLl . r\ctjo  c:' E).:r:"
Enti ;es 
,;,r 
c.C . r-.
Ent!tle
:"7':jf.JG THE S IE 
ot.::  fe, 
on.:;,". ts 
CC r:T Kt,oVi 
i c: 
gc. 
s- P :) r 
::c  rc e  tnS;J-=C :;c"" ir,t-ervie'l S 
: N=  S-E  res::., rtden 
Our conversations with respondents, coupled with our review of

HCFA materials , disclosed that HCFA has increasingly emphasized

greater consistency and stability in program operations and has

intensified efforts to improve communications with the PRO

community and national constituency groups. The HCFA' s push to

First 
improve communication included at least three components. 
since 1  , HCFA has held periodic, high-level meetings with

representatives of the PRO community and national constituency

groups of providers , physicians, and beneficiaries to discuss

issues related to the PRO program. These discussions have

resulted in a formal PRO action plan that summarizes specific

actions for improving program administration in such areas as

communications, monitoring, and evaluation, and for improving the

PROs. 
effectiveness of the  Second, in September 1987 , HCFA
ini tiated 
its first national smteaeftifn.g  with PRO staff and HCFA-
regional and central office  Third, senior HSQB staff now 
participate regularly in biweekly conference calls for all PROs

convened by the American Medical Peer Review Association (AMPRA). 
In addition, HCFA' s increasing emphasis on consistency and
 effectiveness has led it not only to seek more input into policy-

but to undertake a

making from both the PROs and outside experts 
number of research initiatives to strengthen the PRO program. 
Finally, it has taken measures to improve program operations by

formalizing the policy interpretation process, allowing the PROs

30 days' advance notice of program changes and implementing a

tracking system to monitor contract modifications.

The HCFA' s success in improving communication with the PROs was

indicated by the fact that 84 percent of the PRO CEOs and staff

and 93 percent of the HCFA regional staff noted at least "moder
ately effective" relationships between the PROs and HCFA regional

office staff. On the other hand, HCFA' s central office staff

either assigned lower ratings to the regional office/PRO

relationships or offered no opinion of tnose relationships.

A smaller percentage (43 percent) of fRO CEOs and other PRO staff

viewed their relationships with HCFA' ss tcaefntfr.a i office as posi

tively as those with regional office  In fact, many did 
not consider that they had enough of a relationship with HCFA

it. 
central office staff even to comment upon 
When asked for their observations on the changing nature ox PRO

HCFA relations, 66 percent of PRO CEOs , 90 percent of HCFA

central office staff and 48 percent of HCFA regional office staff

getting better. 
reported that those relationships were "  On the 
noted 
other hand, a small percentage (14 percent) of the PRO CEOs 
that their relationships with HCFA were "getting worse.  Those 
regions. In 
individuals represented 6 PROs from 5 of the 10 HCFA 
explaining the reasons for the deteriorating relationships , all

commented on the fact that the regional HCFA staff were Ifni t-

fact 
picky" and offered inadequate technical assistance. The 
that only one HCFA region received more than one of these

negati ve ratings suggests that the PROs' perceptions of HCFA' 
effectiveness may be largely influenced by their relationships

with particular project officers. 
The HCFA' s ability to assess PRO Derformance has been limited bv

the lack of refinement and intearation of its evaluation tools

ini
When we began this study, we expected to be able to draw def 
tive conclusions about PRO effectiveness by supplementing our

data. 
interview-
generated data with HCFA evaluation  But, because 
of certain limitations inherent in HCFA' s various tools, such an

impossible. 
analysis has been 
Al though half of the total respondents thought that HCFA' s 
process for evaluating the PROs was at least moderately effec

tive, a quarter of the PRO CEOs rated the process as minimally or

not effective (see figure V) 
 ~. ~. ~"~~~~~
;:;:," :::  '\ / '\ / '\ "/  .," / 

FIGURE V

PERCEPTIONS OF THE EFFECTIVENESS OF

HCFA' S EVALUATION PROCESS 
f-Ji!, EFFECTIVE 
t: PRO Chief Exe: utive 
Officers (CEGs)
f' 
Case Study PRO Std' 
and Boord Me  ber 
b;'  HCFA Reg;o"'J: or.
MOD'"'  b6 
Central Ofke  t.:: 
ionol External

S' ,  rU7 t.r.. 
tles 
MIN t VII" I 
0(O! Externol

Entities 
NO"" tFS':;'  Total for 0!1
Res;)ondents 
60 
D: t  JT Kt\8Vw  4C 
100 
PERC::  ;Ac;::  OF CATEGOPY RESPONDING 
::: :r.s:o:?,:  i0n !:1ter\r'eVwS 
:-e r'  :-.: reSDo:" .:e;. 
The HCFA employs three general tools for evaluating the PROs

performance--the PRO Monitoring Protocol and Tracking System

(more commonly referred to as PROMPTS-2), SuperPRO and periodic

(monthly and quarterly) data reports (see appendix III for

description). Our interviews and review of a sample of those

documents revealed both individual and collective problems which

will be summarized below.

PROMPTS-2 
Both PRO and HCFA respondent groups rated the PROMPTS-2 as a more

effective evaluation tool than Super PRO (see table 4

appendix X). Sixty-two percent of respondents thought that the

surprisingly, 
PROMPTS-2 was at least moderately effective. Not 
HCFA staff were more enthusiastic in their ratings of PROMPTS-2

than were the PROs. 
Our interviews and review of the PROMPTS-2 protocol revealed the

following concerns about PROMPTS-2:

PROMPTS-2 has focused primarily on process versus

outcome evaluation of the PROs' performance. As one

HCFA official  easures would be nice 
but we don 't hanovtede, " Outtchomee  m. They must be developed.  PROMPTS-2' s design encourages potential subjectivity

and inconsistency among regional office reviewers

because it lacks standards for determining PRO

performance. 
Furthermore, it fails to capture 
gradations in PRO performance because it has only

yes/no rating categories.

The medical review component of PROMPTS-2, in which

regional office staff draw a sample of records reviewed

by the' PRO and validate the PROs' judgments in these 
cases , is seen as problematic. Specifically, some 
respondents questioned the expertise of regional office

medical reviewers and the validity of their

determinations. Many others commented that the medical

review element of PROMPTS-2 duplicates the SuperPRO

process. 
SuperPRO Review

The PRO CEOs and staff were the strongest critics of SuperPRO

(see table 4 , appendix X). Furthermore, while most HCFA staff

considered SuperPRO a nationally consistent tool for validating

medical decisions of all PROs, a number of HCFA regional office

staff echoed the PRO respondents ' uncertainty about how SuperPRO

resul ts have been used in evaluating the performance of

individual PROs. In addition, 
our interviews and a review of 
concerns: 
SuperPRO data revealed the following 
The HCFA has not established guidelines for acceptable

levels of PRO/SuperPRO discrepancies, thereby making it

Nor has 
impossible to assess significant concerns. 
HCFA undertaken an analysis of those PRlOesv weiltsh.  close or

only 
strongly disparate SuperPRO agreement 
analysis computation of national average rates of

PRO/SuperPRO disagreement has been made.

The HCFA has not undertaken any trend analysis of the

types of quality problems identified by SuperPRo.

SuperPRO was viewed as unresponsive by many PROs

bec  use of the lack of contact between PRO and SuperPRO 
reviewers and the SuperPRO' s limited use of local

reviewers. 
The 6-month lag time between SuperPRO review and HCFA I S 
receipt of SuperPRO reports has made it difficult for

HCFA regional staff to integrate the findings into its

own (PROMPTS-2) review of the PROs. For instance 
SuperPRO cycle 4 reports were unavailable in time to

review. 
integrate into the first PROMPTS-2  The HCFA has required SuperPRO to sample approximately

400 records from every PRO' s review area every 6 months

regardless of size or general performance; several PROs

have complained about the excessive staff resources

involved in meeting such demands. 
Despi te the considerable concerns raised about the SuperPRO com

ponent, however, a recent study by the OIG corroborated SuperPRO

admissions. 12 
rather than PRO rates of unnecessary  In that 
review of 7 050 Medicare records from 239 hospitals, OIG found

that 10. 5 percent of the hospital admissions were unnecessary, as

opposed to the 2 percent to 2. 5 percent noted by the PROs. 
Data Reporting

The data-reporting process used to monitor ongoing PRO perfor

mance has been problematic for both the PROs and HCFA since the

beginning of the program.  generated 
In the absence of HCFA-
software and computer system specifications, the PROs have

struggled on their own to develop and refine the sophisticated

data systems required for timely and accurate reporting of their

activities to HCFA.

The complexity and large volume of reports required by HCFA has

complicated this process. For example, the 1-page Sanction
Report 
(HCFA-545) has more than 12 pages of instructions. And 
HCFA has required the PROs in the second contract cycle to submi 

23 different reports, over 90 percent of which are submitted

ei ther 
monthly or quarterly . The PROs have also faced the added
difficul ty of 
being responsible for ever-changing reporting 
requirements called for either by HCFA or by new legislative

mandates. The requirements changed significantly between the

first and second contract periods, and the reporting process will

1988. 
change again for the third scope of work beginning in  The 
consequences have been PRO data reports that are frequently

inaccurate and untimely and data that are inconsistently reported

from PRO to PRO. Al though nearly three-fourths of HCFA staff 
rated the PRO reports as being at least moderately useful , many

voiced concerndSa atbao.u t the quantity, quality, and complexity of

the reported  In-depth data analysis for the purposes of 
program planning and evaluation has been limited, in part because

data. 
of the volume and unreliability of the reported  Analysis 
has also been limited by HCFA' s emphasis on addressing its

immediate concerns for validating the accuracy of PRO reports and

for remedying reporting problems. 
It is worth noting that HCFA 
has recently taken steps to strengthen its data analysis

capability and is planning to reduce the PROs' reporting

cycle. 
requirements in the third contract  Inteqration of Evaluation Tools

The PROMPTS-2 protocol, the SuperPRO process and the PROs ' data

reports have not been sufficiently integrated into a comprehen

sive, coordinated system for evaluating the PROs ' performance.

For instance, although the PROMPTS-2 protocol has been designed

by HCFA to serve as the basis for monitoring and evaluating the

incorporate- systematically 
PROs' performance, it has failed to 
the disagreement rates of SuperPRO reviews to permit comparison

reviews. In 
with those of the HCFA regional office medical 
addition, although the medical review component of PROMPTS-2

appears to have duplicated SuperPRO activities, the time frames

and sampling methodologies have not been complementary, thereby

two. 
precluding potentially useful comparisons between the 
Furthermore, PROMPTS-2 has failed to integrate appropriate data

report information. For instance, although PROMPTS-2 addresses

the PROs' progress in achieving the quality objectives of their

contracts 
, specific data related to the PROs' activities concern
ing quality of care problems , including sanctions, have not been

incorporated into the evaluation document.

Implications for Contract Renewal Process

In the fall of 1987, HCFA issued guidelines to the regional

offices, which were shared with the PROs, that outlined the

process and scoring system to be used for evaluating PROs for

renewal for the third contract period. . The second PROMPTS-2 will 
serve as the basis for assessing PRO performance for contract

renewals. . In 
close-call situations, HCFA will utilize the 
resul ts of SuperPRO review, the PROs' performance in HMO/CMP

review, where applicable, and the PROs ' cost/savings ratios as

computed by HCFA. However, the evaluation guidelines failed to 
include either the- weights or the criteria that HCFA reviewers 
factors. 
will use when considering such 
The process to be followed by HCFA will begin with the prepara

tion by regional offices of an evaluation package for each PRO

consisting of the PROMPTS-2 review, including a total point score

for performance, as well as a recommendation for either renewal

or nonrenewal. 
Evaluation packages will be forwarded to central 
review. 
office panels for further  According to HCFA staff, these 
panels will be ad hoc groups composed of approximately two

neutral" 
central office staff and one staff person from a "
regional office. Panel members will deliberate and make their

decisions by telephone because of limited travel money and tight

deadlines for decision making. 
Should the review panels not 
concur with regional office recommendations or should they make a

recommendation for nonrenewal, a third level of review will be

performed by senior HCFA managers, the proj ect officer, and the

regional branch chief
 ::  ,,..  ,:-:..::-: :;; ;). -):  / ;  ::  \:: y'::; -=,--,:. .:. ::. : ..::
The review panels 
' lack of consistent membership coupled with the 
limitations of HCFA' s evaluation tools make the renewal process

for the third contract period vulnerable to inconsistencies in

evaluation and decision making. 
It is worth noting that GA0 
recognized such inconsistencies in HCFA' s renewal process for the

second contract period, and HCFA assured GAO that proper internal

controls had been instituted to avoid such problems in the

future. 
The HCFA appears to have qi ven the PROs insufficient information 
on its criteria for makinq contract renewal decisions

When asked to identify which of the PROs' activities have been

most important to HCFA when evaluating the PROs ' performance

activities. 
respondents in all groups mentioned a wide range of 
Eighty-three percent of HCFA staff named activities related to

review. In fact, 
the PROs' quality  HCFA staff mentioned quality 
activity.
review almost three times as often as any other 
contrast 
, only 32 percent of the PRO CEOs and 25 percent of other 
PRO staff identified quality review as being the most important

activity to HCFA in evaluating PRO performance.

The PRO CEOs and staff most frequently noted the importance of

fulfilling their contract deliverables. Most of the respondents

from national and local external groups (33 percent and

36 percent, respectively) identified denials and cost containment

activities' as being 
the most important of the PROs' activities tG 
HCFA. Less than 
10 percent of the respondents from these two 
groups mentioned activities related to quality review (see
figure VI) 
FIGURE VI 
PERCEPTIO  S OF THE MOST IMPORTAKT ACTIVITIES 
HCFA E\ EVALUATE,G PRO PERFORMANCE 
f7 PRO Cni
rL 
Y.  C'ffic 
I. " 17v.  , .. I_ X X X X X X X X X X X X X X X X X j  1h". Cose S :"0:; ;: 
ana Bocrc t\i -2r1-I:'-=' 
(X HCcA R-:c,  C": 
(:I 
Centrc:  ,:e  S:-: 
C :)':TF .:-:T DEL IVERAB..  r7 Not,onoi E, te' 
Entities 
Loco: Ex:",,": 
Entities 
C::5T COt\TAiNM::N-:

Totel fo' 01

'"esponoents 
SANCTIONS 
10:: 
PERC::t  :- C4-:E,-:C  C'--ii'  EAC  TC: 
:-=' ':j::. Ir. s::-=:  vt-: -to 
::7.  = iEr: " ';S: : The striking variations and lack of consensus suggested by these

responses . may reflect the different perceptions that exist about 
the PROs' mission. However, they may also suggest that HCFA has

learly and consistently articulated its priorities for

not  PROs. For example, the 
assessing acceptable performance by the 
PROs' second scope of work requirements reflected an increased

emphasis on activities related to quality review. Furthermore, 
14 most PROs identified

as we mentioned in our first report, 
quality review activities as the most important part of their

mission. 
Yet the PROs did not often cite these activities as 
being the most important to HCFA in evaluating PRO performance.

Since our interviews of the late summer and early fall of 1987

HCFA has articulated priori ties for evaluating the PROs' 
eriods. 
Those 
performance during their second contract 

priorities are reflected in the numerical scoring systems to be

used in the final PROMPTS-2 review, which will serve as the

primary determinant in HCFA' s decision to renew PROs ' contracts.

According to that scoring system, four categories of PRO

activities--those related to utilization (and other required

review), to quality review, to data, and to management--will each

receive a maximum of 100 points , yielding an overall maximum of

400 points. Al though that scoring system establishes priori ties 
among the PROs' activities, the system also raises questions

about possible inconsistencies and mixed messages sent to the

PROs. For 
example, quality-related activities under that system 
can . result in a maximum of 100 points, or 25 percent of the total 
score, which may be inconsistent with the heavy emphasis being

Congress. Similarly, 
placed on quality review by HCFA and the 
communi ty outreach acti vi ties, included in the management 
category, can result in only 15 points, or less than 4 percent of

the total score, which seems inconsistent with HCFA' s increased

focus on those acti vi ties during the second contract period. 
year. fixed-
The use of 2- price contracts appears to have created 
inefficiencies in the PRO proqram

Over three-fourths of all respondents believed the 2-year length

of the PROs' contracts has been too short (see figure VII)

nety-f  staff, 
ve percent of the PRO CEOs, 88 percent of HCFA 
and 80 percent of national external groups thought that the P

should have longer contract periods, with IROSt respondents 
favoring a 3- to 4-year period. Respondents from all groups

cited myriad problems with the 2-year contract period. Most were

related to the undue pressure such a quick turnaround time has

placed on the PROs and on HCFA, especially since the PROs have

been given ever-increasing responsibilities since the start of

the program.
 ; .
FIGURE VII 
PERCEPTIO='S OF THE  T\\T  YEAR 
SUITABILITY OF  O-
CONTRACTS FOR THE PRO PROGRA 
\'I 
OPPOSED--7E;' 
DO NOT KNOW--8;' 
Fercent o  Res;:,ondents Favoring

or.j O;:iJcsing Two-Year Contrccts 
s:  ,:-=. (de.. I"sc,-=cti::. r, Inteiv:ews. 
:-=" r  1 =-7 res::'0ndents 
For the PROs, the short contract period has given them very

little time for operating between contract renewal yceycalress.- -some 
For 
have equated it with running for Congress every 2 
HCFA, the short contract period has hampered opportunities for

adequate planning and evaluation since it has had to start

planning for the next scope of work after less than a year'

one. In addition, the final 
experience with the current 
PROMPTS-2 must be conducted by the 17th month of the PROs 

contracts in order for HCFA to evaluate for renewal and to meet

process. 
Moreover, the 
the deadlines imposed by the contracting 
' data reporting 
time lag inherent in the PROs  (PROs have up to 60 
days to review records and  additional days to report to HCFA) 
has meant that the final 17-month PROMPTS-2 reviews may have been

based on only 14 months of review activity by the PROs. 
Finally, HCFA has had only a very short time within which to

evaluate the majority of PRO contracts for renewal and to

work. 
negotiate new ones for the next scope of  Fifty-two of the 
54 current PRO contracts expire within a 6-month period beginning

1988. 
June 30,  During the 6 months preceding each expiration 
each contractor' 
date, HCFA must evaluat  s performance, seek 
competitive contract bids if necessary, review proposal (s) for 
the next scope of work and negotiate a contract before the

period. 
beginning date of the next contract  Moreover, with nine 
groups of PRO contracts expiring over a 6-month period and each

group being at most a month apart , HCFA has had to deal
 simul taneously with the requirements of several different groups 
each of which may be at a different stage in the complicated

evaluation/contracting process.

The recent OBRA 1987 legislation included a provision, supported

by HCFA, to lengthen the contract period to 3 years and to allow

for greater spacing between termination dates of the PROs

contracts. We believe these changes are critically important to

improving the administration of the PRO program. 
They will 
afford HCFA more time both for evaluating the PROs' performance

based on more months of actual review data and for the planning

work. 
and analysis essential for developing subsequent scopes of 
They will also help alleviate the present administrative

nightmare of having to evaluate for renewal so many PRO contracts

time. 
wi thin a short period of 
ALthough fewer respondents directly noted the inefficiencies of

fixed-price contracting , 
many complained about a major by-
product of using such contracts for a constantly changing

program--the consequent need for frequent contract modifications.

According to HCFA staff, by the fall of 1987 , HCFA had made more

than 1, 300 modifications since the first contracts, with 700 of 
them occurring during the first 15 months of the second contract

period. 
The PROs have complained vehemently about the long periods of

During 
time involved in processing these contract modifications. 
the past year, HCFA has initiated several measures to address

these problems. 
Guidelines for the contract modification 
process, developed in cooperation with the PROs, were issued in

September 1987. These guidelines detail the responsibilities of

HCFA and the PROs as well as procedures and time frames for

submission and review of contract modifications. In addition,

the program and contracting components of HCFA responsible for

the PRO program have clarified their respective roles and

responsibili ties and have instituted an automated tracking system

to monitor the status of contract modification proposals.

Furthermore, during our conversations with high ranking HCFA

officials, they noted their intent to experiment with alternative

cycle. 
contracting mechanisms with the third contract 
The HCFA' s second scope of work has been hiqhlv prescriptive and

appears to have provided insufficient incentives for PRO

initiative. 
When asked for their recommendations for improving the PRO

program, 30 percent of all respondents suggested that HCFA give

the PROs greater flexibility in carrying out their review

activities 
(see figure VIII). It is worth noting that 
55 percent of PRO CEOs, 50 percent of the local medical associa

tions 
, and 24 percent of the HCFA regional staff were among those
suggesting such flexibility.  ':: - -: -: ..:.: . ,!  , . \.....:' ".". \.\-. .,  . . . -, :" \; 
 .'):.'.
FIGCRE VIII

RESPO;\DEI\TS RECOl\L\fENDI:\G THE PROS BE ALLO,YED 
IORE FLEXIBILITY II\ PERFOR!\fING THEIR REVIE,YS 
ftL7  PRO Chief Executive

Offic ers (CEOs)
I' 
Case Study PRO Stc 
and Board Mem:)ers

HCFA Regio:loi one 
Centra! Offi:e Sta

Natio:lJ . Extemo: 
Entities 
Enerna 
c.n..lt!es 
otal for- 0::

?espondents 
f.C:  e.c;  100 
I! :-' 
r. .- f". -- - " . . l  c:'\:. 1r- r-.  :. I \j -- ' ,.. r LI. " ' 1=' :
: Oj  jnspect;'J  inte:'viev.'s. 
= 211 res  ond  nts-
Gte: 
One of the major reasons cited for this recommendation was that

HCFA has required the PROs to follow a closely prescribed menu of

activities 
(as summarized in appendix IV) without adequate regard 
for whether the activities are productive in identifying either

quality or utilization problems. 
This concern was reinforced by 
the fact that the PROs identified several activities as having

limited or no effectiveness. While HCFA has either de-emphasized

or eliminated two of those activities (day and cost outliers and

percutaneous 
lithotripsy) in its third scope of work, it has 
retained the other five activities most often considered to be

least effective: 
Medicare code editor, fiscal intermediary 
referrals, preadmission review of pacemaker and four procedures

proposed by the PROs, review of cases involving assistants at

cataract surgery, and claims adjustments resulting in higher-

weighted DRGs. In fact 
, in the third scope of work, HCFA has 
proposed to add further restrictions on the PROs ' preadmission

review by requiring the PROs to choose 8 from a list of 11

procedures determined by HCFA. 
their 
In our discussions with HCFA program staff, we learned that 
process for determining both the second and third scopes of work 
was primarily an internal exercise with some input from the PRO 
communi ty. Al though HCFA staff reported using statistical  consul tat ion in determining the sampling methodologies, we were

unable to determine the reasons for the particular sampling

percentages or trigger points included in the review activities

(e.g., 50 percent sample of day and cost outliers or intensified

review when hospital reaches six cases or 5 percent trigger error

rate) . 
In determining the PROs ' scope of work, HCFA has clearly had to

ting forces for flexibility and uniformity. 
balance competi  While 
the PROs have pushed HCFA to allow them more latitude in deter

mining their approaches to review, Congress has held HCFA

aaccctoiuvntiatibeles .f or maintaining national statistics on the PROs'
However, in reviewing the list of PRO activities 
(appendix IV) with HCFA staff, we learned that most of the

specific requirements for particular types of review derived from

HCFA itself. Many respondents cited the desirability of having

HCFA conduct pilot tests prior to mandating universal review of

PROs. In fact 
activities by all  , HCFA is planning to undertake 
work. 
several pilot tests during the third scope of  Neverthe
less , it is also adding 100 percent review of nine additional

DRGs without first establishing the value of that review. 
Several HCFA central office staff noted that the PROs have been

given latitude to focus on their areas of concern by carrying out

their five objectives. However, many PROs commented that HCFA

has made it very difficult to adjust those objectives when they

have needed to do so. 
Several respondents from each interview category also highlighted

the need for the PROs to have greater flexibility to focus their

review acti vi 
ties on troubled providers rather than performing 
area. 
equal levels of review at all hospitals in their  The HCFA 
is planning to pilot test such an approach in the PROs' third

scope of work.

All these restrictions on the PROs' scope of work seem at least

somewhat to contradict the PRO legislation which stressed the

need to give the PROs more latitude in decision making than PSROs

had.  noted: 
As one medical society representative  "PROs are 
now. 
overwhelmed right  HCFA needs to be more selective with the 
current (review J menu. The tooth of the comb is too fine. 
observed by a PRO review director: "If unnecessary activities

were eliminated 
, PROs could focus attention on areas with higher
yield. " 
The dynamism and broad scope of the PRO proqram have impaired

HCFA' s ability to manaqe the proqram 
Review of the legislative history of the PRO program (see 
appendix III) reveals that, from the beginning, HCFA has had to

implement and administer a complex, ever-changing program which

is of strong interest to the Congress, the Administration, and
 groups. 
diverse national constituency  The PRO program has been 
caught in the cross-currents of debate over national health

policy and of efforts by the Federal Government to ensure high

quality health care while controlling health expenditures.

Controversy over the necessity for and purposes of the PRO

program both within the Federal Government and among various

national constituency groups has surrounded the program from the

beginning. 
The playing out of these various interests has
resul ted 
in a highly dynamic program characterized by multiple 
mandates, shifting emphases , and ever-increasing

responsibilities . 
The work load involved in implementing the PRO program has been

enormous. 
Since the first contracts in 1984 , HCFA has had to 
develop and implement management systems for planning, monitoring

analysis. In addition, 
and evaluation, and data reporting and 
the requirements of the initial 1982 legislation as well as those

of COBRA (1985) and OBRA (1986) have had to be translated into

cycles. 
contract scope of work requirements for three contract 
(For a summary of COBRA and OBRA requirements., see appendix VI.

Two basic contracts have been negotiated for each of the 54 PRO

areas 
, and, as noted earlier, more than 1, 300 modifications to 
the basic contracts of the first two cycles have been required to

keep up with changes in the program. 
In carrying out its complex mission, HCFA' s Health Standards and

Quality Bureau (HSQB) staff havee nhtaidt iteo sc.o ordinate their

activities with those of other  For example, HSQB staff 
have had to coordinate the contracting efforts with HCFA' s Off ice

(OMB). In addition, 
of Management and Budget  HCFA has had a 
close association with the OIG, which has the authority for

sanction determinations and performs contract audits for HCFA. 
And because the PRO program is funded through apportionment fro

the Medicare trust funds, the Executive Office of Management and

Budget 
(EOMB) has been more directly involved in the 
administration of the PRO program than is usual for most

appropriations. 
government programs that are funded through 
HCFA' s large work load has been accompanied by a high degree of 
instability in staff within and outside the bureau responsible

for overseeing the program. 
Not only has the top leadership of 
HSQB and HCFA changed since the spring of 1986, but virtually all

division director positions within HSQB were filled by new

study. 
personnel at least once during the course of our 
, it 
Given the pace of changes in both the work load and the staff 
is not surprising that HCFA' s management of the PRO program

presents a mixed picture. On the one hand, HCFA has been able to

implement a complicated, evolving program and has made

significant strides since the program' s beginning. On the other

hand , HCFA has often been criticized for being crisis-oriented 
and reactive rather than proactive in its management of the PRO
 program. 
This criticism was reflected in the large number of 
respondents who identified a wide range of specific concerns

related to HCFA' s planning for and implementation of legislative

requirements , as well as operational policies and procedures for 
the PRO program. Such concerns were expressed not only by the

PROs and by local and national constituency groups but by HCFA

always 
staff themselves. As one HCFA manager commented, HCFA is "
reacting to crisis and having to do too much.

One manifestation of HCFA' s problems has been its delay in

mandates. For example, the 
implementing certain legislative 
provisions in COBRA 1985 requiring the PROs to secure second

opinions for certain surgical procedures have yet to be imple

mented. On the other hand, timetables established by HCFA for

the PROs to implement new requirements have sometimes been

unrealistically short. 
For example, in late summer 1987, HCFA 
required the PROs to implement by October 1 the OBRA 1986

' complaints about the 
provision that PROs review beneficiaries 
quality of care in skilled nursing facilities (SNFs) and home

heal th agencies (HHAs). Al though HCFA subsequently extended the 
deadline, the review requirement was nonetheless retroactive to

August 1987. 
In addition, HCFA' s instructions and guidelines to the PROs have

sometimes been complex and unclear. For example, several . PROs 
mentioned the confusing and cumbersome procedures for their

nonqoverage. In addition, 
review of hospital-issued notices of 
HCFA has failed to incorporate certain policy changes into the

PRO Manual, which is its official reference for operating

policies. A number of the PROs and regional office staff

commented that HCFA policymakers have lacked sufficient

appreciation of the programmatic and resource implications

associated with the PROs' implementation of HCFA requirements.

They cited as an example the lack of adequate pretesting of the

generic quality screens. As one PRO staff person commented,

"HCFA needs more upfront thinking and planning so the PROs won I t

always have to play catch up.

In addition to inadequate short-term planning, HCFA also appears

to have inadequately identified long-term goals and strategies

for the PRO program. In our interviews and review of HCFA

materials, we saw no indication of HCFA having a long-range

program. 
perspective on the PRO  Preoccupied with keeping up with 
short-term demands of this complex, dynamic program, HCFaAs h:a s

been unable to address such long-term strategic issues  What 
are future directions for the program? 
What role should PROs be 
playing 5 years from now? How can the PROs best affect the 
quality and utilization of health care? How can the impact of 
. That these questions be addressed is 
the PROs be maximized? 
important for the long-term viability of the PRO program and for

directions. 
the ability of HCFA to influence its future  As previously mentioned I Congress recently extended the contract

period for the PRO program, which should help HCFA have more time

for program planning. But OBRA 1987 also assigned new

responsibili ties to the PROs. Al though the PROs' broad 
responsibilities reflect strong public concern for protecting the

integri ty of the Medicare program I several respondents raised

substantial and legitimate concerns that ever-increasing PRO

mandates could precipitate the program' s implosion rather than

facilitate its improvement.
 RECOMMENDATIONS

Our review of the PRO program reflected that HCFA has already

taken some important steps in strengthening its communication

with the PROs and other external entities as well as in improving

its contracting processes and data requirements. We commend

these efforts and suggest that HCFA continue to strengthen

staff, its 
communication between its regional and central office 
programmatic and contracting staff and between itself and the PRO

and external entity communities. We also suggest that HCFA

continue to reduce the volume and complexity of data required of

the PROs , to improve the accuracy, timeliness , and consistency of

that data and to develop strategies and increase resources to

analysis. 
perform comprehensive data 
Our inspection also highlighted several issues related to

evaluation and planning that deserve more attention by HCFA. 
Those issues , along with our recommendations and rationales for 
those recommendations are included below. 
EVALUATION-RELATED ISSUES AND RECOMMENDATIONS

Issues: 
evaluation tools. 
Lack of integration of HCFA' s 
Inadequate communication of evaluation criteria to the PROs. 
High degree of PRO/SuperPRO discrepancy.

data. 
Inefficient use of SuperPRO 
Inadequate gradations and outcome measures within PROMPTS-2. 
Recommendations: 
The HCFA should:

Improve coordination and integration of its evaluation tools

(PROMPTS-2, SuperPRO and PRO data reports) to ensure

comprehensive assessments of PRO performance. This might

include: 
synchronizing the PROMPTS-2 review with the due dates

of SuperPRO reports; and

incorporating SuperPRO results into the PROMPTS-2

document along with data from PRO reports to maximize

evaluation. 
the comprehensiveness of the  Encourage consistency and objectivity in PRO renewal

decisions by:

establishing rating standards for PROMPTS-2 and for

acceptable ranges of variance with SuperPRO;

formally informing the PROs, regional office staff and

review panel members of the weights to be assigned

SuperPRO , HMO/CMP, and cost/savings factors; and 
establishing a standing evaluation panel of HCFA

regional and central office staff to review regional

recommendations. 
Participation by PRO staff before 
closed session action would permit discussion of mutual

concern. 
questions and 
Reexamine the purpose for and the validity of the SuperPRO

review process. This assessment should address such issues

as: 
whether SuperPRO uses appropriate reviewers and

criteria to render valid judgments about local PRO

decisions; 
whether SuperPRO is intended to complement PROMPTS-2

it; 
medical review or to duplicate 
whether SuperPRO is intended to validate PROMPTS-2

review decisions. If so, whether its sample size,
criteria and record selection process permit valid

comparison; and

whether there are better ways to use SuperPRO

expertise 
, such as having SuperPRO perform onsite 
review and provide technical assistance to those PROs

most consistently differing from SuperPRO in th

review decisions or by having SuperPRO assess what

types of quality problems are being identified by PROs. 
Create forums for SuperPRO, HCFA and PRO discussions.

Examine ways to streamline the SuperPRO process to minimize

the administrative burden on PROs and to maximize its

utility to HCFA, such as:

reducing the sample size for small PROs;

revising the schedule for reporting results to reduce

time lag between PROMPTS-2 and SuperPRO reviews; and

facilitating closer interactions between PRO and

SuperPRO reviewers in resolving PRO/SuperPRO

differences.  . . .
Improve the PROMPTS-2 process by: 
developing standards for judging the manner of PRO

performance, i. e., "Are PRO/provider relationships

effective 
, "Is the PRO successfully tracking
problems 
. 7" 
yes/no" 
establishing standards for rating "
acceptability for overall activity category based on

number of "yes/no" ratings of subcategories; and

developing more outcome measureS of PRO effectiveness

through research and demonstrations.

Develop and release comparative PRO performance data to the

parties. 
PROs and other interested 
Rationale: 
As noted in the introduction to this report, the specific

limitation of HCFA' s evaluation tools has made it impossible for

us or other parties to receive an integrated view of PRO

performance. 
Because HCFA' s evaluation system is central to its 
ability to assess the PROs, either individually or collectively,

it should focus time and attention on strengthening its

individual evaluation tools and integrating them more

effectively. 
PLANNING-RELATED ISSUES AND RECOMMENDATIONS

Issues: 
vi ties. 
Inadequate lead time for implementing new PRO acti 
Inconsistency among PROs and HCFA regional staff related to

interpretations. 
HCFA policy 
Lack of coordination and role clarity among the PROs, fiscal

intermediaries and carriers.

Prescriptive nature of the PROs' review activities.

Inadequate long-range planning by 
HCFA. 
Recommendations: 
The HCFA should:

Develop and distribute requirements , instructions , and

policies in a more timely manner to allow sufficient lead-

time for PRO implementation;  Improve the consistency in policy interpretation among PROs

going 
and HCFA staff by updating the PRO Manual on an on-
basis and by providing training and orientation sessions for

both PROs and regional staff as needed; and

Review the current roles of carriers, fiscal intermediaries

and PROs to assess the appropriateness of the current

distribution of responsibilities and assess the coordination

of these entities, such as the timeliness of F. I.

adjustments. 
strengthen long-range planning of future directions and

by: 
appropriate roles for the PRO program 
increasing the emphasis on research and demonstrations

in such areas as review methodologies, patient outcome

and severity measures and data system requirements;

exploring mechanisms to encourage PRO innovations and

reduce the prescriptive nature of the current review

requirements; 
exploring better long-term ways of structuring PROs

acti vi ties so they are cost-effective and complement 
the efforts of other review entities such as state

medical licensure boards and hospital quality assurance

committees. 
For instance, HCFA might explore the 
possibility of having SuperPRO responsible for making

i ty 
DRG determinations and having the PROs focus on qual 
review; and

establishing an advisory group for PRO and other

relevant entity representatives to provide input to

issues. 
HCFA on long-range planning 
Rationale: 
The PRO program has undergone enormous change since its

beginning. The pace of the program, coupled with a lack of HCFA

staff continuity has contributed to a lack of long-range planning

by HCFA. However, now the program contract period has been 
extended to 3 years and PRO contracts will be staggered over the

next 2 years. Therefore, HCFA staff should now have time to

address more long-range planning issues. As mentioned earlier,

HCFA has made important strides in improving communication with

the PRO and external communities. It is imperative that such

input also be sought in determining the future course of the PRO

program.  p. 

APPENDIX I 
ENDNOTES 
From a speech given by Senator David Durenburger at the

American Medical Peer Review Association conference on

May 19 , 1987. 
The utilization and Oualitv Control Peer Review Orqanization

(PRO) Proqram: Oualitv Review Activities  (Control #OAI-01-

88-00570), Office of Analysis and Inspections, Office of

ices, 
Inspector General, Department of Health and Human Serv 
February 1988 , pp. 17-18.

Ibid. 
, pp. 18-20. 
Victory: 
Challenges
Senator John Heinz, "The PRO Funding 
and Growing Expectations,  The AMPRA Review  (The American 
Medical Peer Review Association), January/February 1988, 
Andrew Webber, " A PRO Report Card,  The Internist: Health 
Internal 
Policy in Practice  (The American Society of 
Medicine), July 1986, pp. 7-9. 
Ope cit., OIG Report Control #OAI-01-88-00570, pp. 10-12. 
Ibid. 
, p. 26. 
Medicare Improvinq Qualitv of Care Assessment and Assurance

, u. S. 
Draft Report  General Accounting Off ice, 
December 30, 1987 , p. 3-38.

Ibid. p. 3-31

10.  Ope cit., OIG Report Control #OAI-01-88-00570, pp. 14-15. 
11.  This issue was also raised in the previously cited GAO
report 
, p. 4-30. 
12.  National DRG Validation Studv of Unnecessary Admissions to
Hospitals 
(Control #OAI-01-88-0080), Office of Analysis and
Inspections 
, Office of Inspector General, Department of 
Heal th and Human Services, p. 4.

13.  Medicare Better Controls Needed for Peer Review 
Orqanizations' Evaluation, U. S. General Accounting Office

GAO/HRD 88-13 , October 1987.

14.  Op. cit., OIG Report Control #OAI-01-88-00570, pp. 10-12.  APPENDIX II

BACKGROUND OF THE PRO INSPECTION

The Office of the Inspector General (OIG) is mandated by statute

to provide leadership and coordination wi 
thin the Department of 
Health and Human Services (HHS). The OIG is charged with 
ensuring the economy, efficiency, and effectiveness of HHS

program operations. 
Because of the PRO program' s vi tal role in protecting both the

quality of medical care provided to Medicare beneficiaries and

the financial integrity of the Medicare program, the OIG has

taken a keen interest in and had a close association with the PRO

program since its inception. Among other activities, the OIG has

conducted preaward audits of the PRO and SuperPRO contracts and

of sanction co t estimates for HCFA and has made sanction

determinations on cases referred by the PROs. 
The impetus for this inspection of PRO performance grew out of

the Inspector General' s personal interest in gaining a broad

perspective on the PROs' performance during the second scope of

work. 
Al though several other entities had reviewed various elements of 
the PRO program (see appendix VII), no one had undertaken a broad

evaluation. Hence, in the fall of 1986 , the Inspector General

asked the Office of Analysis and Inspections to conduct an

inspection of the PRO program in the spring and summer of 1987. 
In addition, the OIG' s Office of Investigations asked us to

incorporate a review of the sanction process into our overall

inspection. We designed the PRO inspection to integrate some

original PRO data collection and analysis with other existing

sources. 
PRO-related data collected from primary and secondary 
Because of other OIG priorities, completion of the PRO inspection

fieldw9rk was delayed until the fall of 1987. 
We designed this PRO inspection to provide the Inspector General

and other departmental officials, policymakers, and the public

with a broad perspective on the PRO program and how it has

changed over time. 
In addition to this inspection, the OIG has done and continues to

program. 
do other work related to the PRO  The following is a 
inspections. 
summary of some key audits and 
Past Work

Review of Financial Operations of Peer Review Organiza

tions 
(Audit No. 14-62158), which concluded that most 
PROs made a sizable profit on their initial contracts

and recommended administrative and fiscal procedures
 for HCFA to incorporate into its management of the PRO

program. 
Transfers, 
Inspection of Inappropriate Discharges and 
March 1986, which concluded that many PROs had not

effectively used the authorities or processes available

to address poor quality of care associated with pre

mature discharges and inappropriate transfer. The 
report included recommendations regarding HCFA' 

reporting and evaluation systems and suggested that

the PROs be given authority to deny payments for

substandard care. 
Report on Evaluation of California Medicare Review

Inc. 
(CMRI) Price Proposal for Development of 43
Sanction Cases (Audit No. 09-61658), September 1986, 
which concluded that HCFA failed to include reimburs

able sanction activities in its fixed-price contracts

the PROs. 
wi th  The report recommended that HCFA: (1) 
provide guidance to all PROs on the reporting of and

accounting for sanction costs; (2) require all PROs to

establish adequate cost accounting systems for

summarizing the costs of sanction activities; (3) 
ensure all PROs develop and implement bid-estimating

procedures that more reasonably reflect the estimated

(4) ensure that 
costs of performing sanction activity; 
all PRO contracts contain the n cessary cost-reimburse-
ment provisions required by- the Federal acquisition
regulations 
(FAR) before initiating any reimbursement 
of sanctions; and (5) require CMRI to provide an

unds advanced for sanction activities

accounting of 
and return any excess funds to the Federal Government.

A subsequent report (Audit No. 09-8661662) was issued

in May 1987 on CMRI' s price proposals for its first 117 
sanction cases. 
That report reinforced the recommenda
tions of the first report and also recommended that

price 
HCFA issue modifications to the PROs' fixed-
contracts to make only the PROs' incremental sanction

costs reimbursable.

Current Work

The Region VII Office of Audit is conducting a national

cost-benefit analysis of five types of PRO review

including retrospective admission, DRG validation, day

outlier, cost outlier, and preadmission. The audit is

focusing on 14 PROs and is expected to be completed

soon. 
(OAI) is 
The Office of Analysis and Inspections 
coordinating the national DRG Val idation study, an

analysis of medical record data collected from 239 PPS
 hospi tals (for the period October 1, 1984 to March 31, 
1985) for DRG validation and identification of quality

of care problems. 
The OAI central office staff are 
coordinating the review but regional staff are respon

sible for analysis of the PRO-related data on DRG

validation. 
That PRO-related report is expected to be 
fall. 
completed by this  APPENDIX III

BACKGROUND OF THE PRO PROGRA 
Creation of the PRO Proqram

The Utilization and Quality Control Peer Review Organization

(PRO) program was created by the Peer Review Improvement Act of

1982 , Title I, subtitle C of the Tax Equity and Fiscal Respon
sibili ty 
Act of 1982 , Public Law 97-248. 
Peer Review organizations (PROs) succeeded Professional Standards

Review Organizations (PSROs) in the provision of Medicare peer

review.  (in 
The PSRO program had been established by Congress 
Part B of Title XI of the Social Security Act) in 1972 to ensure

that health care services provided under the Medicare, the

Medicaid, and the Maternal and Child Health and Crippled

Children' s programs were " medically necessary, conformed to 
appropriate professional standards , and were delivered in the

possible. ,,1 The PSRO 
most efficient and economical manner 
program was a response to increasing Medicare and Medicaid costs

and the failure of existing utilization and claims review

mechanisms to deal with widespread inappropriate usage of health

care serv ices. 
The congressional rationale for replacing the PSRO program with

the PRO concept was based on the fact that the PSRO program had

"been faced with certain structural problems: overregulation and

laws 
too detailed specification in  (had) restricted innovation in 
new approaches to review. " 2 
The PRO legislation emphasized greater accountability by requir

ing PROs to have performance-based contracts with specific

measurable objectives. (For the complete PRO statute , see

42 U.  C. Sec. 1320C.  The PRO legislation addressed concerns

prospective
about the potential negative incentives' of the 
payment' system (PPS) for increases in hospitalizations and 
reductions in the quality of care provided to Medicare

beneficiaries.  systems, 
Compared with former cost-reimbursement 
PPS gave hospitals much stronger incentives to increase Medicare

payments by increasing their number of admissions and to reduce

earlier. 
costs by limiting services or discharging patients 
Hence, the PROs were charged with monitoring the system to

protect against potential provider abuses such as unnecessary

admissions , substandard care, and premature discharge. 
In authorizing the Secretary of the Department of Health and

Human Services to enter into contracts with PROs, Congress 
specified that such organizations must be composed of or have

available to them a substantial number of licenseda rdeocat.o rs of

medicine or osteopathy who are practicing in the  Hence, 
the PRO program is composed of both physician-sponsored and

physician-access organizations.  First Scope of Work (1984-86) 
The PRO program was implemented in 54 states and territories

through 2-year, fixed-price contracts with " 
peer review 
organizations. " Each of the 50 States, the District of Columbia

Puerto Rico, and the Virgin Islands was designated as a separate

Islands, 
PRO area. Guam, American Samoa, the Northern Marianna 
and the Trust Territory of the Pacific Islands were designated a

single PRO area.

The first contracts, which became effective over a 5-month period

from July to November 1984, emphasized detection of inappropriate

system. To that 
utilization and payments under the new PPS  end, 
contract goals included reducing unnecessary admissions, ensuring

that payment rates matched the diagnostic and procedural

' in patient 
information contained  records, and reviewing patients 
transferred or readmitted wi thin 7 days of discharge to determine

whether readmission was for the same condition as the first

hospital visit. In addition, each PRO contract included at least

five objectives: reducing unnecessary readmissions because of 
substandard care during the prior admission, ensuring the

provision of medical services critical to avoidance of

unnecessary patient compl ications, reducing un

rist 
necessary surgery or other invasive procedures, reducing the 
of mortality, and reducing avoidable postoperative or other

complications. The PROs were also expected to develop and

analyze Medicare patient data to identify instances and patterns

of poor quality.

When the PROs identified problems with given physicians or

hospi tals, they were expected to address those problems through

education and consultation, intensified review, or denial of

paYment for care that was not reasonable or was provided in an

inappropriate setting. The PROs were also authorized to recom

mend the sanction of physicians or providers in cases of a

substantial violation" in a "substantial number of cases" or a

case. 
gross and flagrant" violation even in a single  Such cases 
were referred to the Inspector General' s Office for review and

sanction determination. 
(A further discussion of the sanction 
process appears later in this appendix.

Second Scope of Work (1986-88) 
During the first contract period, several entities, including the

General Accounting Office, the Inspector General of the Depart

ment of Health and Human Services, and the Rand Corporation,

studied the PROs' performance and recommended that their quality

review be strengthened. 
(See appendix VII for a summary of PRO-
related studies. ) 
In response to these findings and general
pressure from wi thin and out of the Department, HCFA strengthened 
the quality review requirements in the second PRO contracts which
 1986. In 
began in July  those new contracts, the 44 PROs that 
were responsible for review in the 54 PRO areas (see 
contract)
appendix VIII for a summary of PROs with more than one 
had the following requirements: 
review of readmissions to the same hospital within 15

days; 
review of a sample of discharges to assess whether

there was evidence of premature discharge or transfer;

review of hospitals with unexplained statistical

outliers in the PRO data on high mortality rates or

utilization patterns;

application of a standard set or quality-related

criteria 
(called generic quality screens) to all cases 
review. 
selected for PRO  These six generic quality 
screens included adequacy of discharge planning,

medical stability of . 
patient at discharge, deaths 
nosocomial infection, unscheduled return to surgery, 
and trauma suffered in the hospital; and

development and implementation of community

outreach programs.

Thus , the second scope of work intensified the PROs ' review
requirements. In 
addition to generic quality. screens , all 
records selected for retrospective review for any reason were

also subjected to admission review, DRG validation, coverage

review , and discharge review. 
(See appendix IV for a summary of 
PRO activities for the second contract period.

The Senate Finance Committee staff in a background paper, 
Quality and Access to Health Care Under Medicare' s Prospective

Payment System " noted that "these changes in the PRO review

effort were designed to increase detection of premature

discharges; to improve review of care in the hospital, particu

larly the detection of situations where under-service may impact

the quality of patient cases; and to improve the patients' under

standing regarding their rights and appeals under the system.
 " 3

(See appendix V for a summary of the differences between the

first and second scopes of work.

COBRA and OBRA 1986 provisions:

The PROs' responsibilities were substantially increased through

provisions of the Consolidated Omnibus Budget Reconciliation Act

of 1985 (P. L. 99-272, commonly referred to as COBRA) and the 
Omnibus Budget Reconciliation Act of 1986 (P. L. 99-509 , commonly 
1986). 
referred to as OBRA  The COBRA legislation required the  implementation of preadmission review for 10 surgical procedures

and preprocedure review of any cases involving assistants at

cataract surgery. It also gave PROs the authority to deny

payment for quality of care concerns, with the expectation that

this authority would complement rather than conflict with the

PROs' sanction authority. The OBRA 1986 legislation extended the

PROs ' review from only inpatient hospital settings to ambulatory 
and posthospital settings. Over the next several years, the PROs

will be expected to review care delivered in hospital outpatient

departments, ambulatory surgical centers, skilled niucresisn.g 

facil i ties  , and doctors' off 
, home health settings 
addition, in certain States,  PROs are performing quality review

of health maintenance organizations (HMOs) and competitive

(CMPs).  sumary of 
medical plans  (See appendix VI for a  COBRA 
and OBRA 1986 provisions.

Visibility and Vulnerability of the PRO Proqram

As reflected in the legislative history, the scope of the PRO

program significantly expanded after its inception. That

expansion has been accompanied by extensive scrutiny from many

oversight entities within Government and from provider and

consumer groups outside Government. To date, Congress has held

numerous hearings related to the PRO program and numerous

research and oversight entities have conducted PRO-related

evaluations (see appendix VII) . 
The complex identity and inherent vulnerability of the PROs were

sumed up by 
one PRO spokesman: 
It is clear from my vantage point that PROs are quickly

Inspector 
becoming all things to all people.... The 
General of the Department of Health and Human Services

is searching vigorously for a policeman of the

marketplace. The Executive Office of Management and

Budget is looking hard for cost contai ment services, 
particularly to hold the line on Medicare admissions. 
The Medicare beneficiary community earnestly desires a

protector of quality as the incentives of diagnosis

related group (DRG) payment and capitated arrangements

invi te under-service. 
Heal th care consumers seek ready 
access to the information that review activities can

generate. How else will a competitive marketplace

work? Local practicing physicians remain wedded to a

responsibility to monitor and evaluate their own

practice behavior. All this and more for one fifth of

1 percent of the Medicare hospi tal trust fund (the PRO

budget for a single year) . . . . Can there be any doubt 
that PROs will surely fail on someone' s scorecard?4
 The PROs' Ouality ReVlew and Intervention Procedures

As part of their ongoing quality assurance efforts, the PROs draw

a sample of hospital records for a review of both quality and

utilization elements. 
Those records are reviewed on a case-by-
case basis, using six HCFA-generated generic quality screens and

discharge criteria, as well as PRO-specific screens. The PROs 
employ nurse reviewers, or other health care professionals, to

perform the initial review of records. They refer any potential

quality cases to physician reviewers for final determination.

The PROs also identify potential quality problems through

profiling, 
" in which they use their data system to identify 
care. 
patterns of inappropriate 
Once the PROs have determined that a qual i ty problem exists  they
are required to initiate corrective action which may include the
following: intensified r view , alternate timing of review 
education, and sanctions.

Intensified review involves sampling a larger percentage of a

s records (often 
particular physician' s or provider'  100 percent) 
in the subsequent quarter to verify whether or not the identified

problem has continued. Intensified review may also be used after

contact with the physician to ensure that the particular problem

has been corrected. 
Although most PRO review i  done on a retrospective basis, the

PRO may choose to alter the timing of that review to address

particular problems. 
For instance, if a physician has a large 
number of unnecessary admissions, the PRO might initiate

preadmission review of the physician' s patients.

The PRO may also require that a doctor enroll in continuing

medical education. This could include the physician' s taking

specialized courses or possibly retaining a physician consultant

to rev iew his or her cases.

If the PRO determines that corrective action has failed to

address the quality problem adequately, the PRO is expected to

recommend the physician or provider for sanction. 5

The Sanction Process

The Secretary of Health and Human Services is authorized to

impose sanctions on Medicare-reimbursed physicians or providers

substantially" 
if they have "grossly and flagrantly" violated or "
failed in a "substantial number of cases" to comply with their

statutory obligations to provide (1) services " 
economically and 
only when, and to the extent they are medically necessary,  (2)
services that are " 
of a quality which meets professionally
recognized standards of health care, 
" and (3) services that are 
properly documented. The Secretary may impose one of two
 " .
sanctions: 
actual 
(1) a monetary penalty for no more than the "
or estimated cost of the medically improper or unnecessary

services so provided" or (2) exclusion from the Medicare program

time. 
for a specified period of  (For detailed sanction
provisions 
, see 42 U.  C. Sec. 1320C-5. 
The PRO must provide the practitioner or provider with

reasonable notice and opportunity for discussion" before making

its recommendation to the Secretary. Under the regulations,

(42 CFR part 1004) the provider or practitioner is entitled to an

opportunity to submit additional information and/or meet with the

PRO to discuss an allegation of "gross and flagrant" viola

with an allegation of " 
tion (s).  substantial" violations , the 
physician or provider is entitled to submit additional informa

tion and to receive two notices of potential violation and two

PRO. In either 
opportunities to meet with the  case, if the PRO 
recommends the imposition of a sanction, the physician or

provider must be given 3P days' notice and an additional oppor

tuni ty to submit written comments to the Secretary.

The Secretary has delegated the authority for sanction determina

tions to the Office of Inspector General (OIG). Upon receipt of

the PRO' s sanction recommendation, the Inspector General must

determine whether he agrees with the recommendation and whether

the physician or provider has "demonstrated an unwillingness or

lack of ability sUbstantially to comply with statutory obliga

tions.  , rej ect, 
The Inspector General may accept  or modify the 
sanction recommendation forwarded by a PRO. In cases where the 
PRO has recommended exclusion, the OIG must act on that recommen

dation within 120 days or the exclusion automatically goes into

effect pending final determination by the OIG. 
The peer review statute and implementing regulations related to

the sanction process have attempted to balance the competing

priorities to protect both the rights of Medicare beneficiaries

to receive high-quality care and the rights of physicians and

process. 
providers to receive adequate due  That delicate balance 
has meant that although a physician or provider has had an

opportunity to have at least two administrative entities (the PRO

and the OIG) review a case prior to the imposition of a sanction

the process has deferred a full evidentiary hearing until after

the sanction has been imposed. 6

The PROs' sanction procedures have precipitated ongoing debate

among all parties associated with the PRO program. Organized 
medicine has argued vociferously that the PROs should provide

physicians and providers with stronger due process protection. 
In response to such concerns , HCFA and the OIG held discussions

last spring with the American Association of Retired Persons

(AARP) and the American Medical Association (AM) and developed 
sanction procedures that strengthened PRO notice procedures

clarified the role of an attorney for the physician or provider
 at the PRO discussions, ensured that physicians or providers

would be provided records of the PRO proceedings, and permitted

expert witnesses to provide relevant medical evidence at the PRO

discussions with the physician or provider. 
Over the last year, both organized medicine and Medicare benefi

ciaries from some affected communities have also argued that the

sanction process has adversely affected rural communities since

sanctioned physicians and providers have been excluded from the

Medicare program pending their ALJ hearings. Such concerns led

to a recent legislatively mandated requirement (Section 4095 of

the Omnibus Budget Reconciliation Act of 1987 , P. L. 100-203

commonly referred to as OBRA 1987) for ALJ review prior to the

imposition of an exclusion for any physician or provider who is

practicing in a " 
rural health manpower shortage area" (HMSA) or
in -a county with a population of 70, 000 or less, unless it is 
serious risk" 
determined that the physician or provider poses a "
to Medicare beneficiaries. In addition,' Congress has directed 
the Secretary of HHS to conduct a year-end study of how the PROs

new standardized due process procedures have impacted the PRO

program. 
As of December 31, 1987 , the OIG had received 151 referrals from

areas. 
38 of the 54 PRO  For a detailed discussion of sanction 
issues and statistics , see our recent report on the PROs'
sanction activities. 
Administration and Oversiqht of the PRO Proqram

The HCFA is responsible for administering and overseeing the PRO

program through its Office of Medical Review in the Health

Standards and Quality Bureau (HsStQaB)f.f .T hese functions are shared

by central and regional office  The former group is 
responsible for establishing the operational and evaluation

policies and mechanisms for the program and for negotiating the

PRO contracts. The latter group is responsible for implementing

program requirements and providing regular oversight and techni

cal assistance to the PROs in their respective regions. 
Since the inception of the PRO program, both HHS and HCFA

leadership have changed. In 
response to substantial concerns 
raised about their predecessors' management of the program, the

Secretary of HHS and the Administrator of HCFA have met periodi

cally since 1986 with physician, hospital, consumer, and PRO

representatives to hear their concerns about and suggestions for

improving the PRO program. In response to those meetings, a "PRO 
action plan" has been developed to improve both HCFA' s management

of the program and the PROs' performance and effectiveness. The 
action plan has serVed as a resource for HCFA in its ongoing

efforts to strengthen the PRO program.  The HCFA has faced numerous challenges in overseeing the PRO

program. Like 
the PROs, HCFA has responded to competing expecta
thin and outside the Government. 
tions from wi  For instance, HCFA 
has juggled pressures to make the PROs accountable for quantifi

able outputs with those to give the PROs the proper flexibility

to carry out their mission in an efficient and effective manner.

In addition, HCFA has had to balance the expectation that

addi tional PRO provisions (such as COBRA and OBRA) would be 
implemented quickly with the pressures to follow formal mechan

isms. All 
the while allowances had to be made for the limita
tions of available quality review technology. 
The HCFA has also 
had to juggle its mandates to carry out congressional intent for

the PRO program and to operate within the apportioned funding

levels prescribed by the Executive Office of Management and

Budget. 
During the second contract period, HCFA has used three mechanisms

for evaluating PRO performance. For ongoing monitoring of their

activities 
, HCFA has required the PROs to submit over 20 separate
reports 
, 90 percent of which are submitted either monthly or
quarterly. 
For more overall evaluation of the PROs , HCFA employs 
the PRO Monitoring Protocol Tracking System (more commonly

referred to as PROMPTS-2), which HCFA regional staff administer

twice during the course of the contract period. PROMPTS-2 is

structured to assess the PROs' performance in 17 different areas

such as medical review activities, community outreach, data

requirements 
, sanctions , and management internal control through
a series of yes/no questions. 
As part of the PROMPTS process 
HCFA regional office staff draw samples of cases reviewed by each

' determinations. 
PRO to validate the PROs 
In addition to the regional staff review of PRO ae erminations 
HCFA has contracted with Systemetrics, Inc. (more commonly

referred to as SuperPRO) since the PROs ' first contract period to

biannually re-
review a sample of approximately 400 cases from 
areas. In its 
each of the 54 PRO  blind re-review of these cases 
SuperPRO uses the generic quality screens required by HCFA along

with the PROs' individual criteria and then compares its deter

PRO. 
minations to those previously made by each  More specifi
cally, SuperPRO validates the PROs' admission review, discharge

review , DRG determination, and quality review determinations and 
' medical review 
assesses the PROs  criteria and the appropriate
ness of referrals made to PRO physician reviewers by their nurse

reviewers. 
SuperPRO issues draft and final reports of its 
findings to HCFA and the PROs and regional HCFA staff are

expected to follow-up on discrepancies identified between PRO and

SuperPRO determinations.
 The PROs' Relationship to HCFA' s Fiscal Intermediaries and

Carriers. 
In addition to its contracts with the PROs, HCFA contracts with

two other types of entities that are important to the PRO

program. Fiscal intermediaries (FIs) process claims for services
covered by Medicare Part A, including inpatient hospital, skilled

nursing home, hom  health, and hospice services. Carriers 
process claims for services covered by Medicare Part B including

physician, and laboratory and diagnostic services. Like the 
PROs, the carriers and FIs are both required to conduct utiliza

tion review activities, and the intermediaries are required to

conduct quality-related reviews of skilled nursing facility and

home health claims.

The PROs are dependent on the fiscal intermediaries to forward

them the hospital billing information which triggers the PROs

selection of cases for review. The FIs are also expected to make

taipoprnopsr.i ate payment adjustments based on the PROs ' DRG determina

In addition, the carriers are expected to make adj ust
ments to physician billing in response to all hospital and

skilled nursing facility denials forwarded to them by the FIs. 
As the PROs begin to conduct nonhospi tal reviews, their need for

intensify. 
close coordination with the carriers will 
Al though the 
FIs and carriers are overseen by a different bureau 
within HCFA than the one responsible for the PROs, it is

obviously important for HCFA to ensure tne close coordination of

these review entities.
 ENDNOTES 
Federal Reqister, Vol. 49, No. 39, February 27 , 1984

p. 7202.
Tax Equity and Fiscal Responsibility Act of 1982, Commerce

37D. 
Clearing House, Inc. p.  Senate Report, 97-494 at 41. 
"Examination of Quality of Care Under Medicare' s Prospective

, II Hearing before the 
PaYment System  Committee on Finance, 
Uni ted states Senate, June 3, 1984, p. 44. 
Andrew Webber  II A PRO Report Card  The Internist: Health 
Policy in Practice  (The American Society of Internal

Medicine), July 1986, pp. 7-9.

Material on the PROs ' quality review process was  taken from

Conqressional Research Service Report for Conqress: The

the 
Peer Review Orqanization Proqram, The Library of Congress,

October 23,  1987 , pp. 20-21. 
Material for the description of the sanction process was

taken from the transcript of Inspector General Richard

Kusserow' s testimony at a hearing before the Subcommittee on 
Intergovernmental Relations and Human Resources, Committee

, united States 
on Government Op2e0ra,t.i o1ns9 &7.  House of Representa
tives 
, October 
The Utilization and Qualitv Control Peer Review Orqanization

(PRO) Proqram: Sanction Activities  (Control # OAI-Ol-88-

00751), Office of Analysis and Inspections , Office of

Inspector General, Department of Health and Human Services 

March 1988.  -;,   ... :_- ---.__. ..--- ...... .._.,.
APPE;\DIX IV 
PRO ACTIVITIES

Second Scope of Work 
(198&-1988) 
(C) DRG VAUDAT10N (D) ADMISSION  (E) COVERAGE
I.  (A-)dR DtEeIVScIC EHWA RGE (-d8)i SGCESNaECRrIgRCe QE UpAElUaTNnnSin g  ---.PrcPS.  pCa)O'ta  .4R8:E1Ve8IE.W -fB.nIEErtWe
meic
pdreimgateurse/  edic stilit  8ttion  ttrnan
-atd diSsCarge
..W8 inons  f'uiremems
ia 
ule nrm to surgery
-tuma in 
RETROSPECTIVE  OTHER ACTIVITIES r8 
pr 
1. Re 3% random .aple from e.ch PPS hoitl  18. iDd8  ab of da  on a qua bas to 
Induding a 6 mont report of short -ys (1-2 days)  pr, 
2. Reew.n cas rerred by the Fisc Inrmiary for  20. Implment comuanitp ou ealeucn prras
ne 
OOrage isaues  21. Impl HCA idnt  -pi whcnoevvere  
cas  an t8e 
3. Review all  Identied by the Meice Coe Editr  quait 1881 are

coes)  acons:
(9 ICD-9 
education 
4. Review an percutneous lithotripsy claims in hospitals with 
.xvacorporeal shocave litotripters  uInntcentsoinie d review 
5. Reew an submitted claims adjustments resulting in a higher  22. Mae waivr of liabilty determinatons 
Re 
weighted DRG (review prior to payment) 
23.  HMOs and CMPs for quality and approprateness o 
6. Review 15% random sample of discharges from PPS exempt  Inpatient and outent servces (COBRAOBRA) 
hositls and swing be unit to which beneficiary directly 
admited"  24. Reew wren coplaint by beneciares conceming qualir:. 
heal 
of cae at hosits. SNFs, home  agencies, and 
7. Review all transfers from PPS to other PPS hospitals  HMO/CMPs(OBRA 
8. Reiew SO% random sample of day and oost outliers 
PENDING ACTIVITIES 
9. Conduct fose DRG reviews: 46.46. 
10. Review notices of non0(rage issued by PPS and  Reew ten surgica prOOures on preadmisson! 
non-PPS hospitals:  p!eprOOure bas to determine medical nessity 
(CORA) 
100\: of cases where physician disagrees 
100\: of cases where patient disagrees 
Deny payment for substandard quality of care (COBRA
100% of cases of patient cost liability 
10% random sample of remainder  Reew alille nursng facilits and home health agencies 
11. Conduct Intensified Review when hospitl reaches  for qualitiyn of fcoa. r( OBRA) 
6 cases or 5% trigger error rate  Prde  on spfic cas or patterns of sub. 
12. Review 10% or 120 discarges in quarer for validation of  ataacrdi tctain. t ob 8otes l icensing. stte certication, or natlor-
obecivs- aI  (OBRA) 
ho 
Re  Re ..rvl 
proed 
13. ac  all related readmissons wiin 15 days from a PPS  me an  In  O\tient depar
Incaaet b e to any ace hospit (pPS or non.PPS) for ambulatry surgery ceters COBRA
Re  cuasd mwi8i8 nonoa   C(OouBc sml ar Mas in at I8 twlv PR areas 
14.  all  red admiuions 
adv  Ae 
15. gapPen8rn re/frrrpimec ru a5ace o ceunbo,ne nbncyr sep,  rrtadoyn ee drla/i mbpdymrina Daic8tteR8t io)o nnle arn.  apnorduO btOymu DreResGs  )( b(byy   fIPanlPri tvheisn aooglrl   lroce uaafdot mpi Inlrn8td8 pdaer wo liIpinp nhe a3 o1a, dsd maf.yio as a actloni laPlo eaP nn  ua rrnseenn entgewo r e.ad n , 
Re  In afr Jaua 1,1887 (OBRA 
16.  transfers to non-PPS, exempt distnct unit of acue 
hoitls
re 
5500  rraadnomm  ssaappllee  ttoo  eNxicnlugd bede saloool/dNg abuse unit  KEY  Admi88 an OOrage  only 
2S% random saple to excluded rehablitation unit  In lieu of PR validatin, f8ew meica nessty 
To excluded psyc unit:  an lengt of sty 
100% of recrds showing invalid ICD-9 coes 
100% of recrds showing organic brain conditions  -- Reew for ap.propriateness of transfer , admissior
and length Of sty, ss well as determining the medica:
10% random sample of remainder  necssity of length of sty. Do not perform DRG
validation 
PROSPECTIVE 
-.. Discntinued during the aend scpe of work 
I .  17. Preadmission review of pacemaker and 4 procdures  Noe:  This chart was cre.ted by Region I DHHS/OIG/OAJ
propose by PRO that add up to 2.5% of sttewide  in consulttion wih HCFA. 
discharges. 
18. Reew all cases wi assstnts to cataact surgery to
"'_6__  ;.A___.... -' . ....i-- APPENDIX V

COMPARISON OF 1984 SCOPE OF WORK TO 1986 SCOPE OF WORK

Review Area
 1984  1986 
Objectives  3 Admission Objectives  5 Obj ecti ves 
5 Quality Objectives
 Based on PRO data

All proposed and vali
 from first 90 days

dated by PROs. 
Very  of generic quality

limited areas for
 screen review. 
focusing objectives  HCFA-identified 
outl iers . 
Broader obj ecti ves 
Random Samples  5% Admission Sample
 3% random sample

DRG Sample ranging
 (includes 1- and 2

day stays) 
from 3% to 100%

based on hospital

discharge size 
Preadmission Review
 5 Procedures proposed
 Pacemakers plus 4

by PRO
 procedures pro
posed by PRO

Pacemakers 
100% retrospective
 100% preadmission

(see above)

Transfers 
From PPS to another
 Same but lower

hospi tal, exempt level of rev iew 
unit 
, swing bed 
Readmissions 
All readmissions
 All readmissions

within 7 days
 within 15 days

Medicare Code
 100% of 9 diagnoses
 Same 
tor 
Edi 
Focused  468  468, 462, 088

(462 added during 
DRGs 
contract period)

Outl iers
 100% (reduced to 50%
 50% 
during contract
period) 
Percutaneous 
Not in contracts
 Review all claims

thotripsy 
Li  for percutaneous

lithotripsy in

hospitals which

have an extra

corporeal shock wave

lithotripter  Review Area
 1984  1986 
Validation of 
Not in contracts
 Sample of one

Objectives 
quarter' s dis
charges to validate

obj ecti ve performance 
Notices 
Hospi tal  100% where patient
 Same 
or physician dis

agrees. 
100% where
patient is liable. 
10% of remaining

Hospital
Special ty  Proposed by each PRO
 15% of discharges

Review 
Admission Pattern
 Discontinued during Not in Scope of Work

toring  contract 
Moni 
Intensified Review
 Tr igger:  Trigger: 
5% or 3 cases
 5% or 6 cases

(whichever is greater)
 (whichever is 
of cases reviewed
 greater) of cases

to:  reviewed. 
Review increased  Review 
100% or subsets
 increased to: 50% or

quarter; 
subsets (first 
100% or subsets (two c: 
more consecutive

quarters) 
Outreach 
Communi ty  Not in contracts
 All PROs to propose

program 
*All cases reviewed

are subject to generic

screens,
qual i ty 
discharge rev iew , 
admission review, DRG

validation, and coveras 
review. 
Source: 
HCFA  APPENDIX VI 
SUMRY OF RECENT PRO-RELATED 
LEGISLATIVE PROVISIONS

STATUS OF IMPLEMENTATION OF COBRA 1985 PROVISIONS

Assistants at Cataract Surqerv

Provides that no Medicare payment may be made for an

assistant surgeon at cataract surgery unless carrier or

PRO approves use of assistant before procedure is

performed. 
Also prohibits physician from knowingly and 
willfully billing Medicare beneficiary if he or she has

not obtained prior approval and where presence of

assistant surgeon has been found to be unnecessary.

Instructions for PRO review of this activity, effective

wi th assistants proposed to be used after March 

1987 , were issued to PROs on December 30, 1986, and 
review has been implemented. 
PRO Denials for Substandard Care

-determines 
Provides for denial of payment when a PRO 
that the quality of health care services rendered to a

Medicare beneficiary fails to meet professionally

care. 
Also specifies 
recognized standards of health 
that denials for care of substandard quality shall be

made only on the basis of criteria that are consistent

wi th guidelines established by the Secretary.

followed. Regula
Formal rule-making process is being 
tions are in the final stages of departmental clearance

and will be published soon. 
PRO 100 Percent Preprocedure Review

Requires peer review on a preadmission/preprocedure

basis of nonemergency cases for at least 10 surgical

procedures. 
Second opinion will be required if PRO 
cannot make determination as to medical necessity of

services. 
followed. 
Formal rUle-making process will be  Proposed 
regulations are still in process of departmental

clearance.  STATUS OF IMPLEMENTATION OF OBRA 1986 PROVISIONS

Review of services provided in hospital outpatient

Effective 
departments and ambulatory surqical centers 
for PRO contracts entered into or renewed on or after

1987. 
January 1,  This provision has been implemented 
by the Pennsylvania PRO, which entered a new contract

period on July 1, 1987. It will be implemented by

other PROs as they enter their next contract periods.

Review of hospital denial notice  Implemented on

statute. 
December 1, 1986, as required by the 
FIs must provide PROs with "timelv" monthlv informa

tion. or hospitals will be reauired to provide such

Effective April
information directlY to the PROs 
1987. 
Review of at least a sample of readmissions occurrinq

wi thin 31 days of discharqe and any interveninq post
care 
hospi tal  Effective for contracts entered into or 
1, 1987. 
renewed on or after January  This provision 
has been implemented by the Pennsylvania PRO and will

be implemented by the other PROs as they enter their

next contract periods.

A reasonable proportion of PRO funds must be allocated

to review of auality of care provided in all settinqs

The HCFA has no plans ' for separate implementation of 
this provision. It will be implemented as part of

other OBRA provisions.

Review of HMOs/CMPs  The HCFA published a listing of

Federal Reqister

States to be competitively bid in t1he 9
 87. 
on January 5,  Contracts have been awarded in 
applicable. 
those States where it is  Review began July 
1987 
The Secretary is to identifY and make available to PROs

methods of identifyinq those cases that are more likel 

than others to be associated with substandard quality

of care. and to provide at least l2 PROs with data and

data processinq assistance to perform small-area

analysis 
Both provisions effective upon enactment. 
The first is an ongoing acti vi ty . The HCFA has

contracted with the American Medical Review Research

Center (AMC) for. the small-area analysis which will 
12 pilot PROs. Information is 
utilize feedback from 
presently being gathered.
 PRO boards must include at least one consumer

representative  Effective with contracts entered into 
1, 1987. Officially 
or renewed on or after January 
implemented with new contracts, but most PROs have

already implemented this provision.

PROs must respond to beneficiary complaints about poor

aualitv care provided in all settinqs  Implemented 
1987. 
October 1,  PROs had already been required to 
respond to complaints referred to them and will

process. 
continue to do so. Clarifying regulations in 
PROs will be reauired to share (when reauested)

information relatinq to substandard care with State

licensure or certification bodies and with national

Effective April 1, 1987. Clarify
accreditinq bodies 
ing regulations in process regarding exchange of

information with licensure boards. Clarifications 
regarding exchange with state Medicaid agencies have

been publ ished . 
Hospi tals. home health aqencies. HMOs. and skilled

nursinq facilities will be reauired to have aqreements

with PROs. under which costs of PRO review activities

are to be paid bv the Secretary to the PRO  Effecti ve

October 1, 1987. 
Source: HCFA/HSQB/OMR.

Other PRO-Relevant Leqislation

The Medicare and Medicaid Patient Program Protection

Act of 1987 greatly expanded the sanction and civil

monetary penalty authorities under the Medicare and

Medicaid programs. The bill also required the report

ing of all disciplinary actions made by state medical

licensure boards.

The recently enacted 1987 Omnibus Budget Reconciliation

Act has made the following PRO changes: 
three-
year PRO contracts with staggered expiration 
da tes ; 
a ban on informing Medicare beneficiaries and

fiscal intermediaries of payment denials before

offering providers or physicians the opportunity

for reconsideration;

Federal Reqister of the

publication in the 

standards used for evaluating the PROs and any ne
 policy or procedure that substantially affects the

performance of contract obligations; and

negotiation of appropriate contract modifications

before implementation of additional review

functions not included in the initial or renewed

contract; 
provision by the Secretary of regular performance

reports to each organization comparing its

performance with other PROs'

a prohibition of automatic renewals of PRO

contracts held by out-of-state groups, provided

in-state physician groups wish to compete;

requirement that the hospital notify the Medicare

patient when the hospital requests PRO review

because the hospital and the attending physician

do not agree that inpatient care is no longer

necessary; 
a ban on physicians billing Medicare patients for

assigned claims denied for payment on grounds of

substandard qual i ty 
a requirement that PROs ,. in establishing review
standards 
, take into account the special problems 
associated with delivering care in remote rural

areas 
, the availability of service alternatives to 
inpatient hospitalization, and social factors that

could adversely affect the safety or effectiveness

of outpatient treatment;

mandatory onsite review in at least 20 percent of

rural hospitals in a review area;

requirement that PROs offer for PRO physician to

meet several times a year with medical and

administrative staff of hospitals in their review

area; 
requirement that PROs publish and distribute to

providers and practitioners, at least annually, a

report describing the types of cases the PROs

frequently determine involve inappropriate or

unnecessary care, services rendered in an

inappropriate setting, or substandard care; 
assessment of access provided to Medicare

enrollees in risk-
sharing HMOs and CMPs and  mandatory beneficiary outreach to inform enrollees

about the role of the PRO and their rights;

a provision encouraging PROs to use physician

specialists in initial review of psychiatric and

rehabilitation cases;

emphasis , when evaluating PRO performance, on the 
PROs' activities in educating providers and

practi tioners, 
particularly those in rural areas 
about PRO review and criteria;

demonstration proj ects 
for the instruction and 
oversight of rural physicians, in lieu of imposing

sanctions 
, through video telecommunications 
between Medicare teaching hospitals and rural

hospitals ; 
entitlement of a provider or practitioner located

in a rural heal th manpower shortage area, or in a

county with a population of less than 70 000, to 
an administrative law judge hearing prior to being

excluded from the Medicare program, to determine

whether the provider or practitioner poses a

serious risk to his or her patients.

a report to Congress to include an assessment of

the sanction due process reforms agreed to by HHS 

the American Medical Association, and the American

Association of Retired Persons , as well as

physician and provider responses to the improved

procedures and an assessment of the appropriate

balance between procedural fairness and the need

for ensuring quality medical care. 
Source: 
Conqressional Record--House, December 21, 1987.  APPENDIX VII

SUMRY OF PRO-RELATED STUDIES 
Because the PRO program is vital to the Medicare program and

arena, several entities
exists within a highly visible political 
have evaluated the program. 
The following is a summary of some 
key studies related to the PROs: 
Past Studies

The Congressional Research Service (CRS): 
"The Peer Review Organization Program " October 23,
1987: 
The study presented a summary of the legislative 
history, program features, and relevant issues of the

PRO program. The CRS report was prepared at the

request of the House Committee on EnergHy aned Caommletrche 
 and the Environment. 
Subcommi ttee on  It revised 
a prior report prepared at the request of the Senate

Committee on Finance. 
The General Accounting Office (GAO): 
"Strategies for Assessing Medicare Health Care

Quality, " December 30, 1987: The study evaluated the 
systems for assessing quality of care in the Medicare

program (i. e., carriers, ' intermediaries 
, and PROs) and 
identified short- and long-term strategies for measur

care. 
ing and monitoring quality of  Among other
suggestions  review the 
, the GAO recommended that HCFA: 
PROs' methods for dealing with quality issues; evaluate

the spheres of responsibilities of the PROs, FIs, and

carriers to determine that their responsibilities are

appropriately divided; requir  the PROs, FIs, and

carriers to maintain data related to quality; require

that patient diagnoses be recorded on Medicare out

patient Part B claims and develop HCFA data files of

that Part B information; and develop a mechanism to

allow SuperPRO to evaluate PRO cases that were selected

through both the PROs' random sample and specific

samples of hospital records.

"Better Controls Needed for Peer Review Organizations

Evaluations  1987: 
" October 8,  The study assessed 
HCFA' s evaluation process for the 1986-88 contract 
awards and concluded that HCFA' s process was fraught

with inconsistent and inadequate documentation and

improper application of instructions. Al though 
GAO 
found no evidence of inappropriate contract decisions

it recommended that HCFA develop sufficient internal

controls for PRO evaluation, provide better ongoing
 , " 

moni toring to the PROs, and collect and use adequate 
cost and performance data to set each PRO' s contract

funding level. 
"Reviews of Quality of Care at Participating

Hospi tals  September 15 , 1986: The study was based

on a survey of California, Florida, and Georgia PROs,

and focused on the monitoring of inappropriate dis-

charges and profiling of hospital and physician quality

of care problems. 
The GAO recommended that HCFA 
require PROs to include quality of care review data

available from the 1984-86 contract period in their

profiling of hospitals and physicians and that the PROs

review the appropriateness of the discharge

destinations as part of their discharge reviews to

better ensure that patients needing skilled nursing

care are allowed to remain in the hospital while

awaiting placement.

The Prospective Payment Assessment Commission (ProPAC): 
"Report and Recommendations to the Secretary, u. S. 
Department of Health and Human Services " April 1

1986: ProPAC, which is an independent commission

established by Congress to analyze and recommend

changes in the prospective payment system, recommended

in its second annual report that better information

about PPS be provided to beneficiaries, hospitals , and

physicians. 
Pro PAC also recommended that PRO review be 
extended to the overall episode of care, including

skilled nursing facilities, home health care , and

outpatient surgery.

The Rand Corporation: 
Kathleen N. Lohr, "Peer Review Organization: Quality
" July 1985: 
Assurance in Medicare  study focused on 
the first scope of work for PROs' review of qual i ty
PPS. 
during the first 2 years of  Rand recommended that 
the quality objectives in the 1984-86 PRO contracts be

broadened .to include the use of generic screens and 
that quality review be given greater weight in the PRO

review activities. Rand also recommended that the

PROs' quality review be extended beyond the hospital to

include the Medicare beneficiary' s entire episode of

care. 
Current and Future Studies bv the GAO

The Financial Integrity Act Group at GAO is currently

reviewing the internal controls for payments by

Medicare intermediaries. As part of that study, the
 group is assessing the effectiveness of the SuperPRO

a control mechanism for PRO performance. A draft

soon. 
report is expected  (Herb Dantzler--Project
Leader) 
At the request of the Senate Aging Committee, the

Program Evaluation and Methodology Division (PEMD) is

exploring how outcome data can be. used to monitor 
care. 
quality of  That study will include a review of 
how PROs use available data in their profiling of

providers. A final draft is 
expected ' by mid-June
1988. 
(Eric Peterson--Team Leader) 
At the request of the House Ways and Means Committee, 
the PEMD has designed a study to evaluate the PROs' 
issues. 
handling of quality of care  Preliminary
descr ipti ve findings are expected in 1 
ate 1988. 
(Jill Bernstein--Team Leader) 
The Human Resources Division (HRD) , is undertaking a

program:
brief review of two aspects of the PRO 
analysis of the lack of information exchange among

PROs, Medicare carriers, state Medicaid agencies, and

State licensure boards , as well as atni eansal.y sis of OIG

practices in imposing monetary penal  The report 
1988. 
is expected to be issued in the fall of  The HRD 
is also contemplating an evaluation of HCFA' 

anagement of the PRO program. (steven Fox--project

Leader) 
See appenQix II for a summary of the OIG' s past and current work

related to the PRO program.
 APPENDIX VIII

THE PROS WITH MORE THAN ONE CONTRACT*

tional 
Addi 
Orqanization Name/Location  PRO Areas Reviewed

Professional Review Organization.  Alaska 
for Washington Idaho
Seattle 
, WA 
Institute, 
West Virginia Medical  Delaware
Inc. 
Charleston, WV

Delmarva Foundation for Medical Care  Inc  District of
Easton 
, MD  Columbia 
Hawaii Medical Services Association 
Guam/American Sa
Honolulu 
, HI 
PEERVIEW , Inc  Kentucky
Carmel, IN

Heal th Care Review, Inc.  Maine
Providence 
, RI 
Iowa Foundation for Medical Care. .  Nebraska 
West Des Moines, IA

New Hampshire Foundation for Medical Care  Vermont 
Dover, NH

for. 
Montana-Wyoming Foundation  Wyoming 
Medical Care

Helena 
, MT 
*Note: Eight PROs hold two contracts; one PRO holds three contracts.  APPENDIX IX 
MEMORABLE  " PRO- ISMS" 
We appreciated the candor and thoughtfulness with which

questions. In 
individuals responded to our  an effort to share 
more of .those diverse opinions than could be integrated into the 
main body of this report, we offer the following examples of

memorable opinions we heard that were related to the PRO program

(i. 
, "PRO-isms" 
Regarding the PROs' effectiveness:

(against 
"This program stands almost alone as the buffer 
poor medical care) for the American public" (from a PRO

medical director).

"We' re spinning our wheels, we' re not moving forward" (from 
a PRO CEO).

are, 
"I wish that I could tell you how effective (the PROs) 
but I don 't know" (from a congressional staff person) 

Regarding HCFA' s evaluation methods:

"There' s a lot more to a PRO than not getting a chart done 
in 30 days" (from a PRO staff member).

"They 
(HCFA) are starting to do some things right" (from a 
PRO staff member) 

They' re looking at numbers and numbers only" (from a PRO 
staff member) 

Regarding factors constraining PRO performance: 
"Some (PRO) responsibilities are exercises in futility" 
(from a PRO CEO).

"The government is like a monstrous behemoth with its legs

mired in the swamp" (a PRO medical director).

period: 
Regarding the length of the PRO-contract 
"It' 
s 4 years for the President" (from a PRO CEO). 
"Give us time to get a little bit better (seated) in the

saddle. " 
For other "PRo-isms" related to the PROs' sanction and quality

review activities, see our previous two reports on those

subj ects .  APPENDIX X

TABLE 1 
PERCEPTIONS OF PRO EFFECTIVENES S  BY CASE STUDY SITE 
CASE  HOW  NUBER OF RESPONDENTS NOTING  EFFECTIVENESS LEVEL 
STUDY  EFFECTIVE  PRO  PRO Staff  HCFA  Hosp.  Medical  Licensure 
SITE  CEO  & Board  Ass  Society  Board 
VERY 
MODERATELY 
VERY 
MODERATELY 
MINIMALLY 
DO NOT KNOW

VERY 
MODERATELY 
MINIMALLY 
DO NOT KNOW

VERY 
MODERATELY 
MINIMALLY 
DO NOT KNOW

VERY 
MODERATELY 
MINIMALLY 
VERY 
MODERATELY 
MINIMALLY 
VERY 
MODERATELY 
VERY 
MODERATELY 
VERY 
MODERATELY 
MINIMALLY 
DO NOT KNOW

VERY  NOT 
MODERATELY  AVAILABLE 
MINIMALLY 
DO NOT KNOW

VERY 
MODERATELY 
MINIMALLY 
NOT 
DO NOT KNOW

VERY 
MODERATELY 
DO NOT KNOW
 APPENDIX X

TABLE 2 
SUPERPRO - PRO DISAGREEMENT RATES FOR FIRST FOUR CYCLES 
CYCLE  RAGE  SIZE OF  AVERAGE 
DIFFERENCES  (1)  RAGE  DIFFERENCE (2) 
DRG CHANGES  - 20.  16.  10. 
17. 
11. 3  11. 5 
16.  15. 
1. 3
14.  13. 
1. 3
ADMISSION  -9.  - 23.  33. 
REFERRLS  (3)  -1. 8  - 25.  27. 
-2.  - 20.  22. 
-8.  15.  24. 
NECESSITY  15.  14. 
1. 8 
DENIALS  16.  11. 
16.  14. 
1. 9
23. 
1. 9
-1. 5 - 20. 
QUALITY 
PROBLEMS  -3.  11. 7  14. 
-2. 
1. 0
11. 
o. Q 11. 2 
PREMATURE  -1. 0 
DISCHARGES  (4) 
1: Range of differences between the percentage of problems. 
cases. 
found by a PRO and SuperPRO when reviewing same 
Negative values mean PRO found more problems than SuperPRO. 
2: Average difference in percentage of problems found
cases. 
by PROs and by SuperPRO when reviewing same 
3: Percent of cases referred to physician for further review. 
4: Data available only for cycle  APPP;DIX
TABLE 3 
REVIEW  DETERMINATIONS FOR  CASE  STUDY SITES 
DRG  ADMISSIONS  NECESSITY  QUALITY  PREY.A TURE
CASE  CHAGES  REFERRLS  DENIALS'  PROBLEMS  CISCHAGE
STUDY  PRO  DIFF  PRO DI FF  PRO DIFF  PRO D1:FF  PRO DI FF
SITE  CYCLE 
21. B -5.  1.0  14.  1.0 
13. B 21. 6  13. 
19.  1.0  -0.
19.  -0. 
11.8  12. 
1. 7 
12.  1.6 
11.9 
10.  12.  10.  1.6 
10.  21. 
3. B B. 3

1.5
10.  15. 
2 . O 
15.  -0. 
11.4 
13.  15.  1.3
10.  7.. 
12. 
:2. B
 1. 0 
7. B
10. B () . 2 
14. 
2. B :2 . 3 
14.  8. B  12.  1.9  1.4 
14.  11.1 
11. 
1.3 
22.  15. 
17.  16.  12.
16.  14. 
14.  10. 
11.4  13.  11. 
1.3  17.  11. 
11.1  10. 
13.  10. 
14. 
16. 
12.  12. 
12. 
12.  1.8  1.6 
11.1  10. 
1. 3 "1. 
11. 
7. B :2.  10. 
All numbers are percentages.  PRO:  Percent of problems found by PRO. 
DIFF: Difference bet  een percent of problems found by SuperPRO and PRO. 
A necative value means the PRO found more problems than SuperPRO. 
* PROs seen as at least moderately effe tive by all respondents groups.
 APPENDIX X

TABLE 4 
COMPARISON OF THE PERCEPTIONS OF THE EFFECTIVENESS OF

SUPERPRO .AND PROMPTS IN EVALUATING PRO PERFORMNCE 
RESPONDENT  EVALUATION  PERCENT NOTING HOW EFFECTIVE

GROUP  TOOL  Very  Moderately  Minimally  Not Don't Kn 01: 
PRO  PROMPTs 
CEOs  SuperPRO 
PRO  PROMPTs
Staff 
SuperPRO 
HCFA Regional  PROMPTs
Office 
SuperPRO 
HCFA Central  PROMPTs 
Off ice  SuperPRO 
Total for All 
PROMPTs 
Respondents  SuperPRO  APPENDIX XI 
METHODOLOGICAL NOTES

Summary of Interviews

Because we wanted to examine how both the PROs and HCFA viewed

themselves and were viewed by others, we conducted in-depth

(approximately two-hour) interviews with a variety of people

associated with the program. 
Those 211 individuals included the
following: 
PRO chief executive officers (i. e. 
, all PRO CEOs , as 8
of the 44 PROs manage 2 PRO areas and 1 PRO manages

areas; 
other PRO staff and board members (i. e., 
the medical
directors 
, program directors , rev iew directors , board
chairs 
, and consumer representatives from the 12 PROs 
selected for case study site visits) ; 
national external enti ty representatives (i. e. 
, the
American Association of Retired Persons
, the American

Hospital Association, the American Medical Association,

the American Medical Peer Review Association
, and the

Public citizen Health Research Group as well as the

Department of Health and Human Services, Executive

Office of Management and Budget, and congressional
committee staff);

local external entity representatives (i. e. 
, state
medical societies 
, medical licensure boards , hospital
associations 
, fiscal intermediaries , and the American 
Association of Retired Persons chapters associated with

the 12 PROs selected for case study site visits) ; 
Heal th Care Financing Administration (HCFA) central

office staff (i. 
e., from the Health Standards and 
Quality Review Bureau, the Office of Management and

Budget , and the Bureau of Program Operations); and 
HCFA regional office staff (i. e. , all 10 Associate 
Regional Administrators for Health Standards and

Quali ty, all 10 branch chiefs, and a sample of the

proj ect 
officers in the Medical Review Branch) . 
Case Study Selection 
In an effort to gain a firsthand perspective on the 
PROs'
operations 
, we made 3- to 4-day site visits to at least 1 PRO 
from each of the 10 HCFA geographic regions. As part of that

case study effort 
, we planned to compare those case study 
assessments to HCFA' s PRO-specific evaluation documents (i. e. ,  PROMPTS and SuperPRO) for the second scope of work. Hence , we

eliminated those PROs with a November 1, 1987 contract start date

(i. e.
, Group 5 PROs) from the case study selection pool since we 
would be unable to obtain their corresponding HCFA evaluation

documents in time for review. We also eliminated the Pennsyl

vania PRO from the selection pool since its second contract

period began only on July 1, 1987.

We then drew a judgmental sample of the PROs that was based on

size 
the following criteria:  (as reflected by funding level),
geographic location 
, and sanction activity level. We divided the 
PROs into four groups according to their Medicare contract awards

($2. 9 million or less, $3-5. 9 million, $6-8. 9 million, and $9 
million or more) and calculated the appropriate number of PROs to

select from each funding category. That selection of particular

PROs focused on ensuring a group of' PRO sites with a distribution 
of geographic areas (i. e. , at least one PRO from each of the 10 
HCFA regions) and of sanction activity levels and with at least

some representation of PROs that had both Medicare and Medicaid

contracts. 
The final group of 12 organizations selected for site visits in

the 10 HCFA regions were as follows: 
HCFA Reqion  PRO Area  Orqanization 
Massachuset ts  Massachusetts Peer Review

Organization 
, Inc.
Waltham , MA 
Rhode Island  Heal th Care Review, Inc.

Prov idence , RI

New York  Empire state Medical, Scientific anj

Educational Foundation, Inc. 
Lake Success, NY

Delaware  West Virginia Medical
Insti tute 
, Inc. 
Charleston, WV

Florida 
Professional Foundation for Health 
Care, Inc. 
Tampa , FL 
Georg ia  Georgia Medical Care Foundation

Atlanta, GA
 HCFA Re  ion  PRO Area  Orqanization 
Indiana 
PEERVIEW , Inc.
Carmel , IN 
Texas  Texas Medical Foundation 
Austin, TX 
Iowa  Iowa Foundation for Medical Care 
West Des Moines , IA 
Colorado  Colorado Foundation for 
Medical Care 
Denver, CO 
California 
California Medical Review, Inc.
San Francisco 
, CA 
Oregon  Oregon Medical Professional Review
Organization 
Portland, OR 
In the case study selection process, we opted to choose PROs

according to their individual contracts with HCFA
, rather than

combining multiple contracts held by one PRO for different PRO

areas. We chose Delaware as a PRO site to visit, although the

West Virginia PRO actually holds the contract for Delaware.
Hence , we refer to the West Virginia PRO in listing the case 
study PROs but note parenthetically that discussions focused on

the Delaware contract. On the other hand, our site visit to the

Rhode Island PRO focused on its operation in that state
, although

the Rhode Island PRO also holds the PRO contract for Maine. 
Discussion Guides

We designed seven separate but interrelated discussion guides to

capture the perspectives of PRO executive directors
, national

external entities, other PRO board and staff, local external

staff. 
enti ties and HCFA central and regional office  The 
discussion guide questions were structured so that we could later

compare responses within and across groups. All discussion

categories: 
guides grouped questions under three or four 
PRO
assessment 
, quality review and sanctions , HCFA oversight, and in 
, descriptive material. 
some cases  The discussion guides included
about equal numbers of closed and open-ended questions , but most

of the closed questions had an open-ended probe following them. 
Interview Approach

We conducted approximately half the 211 interviews by phone and

person. 
the other half in  For methodological consistency, we 
chose to interview all 44 PRO chief executive officers 
(CEOs) by

telephone and held subsequent additional on-site interviews with
 sites.
those CEOs associated with the 12 case study PRO 
addi tion to the 12 case study PRO sites, we conducted on-site 
rather than telephone interviews with most of the national and

staff. 
local external entities and with HCFA central office 
The primary PRO inspection team consisted of four individuals

from Region I who conducted 95 percent of the telephone inter

ones. 
views and 80 percent of the on-site  Four additional field
team members (two from Region I and two from OAI' s central 
interviews. At least 
office) conducted the other  two team 
mveimsbeirtss .p articipated in each of the 12 PRO case study site

Inspection interviews ranged up to five hours with an 
average length of two hours. We informed all participants

interviewed for this study that the confidentiality of their

specific responses to questions would be maintained, unless

otherwise cleared by them. 
As part of our quality control plan, the project leader assigned

one person to be the proj  s administrative coordinator. 
ect '  That 
individual developed and maintained a tracking system for all

discussion guides , correspondence, supplementary materials , and a

master schedule of team interviews. 
Codinq and Anal vsis 
We designed three primary and six relational data files
, using

dBASE III PLUS, to store and tabulate interview responses. We

developed codes for all questions and one team member usually

coded all questions in a given file to maximize coding

consistency. In 
addition, a different team member checked at 
least a 20 percent random sample of the files to ensure accuracy.
As part of the PRO inspection team' s qual i ty control plan, the 
project leader assigned one team member to be the project' s data

coordinator. In 
addition to having primary responsibility for 
designing the PRO data base, that individual was also responsible

for developing and enforcing data-related quality control

procedures. 
We used dBASE III PLUS to tabulate all interview data by respon

dent group (i. e., 
PRO CEOs, other PRO staff and board, HCFA, 
national external entities, and local external entities).

Other PRO-Related Data

In addition to interview data, we collected and analyzed other

PRO-related data including HCFA' s monthly and quarterly data
summary reports for all PROs and HCsFiAt'e ss P.R OMPTS and SuperPRO

reports for the 12 PRO case study  Because complete and
accurate sanction data was unavailable from HCFA, the team used

information provided by the OIG' s Office of Investigations as a

basis for its sanction data analysis. The team classified rural

and urban sanctions according to whether the particular

physician' 
s address fell within a Standard Metropolitan  , "

statistical Area (SMSA) as defined by the U. S. Bureau of Census.

Information on the numbers of rural and urban physicians for 1985

was provided by the Public Health Service, Bureau of Health

Professions. 
Information on the number of physicians eligible 
1, 
for Medicare reimbursement as of January  1987' was obtained 
from HCFA, Bureau of Program Operations. 
We also collected and reviewed a wide array of other materials

concerning PROs, including newspaper and journal articles

congressional hearings, and GAO, Library of Congress, OIG, and

audits. 
other studies and 
Methodoloqical Considerations in Intercretinq PRO Interview Data

The reader should keep three caveats in mind when reviewing this

report. 
First, because we wanted to give as comprehensive a view 
of ?ROs as possible, we have integrated the case study data with

the universal data. The case study data is generally used to

amplify broader-based findings 
, and such data is always clearly
labeled. 
Although we used a judgmental rather than randQm 
sampling methodology for choosing the case study sites
, it is

worth noting that those sites are broadly representative of PROs

by size, geographic location, and sanction activity level.

A second caveat to keep in mind is that although we interviewed a

total of 211 individuals, a given question may have been directed

universe. 
to only a subset of that  Therefore, in this report, we 
have sought to clarify the number of people responding to a given

question by noting the universe of respondents (N=  ) in all

relevant summary tables and figures. 
The third but perhaps most important consideration to highlight

is that much of the information gathered in this study came fro

questions with both closed and open-ended parts (e. g.  Do you
have any recommendations to the Federal Government regarding

actions it might take that would help PROs be more effective in

addressing quality of care issues?" Explain.  Because we chose

not to distribute the discussion guides prior to the interviews

the open-ended questions required the respondent to spontaneously

answers. 
formulate his or her  Therefore, the percentages of
people noting any particular answer vary much more than if the

respondents had been presented with limited response options or

had reviewed the discussion guides prior to the interviews.  APPENDIX XII

COMMENTS ON THE DRAFT

REPORT AND OIG RESPONSE

TO THE COMMENTS

Within the Department of Health and Human Services we received

comments on the draft report from the Health Care Financing

Administration 
(HCFA) and the Assistant Secretary of Planning and
Evaluation (ASPE). In addition, 
we received comments from the
American Hospital Association (AH). 
In the sections that
follow 
, we offer these comments, in full , and our response to
them. 
AS PE COMMENTS 
Thank you for the opportunity to review the draft report entitled

"The utilization and Quality Control Peer Review organization

(PRO) Program: An Exploration of Program Effectiveness.

primary concern is that the document, a though well-written and

informative 
, needs to be expanded to incorporate more information 
on the third scope of work (1988-1990). Specifically: 
Page one of the report states that the study

focused on "the implications of the changes in the

PROs' scope of work from the first to second

contract period.  Since the report also discusses

changes from the second to third contract period

this should be referenced in the Introduction.

The last sentence on page 18 states that "HCFA is

planning to reduce the PROs' reporting require

ments in the third contract cycle.  Some detail

should be included on the nature of these changes.
How will the contract cycle compare to the 

reports required in the second cycle, 90 percent

of which are submitted either monthly or

quarterly. 
Appendices 
III, IV and V contain summaries and 
comparisons of the first and second scopes of

work. 
These appendices should be expanded to 
include information on the third scope of work as

well, particularly given that the body of the

report includes discussion of third round require

ments (e.g. 
, pp. 24-25). 
Finally, I have one minor suggestion with respect to the format

of the report. Pages iv through vii of the executive summary

OIG. 
list recommendations made by the  Pages 29 through 32 of the
report reiterate the same recommendations. Rather than repeat
the same recommendations verbatim, I recommend that they be put
 in the body of the report and then just summarized in the

executive summary.

OIG RESPONSE

The comments for further information on the third scope of work

and or HCFA changes concerning reporting requirements are well

taken. 
comments. 
In part, these are reflected in HCFA' s  More 
detailed explanation, we think, is best offered in other forums. 
This report essentially focused on the second scope of work. 
AHA COMMENTS

The following are the American Hospital Association (AH) 
comments on OIG draft report entitled "The utilization and

Quality Control Peer Review Organization (PRO) Program: 

Exploration of Program Effectiveness.

SUMY 
EXECUTIVE 
My only comment is with the statement on page iii pertaining to

PRO data reporting. The statement reads "However, HCFA has

recently taken steps to strengthen its data analysis capability

and is planning to reduce the PROs' reporting requirements in the

third contract cycle.  Based on my review of the PRO data

chapter and in conversations with PRO data staff, it would appear

(i. e. in
that HCFA is increasing the PROs data requirements 
clusion of provider and beneficiary identifiers, and information

pertaining to generic quality  atement describing
screens). 
HCFA' s expectations as a result of strengthening the PRO data 
capabilities would be helpful.

GENERAL COMMENTS

In general, the study was informative and easy to follow. The

findings and recommendations presented in the study were quite

accurate. The appendices provided useful background information

about the PRO program. It would be helpful to the reader if some

of the key requirements of the third scope of work could be

incl uded in the report. 
OIG RESPONSE

Our response to the ASPE comments also appl ies to the 

comments above. Addi tional , detailed explanations concerning

HCFA' s expectations and the third scope of work would provide 
helpful context to our report, but we feel are best provided by

HCFA in other forums.  HCFA COMMENTS

We have reviewed the draft report on the effectiveness of the PRO

program and we have two general comments: 
We are concerned that the findings are listed in

the report as having been identified by the OIG. 
In fact, they were identified by HCFA and the

recommendations 
, for the most part, were already
under consideration by HCFA and were discussed

with OIG staff during the inspection. We raised 
this concern in our comments on the  draft 
OIG' s 
report on Quality review Activities (the second in

this series of reviews).

Although the methodology for the review and its

conclusion appear valid, the findings are. based 
questions presented to PRO representatives and

other groups associated with the PROs. The 
suspect. 
validity and reliability are  Questions
used in the survey were ambiguous and in some

instances the analysis of the responses were not

defined. 
For example, what definition or standard
of " 
effectiveness" was used in presenting
questions or evaluating responses? 
Are effective 
review acti vi ties those that result in a high

number of denials , generate significant problem

identify substantial quality of care

savings 
problems or meet some other undelineated

cri terion? 
In addition, there is no examination 
of variables such as familiarity with the system,
philosophy of medical review, and association with

HCFA and the PROs for each category of respondent. 
Our response to the specific recommendations, as well as other

, are attached for your consideration. 
general comments 
We have 
addressed the recommendations in- the order in which they appear
on pages iv through vii of the report. 
OIG Recommendation

Improve coordination and integration of its evaluation tools

(PROMPTS-2, SuperPRO and PRO data reports) to ensure

comprehensive assessments of PRO performance. This might 
incl ude 

synchronizing the PROMPTS-2 review with the

due dates of SuperPRO reports; and
 incorporating SuperPRO results into the

PROMPTS-2 document along with data from PRO

reports to maximize the comprehensiveness of

the evaluation.

HCFA COMMENTS

During the second Scope of Work, the PROMPTS-2 was designed

primarily as a PRO monitoring and performance tracking system.
HCFA employs a broad based PRO evaluation process that reviews a

variety of indicators , including PROMPTS-2.

We do not feel that PROMPTS-2 should be evaluated separately from

the corrective action plan (CAP) process. The combination of 
PROMPTS-2 and the CAPs process is keyed to compliance with

contract requirements and the specifics of the PRO' s own contract

proposal. We are not surprised that the PRO' s interviewed would

characterize HCFA' s insistence on contract compliance as "process
oriented. " We would expect the OIG to rej ect such a 
characterization of the routine monitoring that is prudently

required with large 
, fixed-price federal contracts. 
Finally, the SuperPRO contract for the third Scope of work will

be recompeted and we are giving serious consideration to in

tegrating SuperPRO findings with the PROMPTS-2. 
OIG Recommendation

Encourage consistency and objectivity in PRO renewal decisions

by: 
establishing rating standards for PROMPTS-2 and

for acceptable ranges of variance for SuperPRO. 
formally informing the PROs, regional office staff

and review panel members of the weights to be

assigned SuperPRO, HMO/CMP, cost/savings ratio

factors; and

establishing a standing evaluation panel of HCFA

regional and central office staff to review

regional recommendations. Participation by PRO

staff before closed session action would permit

discussion of mutual questions and concerns.

HCFA COMMENTS

HCFA' s criteria for contract renewal have always been: 
strict contract compliance;

appropriate medical review determinations; and

a fair price.
 The final evaluation for contracts under the second Scope of .Work 
takes into consideration regional office review and recommenda

tions. 
The PROs and regional offices were formally notified as 
to what constitutes acceptable performance. We use evaluation 
panels to review regional office renewal recommendations to

assure that performance standards are applied consistently on a

national basis and to assure that all PROs are treated equally

and fairly. As mentioned previously, we are developing SuperPRO

standards and are currently planning the integration of SuperPRO

and PROMPTS-2 during the next contract cycle. 
As required by statute, HCFA will publish its evaluation criteria

for the next contract cycle and will consider all comments

received . 
OIG Recommendation

Re-examine the purpose for and validity of the SuperPRO review

process. 
This assessment should address such issues as: 
whether SuperPRO is intended to complement

it; 
PROMPTS-2 medical review or to duplicate 
whether SuperPRO is intended to complement

PROMPTS-2 review decisions. If so, whether

its sample size, criteria and record selec

tion process permit valid comparison; and

whether there are better ways to use SuperPRO

expertise, such as having SuperPRO perform

onsi te review and provide technical assis

tance to those PROs most consistently

differing from SuperPRO in their review

decisions or by having SuperPRO assess what

types of quality problems are being iden

tified- by PROs.

HCFA COMMENTS

As discussed with OIG staff during the inspection, HCFA has

already recognized the existence of significant variation in

SuperPRO disagreement rates. We responded by establishing a

system of rereviews of disagreements by regional and central

office medical staff and consultant physicians.

SuperPRO was originally intended as an educational tool for PROs

teox iidsetn.t ify and correct areas of performance where deficiencies

The regional offices use SuperPRO results to monitor PRO 
CAPs. 
performance and establish  SuperPRO does not use local 
physicians and would not necessarily apply local practice

patterns. If 
we have SuperPRO "make " the PROs conform to  national standards , the whole concept of local peer review would

be lost. 
As previously mentioned, we are reevaluating the SuperPRO process

and are planning the incorporation of SuperPRO findings into the

PROMPTS-2. While the report does not explain whether onsi 

review would occur at the PRO or at the health care facility, we

believe that having SuperPRO go onsite would decrease the

effectiveness and objectivity of SuperPRO review and would not be

cost-effective. 
OIG Recommendation 
Create forums for SuperPRO, HCFA and PRO discussions.

HCFA COMMENTS

HCFA already maintains active liaison with the American Medical

Peer Review Association (AMPRA) , the industry group that

represents the PRO contractors. HCFA and AMPRA representatives

meet on a regular basis to provide an effective exchange of

information. In addition, HCFA representatives attend various

regional and topical meetings sponsored by AMPRA.

We conduct annual meetings with PRO Executive Directors and

Medical Directors and routinely convene ad hoc industry task

force groups Mhen we need to implement new legislation/policy

(e. g., industry on HHA, SNF, hospital outpatient review, etc.

Finally, we are considering the development of a regular forum of

PRO , SuperPRO and regional office staff to consider SuperPRO
issues. 
OIG Recommendation 
Examine ways to streamline the SuperPRO process to minimize the

administrative burden on PROs and to maximize its utility to

HCFA , such as: 
reducing the sample size for small PROs; 
revising the schedule for reporting results

to reduce time lag between PROMPTS-2 and

SuperPRO reviews; and

facilitating closer interactions between PRO

and SuperPRO reviewers in resolving

PRO/SuperPRO differences.

HCFA COMMENTS

In preparation of the new Scope of Work for the next SuperPRO

contract, we are considering changes to the SuperPRO model.  ?" ;

include: 
Areas currently under consideration  refinement of 
sampling methodology; flexibility with respect to sample size,
especially for smaller PROs; continued analysis of SuperPRO data;
and effective ways to use SuperPRO findings
, including

integration with PROMPTS-2.

We are also considering having the regional offices act as the

final authority to resolve disagreements between PROs and

SuperPRO. 
OIG Recommendation 
Improve the PROMPTS-2 process by

developing standards for judging the manner of PRO

performance, i. e. 
, "Are PRO/provider relationships
effective. . . 
, "Is the PRO successfully tracking
problems. . . 
establishing standards for rating "yes/no"
acceptability of overall activity category based

on number of " 
yes/no" ratings of subcategories;
and 
developing more outcome measures of PRO

effectiveness through research and demonstrations.

HCFA COMMENTS

The evaluation of peer review, like peer review itself, can never

be totally objective. HCFA has responded to criticisms of the

first PROMPTS by making the PROMPTS-2 easier to complete,
including " 
yes/no" answers where appropriate, and by taking steps 
to develop computer assisted analyses of PROMPTS-2 results. 
Ultimately, we believe that the effectiveness or success of the

PRO, or the impact of subcategory ratings on category ratings 
, is

best determined by the professional judgment and expertise of

HCFA central and regional office staff. HCFA has emphasized

regional office training, regional and central office communica

tion, as well as the participation of senior HCFA management in

evaluation decisions to ensure national consistency. 
PROMPTS-2 
like the original PROMPTS protocol , is currently being reexamined

based on regional office experience. HCFA is always open to

suggestions that will simplify the completion of the PROMPTS 
, but

still allow HCFA to take advantage of the considered judgment of

staff. 
regional office 
As we discussed in detail under our response to the long-term

planning recommendation and have previously discussed with OIG

personnel, we have already initiated pilot studies and research

efforts to develop outcome measures of PRO effectiveness.
 OIG Recommendation

Develop and release comparative PRO performance data to the PROs

and other interested parties.

HCFA COMMENTS

We currently release PRO data reports on a regular basis and will

continue to do so. We are also considering the development and

release of other PRO performance data reports. 
OIG Recommendation 
requirements, instructions 
Develop and distribute  , and pol icies
in a more timely manner to allow sufficient lead time for PRO

implementation. 
HCFA COMMENTS

We recognize that there have been problems in this area in the

past. We have already implemented procedures whereby PROs are

given sufficient lead time (at least 30 days)  prior to implemen

tat ion of 
rev ised contract requirements. 
Federal
RIn aeddiqtioins, tOBeRA r87  requires that we publish in the 

any new policy or procedure that affects the performance 
of contract obligations at least 30 days before the policy or

effective. It 
procedure becomes  also requires that contract 
modifications requiring funding for these activities are

negotiated prior to implementation. 
OIG Recommendation

Improve the consistency in policy interpretation among PROs and

HCFA staff by updating the PRO Manual on an on-going basis and by

providing training and orientation sessions for both PROs and

regional staff as needed.

HCFA COMMENTS

We agree that revisions to the PRO manual must be issued on a

more timely basis. We hasvoe .i mproved our performance in this area

and will continue to do  We also note that the third Scope of 
Work is considerably more detailed and should greatly assist PROs

in the conduct of their contractual obligations. In addition, 
HCFA has emphasized PRO/regional office training to ensure

national consistency. We have made improvements by having at

least annual meetings to train regional office staff who in turn

train the PROs. We also participate at AMPRA conferences and

hold an annual conference for PRO executives.  OIG Recommendation

Review the current roles of carriers, fiscal intermediaries (FIs) 
and PROs to assess the appropriateness of the current

distribution of responsibilities and assess the coordination of

, such as the timel iness of FI adj ustments. 
these entities 
HCFA COMMENTS

interface. 
There have been a variety of problems with the FI/PRO 
In the current Scope of Work, the actual communications problems

which existed in the first Scope of Work were corrected. The 
problems which exist with the processing of adjustments have been

identified and analyzed.  effort, 
HCFA has established a joint 
with representatives from the regional offices, FIs and PROs,

areas. 
which will provide a plan for correcting problem 
OIG Recommendation

Strengthen long-range planning of future directions and

appropriate roles for the PRO program by: 
increasing the emphasis on research and demonstra

tions in such areas as review methodologies,

patient outcome and severity measures and data

system requirements; 
exploring mechanisms to encourage PRO

innovations and reduce the prescriptive

nature of the current review requirements;

exploring better long-term ways of

structuring PROs' activities so they are cost-

effective and complement the efforts of other

review entities such as state medical licensure

boards and hospital quality assurance committees.

For instance, HCFA might explore the possibility

of having SuperPRO responsiple for making DRG

determinations and have the PROs focus on quality

review; and

establishing an advisory group of PRO and other

relevant entity representatives to provide input

issues. 
to HCFA on long-range planning 
HCFA COMMENTS

We believe that long-range planning has been strengthened. As 

advised OIG staff, we believe that is important for the PROs to

efforts. 
become involved in long-range planning  In fact, several 
efforts already are underway. As part of their medical review

eight PROs are currently engaged in a special collection of
 clinical data which will be linked with Medicare data on out

comes. This 
data will be used to assess the impact of interven
tions in six areas--coronary revascularization, cholecystectomy,

prostatectomy, myocardial infarctions , heart failure and pul

monary disease--on mortal i ty, morbidity, disabil i ty and cost. 
HCFA also has developed and begun testing a method for collecting

data. 
clinical data from PROs which can be linked to claims 
The
clinical information will lead to the development of a Uniform

Clinical Data set which will assist in the collection of the key

data elements necessary for effectiveness research. This will 
result in HCFA having the capability to measure patient outcomes

and, as a result, better evaluate the impact of peer review

direct future review activities and guarantee more effective PRO

review. 
Also , as cited elsewhere in the OIG report, HCFA will be 
conducting a number of pilot studies at the PROs directed at

developing methodologies in a number of areas including the

review of both utilization and quality of physician services. 
Finally, we are giving serious consideration to establishing

appropriate technical advisory groups to complement our other

industry liaison efforts. 
It must be noted that much of PRO review is legislatively

mandated. Also 
, if we were to allow greater flexibility to the
PROs , we would have even more complaints about consistency State-
to-state in both the types of reviews and the overall level of

review , and the resulting data would not be comparable. We do 
not believe that having SuperPRO conduct DRG determinations would

be cost-effective and it could have negative program 
results. We

wOuld have to pay for duplicating records for PROs (where
applicable) as well as SuperPRO on a much larger basis and this

would negate all our efforts for increased onsi te review. Also

it is a much lower burden to have the PRO do the DRG validation

while it is already reviewing the medical record than to pay both

the PRO and SuperPRO to review the entire record. 
GENERAL COMMENTS ON THE REPORT

Page 9

The least "useful" reviews listed are probably dependent on how

the question was phrased and the respondent' s idea of what the

question meant. We have deleted percutaneous lithotripsy revie 
as we found it non-effective (i.  , generated few denials or

changes in DRGs). 
Conversely, the Medicare Code Editor review is 
most  effective review we have, with a large admission denial 
the 
rate and a large amount of coding changes. The number of cases

however , is very small , thus not leading to a large amount of 
savings in the aggregate.
 Page 10

The OIG indicates that outlier review and assistants at cataract

surgery review are not " effective. 
In addition, OIG states that 
several PROs complained about extending review of readmissions to

days. 
those occurring within 30  The OIG points out that HCFA 
should reassess the need for these reviews. The HCFA did not

administratively decide to have PROs perform these reviews; these

are legislatively mandated reviews. 
Page 12

Last paragraph, line 9 , we assume should be "physician
referrals,  not 
admission referrals. 
In th  first paragraph OIG lists five activities which are

considered least effective but does not cite whether these "least

effective" activities are, in fact, ineffective (based upon data)
or whether this is a conception based on misinformation on the

part of the respondents. 
Also, in the last sentence of this 
paragraph, the OIG states that in the third Scope of Work we

require PROs to choose from a list of 10 procedures determined by

HCFA. Instead , we require the PROs to choose 8 from a list of 11 
(2 additional ones are mandated for review). In addition, we do

allow the PRO to "go outside the list" if it furnishes data to

support the choices (s) . 
Page 25

The first full paragraph, last sentence, states that we are

mandating review of 9 more DRGs without establishing the value of

review. 
Seven of those codes are newborn codes and amount to 100 
cases nationally. In analyzing the data, these cases are not

truly " clean-up" the
newborn" cases, and HCFA sees the need to "
data 
(since Medicare does not pay for newborns) 
Page 26

In paragraph 3, the OIG states that virtually all division

director jobs changed in HSQB. Since the study was of the PRO 
program only, we assume the reference is to the two division

director jobs in the Office of Program Review. At the time of

the study, one Division Director retired and the other was

Director. 
promoted to Deputy Office 
Page 27

The first full paragraph discusses the fact that HCFA has not

implemented the 100 percent review and opinion portions of COBRA. 
The 100 percent review will be implemented in the third Scope of

Work as will the preadmission review of 10 procedures. We are  already in the process of issuing implementing regulations for

full 100 percent review and the second opinion requirements.

Page 39

Bullet 5 is not accurate. 
PROs report on short stays separately 
review. 
but do not do any extra  (We have made this comment
before. ) 
Page 52

In bullet 
4, the last sentence should be deleted. The MOUs were 
1987. 
implemented on October 1,  This is not one of the 
provisions to be implemented with the Scope of Work. 
OIG RESPONSE

We are concerned with the general thrust of HCFA' s response. 
First, we must disagree with HCFA' s assertion that the findings

in the report were identified by HCFA rather. than the OIG. The 
findings are the result of an intensive, broadly based inquiry

involving many sources outside of as well as within HCFA. If, 

a general level, the OIG' s findings tend to parallel observations

and conclusions already reached by HCFA, they still, quite

findings. 
properly, are presented as OIG  Moreover, as is 
apparent in the report and in HCFA' s response to it, the OIG and

HCFA have a number of differences with respect to many of the

specifics underlying the general findings and many of the

implications flowing from them. 
Second , we must emphasize that because a particular line of 
action is under consideration in HCFA does not necessarily mean

it will be enacted or that an OIG recommendation in concert with

it is inappropriate. We did have discussions with HCFA staff

about the possible implications of our findings. At the time,

those discussions seemed to reflect a good deal of agreement on

the kind of actions that ought to be taken. 
In those instances 
where HCFA had already initiated action at the time our report

was written, we strove to provide that information in the report

itself. 
Third 
, we must express concern about HCFA' s observation that the 
validity and reliability" of the findings "are suspect" and that

the questions used in the survey were "ambiguous.  We carefully 
explained our methodology in the report. The information in the

report is limited, but represents the perspective of a carefully

selected sample of respondents who are directly involved with the

PRO program at various levels. We believe that their

observations and opinions represent valuable feedback on the

performance of the PRO program. We feel it is important for the

Department to take into account the accumulated, operational

wisdom that underlies this feedback.  With respect to the recommendations, we are concerned that HCFA

did not convey a more posi ti ve 
response to our suggestions
concerning the refinement and integration of evaluation tools.
We feel that improvement in this area is vi 
tal to the continued

improved performance of the PRO program. 
We are also concerned

that HCFA did not respond to our call for strengthened long-
range
planning. Our frame of reference in making that recommendation

involved HCFA i tsel f , not the indi vidual PROs. HCFA' s response
focused more narrowly on the 
PROs.