Department of Health and Human Services

OFFICE OF 

INSPECTOR GENERAL 

 

Daniel R. Levinson 

Inspector General 

June 2008 

OEI-01-06-00510 

 

EXTERNAL QUALITY REVIEWS IN 

MEDICAID MANAGED CARE 
 

 

 

 

 

 

Office of Inspector General
 http://oig.hhs.gov 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 

amended, is to protect the integrity of the Department of Health and Human Services 

(HHS) programs, as well as the health and welfare of beneficiaries served by those 

programs. This statutory mission is carried out through a nationwide network of audits, 

investigations, and inspections conducted by the following operating components: 

Office of Audit Services 

The Office of Audit Services (OAS) provides auditing services for HHS, either by conducting 

audits with its own audit resources or by overseeing audit work done by others. Audits 

examine the performance of HHS programs and/or its grantees and contractors in carrying 

out their respective responsibilities and are intended to provide independent assessments of 

HHS programs and operations. These assessments help reduce waste, abuse, and 

mismanagement and promote economy and efficiency throughout HHS. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) conducts national evaluations to provide 

HHS, Congress, and the public with timely, useful, and reliable information on significant 

issues. These evaluations focus on preventing fraud, waste, or abuse and promoting 

economy, efficiency, and effectiveness of departmental programs. To promote impact, OEI 

reports also present practical recommendations for improving program operations. 

Office of Investigations 

The Office of Investigations (OI) conducts criminal, civil, and administrative investigations 

of fraud and misconduct related to HHS programs, operations, and beneficiaries. With 

investigators working in all 50 States and the District of Columbia, OI utilizes its resources 

by actively coordinating with the Department of Justice and other Federal, State, and local 

law enforcement authorities. The investigative efforts of OI often lead to criminal 

convictions, administrative sanctions, and/or civil monetary penalties. 

Office of Counsel to the Inspector General 

The Office of Counsel to the Inspector General (OCIG) provides general legal services to 

OIG, rendering advice and opinions on HHS programs and operations and providing all 

legal support for OIG’s internal operations. OCIG represents OIG in all civil and 

administrative fraud and abuse cases involving HHS programs, including False Claims Act, 

program exclusion, and civil monetary penalty cases. In connection with these cases, OCIG 

also negotiates and monitors corporate integrity agreements. OCIG renders advisory 

opinions, issues compliance program guidance, publishes fraud alerts, and provides other 

guidance to the health care industry concerning the anti-kickback statute and other OIG 

enforcement authorities. 
 

 

 

 

Δ EXECUTIVE SUMMARY 

OBJECTIVES 

To determine: 

1. the extent to and ways in which States utilize external quality 

reviews of Medicaid managed care, 

2. the extent to which external quality review organizations (EQRO) 

provided States with required information, and 

3. limitations that States have identified with the external quality 

review process. 

BACKGROUND 

Medicaid is a joint Federal and State program that provides medical 

assistance to low-income groups, such as children, senior citizens, and 

people with disabilities. Many States utilize managed care to respond 

to Medicaid expenditures that have more than doubled since 2000. As 

of 2006, 65 percent of the 45.6 million Medicaid beneficiaries were 

enrolled in managed care. 

The Social Security Act requires States that operate Medicaid managed 

care programs to provide for an external, independent review of their 

managed care organizations. States may contract with an independent 

entity called an EQRO to conduct the external quality review. The 

Centers for Medicare & Medicaid Services (CMS) provides States with 

matching Federal funds for review expenditures, including the 

production of results, depending on the qualifications of their EQRO 

contractors. Federal regulations require three mandatory activities as 

part of this review and offer States the choice of requiring up to five 

optional activities. Regulations also require the review to produce 

five specific deliverables, including a detailed technical report. 

We reviewed EQRO reports for the 37 States that arrange for external 

quality reviews. We also reviewed the contracts and requests for 

proposals between these States and their EQROs. We then surveyed 

these 37 States regarding their use of EQRO reports and their 

experience with the review process. We received a 100-percent response 

rate. We also conducted structured interviews with representatives 

from six States. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE i 
 

 

 

 

 

E X E CUTIVE S U MMAR Y 
 

FINDINGS 

Most States are using the results of EQRO reviews. For example, 

33 of the 37 States required their managed care plans to make changes 

based on EQRO reports, such as how plans document and conduct 

performance improvement projects and how plans meet State standards 

regarding members’ access to care. Twenty-two States used EQRO 

reports to share knowledge across plans, for example, by targeting 

technical assistance to the plans and sharing best practices across 

plans. Fourteen States reported amending their contracts with plans 

based on EQRO reports and 16 States took other actions, such as setting 

new performance standards. Overall, 21 States rated the EQRO reports 

to be very useful. Furthermore, 25 States required their EQROs to 

conduct optional review activities during their last annual review 

period. 

Some EQRO reports did not include all required information, despite 

the States’ oversight. EQRO reports for 15 States were missing at least 

one of the five required deliverables. The two most common missing 

deliverables were an assessment of managed care plans’ responses to 

previous EQRO recommendations and comparative information about all 

of the State’s plans. EQRO reports for eight States were missing 

information on at least one of the three mandatory activities. All States 

reported regularly monitoring their EQROs through ongoing 

communication, written status reports, and contract provisions. Of the 

37 contracts and requests for proposals that we reviewed, 34 included 

provisions to ensure EQRO compliance with Federal regulations. 

More than half of the States cited concerns with the external quality 

review process. Sixteen States identified staffing concerns with the 

external quality review process, particularly turnover and training 

issues. Thirteen States raised concerns with the quality of EQRO 

reports, especially their timeliness and the degree to which their 

findings and recommendations could be implemented. Five States 

reported finding the mandatory activities redundant to other 

monitoring activities, such as those conducted by private accreditors. 

Many States requested technical support from CMS to help address 

these issues. 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE ii 
 

 

EXECUTIVE SUMMARY 
 

RECOMMENDATIONS 

CMS should work with States to ensure that EQROs are providing 

complete information. CMS could, as part of its review of the contents 

of the EQRO reports, inform States if it finds that any of the required 

deliverables are missing. Further, CMS could amend its regulations to 

describe, as a condition for receiving enhanced reimbursement, the steps 

that States must take to ensure that all the required deliverables are 

included in the quality review results. 

CMS should provide additional technical assistance and written 

guidance to States. CMS could do this by organizing teleconferences, 

offering written guidance, and sharing best practices. It should schedule 

these teleconferences regularly and make any written guidance available 

online. CMS could use a format similar to a teleconference it held with 

States in 2006. CMS should consider two priorities for its technical 

assistance and written guidance: minimum standards for EQRO report 

organization and content and use of accreditation data in external quality 

reviews. 

AGENCY COMMENTS 

CMS agreed with both of our recommendations and cited actions that it 

has taken in response to each. It provided feedback to Medicaid 

Directors in 15 States regarding deficiencies in EQRO reports. CMS 

also focused one of its triannual audio conferences, available to States 

and EQROs, on external quality reviews. Further, CMS assisted a 

managed care accrediting organization in creating a document for 

States on using accreditation data in their reviews. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE iii 
 

 

 

 

Δ TABLE OF CONTENTS 

EXECUTIVE SUMMARY ..................................... i 
 

INTRODUCTION ........................................... 1 
 

FINDINGS ................................................. 7 
 

Most States are using results of EQRO reviews ............... 7

 EQRO reports for some States were missing required elements. . . 9 

More than half of States had concerns with the review process . . 11 

RECOMMENDATIONS ..................................... 14 
 

APPENDIXES ............................................. 16 
 

A: Status of States’ Arrangements for Reviews .............. 16

 B: Time Period of EQRO Reports Reviewed ................. 17

 C: Agency Comments ................................... 18 
 

ACKNOWLEDGMENTS .................................... 21 
 
 

INTRODUCTION 

 

 

 

Δ INTRODUCTION 

OBJECTIVES 

To determine: 

1. the extent to and ways in which States utilize external quality 

reviews of Medicaid managed care, 

2. the extent to which external quality review organizations (EQRO) 

provided States with required information, and 

3. limitations that States have identified with the external quality 

review process. 

BACKGROUND 

Medicaid and the Growth of Managed Care 

Medicaid is a joint Federal and State program that provides medical 

assistance to low-income groups, such as children, senior citizens, and 

people with disabilities. Individual States establish eligibility 

requirements, benefits packages, and Medicaid payment rates under 

standards set by the Centers for Medicare & Medicaid Services (CMS). 

Medicaid spending has increased at both the Federal and State levels. 

Federal expenditures increased from $118 billion in 2000 to $180 billion 

in 2006.1 State expenditures rose by almost 60 percent, from $89 billion 

in 2000 to $142 billion in 2006. These expenditures account for almost 

one-quarter of all States’ expenditures.2 

In response to this growth, States have increasingly utilized managed 

care to deliver Medicaid services to beneficiaries. Managed care is a 

health care delivery system that aims to maximize efficiency by 

negotiating rates, coordinating care, and managing use of services. 

From 2000 to 2006, enrollment in Medicaid managed care grew 

59 percent, from 18.8 million to 29.8 million beneficiaries. As of 2006, 

65 percent of the 45.6 million Medicaid beneficiaries were enrolled in 

managed care.3 

1 Department of Health and Human Services, “Budget in Brief,” 2002 and 2008. 

Available online at http://www.hhs.gov/budget/docbudget.htm. Accessed on 

May 20, 2008. 

2 National Association of State Budget Officers, “2006 State Expenditure Report,” 

2007. 

3 CMS, “Medicaid Managed Care Enrollment Report as of June 30, 2006.” 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 1 
 

 

 

 

 

INTRODUCTION 

Concerns About Medicaid Managed Care Quality of Care 

The Government Accountability Office found that the capitated (i.e., a 

predetermined amount per beneficiary) nature of managed care could 

create fiscal incentives to underserve beneficiaries. It recommended 

that States carefully monitor access to and quality of care delivered to 

Medicaid managed care populations.4 Other published studies have 

suggested that managed care reduces the quality and accessibility of 

health care, particularly in specific vulnerable subpopulations.5 

State Oversight of Medicaid Managed Care 

The Social Security Act requires contracts with Medicaid managed care 

organizations (MCO) to provide for an external independent review. 

Specifically, it mandates that each MCO contracted with the State must 

be reviewed annually by a “qualified independent entity.” This review 

must evaluate the MCO’s “quality outcomes and timeliness of, and 

access to, the items and services for which the organization is 

responsible.”6 

Requirements for External Quality Review Organizations 

In January 2003, CMS issued a final rule to further specify 

requirements for external quality reviews and EQROs.7 To qualify as 

an EQRO, an organization must have knowledgeable staff and sufficient 

physical, financial, and technological resources. An EQRO and any of 

its subcontractors must also be independent from the State Medicaid 

agency and the MCOs under review. States are responsible for ensuring 

that EQROs meet these requirements.8 Some EQROs hold external 

quality review contracts with several States. For example, the Health 

Services Advisory Group holds EQRO contracts in 11 States.9 

4 Government Accountability Office, GAO/T-HRD-93-10, “Medicaid: States Turn to
 

Managed Care to Improve Access and Control Costs,” 1993. Available online at 

http://archive.gao.gov/d42t14/148756.pdf. Accessed on May 20, 2008. 

5 E.C. Schneider, et al., “Racial Disparities in the Quality of Care for Enrollees in 

Medicare Managed Care,” Journal of the American Medical Association, 2002, 

Volume 287, pp. 1288–1294. D. G. Clement, et al., “Access and Outcomes of Elderly 

Patients Enrolled in Managed Care,” Journal of the American Medical Association,
 

1994, Volume 271, pp. 1487–1492. 

6 Social Security Act § 1932(c)(2)(A)(i).
 

7 42 CFR pt. 438, subpart E.
 

8 42 CFR § 438.354. 

9 “Florida Medicaid Selects HSAG for External Quality Review Services,” 

May 25, 2006. Available online at http://www.hsag.com/news/2006_0525.asp. 

Accessed on May 20, 2008.
 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 2 
 

 

INTROD UCT ION 

Regulations also permit the State to conduct the review.10 Most States 

choose to contract with independent EQROs to conduct the required 

activities; however, some States perform one or more of the activities 

themselves. Even States conducting their own reviews usually contract 

with EQROs to write the final technical reports. 

States receive an enhanced Federal match of 75 percent for their review 

expenditures, including the production of results, if they contract with 

qualified EQROs. They receive a Federal match of 50 percent for any 

activities conducted by nonqualified entities.11 States are responsible 

for contracting with EQROs and ensuring that EQROs are producing 

the required deliverables.12 

Mandatory and Optional Review Activities 

Federal regulations require that external quality reviews include three 

mandatory activities. For each MCO or prepaid inpatient health plan13 

(hereafter referred to as plans), the external quality review must 

include the following activities: 

1. validation of performance improvement projects (PIP) required by 

the State and underway in the preceding 12 months; 

2. validation of plan performance measures required by the State and 

reported by the MCO in the preceding 12 months; and 

3. a review, conducted in the previous 3-year period, to determine the 

plan’s compliance with State standards for access to care, structure 

and operations, and quality measurement and improvement.14 

In addition, Federal regulations allow States to require up to five 

optional activities: 

1. validation of encounter data reported by a plan, 

2. administration and validation of consumer and provider surveys on 

quality of care, 

3. calculation of additional performance measures, 

10 42 CFR § 438.358(a). 

11 42 CFR § 438.370.
 

12 42 CFR §§ 438.356 and 438.364.
 

13 These are health plans that provide inpatient hospital or institutional services on
 

an at-risk or other than State plan reimbursement basis. 

14 42 CFR § 438.358(b). 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 3 
 

INTRODUCTION 

4. conduct of additional PIPs, and 

5. conduct of studies on quality focused on a particular aspect of clinical 

or nonclinical services at a point in time.15 

In May 2002, CMS issued a series of protocols for use in conducting 

Medicaid external quality review activities. These serve as guidelines 

for EQROs in conducting mandatory and optional external quality 

review activities. 

External Quality Review Deliverables 

After completing the review activities, an EQRO must produce the 

following information (hereinafter referred to as deliverables): 

1. a detailed technical report describing the data aggregation and 

analysis and the way in which conclusions were drawn as to the 

quality, timeliness, and access to the care furnished by the plans; 

2. an assessment of each plan’s strengths and weaknesses with respect 

to quality, timeliness, and access to care; 

3. as the State determines, methodologically appropriate, comparative 

information about all plans; 

4. recommendations for improving the quality of health care services 

furnished by the plans; and 

5. an assessment of the degree to which each plan has addressed 

effectively the quality improvement recommendations made by an 

EQRO during the prior year’s review.16 

Some EQROs issue one technical report encompassing all of the plans 

in the State; others issue multiple reports, each specific to one plan or 

activity. Generally, these reports are organized around the 

mandatory activities. However, even if a report is missing 

information on one of the mandatory activities, it may still contain all 

of the required deliverables. Regulations require States to make 

these reports available to interested parties upon request.17 Usually 

the States also provide CMS with copies of the deliverables. 

15 42 CFR § 438.358(c). 

16 42 CFR § 438.364(a). 

17 42 CFR §§ 438.364(b) and 438.350(f). 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 4 
 

 

 

 

 

 

INTROD U C T I O N 

CMS Efforts To Improve the External Quality Review Process 

In October 2006, CMS held an audio conference with State Medicaid 

agencies addressing the assessment, utility, and improvement of EQRO 

reports. During this conference, CMS presented its review and 

evaluation of reports from 22 States. According to the review, reports 

vary substantially in organization, length, detail, topics addressed, and 

presence and extent of recommendations. CMS found that many 

reports did not comply with at least one external quality review 

regulatory requirement.18 

CMS prepared a “tool kit” based on that review.19 This tool kit offered a 

brief assessment of 2005 reports, identified key opportunities for 

improved reporting, recommended specific sections to be included in 

reports, and suggested a standardized format for submitting reports.20 

CMS provided the tool kit to States, who could then provide it to their 

EQROs. 

METHODOLOGY 

Scope 

This evaluation is national in scope. Forty States, including the District 

of Columbia (hereafter referred to as States), contract with plans and 

thus must arrange for external quality reviews. Three of these States 

(Georgia, Maine, and New York) are currently working with CMS to 

implement the review process, so we excluded them from our analysis.21 

We included the 37 States that have implemented the review process. 

States that do not contract with managed care organizations are not 

required to conduct reviews. See Appendix A for a list of all States and 

whether they contract with plans. 

Data Sources 

We collected and analyzed data from the following sources: 

EQRO reports. We obtained from CMS and the States copies of all 

reports that EQROs produced pursuant to regulations. Because review 

periods differed across States, we could not review reports for all States 

18 CMS, “State External Quality Review Tool Kit for State Medicaid Agencies,” 

October 2006. 

19 Ibid.
 

20 Ibid. 

21 Since the completion of our review, Maine has discontinued its Medicaid managed
 

care program and will not be implementing an external quality review process. 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 5 
 

 

 

 

 

 

INTROD U C T I O N 

for the same time period. See the table in Appendix B for the time 

periods covered by the reports we reviewed. 

Contracts. We obtained from the States copies of their EQRO contracts 

or requests for proposals covering 2005. 

Surveys. We sent surveys by mail to the Medicaid Directors of the 

37 States and received a 100-percent response rate. The surveys asked 

States how they conduct oversight of their EQROs and utilize the 

results of previous EQRO reports. The surveys also asked about States’ 

general experiences in implementing the external quality review 

process. 

Interviews. We conducted structured telephone interviews (and one 

in-person interview) with representatives from Medicaid offices in six 

States. We purposively selected these States based on their survey 

responses. 

Analysis 

We reviewed each EQRO report to determine whether it included all 

elements required by Federal regulations. We counted elements as 

missing if they were not present in the report and the report or survey 

data did not explain why. For example, we did not count the 

assessment of prior recommendations as missing if the plan’s report 

noted that the plan had not contracted with the State the previous year 

or if the State’s survey reported that the State was currently only in its 

second review period. We also did not count any section as missing if 

the State was still in the process of implementing that aspect of the 

review, for example, if the State had not initiated any PIPs in the 

previous 12-month period. 

Limitations 

Data from the surveys and follow-up interviews were self reported by 

States. We did not independently verify these data. 

We did not determine whether assessments of plan strengths and 

weaknesses were present in EQRO reports. Because the general scope 

of an external quality review is to examine plan strengths and 

weaknesses with respect to quality, timeliness, and access to care, all 

data in the report relate to these topics. 

Standards 

We conducted this study in accordance with the “Quality Standards for 

Inspections” issued by the President’s Council on Integrity and 

Efficiency and the Executive Council on Integrity and Efficiency. 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 6 
 

 

 

 

 

Δ FINDINGS 

Most States are using the results of Of the 37 States, 29 rated the 

EQRO reviews external quality review process as 

very important in overseeing their 

Medicaid managed care programs. States cited several benefits of the 

review process, including identifying and documenting concerns with 

plans, providing a comprehensive review of the State Medicaid 

programs and their plans, and providing additional monitoring that the 

States are unable to do on their own. Similarly, 21 States rated the 

EQRO reports as very useful, citing the data in the reports and the 

independence of the reviews. 

Thirty-three of thirty-seven States required their managed care plans to 

make changes based on EQRO reports 

The plans developed and implemented these State-required changes. 

The most common changes, required by 28 States, called for improving 

the documentation and conduct of PIPs. For example, plans had to 

document that they tested the statistical significance of reported 

improvements. They also had to create templates for developing and 

submitting PIPs. Twenty States reported requiring changes in how 

plans complied with State and Federal standards. These changes 

typically addressed standards regarding member grievances and 

appeals or access to care. See Table 1 for a list of changes States 

required plans to make based on EQRO reports. 

Table 1: Managed Care Plan Changes Required by State 

Area in Which State Required Managed Care Plan Change 

Number of States 

That Required 

Change* 

Documentation and Conduct of Performance Improvement Projects 

Compliance With State and Federal Standards 

Calculation and Reporting of Performance Measures 

Improvement in Information Systems 

Other Change 

28 

20 

16 

10 

4 

Source: Office of Inspector General analysis of State Medicaid Agency survey data, 2007. 

* The numbers do not total 33 because some States required more than one change. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 7 
 

 

F I NDING 
 S 

States typically required the managed care plans to develop and 

complete corrective action plans (CAP) to address these changes. States 

also reported providing technical assistance to the plans to help them 

develop their CAPs. States used several methods to ensure that the 

plans made these changes, such as including contractual provisions, 

monitoring CAPs, and delegating oversight to EQROs. 

According to eight States, their managed care plans initiated changes 

based on EQRO report data without the States requiring them to do so. 

For example, one plan made changes to its internal information systems 

after the EQRO pointed out shortcomings. 

Twenty-two States used EQRO reports to share knowledge across managed 

care plans 

Knowledge sharing included States targeting technical assistance to the 

plans (reported by 21 States). Eight of these States focused their 

assistance on improving PIPs. Often, EQROs provided assistance to 

plans (reported by 15 States). States also reported sharing best 

practices across their plans. Typically, these States convened periodic 

meetings with representatives from the States, EQROs, and all plans. 

Some States cited the potential value of identifying and sharing best 

practices at the national level. 

Fourteen States reported amending their contracts with managed care plans 

based on EQRO findings 

Of these States, 12 inserted new provisions into their contracts. The 

changes included strengthening and clarifying contract requirements, 

using external quality review results to justify contract extensions and 

renewals, and adjusting PIPs. 

Sixteen States also reported taking other actions based on EQRO reports 

Other commonly reported State changes included modifying the 

performance measures. Eleven States reported changes such as adding 

new measures and establishing new benchmarks for plans to meet. 

Additionally, nine States reported making changes such as 

standardizing quality assurance processes and implementing a tracking 

system for identifying and responding to adverse events. 

Furthermore, 25 States required their EQROs to conduct optional review 

activities during their last annual review period 

States reported requiring their EQROs to conduct at least one of the five 

optional review activities to provide additional information for States’ 

quality improvement efforts. One of the two most common optional 

activities States reported was administering or validating surveys of 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 8 
 

 

 

F I NDING S 

quality of care either to consumers or to providers. In the reports we 

reviewed, many EQROs used the Consumer Assessment of Healthcare 

Providers and Systems survey instrument. This survey stems from a 

public-private initiative that developed standardized surveys of 

patients’ experiences. The second of the two most common optional 

activities was conducting focused quality studies on particular aspects 

of care. Among the focused studies we saw, common topics included 

determining childhood immunization rates at a specific age and 

assessing trends in adolescent well care. See Table 2 for the number of 

optional activities States required their EQROs to conduct. 

Table 2: Optional Review Activities Required by States 

Optional Activity 

Number of States 

That Required 

EQROs To Conduct 

Activity 

Administering or Validating Surveys on Quality of Care 

Conducting Focused Quality Studies 

Validating Encounter Data 

Calculating Additional Performance Measures 

Conducting Additional Performance Improvement Projects 

14 

14 

12 

7 

6 

Source: Office of Inspector General analysis of State Medicaid Agency survey data, 2007. 

Federal regulations require EQROs Some EQRO reports did not include all required to produce five deliverables based information, despite the States’ oversight on their review activities and 

provide them to the States. 

EQRO reports for 15 States were missing at least one of the five required 

deliverables 

Reports for 12 States were missing the assessment of managed care 

plans’ responses to previous EQRO recommendations, and reports for 

7 States were missing comparative information about all of the State’s 

plans. 

These deliverables can provide useful information. For example, some 

reports displayed the previous EQRO recommendations, plan responses 

and actions, and the new recommendations in one chart, thereby 

enabling the States to see a comprehensive history at a glance. The 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 9 
 

 

 

 

F I NDING 
 S 

comparative information often included tables presenting, for all plans, 

performance measure scores, PIP ratings and scores, as well as charts 

showing those plans not in compliance with each of the reviewed State 

and Federal standards. In addition, some reports included a 

comparison of the States’ plans with national averages. 

Reports for 22 States included all of the required deliverables. Seven of 

these States contracted with the same EQRO to produce their reports. 

This EQRO structures its reports similarly, organized around each 

required deliverable. 

EQRO reports for eight States were missing information on at least one of 

the three mandatory activities 

External quality reviews must include three mandatory activities: 

validation of PIPs, validation of performance measures, and reviews of 

compliance with State standards. EQRO reports from five States were 

missing information on one mandatory activity, reports from two States 

were missing information on two activities, and a report from one State 

was missing information on all three activities. Because PIPs are 

typically multiyear endeavors, many of the reports that did include data 

on them could not draw full conclusions about the PIPs’ efficacy. In 

reports we reviewed, plans completed only about 20 percent of the PIPs. 

In these cases, the reports included a summary of the PIPs’ progress at 

that point and a review of the managed care plans’ methodology. 

All States reported regularly monitoring their EQROs 

States rely on ongoing communication, written status reports, and 

contract provisions to oversee their EQROs. 

All but 1 of the 37 States reported regularly communicating with their 

EQROs through telephone calls, e-mail, and in-person meetings.22 

Typically, contact occurred weekly and involved the EQRO updating the 

State on its progress in completing particular review activities and how 

the EQRO was performing toward meeting its timeline. This gave both 

parties an opportunity to discuss any problems that may have arisen. 

Staff from one State told us that they had increased contact with their 

EQRO to weekly calls because of concerns over its performance. 

22 The State that reported no ongoing communication with its EQRO performs the 

external quality reviews itself. This State contracts with EQROs only to produce the 

annual report. 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 10 
 

 

F I NDING S 

Twenty-four States also require their EQROs to submit ongoing status 

reports during the review period. These reports typically include 

information on the EQROs’ progress toward completing deliverables. 

Contracts for 22 States require EQROs to submit written status reports 

or draft reports. 

Of the 37 EQRO contracts and requests for proposals we reviewed, 

34 included provisions to ensure EQRO compliance with Federal 

regulations. Contracts in 24 States require EQROs to show that they 

(1) have sufficient and qualified staff, (2) are entities independent of the 

State Medicaid agencies, and (3) possess sufficient resources to conduct 

the external quality reviews. One of these States required its EQRO to 

pass a readiness review prior to beginning review activities. That State 

designed its review to ensure that the EQRO had in place the 

infrastructure to successfully conduct external quality reviews. 

Even though States had contractual remedies available to them, EQRO 

reports for 19 States were still missing either required deliverables or 

information on mandatory activities. Among the remedies were those 

allowing States to withhold payment from EQROs or requiring them to 

purchase a performance bond to be forfeited to a State if an EQRO 

failed to fulfill its contract. When asked whether they would change the 

review process based on their experiences to date, only one State 

specifically addressed the missing elements. That State reported that it 

was developing stricter requirements for the delivery of external quality 

review products. 

More than half of the States cited concerns with Twenty-four of the thirty-seven 

States reported three main 

concerns: staffing, EQRO report 

quality, and redundancy with other monitoring efforts. 

the external quality review process 

Sixteen States identified staffing concerns with the external quality review 

process 

Among these States, 15 reported that staff turnover at the managed 

care plans, State Medicaid agencies, or EQROs prevented them from 

implementing EQRO report recommendations because of the time 

needed to train new staff. Seven States specifically identified training 

EQRO staff as a barrier. These States reported that to get acceptable 

reports, they had to provide EQROs with technical assistance, thereby 

using States’ time and resources that otherwise would have been spent 

on their own oversight. 

O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 11 
 

FINDING 
 S 

Thirteen States raised concerns with the quality of EQRO reports 

States identified concerns ranging from the timeliness of the reports to 

the degree to which the findings and recommendations could be 

implemented. In our review of the reports, the substance of 

recommendations varied widely. Some were extremely broad and 

difficult to interpret. For example, one report recommended: 

Overall, and as seen from the data, tables, and graphs 

presented herein, the plan is responsible for addressing 

opportunities for quality improvement through the 

corrective action plan process established by the State. 

Although the plan is empowered to design and implement 

a corrective action plan that most suitably addresses 

substandard performance . . . . It will be imperative that 

the plan follow completely through with its corrective 

action plan(s) already approved by the State, which will 

also be monitored by the State. 

Similarly, another EQRO recommended that a plan “monitor 

performance of health care utilization and health indicators, identify 

areas for improvement, implement intervention strategies, and monitor 

the effectiveness of the interventions” without offering any more specific 

information. Furthermore, two other States reported that, to get an 

acceptable report, they had to use their own resources to provide 

technical assistance to their EQROs. Finally, one State found that the 

EQRO wrote the report with CMS as its primary audience, thus 

diminishing the report’s value to the State. 

Eleven States requested additional technical guidance and feedback 

from CMS, particularly regarding EQRO reports. Staff we spoke with 

from one State wanted more guidance from CMS at the national level 

rather than the regional, particularly regarding minimum standards for 

report organization. 

Five States reported finding EQRO mandatory activities redundant to other 

monitoring activities 

Many States require plans to use specific performance measures, known 

as the Healthcare Effectiveness Data and Information Set, which 

EQROs then validate. The National Council on Quality Assurance, 

which accredits almost 30 percent of the plans for which we reviewed 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 12 
 

 

FINDINGS 

reports, validates those same measures.23 In other cases, the States 

conduct their own reviews as part of their contract with the plans. 

These duplicate activities can place a burden both on plans, which must 

provide similar data to EQROs, States, and/or accreditors, and on 

States, which must use their resources to ensure that plans submit data 

to EQROs, thus reducing time spent on other oversight issues. 

Federal regulations permit States to use data from a Medicare or 

accreditation review in place of the EQRO review of plan compliance 

with State standards.24 However, States reported receiving conflicting 

guidance from CMS, particularly from the regional offices, on this 

provision. A staff member from one State told us that she relied on 

their EQRO to determine what data it used in its review. Because of 

the States’ uncertainty, six States requested increased guidance or 

flexibility from CMS regarding use of accreditation data in external 

quality reviews. 

23 National Council on Quality Assurance, “MCO Accreditation.” Available online at 

http://web.ncqa.org/tabid/67/Default.aspx. Accessed on May 20, 2008. 

24 42 CFR § 438.360. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 13 
 

 

Δ RECOMMENDATIONS 

We found that although most States are using the results of EQRO 

reviews and find value in the review process, more than half of the 

States raised concerns regarding the external quality review process, 

including concerns about staffing, the quality of reports, and redundant 

data reporting. States oversee the external quality review process 

through a variety of means, yet EQRO reports for some States are 

missing required information. To improve the quality and usefulness of 

the external quality review process, States requested additional 

guidance and feedback from CMS on these and other issues. 

To help States ensure that all required elements are present and 

maximize the usefulness of external quality reviews, CMS should: 

Work With States To Ensure That EQROs Are Providing Complete 

Information 

CMS could, as part of its review of the contents of the EQRO reports, 

inform States if it finds that any of the required deliverables are 

missing. Further, CMS could amend its regulations to describe, as a 

condition for receiving enhanced reimbursement, the steps that States 

must take to ensure that all required deliverables are included in the 

quality review results. 

Provide Additional Technical Assistance and Written Guidance to States 

CMS could do this by organizing teleconferences, offering written 

guidance, and sharing best practices. CMS should continue to use a 

format similar to its 2006 teleconference. It should schedule these 

conferences regularly, perhaps annually or semiannually. 

CMS should provide further written guidance to the States. This 

guidance could build on the tool kit that CMS has already developed. 

CMS should post its written guidance online to make it easily accessible 

to any new State Medicaid agency staff. 

Because CMS usually receives and reviews the EQRO reports, it should 

consider culling best practices to share across the States. It could share 

these during teleconferences. 

Finally, CMS should consider two priorities for its technical assistance 

and written guidance: 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 14 
 

 

RECOMMENDATIONS 

Provide States with minimum standards for report organization and content. 

Further guidance could reduce the likelihood that any of the required 

deliverables would be missing from EQRO reports. Clarifying 

expectations for the content of reports could improve their level of detail 

and usefulness, particularly regarding the assessment of plan strengths 

and weaknesses. CMS could utilize best practices from reports to serve 

as a model for other States. 

Clarify the degree to which Federal regulations allow the data from 

accreditation reviews to be used. Currently, States have received 

differing interpretations from CMS central and regional offices 

regarding whether Federal regulations allow them to use data from 

accreditation reviews in place of external quality reviews. Providing 

consistent and specific guidance could improve efficiency for States, 

plans, and EQROs. 

AGENCY COMMENTS 

CMS agreed with both of our recommendations and cited actions that it 

has taken in response to each. It provided feedback to Medicaid 

Directors in 15 States regarding deficiencies in EQRO reports. CMS 

also focused one of its triannual audio conferences, available to States 

and EQROs, on external quality reviews. Further, CMS assisted the 

National Committee on Quality Assurance in creating a document for 

States on using accreditation data in their reviews. CMS has begun 

working with a second managed care accreditation agency, the 

Utilization Review and Accreditation Committee, on a similar 

document. The complete text of CMS’s comments can be found in 

Appendix C. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 15 
 

 

Δ APPENDIX ~ A 

Status of States’ Arrangements for External Quality Reviews at the Time of Our Review 

States that are required to arrange for external quality reviews (because they have Medicaid 

managed care contracts): 

Alabama Kentucky Oregon 

Arizona Maine Pennsylvania 

California Maryland Rhode Island 

Colorado Massachusetts South Carolina 

Connecticut Michigan Tennessee 

District of Columbia Minnesota Texas 

Delaware Missouri Utah 

Florida Nebraska Vermont 

Georgia Nevada Virginia 

Hawaii New Jersey Washington 

Illinois New Mexico West Virginia 

Indiana New York Wisconsin 

Iowa North Carolina 

Kansas Ohio 

States that are not required to arrange for external quality reviews (because they do not have 

Medicaid managed care contracts): 

Alaska 

Arkansas 

Idaho 

Louisiana 

Mississippi 

Montana 

New Hampshire 

North Dakota 

Oklahoma 

South Dakota 

Wyoming 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 16 
 

 

Δ APPENDIX ~ B 

Time Period of EQRO Reports Reviewed 

Time Period 

Number of States' 

EQRO reports 

Calendar Year 2004 

Calendar Year 2005 

Calendar Year 2006 

Fiscal Year 2005 

Fiscal Year 2006 

Fiscal Year 2007 

Other Time Period * 

2 

10 

3 

4 

4 

1 

13 

Total 37 

Source: Office of Inspector General analysis of State Medicaid agency survey data, 2007. 

* Reports covering other time periods include those covering more than 12 months, 12-month periods that did not 

specify fiscal or calendar years, and periods that we were unable to identify. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 17 
 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 18 

Agency Comments 

Δ APPENDIX ~ C 
 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 19 

APPENDIX ~ C 
 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 20 

APPENDIX ~ C 
 

 

 

Δ ACKNOWLEDGMENTS 

This report was prepared under the direction of Joyce M. Greenleaf, 

Regional Inspector General for Evaluation and Inspections in the 

Boston regional office, and Russell W. Hereford, Deputy Regional 

Inspector General. 

Ivan Troy served as the team leader for this study. Other principal 

Office of Evaluation and Inspections staff from the Boston regional 

office who contributed to the report include Rose Lichtenstein; other 

central office staff who contributed include Cynthia Thomas. 

 O E I - 0 1 - 0 6 - 0 0 5 1 0 E XTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE 21