Department of Health and Human Services
OFFICE OF 
 
INSPECTOR GENERAL 
 
 
 
 
 
 
 
 
EXTERN AL QUALITY REVIEWS IN 
 
M M C  
EDICAID  ANAGED  ARE
 
 
 
 
 
 
 
 
Daniel R. Levinson  
Inspector General 
 
June 2008 
OEI-01-06-00510   
Office of Inspector General

http://oig.hhs.gov  
 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as  
amended, is  to protect the integrity of the Department of  Health and Human Services  
(HHS)  programs, as well as the health and welfare of beneficiaries  served by those 
programs.  This statutory mission  is  carried  out through  a nationwide network  of  audits, 
investigations, and inspections conducted by the  following operating components:  
Office of Audit Services 
The Office  of  Audit Services  (OAS) provides auditing services  for HHS, either by  conducting  
audits  with its own audit resources or by  overseeing  audit work done by others.  Audits  
examine the performance of HHS programs and/or its grantees and contractors in carrying  
out their respective responsibilities and are intended  to provide independent assessments of 
HHS programs and operations.  These assessments help reduce waste, abuse, and  
mismanagement and promote economy and efficiency  throughout  HHS. 
Office of Evaluation and Inspections  
The Office  of  Evaluation and Inspections (OEI)  conducts national evaluations to  provide 
HHS, Congress, and the public with timely, useful, and reliable information on significant 
issues.  These evaluations focus on preventing fraud,  waste, or abuse  and promoting  
economy, efficiency, and effectiveness of departmental programs.  To promote impact, OEI 
reports also present practical recommendations for improving  program operations.  
Office of Investigations 
The Office of Investigations (OI) conducts criminal, civil, and administrative investigations  
of  fraud and misconduct  related to HHS programs, operations, and beneficiaries.  With 
investigators working in all 50 States and the District of Columbia, OI  utilizes its resources 
by actively  coordinating with the Department  of Justice  and other Federal, State, and local 
law enforcement authorities.  The investigative efforts of OI often lead to  criminal 
convictions, administrative sanctions,  and/or  civil monetary  penalties.  
Office of Counsel to the Inspector General 
The Office of Counsel to the  Inspector  General (OCIG) provides general legal services to 
OIG, rendering  advice and opinions on HHS programs and operations and  providing all  
legal support for OIGs  internal operations.  OCIG represents  OIG in all civil and 
administrative fraud and abuse cases involving HHS programs,  including False Claims Act, 
program exclusion, and civil monetary penalty cases.  In  connection with these cases, OCIG 
also negotiates and monitors corporate integrity agreements.  OCIG renders advisory 
opinions, issues compliance program  guidance, publishes fraud alerts, and provides other  
guidance  to  the health care industry concerning the anti-kickback statute and other  OIG  
enforcement authorities.      
  E X E C U T I V E   S U M M A R Y  
OBJECTIVES 
 
To determine: 
1. the extent to and ways in which States utilize external quality 
reviews of Medicaid managed care,  
2. the extent to which external quality review organizations (EQRO)  
provided States with required information, and  
3. limitations that States have identified with the external quality 
review process. 
BACKGROUND 
Medicaid is a joint Federal and State program that provides medical 
assistance to low-income groups, such as children, senior citizens, and 
people with disabilities.  Many States utilize managed care to respond 
to Medicaid expenditures that have  more than doubled since 2000.  As 
of 2006, 65 percent of the 45.6 million Medicaid beneficiaries were 
enrolled in managed care. 
The Social Security Act requires States that operate Medicaid managed 
care programs to provide for an external, independent review of their 
managed care organizations.  States may contract with an independent 
entity called an EQRO to conduct the external quality review.  The 
Centers for Medicare &  Medicaid Services (CMS) provides States with 
matching Federal funds for review expenditures, including the 
production of results, depending on the qualifications of their  EQRO  
contractors.  Federal regulations require three mandatory activities as 
part of this review and  offer States the choice of requiring up to five 
optional activities.  Regulations also require the  review to produce     
five specific deliverables, including a detailed technical report. 
We reviewed EQRO reports for the 37 States that arrange for external 
quality reviews.  We also reviewed the contracts and requests for 
proposals between these  States and their EQROs.  We then surveyed  
these 37 States regarding their use of EQRO reports and their 
experience with the review process.  We received a 100-percent response 
rate.  We also conducted structured interviews with representatives 
from six States.    
 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   i     
EE XX EE CC UU TT II VV EE    SS UU MM MM AA RR YY
    
FINDINGS 
Most States are using the results of EQRO reviews. For example,    
33 of the 37 States required their managed care plans to make changes 
based on EQRO reports,  such as how plans document and conduct 
performance improvement projects and how plans meet State standards 
regarding members access to care. Twenty-two States used EQRO  
reports to share knowledge across plans, for example, by targeting 
technical assistance to the plans and sharing best practices across 
plans. Fourteen States  reported amending their contracts with plans 
based on EQRO reports and 16 States took other actions, such as setting  
new performance standards. Overall, 21 States rated the EQRO reports 
to be very useful. Furthermore, 25 States required their EQROs to 
conduct optional review activities during their last annual review 
period. 
Some EQRO reports did not include all required information, despite  
the States oversight.   EQRO reports for 15 States were missing at least 
one of the five required  deliverables. The two most common missing 
deliverables were an assessment of managed care plans responses to 
previous EQRO recommendations and  comparative information about all 
of the States plans. EQRO reports for eight States were missing 
information on at least one of the three mandatory activities. All States 
reported regularly monitoring their EQROs through ongoing  
communication, written status reports, and contract provisions. Of the   
37 contracts and requests for proposals that we reviewed, 34 included 
provisions to ensure EQRO compliance with Federal regulations. 
More than half of the States cited concerns with the external quality  
review process. Sixteen States identified staffing concerns with the 
external quality review process, particularly turnover and training 
issues. Thirteen States  raised  concerns with the quality of EQRO 
reports, especially their timeliness and the degree to which their 
findings and recommendations could  be implemented. Five States 
reported finding the mandatory activities redundant to other 
monitoring activities, such as those conducted by private accreditors. 
Many States requested technical support from  CMS to help address 
these issues.  
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   ii     
E X E C U T I V E   S U M M A R Y
   
 
RECOMMENDATIONS 
CMS should work with States to ensure that  EQROs are providing 
complete information.   CMS could,  as part of its review of the contents 
of the EQRO reports, inform States if  it finds that any of the required  
deliverables are missing.  Further, CMS could amend its regulations to  
describe, as a condition for receiving enhanced reimbursement, the steps 
that States must take to ensure that all the required deliverables are 
included in the quality review results.  
CMS should provide additional technical assistance and written  
guidance to States.  CMS could do this by organizing teleconferences, 
offering written guidance, and sharing best practices.  It should schedule 
these teleconferences regularly and  make any written guidance available 
online. CMS could use a format similar to a teleconference it held with 
States in 2006.  CMS should consider two priorities for its technical 
assistance and written guidance:  minimum standards for EQRO report 
organization and content and use of accreditation data in external quality 
reviews. 
AGENCY COMMENTS 
CMS agreed with both of our recommendations and cited actions that it 
has taken in response to  each.  It provided feedback to Medicaid  
Directors in 15 States regarding deficiencies in EQRO reports.  CMS 
also focused one of its triannual audio conferences, available to States 
and EQROs, on external quality reviews.  Further, CMS assisted a 
managed care accrediting organization in creating a document for 
States on using accreditation data in their reviews. 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   iii   
  T A B L E   O F   C O N T E N T S  
EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
  
 
 
I N T RO DUC T IO N  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1 
  
 
 
F I N D IN G S  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  7 
  
  Most States are using results of EQRO reviews  . . . . . . . . . . . . . . .  7 

         EQRO reports for some States were missing required elements. . .  9 

  More than half of States had concerns with the review process. .  11 

 
 
R E C O MMEN D A T ION S  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  14 
  
 
 
APPENDIXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  16 
  
  A:  Status of States Arrangements for Reviews  . . . . . . . . . . . . . .  16 

         B:  Time Period of EQRO Reports Reviewed . . . . . . . . . . . . . . . . .  17
 
         C:  Agency Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18
  
 
 
A C K NO WL E D G M E N T S  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21 
      
I N T R O D U C T I O N  
  I N T R O D U C T I O N  
OBJECTIVES 
To determine: 
1. the extent to and ways in which States utilize external quality 
reviews of Medicaid managed care,  
2. the extent to which external quality review organizations (EQRO)  
provided States with required information, and  
3. limitations that States have identified with the external quality 
review process. 
BACKGROUND 
Medicaid and the Growth of Managed Care 
Medicaid is a joint Federal and State program that provides medical 
assistance to low-income groups, such as children, senior citizens, and 
people with disabilities.  Individual States establish eligibility 
requirements, benefits packages, and Medicaid payment rates under  
standards set by the Centers for Medicare & Medicaid Services (CMS). 
Medicaid spending has increased at both the Federal and State levels.  
Federal expenditures increased from $118 billion in 2000 to $180 billion 
in 2006.1  State expenditures rose by almost 60 percent, from $89 billion 
in 2000 to $142 billion in 2006. These expenditures account for almost 
one-quarter  of all States expenditures.2    
In response to this growth, States have increasingly utilized managed 
care to deliver Medicaid services to beneficiaries.  Managed care is a 
health care delivery system that aims to maximize efficiency by 
negotiating rates, coordinating care, and managing use of services.  
From 2000 to 2006, enrollment in Medicaid managed care grew            
59 percent, from 18.8 million to 29.8 million  beneficiaries. As of 2006, 
65 percent of the 45.6 million Medicaid beneficiaries were enrolled in 
managed care.3    
 
 
1 Department of Health and Human Services, Budget in Brief, 2002 and 2008.  

Available online at http://www.hhs.gov/budget/docbudget.htm. Accessed on 

May 20, 2008. 

2 National Association of State Budget Officers, 2006 State Expenditure Report, 

2007. 

3 CMS, Medicaid Managed Care Enrollment Report as of June 30, 2006. 

  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   1     
I N T R O D U C T I O N   
 
Concerns About Medicaid Managed Care Quality of Care  
The Government Accountability Office found that the capitated (i.e., a 
predetermined amount per beneficiary) nature of managed care could 
create fiscal incentives to underserve beneficiaries.  It recommended 
that States carefully monitor access to and quality of care delivered to 
Medicaid managed care  populations.4 Other published studies have 
suggested that managed care reduces the quality and accessibility of 
health care, particularly in specific vulnerable subpopulations.5  
State Oversight of Medicaid Managed Care 
The Social Security Act requires contracts with Medicaid managed care 
organizations (MCO) to provide for an external independent review.   
Specifically, it mandates that each MCO contracted with the State must 
be reviewed annually by a qualified independent entity.  This review 
must evaluate the MCOs quality outcomes and timeliness of, and 
access to, the items and services for which the organization is 
responsible.6      
Requirements for External Quality Review  Organizations 
In January 2003, CMS issued a final rule to further specify 
requirements for external quality reviews and EQROs.7 To qualify as 
an EQRO, an organization must have knowledgeable staff and sufficient 
physical, financial, and technological resources.  An EQRO and any of  
its subcontractors must also be independent from the State  Medicaid 
agency and  the MCOs under review.  States are  responsible for ensuring  
that EQROs meet these requirements.8  Some EQROs hold external 
quality review contracts with several States.  For example, the Health 
Services Advisory Group holds EQRO contracts  in 11 States.9    
 
4 Government Accountability Office, GAO/T-HRD-93-10, Medicaid:  States Turn to
  
Managed Care to Improve Access and Control Costs, 1993.  Available online at 

http://archive.gao.gov/d42t14/148756.pdf. Accessed on May 20, 2008. 

5 E.C. Schneider, et al., Racial Disparities in the Quality of Care for Enrollees in 

Medicare Managed Care, Journal of the American Medical Association, 2002, 

Volume 287, pp. 12881294.  D. G. Clement, et al., Access and Outcomes of Elderly 

Patients Enrolled in Managed Care,  Journal of the American Medical Association,
  
1994, Volume 271, pp. 14871492. 

6 Social Security Act  1932(c)(2)(A)(i).
  
7 42 CFR pt. 438, subpart E. 
 
8 42 CFR  438.354. 

9 Florida Medicaid Selects HSAG for External Quality Review Services,             

May 25, 2006.  Available online at http://www.hsag.com/news/2006_0525.asp. 

Accessed on May 20, 2008.
   
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   2     
II NN TT RR OO DD UU CC TT II OO NN    
 
Regulations also permit the State to conduct the review.10 Most States 
choose to contract with independent EQROs to conduct the required 
activities; however, some States perform one or more  of the  activities 
themselves. Even States conducting their own reviews usually contract  
with EQROs to write the final technical reports.  
States receive an enhanced Federal match of 75 percent for their review  
expenditures, including the production of results, if they contract with 
qualified EQROs. They receive a Federal match of 50 percent for any 
activities conducted by nonqualified entities.11 States are responsible 
for contracting with EQROs and ensuring that EQROs are producing 
the required deliverables.12  
Mandatory and Optional Review  Activities 
Federal regulations require that external quality reviews include three 
mandatory activities. For each MCO or prepaid inpatient health plan13  
(hereafter referred to as plans), the external quality review must 
include the following activities:  
1.  validation of performance improvement projects  (PIP) required by 
the State and underway in the preceding 12 months; 
2.  validation of plan performance measures required by the State and 
reported by the MCO in the preceding 12 months; and  
3.  a review, conducted in the previous 3-year period, to determine the 
plans compliance with State standards for access to care, structure 
and operations, and quality measurement and improvement.14  
In addition, Federal regulations allow States to require up to five 
optional activities: 
1.  validation of encounter data reported by a plan, 
2.  administration and validation of consumer and provider surveys on 
quality of care, 
3.  calculation of additional performance measures, 
 
10 42 CFR  438.358(a). 

11 42 CFR  438.370. 
 
12 42 CFR  438.356 and 438.364.
  
13 These are health plans that provide inpatient hospital or institutional services on 
 
an at-risk or other than State plan reimbursement basis. 

14 42 CFR  438.358(b). 

OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   3     
I N T R O D U C T I O N   
4.  conduct of additional PIPs, and 
5.  conduct of studies on quality focused on a particular aspect of clinical 
or nonclinical services at a point in time.15  
In May 2002, CMS issued a series of  protocols for use in conducting 
Medicaid external quality review activities.  These serve as guidelines 
for EQROs in conducting mandatory and optional external quality 
review activities. 
External Quality Review  Deliverables 
After completing the review activities, an EQRO must produce the 
following information (hereinafter referred to as deliverables):  
1.  a detailed technical report describing the data aggregation and 
analysis and the way in  which conclusions were drawn as to the 
quality, timeliness, and access to the care furnished by the plans;   
2.  an assessment of each plans strengths and weaknesses with  respect 
to quality, timeliness, and access to care; 
3.  as the State determines, methodologically appropriate, comparative 
information about all plans; 
4.  recommendations for improving the quality of health care  services 
furnished by the plans; and 
5.  an assessment of the degree to which each plan has addressed 
effectively the quality improvement recommendations made by an 
EQRO during the prior years review.16  
Some EQROs issue one technical report encompassing all of the plans 
in the State; others issue multiple reports, each specific to one plan or  
activity. Generally, these reports are organized around the 
mandatory activities.  However, even if a report is missing 
information on one of the mandatory activities, it may still contain all 
of the required deliverables.  Regulations require  States to make 
these reports available to interested parties upon request.17 Usually 
the States also provide CMS with copies of the deliverables. 
 
 
 
15 42 CFR  438.358(c). 
16 42 CFR  438.364(a).  
17 42 CFR  438.364(b) and 438.350(f). 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   4     
II NN TT RR OO DD UU CC TT II OO NN    
CMS Efforts To Improve the External Quality Review Process 
In October 2006, CMS held an audio conference with State Medicaid 
agencies addressing the assessment, utility, and improvement of EQRO 
reports.   During this conference, CMS presented its review and  
evaluation of reports from 22 States. According to the review, reports 
vary substantially in organization, length, detail, topics addressed, and 
presence and extent of recommendations. CMS found that many 
reports did not comply with at least one external quality review 
regulatory requirement.18  
CMS prepared a tool kit based on that review.19 This tool kit offered a 
brief assessment of 2005 reports, identified key opportunities for 
improved reporting, recommended specific sections to be included in 
reports, and suggested a standardized format for submitting reports.20   
CMS provided the tool kit to States, who could then provide  it to their 
EQROs. 
METHODOLOGY 
Scope 
This evaluation is national in scope.   Forty States, including the District 
of Columbia (hereafter referred to as States), contract with plans and  
thus must arrange for external quality reviews.  Three of these States  
(Georgia, Maine, and New York) are currently working with CMS to 
implement the review process, so we  excluded them from  our analysis.21   
We included the 37 States that have implemented the review process.  
States that do not contract with managed care organizations are not 
required to conduct reviews. See Appendix A for a list of all States and 
whether they contract with plans. 
Data Sources 
We collected and analyzed data from the following sources: 
EQRO reports. We obtained from CMS and the States copies of all 
reports that EQROs produced pursuant to regulations. Because review 
periods differed across States, we could not review reports for all States  
 
18 CMS, State External Quality Review Tool Kit for State Medicaid Agencies, 

October 2006. 

19 Ibid. 
 
20 Ibid. 

21 Since the completion of our review, Maine has discontinued its Medicaid  managed 
 
care program and will not be implementing an external quality review process. 

OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   5     
II NN TT RR OO DD UU CC TT II OO NN    
for the same time period. See the table in Appendix B for the time 
periods covered by the reports we reviewed. 
Contracts. We obtained  from  the  States copies of their EQRO contracts  
or requests for proposals covering 2005. 
Surveys. We sent surveys by mail to the Medicaid Directors of the         
37 States and received a 100-percent response rate. The surveys asked 
States how they conduct oversight of their EQROs and utilize the 
results of previous EQRO reports.   The surveys also asked  about States 
general experiences in implementing the external quality review 
process.    
Interviews. We conducted structured telephone interviews (and one 
in-person interview) with representatives from Medicaid offices in six  
States. We  purposively selected these States based on their survey 
responses.    
Analysis 
We reviewed each EQRO report to determine whether it included all 
elements required by Federal regulations. We counted elements as 
missing if they were not present in the report and the report or survey  
data did not explain why. For example, we did not count the  
assessment of prior recommendations  as missing if the plans report 
noted that the plan had not contracted with the State the previous year 
or if the States survey reported that the State was currently only in its 
second review period. We also did not count any section as missing if 
the State was still in the process of implementing that aspect of the 
review, for example, if the State had not initiated any PIPs in the 
previous 12-month period. 
Limitations 
Data from the surveys and follow-up interviews were self reported by 
States. We  did not independently verify these data. 
We did not determine whether assessments of plan strengths and 
weaknesses were present in EQRO reports. Because the general scope 
of an external quality review is to examine plan strengths and  
weaknesses with respect to quality, timeliness, and access to  care, all 
data in the report relate to these topics. 
Standards 
We conducted this study in accordance with the Quality Standards for 
Inspections issued by the Presidents Council on Integrity and 
Efficiency and the Executive Council on Integrity and Efficiency. 
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   6     
  F I N D I N G S  
 
Most States are using the results of     
Of the 37 States, 29 rated the 
EQRO reviews 
external quality review process as 
very important in overseeing their  
Medicaid managed care programs.   States cited several benefits of the 
review process, including identifying and documenting concerns with 
plans, providing a comprehensive review of the State Medicaid  
programs and their plans, and providing additional monitoring that the  
States are unable to do  on their own.  Similarly, 21 States rated the 
EQRO reports as very useful, citing the data in the reports and the 
independence of the reviews.   
Thirty-three of thirty-seven States required their managed care plans to 
make changes based on EQRO reports  
The plans developed and implemented these State-required  changes.  
The most common  changes, required by 28 States, called for improving 
the documentation and conduct of PIPs.  For example, plans had to  
document that they tested the statistical significance of reported 
improvements.  They also had to create templates for developing and 
submitting PIPs. Twenty States reported requiring changes in how  
plans complied with State and Federal standards.  These changes 
typically addressed standards regarding member grievances and 
appeals or access to care.  See Table 1 for a list of changes States 
required plans to make based on EQRO reports. 
 
 Ta ble 1:  Managed Care Plan Changes Required by State 
Number of  States 
That Required  
Area in Which State Required Managed Care  Plan Change  Change* 
Documentation and Conduct of Performance Improvement  Projects  28 
Compliance With State and Federal Standards  20 
Calculation and Reporting of Performance  Measures  16 
Improvement in Information Systems  10 
Other Change  4 
  Source:  Office of  Inspector General analysis of State Medicaid Agency survey data, 2007. 
* The  numbers do not total 33 because some States required more than one change. 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   7     
FFF III NNN DDD III NNN GGG
  SSS  
States typically required the managed care plans to develop and 
complete corrective action plans (CAP) to address these changes. States 
also reported providing technical assistance to the plans to help them 
develop their CAPs. States used several methods to ensure that the 
plans made these changes, such as including contractual provisions,  
monitoring CAPs, and delegating oversight to EQROs. 
According to eight States, their managed care plans initiated changes 
based on EQRO report data without the States requiring them to do so.  
For example, one plan made changes  to its internal information systems 
after the EQRO pointed out shortcomings. 
Twenty-two  States used EQRO reports to share knowledge across managed 
care plans 
Knowledge sharing included States targeting technical assistance to the 
plans (reported by 21 States). Eight of these States focused their 
assistance on improving PIPs. Often, EQROs provided assistance to 
plans (reported by 15 States). States also reported sharing best 
practices across their plans. Typically, these States convened periodic 
meetings with representatives from  the States, EQROs, and all plans. 
Some States cited the potential value of identifying and sharing best 
practices at the national level.  
Fourteen States reported amending their contracts  with managed care plans 
based on EQRO findings  
Of these States, 12 inserted new provisions into their contracts. The 
changes included strengthening and clarifying contract requirements, 
using external quality review results to justify contract extensions and 
renewals, and adjusting PIPs. 
Sixteen States also reported taking other actions based on EQRO reports 
Other commonly reported State changes included modifying the 
performance measures.   Eleven States reported  changes such as adding 
new measures and establishing new benchmarks for plans to meet. 
Additionally, nine States reported making changes such as 
standardizing quality assurance processes and  implementing a tracking  
system for identifying and responding to adverse events. 
Furthermore, 25 States required their EQROs to conduct optional review  
activities during their last annual review  period 
States reported requiring their EQROs to conduct at least one of the five  
optional review activities to provide additional information for States 
quality improvement efforts. One of the two most common optional 
activities States reported was administering or validating surveys of 
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   8     
FFF III NNN DDD III NNN GGG SSS  
quality of care either to  consumers or to providers. In the reports we 
reviewed, many EQROs used the Consumer Assessment of Healthcare  
Providers and Systems survey instrument. This survey stems from a 
public-private initiative that developed standardized surveys  of 
patients experiences.  The second  of the two most common optional 
activities was conducting focused quality studies on particular aspects 
of care. Among the focused studies we saw, common topics included 
determining childhood immunization rates at a specific age and 
assessing trends in adolescent well care. See Table 2 for the number of 
optional activities States required their EQROs to conduct. 
 Ta ble 2:  Optional  Review  Activities Required by States 
Number of States  
That Required 
EQROs To Conduct  
Optional Activity  Activity 
Administering or Validating Surveys on Quality of Care  14 
Conducting Focused Quality Studies  14 
Validating Encounter Data  12 
Calculating Additional Performance Measures  7 
Conducting Additional Performance Improvement Projects  6 
  Source: Office of  Inspector General analysis of State Medicaid Agency survey data, 2007.  
 
Federal regulations require EQROs
Some EQRO reports did not include all required 
to produce five deliverables based 
information, despite the States oversight 
on their review activities and  
provide them to the States. 
EQRO reports for 15 States were missing at least one of the five required 
deliverables 
Reports for 12 States were missing the assessment of managed care 
plans responses to previous EQRO recommendations, and reports for     
7 States were missing comparative information about all of the States 
plans. 
These deliverables can provide useful information. For example, some 
reports displayed the previous EQRO recommendations, plan responses  
and actions, and the new recommendations in one chart, thereby 
enabling the States to see a comprehensive history at a glance. The 
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   9     
FF II NN DD II NN GG
  SS  
comparative information often included tables presenting, for all plans, 
performance measure scores, PIP ratings and scores, as well as charts 
showing those plans not in compliance with each of the reviewed State 
and Federal standards. In addition, some reports included a 
comparison of the States plans with national averages. 
Reports for  22 States included all of the required deliverables. Seven of  
these States contracted  with the same EQRO to produce their reports.  
This EQRO structures its reports similarly, organized around each 
required deliverable.   
EQRO reports for eight States were missing information on at least one of 
the three mandatory activities 
External quality reviews must include three mandatory activities: 
validation of PIPs, validation of performance measures, and reviews of 
compliance with State standards. EQRO reports from five States were 
missing information on one mandatory activity,  reports from two States 
were missing information on two activities, and  a report from one State 
was missing information on all three activities.  Because PIPs are 
typically multiyear endeavors, many of the reports that did include data 
on them could not draw full conclusions about the PIPs efficacy. In 
reports we  reviewed, plans completed only about 20 percent  of the PIPs.   
In these cases, the reports included  a summary of the PIPs progress at 
that point and a review of the managed care plans methodology. 
All States reported regularly monitoring their EQROs 
States rely on ongoing communication, written status reports, and 
contract provisions to oversee their EQROs. 
All but 1 of the 37 States reported regularly communicating with their 
EQROs through telephone calls, e-mail, and in-person meetings.22   
Typically, contact occurred weekly and involved the EQRO updating the 
State on its progress in completing particular review activities and how 
the EQRO was performing toward meeting its timeline. This gave both  
parties an opportunity to discuss any problems that may have arisen.  
Staff from one State told us that they had increased contact with their  
EQRO to weekly calls because of concerns over its performance. 
 
22 The State that reported no ongoing communication with its EQRO  performs the 
external quality reviews itself. This State contracts with EQROs only to produce the 
annual report. 
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   10     
FFF III NNN DDD III NNN GGG SSS  
Twenty-four States also require their EQROs to submit ongoing status 
reports during the review period. These reports typically include 
information on the EQROs progress toward completing deliverables.  
Contracts for 22 States require EQROs to submit written status reports 
or draft reports. 
Of the 37 EQRO contracts and requests for proposals we reviewed,     
34 included provisions to ensure EQRO compliance with Federal 
regulations. Contracts in 24 States require EQROs to show that they 
(1) have sufficient and qualified staff, (2) are entities independent of the  
State Medicaid agencies, and (3) possess sufficient resources to conduct 
the external quality reviews. One of these States required its EQRO to 
pass a readiness review prior to beginning review activities. That State 
designed its review to  ensure that the EQRO had in place the 
infrastructure to successfully conduct external quality reviews. 
Even though States had contractual remedies available to them, EQRO  
reports for 19 States were still missing either required deliverables or 
information on mandatory activities. Among the remedies were those 
allowing States to withhold payment from EQROs or requiring them to 
purchase a performance bond to be forfeited to a State if an EQRO 
failed to fulfill its contract. When asked whether they would change the 
review process based on their experiences to date, only one State 
specifically addressed the missing elements. That State reported that it  
was developing stricter requirements  for the delivery of external quality 
review products. 
 
More than half of the States cited concerns  with  Twenty-four of the thirty-seven 
the external quality review process  States reported three main 
concerns: staffing, EQRO report  
quality, and redundancy with other monitoring efforts. 
Sixteen States identified staffing concerns with the external quality review  
process 
Among these States, 15 reported that staff turnover at the managed  
care plans, State Medicaid agencies, or EQROs prevented them from  
implementing EQRO report recommendations because of the time  
needed to train new staff. Seven States specifically identified training 
EQRO staff as a barrier. These States reported that to get acceptable 
reports, they had to provide EQROs with technical assistance, thereby 
using States time and resources that otherwise would have been spent 
on their own oversight. 
OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   11      
F I N D I N G
  S  
Thirteen States raised concerns with the quality of EQRO reports 
States identified concerns ranging from the timeliness of the reports to  
the degree to which the findings and recommendations could  be 
implemented.  In our review of the reports, the substance of 
recommendations varied widely.  Some were extremely broad and  
difficult to interpret.  For example, one report recommended: 
Overall, and as seen from the data, tables, and graphs 
presented herein, the plan is responsible for addressing 
opportunities for quality improvement through the 
corrective action plan process established by the State.  
Although the plan is empowered to  design and implement 
a corrective action plan that most suitably addresses 
substandard performance . . . .  It will be imperative that 
the plan follow completely through with its corrective 
action plan(s) already approved by the State, which will 
also be monitored by the State. 
Similarly, another EQRO recommended that a plan monitor 
performance of health care utilization and health indicators, identify 
areas for improvement, implement intervention strategies, and monitor 
the effectiveness of the interventions without offering any more specific  
information.  Furthermore, two other States reported that, to get an 
acceptable report, they had to use their own resources to provide 
technical assistance to their EQROs.  Finally, one State found that the 
EQRO wrote the report with CMS as its primary audience, thus 
diminishing the reports  value to the State.   
Eleven States requested additional technical guidance and feedback 
from CMS, particularly regarding EQRO reports.  Staff we spoke with 
from one State wanted more guidance from CMS at the national level  
rather than the regional, particularly regarding minimum standards for 
report organization. 
Five States reported finding EQRO mandatory activities redundant to other  
monitoring activities  
Many States require plans to use specific performance measures, known  
as the Healthcare Effectiveness Data and Information Set, which 
EQROs then validate.  The National Council on Quality Assurance,  
which accredits almost 30 percent of the plans for which we reviewed 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   12     
F I N D I N G S  
reports, validates those same measures.23  In other cases, the States 
conduct their own reviews as part of their contract with the plans.  
These duplicate activities can place a burden both on plans, which must 
provide similar data to EQROs, States, and/or accreditors, and on 
States, which must use their resources to ensure that plans submit data 
to EQROs, thus reducing time spent on other oversight issues. 
Federal regulations permit States to use data from a Medicare  or  
accreditation review in place of the EQRO review of plan compliance 
with State standards.24  However, States reported receiving conflicting 
guidance from CMS, particularly from the regional offices,  on this 
provision.  A staff member from one State told us that she relied on 
their EQRO  to determine what data it used in its review.  Because of  
the States uncertainty, six States requested increased guidance or 
flexibility from CMS regarding use of accreditation data in external 
quality reviews. 
 
23 National Council on Quality Assurance, MCO  Accreditation.  Available online at 

http://web.ncqa.org/tabid/67/Default.aspx. Accessed on May 20, 2008. 

24 42 CFR  438.360. 

  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   13     
  R E C O M M E N D A T I O N S  
 
 
We found that although most States are using the results of  EQRO 
reviews and  find value in the review process, more than half of the 
States raised concerns regarding the external quality review process,  
including concerns about staffing, the quality of reports, and redundant 
data reporting.  States oversee the external quality review process 
through a variety of means, yet EQRO reports for some States are 
missing required information. To improve the quality and usefulness of 
the external quality review process, States requested additional 
guidance and feedback from CMS on these and other issues.   
To help States ensure that all required elements  are present and  
maximize the usefulness of external quality reviews, CMS should: 
Work With States To Ensure That EQROs Are Providing Complete 
Information 
CMS could,  as part of its review of the contents of the EQRO reports,  
inform States if it finds that any of the required deliverables are 
missing. Further, CMS could amend its regulations to describe, as a 
condition for receiving enhanced reimbursement, the steps that States 
must take to ensure that all required deliverables are included in the  
quality review results.  
Provide Additional Technical Assistance and Written Guidance to States 
CMS could do this by organizing teleconferences, offering written 
guidance, and sharing best practices.  CMS should continue to use a 
format similar to its 2006 teleconference. It should schedule these 
conferences regularly, perhaps annually or semiannually. 
CMS should provide further written guidance to the States.  This 
guidance could build on the tool kit that CMS has already developed.  
CMS should post its written guidance online to  make it easily accessible 
to any new State Medicaid agency staff. 
Because CMS usually receives and reviews the EQRO reports, it should 
consider culling best practices to share across the States.  It could share  
these during teleconferences. 
Finally, CMS should consider two priorities for its technical assistance 
and written guidance:   
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   14     
R E C O M M E N D A T I O N S  
Provide States with minimum standards for report organization and content. 
Further guidance could reduce the likelihood that any of the required  
deliverables would be missing from  EQRO reports.  Clarifying 
expectations for the content of reports could improve their level of detail 
and usefulness, particularly regarding the assessment of plan strengths  
and weaknesses.  CMS could utilize best practices from reports to serve 
as a model for other States.   
Clarify the degree to which Federal regulations allow the data from 
accreditation reviews to be used. Currently, States have received 
differing interpretations from CMS central and regional offices 
regarding whether Federal regulations allow them to use data from 
accreditation reviews in place of external quality reviews.  Providing 
consistent and specific guidance could improve efficiency for States, 
plans, and EQROs. 
AGENCY COMMENTS 
CMS agreed with both of our recommendations and cited actions that it 
has taken in response to  each.  It provided feedback to Medicaid  
Directors in 15 States regarding deficiencies in EQRO reports.  CMS 
also focused one of its triannual audio conferences, available to States 
and EQROs, on external quality reviews.  Further, CMS assisted the 
National Committee on  Quality Assurance in creating a document for 
States on using accreditation data in their reviews.  CMS has begun 
working with a second managed care accreditation agency, the 
Utilization Review and Accreditation Committee, on a similar 
document.  The complete text of CMSs comments can be found in 
Appendix C. 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   15     
  A P P E N D I X   ~   A
Status of States Arrangements for External Quality Reviews at the Time of  Our Review  
States that are required to arrange for external quality reviews (because they have Medicaid 
managed care contracts): 
 
Alabama  Kentucky  Oregon 
Arizona  Maine  Pennsylvania 
California  Maryland  Rhode Island  
Colorado   Massachusetts  South Carolina 
Connecticut   Michigan  Tennessee 
District of Columbia  Minnesota  Texas 
Delaware  Missouri  Utah 
Florida  Nebraska  Vermont 
Georgia  Nevada  Virginia 
Hawaii  New Jersey   Washington  
Illinois  New Mexico  West Virginia 
Indiana  New York  Wisconsin 
Iowa  North Carolina 
Kansas  Ohio 
 
States that are not required to arrange for external quality reviews (because they do not have  
Medicaid managed care contracts): 
 
Alaska   
Arkansas 
Idaho 
Louisiana 
Mississippi 
Montana 
New Hampshire 
North Dakota 
Oklahoma 
South Dakota  
Wyoming   
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   16     
  A P P E N D I X   ~   B
 
 
   
Time Period of EQRO Reports Reviewed 
Number of States' 
Time Period  EQRO reports 
Calendar Year 2004  2 
Calendar Year 2005  10 
Calendar Year 2006  3 
Fiscal Year 2005  4 
Fiscal Year 2006  4 
Fiscal Year 2007  1 
Other Time Period *  13 
   Total  37 
  Source:  Office of  Inspector General analysis of State Medicaid agency survey data, 2007. 
* Reports covering other time periods include those  covering more than 12 months, 12-month periods that did not 
specify fiscal or  calendar years, and periods that we were unable to identify. 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   17     
 A P P E N D I X   ~   C  
 
Agency Comments 
  OEI-01-06-00510  EXTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE  18     
A P P E N D I X   ~   C  
 
  OEI-01-06-00510  EXTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE  19     
A P P E N D I X   ~   C  
 
 
  OEI-01-06-00510  EXTERNAL QUALITY REVIEWS IN MEDICAID MANAGED CARE  20     
  A C K N O W L E D G M E N T S
 
This report was prepared under the  direction of Joyce M. Greenleaf, 
Regional Inspector General for Evaluation and Inspections in the 
Boston regional office, and Russell W. Hereford, Deputy Regional 
Inspector General.    
Ivan Troy served as the team leader for this study.  Other principal 
Office of Evaluation and Inspections staff from the Boston regional 
office who contributed to the report include Rose Lichtenstein; other 
central office staff who contributed include Cynthia Thomas.   
 
  OEI-01-06-00510   EXTERNAL  QUALITY  REVIEWS IN  MEDICAID  MANAGED CARE   21