DEPARTMENT OF HEALTH & HUMAN SERVICES 
Health Care Financing Administration 
_________________________________________________________________________________ 
Center for Medicaid and State Operations  
7500 Security Boulevard 
Baltimore, Maryland 21244-1850 
 
 
DEC 22, 1998  
 
Dear State Medicaid Director: 
 
This letter transmits the enclosed "Independent Assessment Requirement for Section 1915(b) Waiver 
Programs: Guidance to States." 
 
This guidance was developed by Health Care Financing Administration (HCFA) Central Office and 
Regional Office staff The Office of Management and Budget and the Medicaid Managed Care Technical 
Advisory Group reviewed and commented on this document. 
 
The purpose of the guidelines is to assist States in choosing an appropriate independent assessment 
entity and to specify the appropriate contents for the assessment. We also discuss how the independent 
assessment relates to quality improvement strategies and activities. Unless a specific strategy or 
regulatory reference is cited, these guidelines are to be considered guidelines only. HCFA does not 
intend to portray the attached document as mandatory standards. 
 
I hope that you find these guidelines useful in developing contracts with entities to conduct the 
independent assessment of Section 1915(b) waivers. Any comments or questions regarding the 
document should be directed to Mike Fiore of my staff on (410) 786-0623 or mfiore~hcfa.gov. 
 
Sincerely, 
 
Sally K. Richardson 
Director 
 
Enclosure 
 
cc: All HCFA Regional Administrators 
All HCFA Associate Regional Administrators 
  for Medicaid and State Operations 
Lee Partridge 
American Public Human Services Association 
Joy Wilson 
National Conference of State Legislatures 
Nolan Jones 
National Governors' Association   
 
 
 
 
 
 
 
SECTION 1915(b) 
WAIVER PROGRAM 
INDEPENDENT ASSESSMENTS: 
GUIDANCE TO STATES 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
            US DEPARTMENT OF HEALTH AND HUMAN SERVICES 
                                Health Care Financing Administration 
                                Center for Medicaid and State Operations 
                                  December 1998   
TABLE OF CONTENTS 
 
 
INTRODUCTION.........................................................................................01 
   
CRITERIA FOR ENTITIES CONDUCTING INDEPENDENT ASSESSMENTS.............03 
 
CHOOSING AN APPROPRIATE INDEPENDENT ASSESSMENT ENTITY ......03 
 
DEFINING THE NECESSARY EXPERTISE .............................................04 
 
DEVELOPING A METHODOLOGY .......................................................05 
 
WRITING THE FINAL REPORT ............................................................06 
 
CONTENT OF THE INDEPENDENT ASSESSMENT............................................06 
 
TOOLS FOR MONITORING ACCESS AND QUALITY ................................07 
 
ELEMENT OF THE INDEPENDENT ASSESSMENT...................................08 
 
ACCESS TO CARE ...................................................................08 
 
QUALITY OF CARE ..................................................................10 
 
COST EFFECTIVENESS ............................................................12 
 
RELATED QUALITY IMPROVEMENT STRATEGIES, ACTIVITIES, 
AND REFERENCES.....................................................................................14 
 
STRATEGIES AND ACTIVITIES ..........................................................14 
 
BBA REGULATION: QUALITY ASSESSMENT AND 
IMPROVEMENT STRATEGY......................................................14 
 
QUALITY IMPROVEMENT SYSTEM FOR MANAGED CARE 
(QISMC)................................................................................15 
 
EXTERNAL QUALITY REVIEW..................................................15 
 
FOCUSED CLINICAL STUDIES..................................................16 
 
PERFORMANCE MEASUREMENT..............................................16 
 
REFERENCES...................................................................................17 
 
APPENDIX A ..................................................................................  SECTION 1915(b) WAIVER PROGRAM 
INDEPENDENT ASSESSMENTS: GUIDANCE TO STATES 
 
The Social Security Act (the Act) authorizes the Secretary of the Department of Health and Human Services to 
waive requirements of Section 1902 of the Act to administer specific freedom of choice waiver programs to 
implement Medicaid managed care to the extent he or she finds it to be cost effective and efficient and not 
inconsistent with the purposes of Title XIX (emphasis added). In order for the Secretary to determine that this 
requirement has been met by States with Section 1915 (b) waivers, the States must comply with the specific 
requirements contained in 42 CFR43I.55(b)(2) which are as follows: 
 
In applying for a waiver to implement an approvable project under paragraph (c), (d), (e), or (f) of this 
section, a Medicaid agency must document in the waiver request and maintain data regarding: 
(i) The cost-effectiveness of the project; 
(ii) The effect of the project on the accessibility and quality of services; (iii) The anticipated impact of 
the project on the State's Medicaid program. 
 
Further, pursuant to Section 2111(B) of the State Medicaid Manual States must arrange for an independent 
evaluation or assessment of their waiver program and submit the findings when renewing their waiver 
programs. At a minimum, the Independent Assessment (IA) is a requirement of the first two waiver periods. The 
1915(b) Streamlined Waiver Application further specifies that the IA should be submitted with the waiver 
renewal request ninety (90) days before the expiration of the approved waiver program. 
 
This document provides guidance to States on the IA including the following components: 
 
I. CRITERIA FOR ENTITIES CONDUCTING INDEPENDENT ASSESSMENTS. This 
section discusses the following: 
 
A)  What is meant by Independent." 
 
B)  What expertise is necessary to conduct an IA. 
 
C)  What the IA agreement should contain 
 
D)  How to develop a reliable and valid methodology. 
 
E)  What should be included in the final report. 
 
1  II.  THE CONTENT OF THE INDEPENDENT ASSESSMENT. This section outlines issues related to 
access to services under the waiver, quality of waiver services, the cost effectiveness of the waiver and 
any other specific requirements made by HCFA in its approval of the waiver request. 
 
A)  Beneficiary Access to Services Under the Waiver: A waiver program under 191 5(b) may not 
substantially impair a beneficiary's access to services as compared to accessibility of services 
prior to or without the waiver. The IA should evaluate or measure the availability of services 
under the waiver and compare it to the level of waiver services that existed prior to the waiver. 
Examples of measurements that could be made to demonstrate access are: ratio of primary care 
providers, specialists, hospitals, and pharmacies to beneficiaries; average distance and travel time 
to see a doctor; and waiting time for an appointment. 
 
B)  Quality of Waiver Services: The quality of services under a 19 15(b) waiver program may not 
be less than the quality of services prior to or without the waiver. The IA should evaluate the 
impact of the quality of services provided to beneficiaries under the waiver and define measures 
to improve and ensure quality of care. Examples of tools that could be used to measure quality of 
care include the quality improvement system for managed care (QISMC), beneficiary surveys 
such as CAHPS, provider surveys, State quality improvement (QI) strategies, and encounter 
data. 
 
C)  Cost Effectiveness of the Waiver: The total costs of the waiver, including program benefits and 
administrative costs, must not be greater man the cost of providing like services without a 
waiver. The IA should compare the cost of the waiver program to the estimated cost of the same 
services to an actuarially equivalent population without the waiver. 
 
III  RELATED QUALITY IMPROVEMENT STRATEGIES. ACTIVITIES. AND REFERENCES This 
section provides the following: 
 
A)  A brief description of related QI activities. 
 
B) A reference section on documents that highlight quality improvement and quality assurance (QA) 
efforts. 
 
2  I.  CRITERIA FOR ENTITIES CONDUCTING INDEPENDENT ASSESSMENTS 
 
 
A)  CHOOSING AN APPROPRIATE INDEPENDENT ASSESSMENT ENTITY
 
Independent Assessment entities must be external to and independent of the State Medicaid agency and 
must not have a direct or indirect relationship (e.g. financial) with any program managed care entities 
(MCEs). 
 
Some examples of entities that may qualify to conduct an IA are: 
State universities; 
- 
Research organizations; 
- 
Independent State audit agencies; 
- 
Health care consulting or management consulting organizations (i.e., Big 5 management 
- 
consulting groups or small, specialized Medicaid managed care consulting firms); and 
External quality review organizations (EQROs). 
-  * 
 
Some examples of entities that may not conduct an IA (or portions thereof) are: 
Actuaries contracted by the State to develop rates and do financial analyses; and 
- 
Organizations, including State agencies, State universities and enrollment brokers, which 
- 
provide services (whether administrative or medical) to enrollees in the managed care program 
being assessed. 
 
 
 
 
 
 
 
If an BORO conducts the IA, H~FA suggests that it be an entity other than the prime contractor conducting 
* 
the external quality review (EQR2 of the managed care organizations (MCOs) providing services under the 
1915(b) waiver program. The substitution of KOR reports as the IA is not considered to be value-added to the 
independent assessment process. As noted in this document, the Independent Assessor is asked to review the 
LOR reports 'as one of the tools to use for monitoring access and quality), to determine how the State does as a 
whole  (i.e while the EQRO reviews individual health plans, the independent assessor looks at the totality of the 
State 's managed care program). Regardless of who conducts the IA, the IA activity receives 50% Federal 
financial participation (FFP) 
 
 
3  B)  DEFINING THE NECESSARY EXPERTISE
 
 Independent Assessment entities should have staff who have an understanding of: 
* * 
1.  Health services research methodologies including: 
 
    the principles of scientific research and statistical analysis; 
- 
    the epidemiologic and statistical measurement of health status indicators in 
- 
      defined populations; 
    methodologies of data collection; 
- 
    the principles of data interpretation, including the social and economic 
- 
      factors that affect the interpretation of data (e.g. case load and cost 
      estimates); and 
    survey and focus group design and analysis. 
- 
  2.  Clinical study methodology, including experience and education in: 
    assessing medical and health care services through QA technology; 
- 
    developing, evaluating, and implementing corrective action plans and 
- 
      quality improvement activities; and 
    reviewing MCO and primary care case management (PCCM) programs (to 
- 
      the extent possible). 
  3.  Financial management practices including: 
    generally accepted accounting principles; 
- 
    budgeting and forecasting principles; and 
- 
    financial auditing principles. 
- 
 
4.  Information systems and claims management. 
 
5.  The Medicaid population and Medicaid managed care. 
 
C)  INDEPENDENT ASSESSMENT AGREEMENT
 
  The agreement between the State and the organization performing the IA should include all necessary 
interagency agreements or contract provisions as required in regulation and in the State Medicaid 
Manual. 
 
  The State should clearly define the scope of the IA in the following areas: 
 
 
Section 4705 of the BRA requires that a method for the identification of entities qualified to conduct FOR be 
** 
established This document will be reviewed to ensure consistency with the final regulation developed on EOR. 
 
4  selection of review topics; 
- 
study design features; 
- 
analysis and interpretation of study findings; and 
- 
structure of follow-up work plans. 
- 
 
 The State should define the duration of the IA Contract. 
 
  The State should clearly specify the data (paid claims, eligibility) available, including format, media, file 
layout, time span, and whether and to what extent the State will clean up and/or manipulate data prior to 
sending it to the contractor. 
 
  The State should clearly outline the confidentiality and disclosure requirements of beneficiary and plan 
specific information for non-peer review organization (PRO) entities. PRO contracts should cite Section 
1160 of the Act. 
 
D)  DEVELOPING A METHODOLOGY
 
  The quality, access and cost effectiveness topics should focus on the State level Medicaid managed care 
program. Analysis should include State and contractor functions such as EQR, enrollment broker, 
actuary, MCO and PCCM. 
 
  The Independent Assessor should begin its review by analyzing existing documents, including federal 
waiver approval letters and reviews, EQRO reports, waiver request/renewal applications, complaint and 
grievance logs, any other quality/access reports produced by the State and by meeting with key 
stakeholders. The Independent Assessor should review resolution and QI activities undertaken by the 
State and MCE to ensure completion and compliance. The Independent Assessor should also 
recommend follow-up activities on previously identified issues and other problems identified through 
this discovery process. 
 
  The Independent Assessor should utilize a comprehensive programmatic research design that can 
describe and evaluate the care delivered to Medicaid beneficiaries through the State's managed care 
program. 
  In order to ensure an effective focused study, the State and Independent Assessor should 
  work together to: 
  refine the study question to limit the aspects of care reviewed (also called framing 
- 
    the question); 
  define a point of comparison, e.g. pre-waiver experience or experience of 
- 
    actuarially equivalent population; 
  determine the methodology the Independent Assessor will use; 
- 
 
 
5  define the quality indicators to be assessed; 
  - 
  review conditions with highest prevalence or incidence, or those which have the 
- 
    potential to benefit certain subgroups of the eligible population (e.g. special needs 
    populations); 
  develop objective criteria for assessing care; 
- 
  define the potential for improving health status; 
- 
  study service delivery in multiple settings; 
- 
  ensure statistically valid sampling techniques; and 
- 
  not duplicate EQRO efforts. 
- 
 
  The State should provide input to the Independent Assessor regarding the analysis of the data 
 
E)  WRITING THE FINAL REPORT
  The IA should include a written report that: 
  concludes whether access, quality, and cost effectiveness are better than, equal to, 
- 
    or worse than prior to the waiver; 
  identifies the positive aspects of the State's process for monitoring the program 
- 
    and suggests/recommends processes in State monitoring that can be improved; 
  highlights unresolved issues; 
- 
  identifies instances in which care can be improved; 
- 
  provides a baseline for future assessment; 
- 
  offers recommendations for concrete actions that can be undertaken by the State to 
- 
    improve the health care received through MCOs and PCCM programs; and 
  provides a summary of the findings of the EQRO and the State's quality strategy. 
- 
 
 
II.  CONTENT OF THE INDEPENDENT ASSESSXIENT 
 
The IA document is intended to confirm and supplement programmatic information provided by the State. The 
content of the IA should not duplicate the State's efforts to assess the program as part of the waiver program 
renewal process; the scope of the IA review should emphasize the State's efforts and ability to monitor the 
program. Ultimately, the State should utilize the Independent Assessor's results and recommendations as a tool 
to improve the program. 
 
NOTE: To avoid duplication of effort, for any  of the following components that are included in the State EQR 
or State quality strategy analysis, we suggest using that information where applicable as it relates to the 19 15(b) 
waiver program for the waiver period being evaluated 
 
 
 
6  The following sections regarding access, quality, and cost effectiveness provide suggestions on tools and 
methodologies to be used in gathering programmatic data as well as the elements to be included in the content 
of the IA report. Some of the items are more useful when combined with other information; for example, 
qualitative measures of access (e.g surveys) are more useful when presented with quantitative measures of 
access (e.g. comparisons of provider availability pre and post waiver implementation). 
 
Tools for Monitoring Access and Quality 
 
The following resources can be used by Independent Assessors to gather information on access and quality in 
Medicaid managed care programs: 
 
  HEDIS Current Version. 
- 
  QISMC. 
  Federal and State statutory and regulatory guidelines. 
  EQRO reports. 
  Beneficiary, provider, and subcontractor surveys (separate from those utilized by the State). We suggest 
that the Independent Assessor use the CAHPS Survey. 
  Interviews or focus groups with beneficiaries, beneficiary representatives/care givers, providers, 
subcontractors, and advocacy groups. 
  Claims, utilization, referral and encounter data. 
  Information sent/given to beneficiaries and displayed in provider and intake offices 
  Enrollment procedures. 
  MCO and State complaint, grievance, and fair hearing appeal logs, reports, and summaries. 
  Provider requirements/credentialing requirements. 
  Disenrollment statistics and reasons. 
  State access monitoring reports and studies including provider geographical mapping documents. 
  Medical records. 
  State monitoring of quality. 
  State administrative documents, such as State contracts; requests for proposals (REP) for services, for 
enrollment brokers, with EQROs; waiver requests; and HCFA waiver approval letters. 
  Qualitative data collected as a result of site visits with State officials. 
  State quality of care reports, performance measures (e.g. HIEDIS reports), and review of QA/QI plans. 
 
 
 
 
 
 
7  Elements of the Independent Assessment Report 
 
A)  ACCESS TO CARE 
 
The following elements are important to the assessment of access to care in Medicaid managed care programs 
(see Related Quality Improvement Strategies, Activities, and References). HCFA recommends that States work 
with the Independent Assessor to include as many of these suggested elements as are practical and appropriate. 
The list is not all-inclusive; the State or the Independent Assessor may identify other areas of importance. If all 
elements are not addressed, States should ensure that the IA includes several elements because a single element 
may not give a true measure of access. 
 
Evaluation of the State program's access monitoring and analysis: 
 
Analysis of State RFPs and/or contract provisions which incorporate access standards and protocols. 
- 
State efforts to ensure all covered benefits are available and accessible. 
- 
 
Enrollment information: 
 
Adequacy of MCE provider directories: 
- 
  how current, clarity of the provider directory, when and how often sent Out, includes pertinent 
information such as pharmacies, hospitals, specialists. etc. 
Effectiveness of development and distribution of marketing materials: 
- 
  ensure clarity, appropriate comprehension level, language and cultural appropriateness, and 
availability of information to enrollees with disabilities. 
Methods by which State ensures that marketing plans and materials are accurate and do not mislead, 
- 
confuse, or defraud beneficiaries. 
Analysis of enrollment procedures; 
- 
  who is actually enrolling beneficiaries and helping them choose providers~ 
  what is the average time between enrollment and ability to get care? 
Report of autoassignment rates at the outset of program versus at time of evaluation, if applicable. 
- 
Disenrollment information; 
- 
  rates of disenrollment, both voluntary and non/voluntary, provider/MCO specific information. 
  State tracking and monitoring of disenrollments; reasons. effective time, process to transition 
back to previous type of care. 
Evidence of efforts in place to preserve current beneficiary/provider relationships. 
- 
State standards and efforts to monitor/improve enrollment procedures. 
- 
 
 
8  Education and customer service information: 
 
Overview of State and provider outreach plan, measured effectiveness. 
- 
Beneficiary, provider, subcontractor awareness/comprehension of program and of managed care system. 
- 
Availability of adequate beneficiary hotline. 
- 
Availability of adequate provider hotline. 
- 
Effective education of Social Service Agencies, Social Security Offices, and any other agencies that 
- 
have frequent contact with beneficiaries. 
Evaluation of effectiveness of enrollment broker agency, if applicable. 
- 
State monitoring of education and customer service activities. 
- 
 
Provider Capacity: 
 
Account for duplication of providers across MCO networks or systems. 
- 
Information on provider networks, including specialists, providers not accepting new enrollees, types of 
- 
interpretive services available to non-English speaking beneficiaries, interpretive services available to 
hearing disabled beneficiaries. 
Levels of provider participation/caseload according to discipline, by MCO. 
- 
Availability of providers during convenient hours of operation to enrollees. 
- 
Beneficiary perception of access to providers for preventive and specialty care. 
- 
State efforts to track/improve provider: beneficiary ratios and overall provider capacity. 
- 
Availability of providers experienced in specific disease management, such as HIV/MDS. 
- 
 
Urgent/Emergent care: 
 
Availability of care 24 hours per day. 
- 
Increase or decreased emergency room utilization rates (over time of program). 
- 
Assurances and standards in place for access to and timely authorization of urgent and emergent care. 
- 
Assure compliance with Section 1932(b)(2) of the Act  prudent layperson definition. 
- 
State efforts to monitor/improve access to urgent/emergent care. 
- 
 
Travel and waiting times for primary care and specialty care (suggested at the beginning of the 
program-- under FFS-- and at period of evaluation'): 
 
Average time to schedule an appointment. 
- 
Availability of medical help over the phone. 
- 
Availability of health care professional to obtain urgent or routine care. 
- 
Average time in waiting area in office. 
- 
State standards and efforts to monitor/improve above times. 
- 
 
 
9  Availability of transportation services to medical appointments. 
 
Referrals 
 
Availability of primary care doctor to see assigned enrollees. 
- 
Ease/ability of enrollees to get necessary referrals: 
- 
  referrals to various providers such as: dentists, mental health providers, language interpretation 
services, obstetrical and prenatal care providers. 
Analysis of general referral procedures. 
- 
State efforts to track referrals and address problematic areas. 
- 
Review of process for feedback to providers. 
- 
 
 
B) QUALITY OF CARE 
 
The following elements are important to the assessment of quality of care in Medicaid managed care programs 
(see Related Quality Improvement Strategies, Activities, and References). HCFA recommends that States work 
with the Independent Assessor to include as many of these suggested elements as are practical and appropriate. 
The list is not all-inclusive; the State or the Independent Assessor may identify other areas of importance. If all 
elements are not addressed, States should ensure that the IA includes several elements because a single element 
may not give a true measure of quality. 
 
Evaluation of the State program's Quality monitoring elements and analysis and review of EORO 
reports: 
 
Standards and procedures for assessing quality and appropriateness of care and services furnished to all 
- 
beneficiaries under the program. 
Appropriate application of intermediate sanctions. 
- 
State's management of EQRO contract, including scope of contract compared to final EQRO product. 
- 
Soundness of research methodology used by EQRO  analysis and review of EQRO report. 
-  -- 
An information system that is sufficient to support initial and ongoing operation and review of the 
- 
quality strategy. 
 
 
 
 
 
 
 
 
10  Clinical review of utilization patterns (suggested at the beginning of the program under FF5 and at 
--
period of evaluation): 
 
Analysis of preventive services for children and adolescents (e.g., vaccine-specific and combined 
- 
immunization rates for 2-year-olds, well-child visits in the first year of life, adolescent well-care visits, 
substance counseling for adolescents). 
Analysis of preventive services for adult women (e.g., cervical cancer screening, mammography 
- 
screening). 
Analysis of prenatal care (e.g., low birthweight, prenatal care utilization). 
- 
Analysis of acute and chronic care (e.g., asthma inpatient admission rate, diabetic retinal exam). 
- 
Analysis of mental health care (e.g., ambulatory follow-up after hospitalization for specified mental 
- 
health disorders). 
Analysis of utilization data by service category for change in patterns from FFS to managed care and to 
- 
include patterns of over/under utilization. 
Analysis of changes in prescription drug use. 
- 
Impact of appropriate or inappropriate use of primary services or emergency services. 
- 
Analysis of utilization patterns for select group of beneficiaries. 
- 
State efforts to monitor and improve utilization patterns. 
- 
 
Grievances and appeals: 
 
Analyze the effectiveness of grievance and appeal process. 
- 
Beneficiary, provider, and subcontractor understanding/knowledge of grievance processes. 
- 
State efforts to monitor grievance patterns for MCEs and providers. 
- 
 
 
Beneficiary, provider and subcontractor satisfaction. We suggest that the Independent Assessor use the 
CAHPS Survey for beneficiary experience: 
 
Member rating/satisfaction with primary care doctor, nurse, specialist and MCE. 
- 
Member satisfaction with the courtesy and respect of providers. 
- 
General satisfaction of beneficiaries, advocacy groups, and provider groups with health care received. 
- 
Involvement of beneficiaries and providers in QA activities. 
- 
State efforts to monitor and follow-up on requests to change providers. 
- 
Analysis of turnover in provider network (primary care providers, mental health providers, OB/GYN, 
- 
dentists, etc.). 
Assessment of provider compensation arrangements on provision of satisfactory care. 
- 
State efforts to gather information regarding beneficiary/provider satisfaction with the program. 
- 
 
 
 
11  State Quality improvement measures: 
 
State/NICE incorporation of State quality improvement strategy/MCE Quality Assurance and 
- 
Performance Improvement (QAPI) plan. 
Provider/MCE adherence to clinical practice guidelines and State monitoring of adherence. 
- 
Evaluation of increase/decrease in quality of life, for example, gainful employment, loss of Medicaid 
- 
eligibility, Global Assessment of Functioning (GAF) scale, etc. 
State efforts to evaluate and improve beneficiary quality of life. 
- 
Review of focused clinical studies. 
- 
State efforts to encourage/assist providers to utilize measures such as encounter data. 
- 
Level and timeliness of feedback to providers. 
- 
Analysis of QI systems, collection and analysis of encounter data, monitoring of quality through 
- 
standard reports, use of data analysis for State QI activities. 
 
C)  COST EFFECTIVENESS 
 
This section provides guidelines on the cost-effectiveness analysis that Independent Assessors may conduct to 
evaluate this component of the State's Medicaid managed care program. HCFA recommends that States 
consider the core approaches listed below as basic procedures and processes that should be used by Independent 
Assessors regardless of the program being evaluated. Additional approaches that could strengthen the cost 
effectiveness evaluation are also described below. 
 
Core Cost Effectiveness Approaches 
 
  Calculate the cost-effectiveness of the managed care program for the previous time period of the 
program. If this is the first IA for the waiver, the Independent Assessor should analyze the first full year 
of data. If this is the second IA of the waiver, the Independent Assessor should analyze the first three 
years' of data (even though some of this data is from a different waiver period) to the extent that the data 
is available. Future IAs should analyze at least two years' of data. The cost-effectiveness should be 
analyzed and risk-adjusted by age and sex to the extent available and appropriate (this applies to MCO, 
P1-lIP, and PCCM waivers). If comparable populations are utilized, then a description of the population 
including a description of the similarities and dissimilarities of such factors such as health status, age, 
and sex should be included. The calculation should include the 4-steps methodology outline in Appendix 
A. 
 
  Review of the State's upper payment limit (UPL) calculation, ratesetting, and cost-effectiveness 
monitoring processes including: 
Validation of the UPL's assumptions; 
- 
 
12  Validation of the UPL's methodology; 
- 
Validation of ratesetting methodology; 
- 
Validation of the inflation rate used by the State as well as a comparison of the inflation rate used 
- 
to the actual experience of the State for the same or a similar set of services for a comparable 
population for the same period of time; and 
Validation and evaluation of the State program's cost-effectiveness monitoring and analyses 
- 
examining frequency, effectiveness, accuracy, and similarity of comparable populations (e.g. on 
factors such as health status, age, and sex) used in the State's cost-effectiveness calculations. If 
the State does not monitor cost-effectiveness for program management purposes, this omission 
should be noted. 
  Analysis of the source of the cost savings in the program including: 
  Has utilization changed (i.e., are there fewer inpatient hospitalizations and more 
- 
    outpatient services)? 
  Is there evidence of a decrease in recipients or services being provided? 
- 
 
Additional Cost Effectiveness Approaches 
  Analysis of possible cost-shifting from capitated service utilization to fee-for-service 
  utilization (or PCCM services to fee-for-service utilization), including: 
  Analysis of side-by-side PCCM and MCO programs to determine if biased 
- 
    selection is occurring and high-cost beneficiaries are disproportionately selecting 
    PCCM programs. 
  Analysis of carve-out programs to determine if easily substitutable fee-for-service 
- 
    costs are being utilized to reduce Prepaid Health Plan (P1-lIP) costs. For example, 
    in mental health and substance abuse carve-out models, are pharmaceuticals being 
    used to reduce the number of inpatient admissions? Another example would be if 
    the PHIP is shifting children with severe emotional disturbances into foster care or 
    rehabilitation programs and reducing the costs of the PI-IP. 
  Analysis of risk-comprehensive MCO or PCCM programs to determine the effects 
- 
    of cost-shifting to any fee-for-service services. For example. if pharmacy or 
    mental health services are not included in the MCO or PCCM model, has there 
    been a dramatic increase in the amount of mental health treatments or 
    pharmaceuticals prescribed for the MCO or PCCM enrollees. 
 
  Assess whether HCFA and the State are paying MCOs/PHPs and PCCM providers appropriately for 
services. The Independent Assessor in this analysis would be calculating the overall amount paid to 
MCOs/PHPs or PCCM providers by the State and comparing it against the contractor's costs for 
administration, profit and service delivery. For example, how do the State's incurred costs (the outlay of 
managed care dollars) to the MCOs relate to the actual costs incurred by the MCOs for providing 
services to the Medicaid enrollees? If on average, the State agency is paying an MCO $100 per member 
per month, but the 
 
13  MCO is only expending $75 per member per month, the Independent Assessor should note this in its 
report. 
 
  Perform an analysis of the State's capitation payment system in paying capitation to MCOs. For 
example, does the State have adequate controls in place to assure that the dollars paid to MCOs are 
accurate and do not reflect FFS payments? 
 
III.  RELATED QUALITY IMPROVEMENT STRATEGIES, ACTIVITIES, AND 
REFERENCES 
 
A)  STRATEGIES AND ACTIVITIES 
 
The IA requirement is distinct from other quality oversight activities; some of which result from the Balanced 
Budget Act of 1997 (BBA), some reflect HCFA policy issued through State Medicaid Directors letters, and 
some are products of HCFA contracts with outside entities. It is important to note several fundamental 
differences between the IA and the activities required under BBA. First, as noted throughout the document, the 
IA covers three broad areas of overall program impact including access to care, quality of care, and program 
cost-effectiveness. The BBA provisions only focus on issues related to beneficiary access to and quality of care. 
Second, the IA is required for the initial two waiver periods and is seldom required beyond that time, whereas 
the BBA requirements are ongoing. Third, the IA is a State Medicaid program-level review, whereas many of 
the BBA requirements are MCE focused. For instance, EQRs generally assess MCOs, while the IA analyzes the 
State's overall program. State IA should be consistent with the State's quality strategy as dictated by the BBA. 
 
The following sections describe five quality assurance and quality improvement strategies/activities currently 
underway that relate to the IA requirement. 
 
BBA Regulation: Quality Assessment and Improvement Strategy 
 
Section 4705 of the BBA requires States that contract for Medicaid managed care under Section 1903(m) of the 
Social Security Act (the Act) to develop and implement quality assessment and improvement strategies. At a 
minimum, these strategies must include access standards, measures that examine other aspects of care and 
services directly related to improving the quality of care (e.g., grievance procedures and marketing standards), 
procedures for monitoring and evaluating the quality and appropriateness of care and services that Medicaid 
enrollees receive, and requirements for the provision of data sets (e.g., HEDIS). A workgroup is currently 
developing a regulation that will implement this provision of the law. 
 
 
 
14  Quality Improvement System for Managed Care (OISMC') 
 
QISMC is a HCFA initiative to protect and improve the health and satisfaction of Medicare and Medicaid 
managed care enrollees. The first product of this initiative health care quality improvement standards and 
- 
guidelines for Medicare and Medicaid contracting MCOs provides a model for HCFA and States to establish 
- 
requirements pertaining to quality measurement and improvement and the delivery of health care and enrollee 
services. The QISMC quality standards direct MCOs to: 
 
  Operate an internal program of quality assessment and performance improvement (QAPI) that achieves 
demonstrable improvements in enrollee health, functional status, and satisfaction across a broad 
spectrum of care and services. 
 
  Collect and report data reflecting its performance on standardized measures of health care quality, and 
meet such performance levels on these measures as may be established under its contract with HCFA or 
the State. 
 
  Demonstrate compliance with basic requirements for administrative structures and operations that 
promote quality of care and beneficiary protection. 
 
The standards address only those areas of MCO operation and performance that are closely related to QI or to 
the delivery of health care and enrollee services. Separate regulations, standards and guidelines exist for other 
areas of organization operation, such as marketing and enrollment and claims processing, that HCFA and States 
must also consider in evaluating organizations. Separate mechanisms also exist for external appeals, monitoring 
and enforcement. These mechanisms are part of HCFA's and the State's oversight protocols, not the QISNIC 
standards. 
 
The QISMC standards and guidelines update HCFA's 1993 Health Care Quality Improvement System for 
Medicaid Managed Care (QARI) guidelines for internal QA programs of Health Maintenance Organizations, 
Health Insuring Organizations and P1-liPs contracting with State Medicaid agencies. Other components of 
QARI (a framework for health care quality improvement systems of State managed care programs, and 
guidelines for external quality review) will also be updated as HCFA promulgates BBA regulations and 
additional guidance relating to State quality assessment and improvement strategies and EQR. 
 
External Quality Review (EQR') 
 
Currently, States contracting with Medicaid MCOs under section 1903  of the Act must arrange for an annual 
(in) 
independent, external review of the quality of services delivered by each MCO. Section 1902(a)(30)(c) of the 
Act gives States latitude in defining the scope of the 
 
 
15  reviews. The entities that conduct these reviews are not to be part of the State government, a MCO, or an 
association of MCOs. The entities can be PROs, private accreditation organizations approved by HCFA, or 
PRO-like entities approved by HCFA. 
 
Section 4705 (a) of the BBA (Section 1932 (c)(2) to the Social Security Act) establishes State requirements for 
the EQR. This provision requires States to provide for EQR of quality outcomes and timeliness of, and access 
to, the items and services for which the organization is responsible under the contract. This provision also 
requires the Secretary of the Department of Health and Human Services, in consultation with States, to establish 
a method for identifying entities that are qualified to perform EQR and to contract with an independent quality 
review organization to develop protocols to be used in the independent reviews. A HCFA workgroup is 
currently developing a regulation that will implement this provision of the law 
 
Focused Clinical Studies 
 
HCFA has encouraged States to conduct focused clinical studies through Section 1115 research and 
demonstration waivers, and Section 1915(b) program waiver requirements. In addition QISMC's QAPI domain 
standards specify certain focused areas that must be addressed over time by MCOs (e.g., preventive services, 
care of beneficiaries with chronic conditions). Within these broad areas, MCOs can select particular topics for 
measurement and improvement. 
 
Performance Measurement 
 
The Government Performance and Results Act (GPRA) is the primary legislative framework through which 
government agencies are required to set strategic goals, measure performance, and report on the degree to which 
the goals are met. HCFA's GPRA goal on childhood immunization will track the rate of two-year old children 
enrolled in Medicaid who are fully immunized. 
 
In addition, under a HCFA contract, MEDSTAT has undertaken the Key Operational Indicators Project to 
develop a set of performance measures reflecting key aspects of Medicaid managed care program operations. 
The project is intended to accomplish the following: (1) provide States with benchmarks for performance 
comparisons across programs and with other States; (2) provide HCFA Regional Offices with a standardized, 
systematic way of gathering information on operational performance measures collected by the States; and (3) 
provide HCFA Central Office with a mechanism for tracking operational performance trends across States and 
to formulate technical assistance strategies. 
 
In December 1997, MEDSTAT developed a draft of an open-ended data collection tool. This survey tool was 
piloted in the first quarter of 1998 in four States. The tool has been modified and was administered to an 
additional 10 States in July. The information received from all 14 States was presented through a Picturetel on 
October 30, 1998 to the regions that had State 
 
 
16  participation. The actual tools that were completed by each State will have been sent to each of the 
corresponding regions by the end of November. A report on the findings is being consolidated and is due in 
January 1999. After reviewing the report a decision will be made on the projects next steps. 
 
B)  REFERENCES 
 
Below are references which provide information related to Medicaid managed care quality assessment and 
improvement efforts, and rate setting and risk adjustment methodologies: 
 
Actuarial Research Corporation, Report prepared for the Department of Health and Human 
Services (DI-IHS)/the Health Care Financing Administration (HCFA), Capitation Rate Setting in 
Areas with Eroded Fee-For-Service Base Final Report, 1992. 
 
Actuarial Research Corporation, Setting an Upper Payment Limit Where the Fee for Services Base is 
Inadequate: Final Re port, 1992. 
 
Alpha Center, Report produced for the Robert Wood Johnson Foundation, Risk Adjustment: A Special Report, 
1997. 
 
Ann Arbor Actuaries, Inc., Report prepared for DHSS/HCFA., A Review of Rate Setting Methods of Selected 
State Medicaid Agencies for Prep aid Health Plans, 1991. 
 
Ann Arbor Actuaries, Inc., Report prepared for DHSS/HCFA, Actuarially Sound Rate Setting Methodologies, 
1991. 
 
Conference Report 105-217 to accompany H.R. 2015, the Balanced Budget Act of 1997, (Section 4705 and the 
regulations being developed to implement these requirements). 
 
Foundation for Accountability (FACCT), Foundation for Accountability (FACCT) Guidebook for Performance 
Measurement Prototype Summary, 1995. 
 
Joint Commission for Accreditation of Healthcare Organizations, National Library of Health Care Indicators 
1997. 
 
Massachusetts Medical Society, Quality of Care: Selections from The New England Journal of Medicine, 1997. 
 
Mathematica Policy Research, Inc, The Quality Assurance Reform Initiative (QARI) Demonstration For 
Medicaid Managed Care: Final Evaluation Report, 1996. 
 
 
 
17  MEDSTAT Group, Report prepared for U.S. DHI-IS/HCFA, A Guide for States: Collecting and Analyzing 
Medicaid Managed Care Data, 1997. 
 
MEDSTAT Group, Report prepared for U.S. DHIHS/HCFA, Survey of Key Performance Indicators, 1997. 
 
Medicaid Management Institute of the American Public Welfare Associations, report prepared for 
DHHS/HCFA, Medicaid Primary Care Case Management Programs: Guide for Implementation and Quality 
Improvement, 1993. 
 
Merlis, Mark for National Governor's Association (NGA), Medicaid Contracts with HMOs and Pre Paid Health 
Plans: A Handbook for State Managers, 1987. (**Rate Setting Description still applicable) 
 
National Academy for State Health Policy, Quality Improvement Primer For Medicaid Managed Care, 1995. 
 
National Academy for State Health Policy, Quality Improvement Standards and Processes Used by Select 
Public and Private Entities to Monitor Performance of Managed Care: A Summary, 1995. 
 
National Academy for State Health Policy, Report prepared for HCFA. Quality Improvement System for 
Managed Care (Draft), 1997. 
 
National Committee for Quality Assurance (NCQA), Health Plan Employer Data and Information Set (HEDIS 
 Current Version). 
 
President's Advisory Commission on Consumer Protection and Quality in the Health Care 
Industry, Final report to the President of the United States, Quality First: Better Health Care for 
All Americans, 1998. 
 
U.S. DHHS/HCFA, A Health Care Quality improvement System for Medicaid Managed Care: A Guide for 
States, 1993. 
 
U.S. DHHS/PHS/AHCPR, Conquest 1.1: A Computerized Needs-Oriented Quality Measurement Evaluation 
System, 1996. 
 
U.S. DHHS/PHS/AHCPR, Consumer Assessment of Health Plans (CAUPS) Satisfaction Survey, 1997 
 
U.S. DHI-{S/PHS/AHCPR, Putting Research to Work in Quality Improvement and Quality Assurance: 
Summary Report, 1993, Publication No. 93-0034. 
 
18  U.S. DI-IHS/PHS/AHCPR Research Activities Newsletter, Monthly publication. 
 
U.S. DHI-IS/HCFA and National Committee on Quality Assurance (NCQA), Health Care Quality 
Improvement Studies in Managed Care Settings: Design and Assessment: A Guide for State 
Medicaid Agencies, 1994, Purchase Order #HCFA-92-1279 
 
U.S. DHHS/HCFA American Public Welfare Association (APWA), Monitoring Risk-Based Managed Care 
Plans: A Guide for State Medicaid Agencies. 
 
 
Websites: www.hcfa.gov, www.ahcpr.gov or outside organizations such as www.ncqa.org, www nashp.org. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19  APPENDIX A 
 
THE FOLLOWING 4-STEP TECHNIQUE 
MAY BE USED TO CALCULATE WAIVER COST-EFFECTIVENESS: 
 
Step 1. Create a single set of numbers of managed care eligibles that can be related back 
to the waiver. 
Step 2 Calculate total without waiver costs by first calculating per beneficiary without 
waiver costs. The total without waiver costs are equal to the number of managed care enrolled 
beneficiaries utilizing services at the non-managed care utilization multiplied by non-managed 
care costs per beneficiary. To eliminate the effects of cost-shifting, all significant Medicaid costs, 
including fee-for-service costs, should be included in cost-effectiveness analyses of risk-
comprehensive MCO and PCCM models. Easily substitutable fee-for-service costs should be 
included in cost-effectiveness analyses for P1-liP models (e.g., pharmacy costs in mental health 
carve-outs). The Independent Assessor may want to take actuarial adjustments such as Incurred 
But Not Yet Reported Claims into account to the extent possible. The population sex age 
categories, as well as major service categories (e.g., physician, hospital, pharmacy, other 
services,), should at a minimum be calculated separately before being combined in a summary 
table for each year and for the waiver period 
 
  Method 1: Comparable population cost-effectiveness methodology 
 
Method 1. (managed care enrolled eligible member months x non-managed care 
utilization***) x non-managed care cost per beneficiary per month by category of service 
Total without waiver costs 
= 
 
  Method 2. Trend fee-for-service costs forward methodology 
 
Method 2. Per beneficiary base year costs for all services updated to account for policy 
changes, inflation, etc occurring under the fee-for-service program  without waiver costs 
= 
 
Step 3 Calculate total with waiver costs by first calculating per beneficiary with waiver 
costs. Total with waiver costs are equal to the number of managed care enrolled beneficiaries 
multiplied by the managed care costs per beneficiary plus administrative fees for managed care. 
The Independent Assessor may want to take actuarial adjustments such as Incurred But Not Yet 
Reported Claims into account to the extent possible. The population sex age categories should at 
a minimum be calculated separately before being combined in a summary table for each yea1~ 
and for the waiver period. 
 
 
Utilization is defined as the number of consumers who actually use the service. 
* * *  APPENDIX A 
 
Methods 1 and 2: (managed care enrolled eligible member months x managed care per 
beneficiary costs by category of services) + administration costs of the waiver as listed in Table 2 
with waiver costs 
= 
 
Step 4: Calculate savings per beneficiary and the overall waiver savings for the 
current waiver period. Multiply the per beneficiary costs by the number of beneficiaries for 
both Year I and Year 2. Add the two years costs together. The population~ sex. age categories, 
as well as major service categories (e.g., physician, hospital, pharmacy, other services), should 
at a minimum be calculated separately before being combined in a summary table for each year 
and for the waiver period 
 
Without waiver costs with waiver costs savings 
- 
 
Table 1: Number of Enrollees and Enrolled Member Months 
 
Table I Instructions. Create a table showing the number of enrollees and member months by population, age, 
and sex (for capitation should be each rate cell). Please create tables for each of the rate categories and a 
summary table. These tables should be consistent with the submittal of the cost-effectiveness calculations, 
Section C's, Appendices, and calculations. The table should be tailored to the actual needs of each State. 
 
The following are examples of possible rate cells (if capitated program, use the rate cells 
developed by the actuary): 
  TANF eligible enrollees age 18 and under 
  TANF eligible enrollees age 19 and older 
  SSI eligible enrollees age 18 and under 
  SSI eligible enrollees age 19 and older 
  Medicaid-Medicare dual eligible enrollees 
 
TANF 0-18 Sample Table I 
  Number of Enrollees 
  Enrollees  Member Months 
Year 1     
Year 2     
Current Waiver Total      APPENDIX A 
 
Table 2: Administration Costs for the Waiver 
 
Report administrative costs in a table-like form (see sample below). The table should be tailored to the actual 
needs of each State. One-time only costs associated with program initial implementation should be included. 
The State should document these costs and the effect upon cost-effectiveness in that period. For more 
information, see the 1915(b) streamlined waiver application preprint instructions 5.c and 7 and Appendix C.7 in 
the PCCM renewal and C.6 and Appendix C.7 in the capitated initial form. 
 
Sample Table 2 
  Year 1  Year 2  Current Waiver Total 
       
Contract Administration       
Systems Modification       
Marketing, Educating Recipients       
Handling Grievances/Appeals       
Utilization Review System       
Additional Staff       
Enrollment Broker       
Encounter Data/MMIS       
EQRO       
Hotline Operation       
Other (Peer Education, Health Assessment. and       
Consulting) Please specify. 
Total       
 
 
NOTE: Administrative overhead costs in fee-for-service that would be comparable in managed care should not 
be calculated in waiver cost effectiveness determination (e.g. state Medicaid Director salary. Hoii 'ever, cost 
directly associate with providing administrative  functions applicable to both fee-for-service and managed care 
but which are not equivalent, such as surveillance and utilization reviews, Medicaid management information 
system claims payment, should be noted and the fee-for-service costs should be subtracted from the managed 
care costs. Or, this information could be built into the L/PL.