69418 Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 

LeRoy Richardson, 

Chief, Office of Scientific Integrity, Office of 

the Associate Director for Science, Office of 

the Director, Centers for Disease Control and 

Prevention. 

[FR Doc. 2013–27653 Filed 11–18–13; 8:45 am] 

BILLING CODE 4163–18–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Centers for Medicare & Medicaid 

Services 

[CMS–3288–NC] 

Patient Protection and Affordable Care 

Act; Exchanges and Qualified Health 

Plans, Quality Rating System (QRS), 

Framework Measures and 

Methodology 

AGENCY: Centers for Medicare & 

Medicaid Services (CMS), HHS. 

ACTION: Notice with comment. 

SUMMARY: This notice with comment 

describes the overall Quality Rating 

System (QRS) framework for rating 

Qualified Health Plans (QHPs) offered 

through an Exchange. The purpose of 

this notice is to solicit comments on the 

list of proposed QRS quality measures 

that QHP issuers would be required to 

collect and report, the hierarchical 

structure of the measure sets and the 

elements of the QRS rating 

methodology. In addition, this notice 

solicits comments on ways to ensure the 

integrity of QRS ratings, and on priority 

areas for future QRS measure 

enhancement and development. 

DATES: To be assured consideration, 

comments must be received at one of 

the addresses provided below, no later 

than 5 p.m. on January 21, 2014. 

ADDRESSES: In commenting, refer to file 

code CMS–3288–NC. Because of staff 

and resource limitations, we cannot 

accept comments by facsimile (FAX) 

transmission. 

You may submit comments in one of 

four ways (please choose only one of the 

ways listed): 

1. Electronically. You may submit 

electronic comments on this regulation 

to http://www.regulations.gov. Follow 

the ‘‘Submit a comment’’ instructions. 

2. By regular mail. You may mail 

written comments to the following 

address only: Centers for Medicare & 

Medicaid Services, Department of 

Health and Human Services, Attention: 

CMS–3288–NC, P.O. Box 8016, 

Baltimore, MD 21244–8016. 

Please allow sufficient time for mailed 

comments to be received before the 

close of the comment period. 

3. By express or overnight mail. You 

may send written comments to the 

following address only: Centers for 

Medicare & Medicaid Services, 

Department of Health and Human 

Services, Attention: CMS–3288–NC, 

Mail Stop C4–26–05, 7500 Security 

Boulevard, Baltimore, MD 21244–1850. 

4. By hand or courier. Alternatively, 

you may deliver (by hand or courier) 

your written only to the following 

addresses: 

a. For delivery in Washington, DC— 

Centers for Medicare & Medicaid 

Services, Department of Health and 

Human Services, Room 445–G, Hubert 

H. Humphrey Building, 200 

Independence Avenue SW., 

Washington, DC 20201. 

(Because access to the interior of the 

Hubert H. Humphrey Building is not 

readily available to persons without 

Federal government identification, 

commenters are encouraged to leave 

their comments in the CMS drop slots 

located in the main lobby of the 

building. A stamp-in clock is available 

for persons wishing to retain a proof of 

filing by stamping in and retaining an 

extra copy of the comments being filed.) 

b. For delivery in Baltimore, MD— 

Centers for Medicare & Medicaid 

Services, Department of Health and 

Human Services, 7500 Security 

Boulevard, Baltimore, MD 21244–1850. 

If you intend to deliver your 

comments to the Baltimore address, call 

telephone number (410) 786–9994 in 

advance to schedule your arrival with 

one of our staff members. 

Comments erroneously mailed to the 

addresses indicated as appropriate for 

hand or courier delivery may be delayed 

and received after the comment period. 

For information on viewing public 

comments, see the beginning of the 

SUPPLEMENTARY INFORMATION section. 

FOR FURTHER INFORMATION CONTACT: 

Nidhi Singh Shah, (301) 492–5110, for 

general information. Elizabeth FlowDelwiche, (410) 786–1718, for matters 

relating to the Quality Rating System. 

SUPPLEMENTARY INFORMATION: Inspection 

of Public Comments: All comments 

received before the close of the 

comment period are available for 

viewing by the public, including any 

personally identifiable or confidential 

business information that is included in 

a comment. We post all comments 

received before the close of the 

comment period on the following Web 

site as soon as possible after they have 

been received: http://

www.regulations.gov. Follow the search 

instructions on that Web site to view 

public comments. 

Comments received timely will also 

be available for public inspection as 

they are received, generally beginning 

approximately 3 weeks after publication 

of a document, at the headquarters of 

the Centers for Medicare & Medicaid 

Services, 7500 Security Boulevard, 

Baltimore, Maryland 21244, Monday 

through Friday of each week from 8:30 

a.m. to 4 p.m. To schedule an 

appointment to view public comments, 

phone 1–800–743–3951. 

I. Background 

A. Legislative Background 

The Patient Protection and Affordable 

Care Act of 2010 (Pub. L. 111–148) as 

amended by the Health Care and 

Education Reconciliation Act of 2010 

(Pub. L. 111–309) (collectively referred 

to as the Affordable Care Act) establish 

Affordable Insurance Exchange or 

Exchange (also known as a Health 

Insurance Marketplace or Marketplace) 

within each state. Qualified individuals 

and qualified employers in each state 

will be able to shop for affordable health 

insurance through Exchanges. 

The Department of Health and Human 

Services (the Secretary) holds primary 

responsibility for establishing the 

standards and guidelines for the 

Exchanges. The Affordable Care Act 

provides States with the flexibility to 

establish and operate their own 

Exchange (State-based Exchange). 

However, if a state elects not to establish 

a State-based Exchange or if a state will 

not have an Exchange that is operational 

by January 1, 2014, pursuant to section 

1321(c)(1) of the Affordable Care Act, 

the Secretary will establish and operate 

a Federally-facilitated Exchange in those 

states. The Affordable Care Act and 

applicable Exchange regulations 

establish that health plans offered 

through an Exchange must meet specific 

standards to be certified as QHPs and to 

offer coverage in an Exchange beginning 

in January 2014. 

The Affordable Care Act also requires 

the Secretary to develop a number of 

reporting requirements to support the 

delivery of quality health care coverage 

offered in the Exchanges. Specifically, 

sections 1311(c)(3) and (c)(4) of the 

Affordable Care Act direct the Secretary 

to develop—(1) a system that rates 

qualified health plans (QHPs) based on 

the relative quality and price; and (2) an 

enrollee satisfaction survey system that 

assesses the level of enrollee experience 

(that is, consumer experience) with 

QHPs. Because we believe that QHP 

consumer experience is an important 

part of rating the overall quality of a 

QHP, we intend to use some of the 

information collected from the Enrollee 


TKELLEY on DSK3SPTVN1PROD with NOTICES
Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 69419 

1Patient Protection and Affordable Care Act; 

Establishment of Exchanges and Qualified Health 

Plans; Exchange Standards for Employers, 77 FR 

18310 (Mar. 27, 2012) (to be codified at 45 CFR 

parts 155, 156, & 157). 

2See Report to Congress: National Strategy for 

Quality Improvement in Health Care available at 

http://www.ahrq.gov/workingforquality/nqs/

nqs2013annlrpt.htm. 

Satisfaction Survey in the Quality 

Rating System (QRS). 

In addition to consumer experience, 

we believe that the QRS should provide 

ratings of QHPs based on health care 

quality, health outcomes, and cost of 

care. We intend for all QHP issuers to 

report data at the product level for the 

initial years of QRS implementation (for 

example, at the Health Maintenance 

Organization level or Preferred Provider 

Organization level). We expect QHPs to 

provide product-level quality 

performance data for the QRS in general 

topics, such as clinical effectiveness of 

care, patient safety, care coordination, 

prevention of disease and illness, access 

to care, member experience, plan 

services and efficiency, and cost 

reduction. The QRS ratings should 

demonstrate sound, reliable, and 

meaningful information on the 

performance of QHPs to ultimately 

support informed decisions by 

consumers. 

We have already promulgated 

regulations at 45 CFR 155.200(d) that 

direct Exchanges to oversee 

implementation of the QRS, and 45 CFR 

156.200(b)(5) 1 that directs QHP issuers 

to report health care quality information 

to an Exchange. In this notice, we 

describe the overall QRS framework and 

the factors that guided the development 

of the QRS. We solicit comments on the 

QRS measure sets for QHPs offered to 

adult individuals and families, (QRS) 

and for child-only QHPs (Child QRS), 

the hierarchical structure of the measure 

sets, and the elements of the rating 

methodology. We also solicit comments 

on ways to ensure the integrity of QRS 

ratings, and the identification of priority 

areas for future QRS measure 

enhancement and development. 

In future rulemaking, we intend to 

propose requirements for QHPs and 

Exchanges regarding the collection and 

submission of specific quality-related 

information. In addition, we intend to 

provide future technical guidance for 

QHP issuers and Exchanges related to 

the QRS measure specifications, 

detailed rating methodology guidelines, 

and data reporting and procedures. 

B. QRS Goals and Principles 

We believe that the overarching goal 

of the QRS is based on two fundamental 

tenets: (1) Providing comparable and 

useful information regarding the quality 

of QHPs offered through the Exchanges 

to inform consumer and employer 

choice; and (2) facilitating regulatory 

oversight of QHPs with regard to the 

quality standards set forth in the 

Affordable Care Act. Consequently, we 

believe that the QRS should provide 

QHP ratings based on health care 

quality and outcomes, consumer 

experience, and cost. We developed the 

following five general QRS principles to 

guide the design of the QRS: 

• The QRS should produce QHP 

quality performance information to 

encourage the delivery of higher-quality 

health care services, expand access to 

care, and improve health outcomes for 

QHP enrollees. 

• The QRS should provide sound, 

reliable, and meaningful quality-related 

QHP information, which could be used 

by consumers when comparing health 

plans, by QHPs for quality 

improvement, as well as by Exchanges 

and CMS for QHP certification and 

regulatory oversight activities. 

• The QRS should reflect the goals of 

the National Strategy for Quality 

Improvement in Health Care priorities,2

which includes reporting cross-cutting 

performance areas (that is, patient 

safety, prevention, population health, 

patient engagement, patient experience, 

and efficient resource use). The QRS 

should also facilitate reporting on 

conditions or procedures of significant 

prevalence and importance (for 

example, heart disease or breast cancer 

screening). 

• The QRS measures set should be 

evidence-based and align, to the 

maximum extent possible, with priority 

measures currently implemented in 

federal, state, and private sector 

programs to minimize QHP issuer 

burden. We have drawn on our 

experience administering the Medicare 

Advantage 5-star rating system in 

developing this framework, and intend 

that the development and evolution of 

the QRS should be public and 

transparent and should allow for 

flexibility to incorporate changes in 

measures and methodologies as medical 

treatments and technology evolve and 

the Exchanges mature. 

C. QRS Framework 

We have developed a framework for 

creating, implementing, maintaining 

and revising the QRS. The overall 

framework consists of the following 

components that are guided by the QRS 

goals and principles: 

• Performance Information 

• Rating Methodology 

In total, there are ten associated 

elements that further clarify the 

Performance Information and Rating 

Methodology components (see Table 1 

below). 


TKELLEY on DSK3SPTVN1PROD with NOTICES
69420 Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 

3 In addition to the programs and measure sets 

mentioned above, CMS included the following 

program measure sets in the environmental scan: 

eValue8, Consumer Reports Health Plan Rankings, 

Office of Personnel Management Federal Employee 

Health Benefit Program; Health Plan Accreditation 

programs: URAC, National Committee for Quality 

Assurance, Accreditation Association for 

Ambulatory Health Care; State Health Monitoring 

Programs: Maryland HealthChoice Consumer 

Report Card, California Healthcare Quality Report 

Card, NY Electronic Quality Assurance Reporting 

Requirements, Maryland Health Plan Report Card, 

California Medi-CAL Health Plan Quality Ratings; 

State Based Exchanges: Oregon Health Insurance 

Exchange, New York State Health Benefit Exchange 

California Health benefits Exchange 

4National Quality Forum. ‘‘Measure Evaluation 

Criteria, November 2012.’’ accessed January 23, 

The goals and principles for the QRS 

serve as the common thread throughout 

the QRS framework. The Performance 

Information component consists of four 

elements: (1) Measures Selection; (2) 

Hierarchical Structure; (3) Organization 

of Measures; and (4) Data Strategy. The 

Measures Selection element represents 

the process for selecting and evaluating 

the measure sets of the QRS. The 

Hierarchical Structure element 

establishes how the QRS measure sets 

are organized for scoring, rating, and 

reporting purposes. The Organization of 

Measures element establishes the 

approach to create composites, domains, 

and summary indicators ratings. The 

Data Strategy element, which is 

discussed in section IV, refers to the 

procedures for how the measures data 

will be collected, calculated, submitted 

and will help to inform how data will 

be displayed. 

The Rating Methodology component 

aims to define how QHPs will be scored 

and compared, and as proposed, 

consists of six elements: 

• Aggregation Rules would be used to 

determine how measures should be 

combined to create useful quality 

information on health care areas such as 

diabetes care or preventive health care. 

• Sampling and Attribution would 

establish the selection criteria for 

determining appropriate population 

samples that yield reliable and valid 

information. 

• Scoring would be the process used 

to convert the raw QRS measures data 

to points or percentiles on a common 

numeric scale. 

• Performance Classification would 

be used to assign values to the QHP 

scores; these values would then be used 

to categorize the QHP’s performance. 

• Population and other adjustments 

would refer to changes made to raw data 

or measures to remove potential bias 

introduced by factors that are not 

modifiable by the QHP. 

• Peer Groups would be used to 

establish a benchmark dataset for 

comparison of the individual QHP in 

the performance classification work, 

most often based on the geographic and 

time period considerations (for example, 

current annual distribution of all plans 

nationally). 

II. Performance Information 

Component 

A. Measures Selection 

The process used to select the QRS 

measure sets included a review of 

existing health plan measures, so that 

the QRS measures promote consistency 

and harmonization across State, Federal 

government entities (for example, CMS) 

and private-sector efforts. Our review 

included national measure sets that 

were relevant to the intended purpose of 

the QRS and incorporate health plan 

measures such as the Initial Adult 

Medicaid Core Set of Health Care 

Quality Measures, Initial Core Set of 

Children’s Health Care Quality 

Measures, Clinical Quality Measures for 

Eligible Professionals, and Medicare 

Part C and Part D Reporting 

Requirements, as well as a variety of 

other quality measurement programs, 

including health plan accreditation 

programs.3 We believe it’s important 

that measures, in the initial years, be 

specified for health plans (rather than 

specified for health care providers) to 

ensure reliable data, reduce QHP burden 

and facilitate consumer use and 

comprehension. 

Measures selection and measure set 

evaluation criteria were developed 

using the National Quality Forum (NQF) 

Measure Evaluation Criteria and the 

Measures Application Partnership 

(MAP) Measure-Selection Criteria.4 5


TKELLEY on DSK3SPTVN1PROD with NOTICESEN19NO13.003</GPH>
Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 69421 

2013, http://www.qualityforum.org/docs/measure_

evaluation_criteria.aspx. 5Measure Applications Partnership. ‘‘MAP 

Working Measure Selection Criteria and Working 

Guide.’’ National Quality Forum, December 2012. 6Request for Information Regarding Health Care 

Quality for Exchanges: http://www.gpo.gov/fdsys/

pkg/FR–2012–11–27/pdf/2012–28473.pdf. 7Patient Protection and Affordable Care Act; 

Standards Related to Essential Health Benefits, 

Actuarial Value, and Accreditation; Final Rule 78 

FR 12834 (Feb. 25, 2013) (to be codified at 45 CFR 

parts 147, 155 and 156). 

8 Initial Core Set of Health Care Quality Measures 

for Adults Enrolled in Medicaid (Medicaid Adult 

Core Set). February 2013. 

9SHO: #13–002. Letter to State Health Official 

and State Medicaid Director. Re: 2013 Children’s 

Core Set of Health Care Quality Measures. January 

24, 2013. 

10http://www.consumerreports.org/health/

insurance/health-insurance-plans.htm. 

The measure selection criteria, which 

represent industry-tested criteria and 

were supported as measure inclusion 

criteria based on discussions with 

stakeholders and public comment 

received in response to a Request for 

Information (RFI),6 focuses on the 

following areas: 

• Importance: the extent to which the 

measure is important to making 

significant gains in health care quality, 

improving health outcomes, has a high 

impact (high priority) and is relevant to 

the Exchange population and benefits 

covered by QHPs. 

• Performance Gap: the extent to 

which the measure demonstrates 

opportunities for performance 

improvement based on variation in 

current health plan performance. 

• Reliability and Validity: the extent 

to which the measure produces 

consistent (reliable) and credible (valid) 

results. 

• Feasibility: the extent to which the 

data related to the measure are readily 

available or could be captured without 

undue burden and can be implemented 

by QHPs. 

• Alignment: the extent to which the 

measure is included in one or more 

existing federal, state or private sector 

health plan quality reporting programs. 

The QRS measure set evaluation 

criteria were applied to identify 

measurement gaps in the QRS measure 

sets and helped to ensure that the 

proposed QRS measure sets as a whole 

would best meet the needs of consumers 

and the Exchanges. 

The draft QRS measure sets were 

evaluated to determine the extent to 

which the measures were NQF-endorsed 

and aligned with the NQS priorities. 

Relevance to the Exchange consumer 

was evaluated by assessing whether the 

measure set addressed clinical 

conditions of moderate or high 

prevalence or high disease burden 

(applicable only to the clinical care 

measures) and whether the measure sets 

identified the needs of the consumer 

related to health-plan operations and 

satisfaction. Relevance of the QRS 

measure sets to QHPs was evaluated by 

assessing how well each of the sets 

addressed the benefit categories 

required of QHPs as part of the 

Affordable Care Act essential health 

benefits requirement; 7 and if the sets 

complemented other information used 

by the Exchange to support consumer 

comparison of health plans or to assist 

with QHP certification and plan 

monitoring. The comprehensiveness of 

the draft QRS measure sets were 

assessed by examining the measures and 

ensuring that, to the extent possible 

based on the availability of health-plan 

specified measures, the sets included an 

appropriate mix of clinical care measure 

types, such as structure, process and 

outcome measures; experience of care 

measures; and measures that assess 

cost/resource use/appropriateness of 

care and plan management. The draft 

QRS measure sets were evaluated for the 

degree to which they promoted 

equitable access and treatment by 

considering healthcare disparities, and 

ways in which the measure sets can 

capture data to promote strategies that 

address variations in care. In addition, 

the draft QRS measure sets were 

evaluated based on the percentage of 

measures that demonstrated parsimony, 

an efficient use of resources, 

including—(1) the ready availability of 

automated data (available through 

existing claims, administrative, survey, 

and health plan management databases); 

or (2) whether the measures are publicly 

reported or currently in use as 

contractual performance standards 

between plans and public/private 

purchasers or between plans and 

provider organizations or as in 

accordance with statutory or regulatory 

requirements. 

The draft measure sets were revised 

and the proposed QRS measure sets 

were created following this evaluation. 

The proposed QRS measure sets were 

also evaluated and reviewed internally 

by CMS, externally by industry and 

stakeholders and in a field test using 

available health plan data. Listening 

sessions were also conducted for 

insurers, states and consumer groups. 

Although the measures contained in 

the QRS are consistent with the state-ofscience for measuring health care 

quality, science and technology do not 

yet allow us to measure or represent the 

quality of all care delivered through the 

QHPs. Therefore, the QRS measure set 

should not be viewed as representative 

of all care delivered by QHPs. 

B. Individual Measures for QRS and 

Child-Only QRS 

QHPs offered in the Exchange may 

provide family/adult self-only coverage 

or child-only coverage (child-only 

QHPs) and therefore, there are two 

proposed measure sets; the QRS 

measure set (for family and adult selfonly coverage) and a Child-only QRS 

measure set. Both measure sets were 

selected based on the above described 

key criteria. We solicit comments on the 

proposed measures in the QRS and 

Child-only QRS listed below in Table 2. 

The proposed QRS measure set for 

family/adult self-only coverage consists 

of a total of 42 measures—29 clinical 

measures, which encompass health care 

topics of clinical effectiveness, 

prevention, access and efficiency; and 

13 Consumer Assessment of Healthcare 

Providers and Systems® (CAHPS) 

survey measures, which encompass 

topics such as member experiences with 

the QHP, providers and health care 

services, including preventive care. The 

QRS measure set addresses the essential 

health benefits for which health plan 

measures are currently available. The 

majority (76 percent) of the measures 

are presently NQF-endorsed and 

address all six National Quality Strategy 

priorities. Approximately, 83 percent of 

the QRS measures are included in at 

least one of the reviewed Federallyestablished measure sets (for example, 

Office of Personnel Management Federal 

Employee Health Benefit (OPM FEHB), 

CMS Medicare Stars, CMS Adult 

Medicaid Core Set,8 CMS Initial 

Children’s Core Set,9 Medicare Part C&D 

Plan Reporting). The remaining 

measures are used in other state based 

and private sector health plan reporting 

programs such as Consumer Reports 

Health Plan Rankings 10 or through 

accreditation. QHPs offering family or 

adult self-only coverage would be 

required to report on all 42 measures in 

the QRS measure set. 

The Child-only QRS measure set 

consists of a total of 25 measures—15 

clinical measures and 10 CAHPS 

measures. The Child-only measure set 

includes a combination of process and 

outcome measures. The Child-only QRS 

measure set addresses many of the 

essential health benefits. The majority of 

the measures (84 percent) are NQFendorsed and largely address the six 

National Quality Strategy priorities. 

Approximately 80 percent of the 

measures are included in either the 

OPM FEHB Set or the CMS Initial 

Children’s Core Set. As with the QRS 

measure set, the remaining measures in 


TKELLEY on DSK3SPTVN1PROD with NOTICES
69422 Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 

the child-only set are used state based 

and private sector health plan reporting 

programs. Child-only QHPs would be 

required to report on all 25 measures in 

the Child-only QRS measure set. 

TABLE 2—PROPOSED MEASURE SETS FOR THE QRS AND CHILD-ONLY QRS 

Measure title NQF ID 11 QRS Child-only QRS 

Adolescent Well-Care Visits .............................................................. Not currently endorsed ................ X X 

Adult BMI Assessment ...................................................................... Not currently endorsed ................ X ..............................

Adults’ Access to Preventive and Ambulatory Health Services ........ Not currently endorsed ................ X ..............................

Annual Dental Visit ............................................................................ 1388 ............................................ X X 

Annual Monitoring for Patients on Persistent Medications ............... Not currently endorsed ................ X ..............................

Antidepressant Medication Management .......................................... 0105 ............................................ X ..............................

Appropriate Testing for Children With Pharyngitis ............................ 0002 ............................................ X X 

Appropriate Treatment for Children With Upper Respiratory Infection.0069 ............................................ .............................. X 

Avoidance of Antibiotic Treatment in Adults with Acute Bronchitis .. 0058 ............................................ X ..............................

Breast Cancer Screening .................................................................. Not currently endorsed ................ X ..............................

CAHPS—Aspirin Use and Discussion ............................................... Not currently endorsed ................ X ..............................

CAHPS—Coordination of Members’ Health Care Services .............. Not currently endorsed 12 ............ X X 

CAHPS—Cultural Competency ......................................................... Not currently endorsed 13 ............ X X 

CAHPS—Customer Service .............................................................. 0006 ............................................ X X 

CAHPS—Flu Shots for Adults ........................................................... 0039 ............................................ X ..............................

CAHPS—Getting Care Quickly ......................................................... 0006 ............................................ X X 

CAHPS—Getting Needed Care ......................................................... 0006 ............................................ X X 

CAHPS—Global Rating of Health Plan ............................................. 0006 ............................................ X X 

CAHPS—Medical Assistance With Smoking and Tobacco Use 

Cessation.

0027 ............................................ X ..............................

CAHPS—Plan Information on Costs ................................................. 0006 ............................................ X X 

CAHPS—Rating of All Health Care ................................................... 0006 ............................................ X X 

CAHPS—Rating of Personal Doctor ................................................. 0006 ............................................ X X 

CAHPS—Rating of Specialist Seen Most Often ............................... 0006 ............................................ X X 

Cervical Cancer Screening ................................................................ 0032 ............................................ X ..............................

Child and Adolescent Access to PCPs ............................................. Not currently endorsed ................ .............................. X 

Childhood Immunization Status ......................................................... 0038 ............................................ X X 

Chlamydia Screening in Women (Ages 16–20) ................................ 0033 ............................................ .............................. X 

Cholesterol Management for Patients With Cardiovascular Conditions: LDL–C Control (<100 mg/Dl).Not currently endorsed ................ X 

Cholesterol Management for Patients With Cardiovascular Conditions: LDL–C Screening.Not currently endorsed ................ X 

Colorectal Cancer Screening ............................................................. 0034 ............................................ X ..............................

Controlling High Blood Pressure ....................................................... 0018 ............................................ X ..............................

Diabetes Care: Eye Exam (Retinal) Performed ................................ 0055 ............................................ X ..............................

Diabetes Care: Hemoglobin A1c (HbA1c) Control <8.0% ................ 0575 ............................................ X ..............................

Follow-Up After Hospitalization for Mental Illness: 7 days ................ 0576 14 ......................................... X ..............................

Follow-Up Care for Children Prescribed ADHD Medication: Initiation Phase.0108 15 ......................................... X X 

Follow-Up Care for Children Prescribed ADHD Medication: Continuation and Maintenance Phase.0108 ............................................ .............................. X 

HPV Vaccination for Female Adolescents ........................................ 1959 ............................................ .............................. X 

Immunizations for Adolescents .......................................................... 1407 ............................................ X X 

Medication Management for People With Asthma ............................ 1799 ............................................ X ..............................

Medication Management for People With Asthma (Ages 5–18) ....... 1799 ............................................ .............................. X 

Plan All—Cause Readmissions ......................................................... 1768 ............................................ X ..............................

Prenatal and Postpartum Care: Postpartum Care ............................ 1517 ............................................ X ..............................

Prenatal and Postpartum Care: Timeliness of Prenatal Care ........... 1517 ............................................ X ..............................

Relative Resource Use for People with Cardiovascular Conditions—Inpatient Facility Index.1558 ............................................ X ..............................

Relative Resource Use for People with Diabetes—Inpatient Facility 

Index.

1557 ............................................ X ..............................

Use of Imaging Studies for Low Back Pain ...................................... 0052 ............................................ X ..............................

Weight Assessment and Counseling for Nutrition and Physical Activity for Children and Adolescents.0024 ............................................ .............................. X 

Weight Assessment and Counseling for Nutrition and Physical Activity for Children and Adolescents: BMI Percentile Documentation.0024 16 ......................................... X ..............................

Well-Child Visits in the First 15 Months of Life ................................. 1392 ............................................ .............................. X 

Well-Child Visits in the Third, Fourth, Fifth, and Sixth Years of Life 1516 ............................................ X X 


TKELLEY on DSK3SPTVN1PROD with NOTICES
Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 69423 

11 Definitions of NQF endorsed measures can be 

found here: http://www.qualityforum.org/

Home.aspx. 

12Only one question within the CAHPS 

Coordination of Members’ Health Care Services 

composite is currently endorsed (#0007): ‘‘Did your 

personal doctor seem informed and up-to-date 

about the medical care you got?’’. The remaining 

questions in the composite are new and have not 

yet been endorsed. 

13One of the questions within this CAHPS 

composite was modified from CAHPS Clinician and 

Group 2.0, Adult Supplemental (NQF #1904) and 

the other question is new. 

14Measure includes only one indicator of the 

NQF-endorsed measure. 

15Measure includes only one indicator of the 

NQF-endorsed measure for the child-only QRS. 

16Measure includes only one indicator of the 

NQF-endorsed measure. 

C. Organization and Hierarchical 

Structure of the QRS Measures 

The Performance Information 

component of the QRS framework 

guided the proposed structure and 

hierarchy, as well as the measures that 

will be included within each level of the 

hierarchy. In order to be most useful to 

consumers, rating systems that can 

present a large collection of measures 

must be organized into a hierarchical 

structure. We considered organizing the 

measures in a manner to maximize the 

approachability and understandability 

of the information provided by the QRS. 

We are proposing hierarchical structures 

for the QRS and Child-only QRS that 

allow consumers to easily use 

information from the QRS in their 

health plan comparisons for selection of 

a QHP in the Exchange. We solicit 

comments on the proposed hierarchical 

structures outlined in Tables 3 and 4 

below. 

The fundamental building block of 

the QRS structure is the individual 

indicator or measure. The hierarchical 

structures include composites, which 

represent the combination of two or 

more individual indicators or measures 

that result in a single score. Measures 

are grouped into composites so large 

amounts of information can be 

streamlined and reported in formats that 

are easy for consumers to comprehend. 

Grouping measures into composites also 

helps to reduce random variability, 

differentiate performance across health 

plans and provide meaningful 

information to the consumer. Not all 

measures in the QRS are part of a 

composite. Table 3 provides the 

organization of the proposed QRS 

measure set for family/adult self-only 

coverage. The QRS organizes measures 

and composites into a set of eight 

domains that represent unique and 

important aspects of quality: (1) Clinical 

Effectiveness, (2) Patient Safety, (3) Care 

Coordination, (4) Prevention, (5) Access, 

(6) Doctor and Care, (7) Efficiency and 

Affordability (8) Plan Services. The 

domains are grouped into three 

summary indicators which align with 

CMS priority areas: (1) Clinical Quality 

Management; (2) Member Experience; 

and (3) Plan Efficiency, Affordability 

and Management. The summary 

indicators organize the domains into 

broad categories that the consumer may 

use when evaluating health plan 

options. All three summary indicators 

would then be grouped into a single 

Global Rating. The Global Rating is a 

score that summarizes all measures, 

composites and domains in the 

hierarchical structure of the QRS. 

TABLE 3—PROPOSED QRS STRUCTURE

QRS summary indicator QRS domain QRS composite Measure title 

Clinical Quality Management Care Coordination ............. No Composite .................... CAHPS—Coordination of Members’ Health Care Services. 

Clinical Effectiveness ........ No Composite .................... Medication Management for People With Asthma. 

Behavioral Health .............. Antidepressant Medication Management. 

Follow-Up After Hospitalization for Mental Illness: 7 

days. 

Follow-Up Care for Children Prescribed ADHD Medication: Initiation Phase. 

Cardiovascular Care .......... Cholesterol Management for Patients With Cardiovascular Conditions: LDL–C screening. 

Cholesterol Management for Patients With Cardiovascular Conditions: LDL–C control (<100 mg/Dl). 

Controlling High Blood Pressure. 

Diabetes Care ................... Diabetes Care: Eye Exam (Retinal) Performed. 

Diabetes Care: Hemoglobin A1c (HbA1c) Control 

<8.0%. 

Patient Safety .................... No Composite .................... Annual Monitoring for Patients on Persistent Medications. 

Plan All-Cause Readmissions. 

Prevention ......................... Checking for Cancer ......... Breast Cancer Screening. 

Cervical Cancer Screening. 

Colorectal Cancer Screening. 

Maternal Health ................. Prenatal and Postpartum Care: Postpartum Care. 

Prenatal and Postpartum Care: Timeliness of Prenatal 

Care. 

Staying Healthy Adult ........ Adult BMI Assessment. 

CAHPS—Aspirin Use and Discussion. 

CAHPS—Flu Shots for Adults. 

CAHPS—Medical Assistance With Smoking and Tobacco Use Cessation. 

Staying Healthy Child ........ Annual Dental Visit. 

Childhood Immunization Status. 

Immunizations for Adolescents. 

Weight Assessment and Counseling for Children and 

Adolescents: BMI Percentile Documentation. 

Member Experience ............ Access ............................... Access Preventive Visits ... Adolescent Well-Care Visits. 

Adults’ Access to Preventive and Ambulatory Health 

Services. 


TKELLEY on DSK3SPTVN1PROD with NOTICES
69424 Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 

TABLE 3—PROPOSED QRS STRUCTURE—Continued 

QRS summary indicator QRS domain QRS composite Measure title 

Well-Child Visits in the Third, Fourth, Fifth, and Sixth 

Years of Life. 

Access to Care .................. CAHPS—Getting Care Quickly. 

CAHPS—Getting Needed Care. 

Doctor and Care ................ Doctor and Care ................ CAHPS—Cultural Competency. 

CAHPS—Rating of All Health Care. 

CAHPS—Rating of Personal Doctor. 

CAHPS—Rating of Specialist Seen Most Often. 

Plan Efficiency, Affordability 

and Management.

Efficiency and Affordability Efficient Care ..................... Appropriate Testing for Children With Pharyngitis. 

Avoidance of Antibiotic Treatment in Adults with Acute 

Bronchitis. 

Relative Resource Use for People with Cardiovascular 

Conditions—Inpatient Facility Index. 

Relative Resource Use for People with Diabetes—Inpatient Facility Index. 

Use of Imaging Studies for Low Back Pain. 

Plan Service ...................... Member Experience with 

Health Plan.

CAHPS—Customer Service. 

CAHPS—Global Rating of Health Plan. 

CAHPS—Plan Information on Costs. 

The hierarchical structure for the 

proposed Child-only QRS is similar to 

the proposed QRS. The 25 measures of 

the Child-only QRS provide the basic 

foundation of the structure. Not all 

measures in the Child-only QRS are part 

of a composite. Table 4 below provides 

the organization of the proposed Childonly QRS measure set. The Child-only 

QRS organizes measures and composites 

into a set of seven domains: (1) Care 

Coordination, (2) Clinical Effectiveness, 

(3) Prevention, (4) Access, (5) Doctor 

and Care, (6) Efficiency and 

Affordability (7), and Plan Service. The 

domains are grouped into the same 

three summary indicators as the QRS: 

(1) Clinical Quality Management; (2) 

Member Experience; and (3) Plan 

Efficiency, Affordability and 

Management. All three summary 

indicators would then be grouped into 

a single Global Child-only Rating. 

TABLE 4—PROPOSED CHILD-ONLY QRS STRUCTURE

Child-only summary 

indicator Child-only domain Child-only composite Measure title 

Clinical Quality Management Care Coordination ............. No Composite .................... CAHPS—Coordination of Members’ Health Care Services. 

Clinical Effectiveness ........ No Composite .................... Medication Management for People With Asthma 

(Ages 5–18). 

Behavioral Health Child ..... Follow-Up Care for Children Prescribed ADHD Medication: Initiation Phase 

Follow-Up Care for Children Prescribed ADHD Medication: Continuation and Maintenance (C and M) 

Phase. 

Prevention ......................... Staying Healthy Child ........ Annual Dental Visit. 

Childhood Immunization Status. 

Chlamydia Screening in Women (Ages 16–20). 

Immunizations for Adolescents. 

Weight Assessment and Counseling for Children and 

Adolescents. 

HPV Vaccination for Female Adolescents. 

Member Experience ............ Access ............................... Access Preventive Visits 

Child.

Adolescent Well-Care Visits. 

Child and Adolescent Access to PCPs. 

Well-Child Visits in the First 15 Months of Life. 

Well-Child Visits in the Third, Fourth, Fifth, and Sixth 

Years of Life. 

Access to Care .................. CAHPS—Getting Care Quickly. 

CAHPS—Getting Needed Care. 

Doctor and Care ................ Doctor and Care ................ CAHPS—Rating of All Health Care. 

CAHPS—Rating of Personal Doctor. 

CAHPS—Rating of Specialist Seen Most Often. 

CAHPS—Cultural Competency. 

Plan Efficiency, Affordability 

and Management.

Efficiency and Affordability Efficient Care Child ........... Appropriate Testing for Children With Pharyngitis. 

Appropriate Treatment for Children With Upper Respiratory Infection. 

Plan Service ...................... Member Experience with 

Health Plan.

CAHPS—Customer Service. 

CAHPS—Global Rating of Health Plan. 


TKELLEY on DSK3SPTVN1PROD with NOTICES
Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 69425 

TABLE 4—PROPOSED CHILD-ONLY QRS STRUCTURE—Continued 

Child-only summary 

indicator Child-only domain Child-only composite Measure title 

CAHPS—Plan Information on Costs. 

III. QRS Rating Methodology 

Component 

Once the QRS measures are organized 

and the hierarchical structure is 

established, the QRS rating 

methodology would combine health 

plan measure scores into performance 

ratings using a set of rules and formulae. 

We solicit comments on the proposed 

six elements of the Rating Methodology 

component that will guide the 

calculation of the ratings (refer to 

Section I for the definitions of the 

elements of the Rating Methodology 

component). The six elements of the 

proposed Rating Methodology are 

grouped within three broad categories 

(Measure Scoring Rules, Aggregation 

Rules, and Reference Standards). See 

Table 5. 

TABLE 5—RATING METHODOLOGY CATEGORIES OF ELEMENTS

Category Rating category elements 

Measure scoring rules .................................................................................................................................. Sampling and Attribution. 

Scoring. 

Aggregation Rules ........................................................................................................................................ Aggregation Rules. 

Reference Standards .................................................................................................................................... Performance Classification values. 

Population and Other Adjustments. 

Peer Groups. 

Measure Scoring Rules will 

standardize the individual measure 

scores so that scores are on the same 

scale (for example, all percentiles) and 

can be combined meaningfully. 

Aggregation Rules will be used to 

combine measures to create quality 

constructs, such as diabetes care or 

preventive health. Reference Standards 

will determine how scores are converted 

to categorical ratings (for example, star 

groups on a scale of one to five) that can 

be easily understood, compared, and 

used by consumers. We intend to 

publish, for review and comment, 

technical guidance that identifies 

further details regarding the Rating 

Methodology component, elements and 

measure specifications. 

IV. QRS Data Strategy 

The QRS data strategy refers to how 

QRS data are collected, calculated, and 

submitted and will help to inform how 

data is displayed. We intend to develop 

a data strategy that would facilitate 

consistent data collection and 

calculation across QHPs; and help to 

ensure the integrity and accuracy of 

QRS ratings. We solicit comments on 

potential ways to enhance the QRS data 

strategy for QHP issuers. We intend to 

direct QHP issuers to submit validated 

data to ensure that QRS data displayed 

for public reporting are accurate, valid 

and comparable, and to allow 

consumers objective and meaningful 

comparisons of the QHPs’ quality data. 

We believe that the ratings assigned 

must reflect true differences in quality. 

We intend to display Global Ratings 

using a five-star scale. While it is our 

intention for all QHPs in Exchanges to 

have publicly available ratings, some 

QHPs may have missing data due to 

data quality issues or low enrollment in 

the initial years. 

We plan to use a full-scale rule at the 

global and summary indicator levels, so 

that these scores are true representations 

of what they are intended to represent. 

This method allows the consumer to 

compare Global Ratings with the 

important concepts at highest levels of 

the hierarchy represented (refer to Table 

3 for proposed QRS structure). 

Therefore, we are considering that, for 

QHPs that are missing any of the 

domain ratings used for creating the 

Member Experience or Plan Efficiency, 

Cost Reduction and Management 

summary indicators would not have an 

associated summary indicator rating 

publically displayed. For the Clinical 

Quality Management indicator, QHPs 

must have the Care Coordination, 

Clinical Effectiveness, and Prevention 

domains present to have the summary 

indicator rating publically displayed. 

We have conducted preliminary testing 

that demonstrates that a Clinical Quality 

summary indicator can be reported as 

long as Care Coordination, Clinical 

Effectiveness, and Prevention domains 

are present even if the Patient Safety 

domain is not reportable because this 

domain did not impact QHP 

comparability. We believe that Patient 

Safety is important to measure and it is 

a CMS priority. We plan to further 

develop this domain of the QRS as more 

health-plan patient safety measures 

become available. We are also proposing 

that a Global Rating will be displayed 

only when all three summary indicator 

ratings are available. For the lower 

levels of the hierarchy, the half-scale 

rule would be applied, meaning that at 

a minimum, half of the components of 

the domain or composite must be 

present for the rating to be displayed. 

Thus, if a domain is composed of three 

composites, two would have to be 

present for it to be displayed or if a 

composite is composed of two measures 

at least one would have to be present for 

it to be displayed. Specifically, we 

solicit comment to inform future 

technical guidance regarding the fullscale and half-scale rules described as 

well as any additional ways to address 

data quality issues or potential low 

enrollment in QHPs in the initial years. 

V. Future Considerations 

We solicit comments to inform future 

technical guidance on priority areas for 

additional measure enhancements and 

development of the QRS. We intend to 

continually monitor the QRS and make 

necessary adjustments to ensure that the 

methodology and measures remain 

consistent with the intended goals and 

principles of the QRS. As advancements 

in health plan quality measurement and 

reporting are made, we will consider 

ways in which the QRS may evolve 

(such as the potential selection of 

measures that are reportable through 

disease registries or all-payer claims 

databases). In addition, we will consider 

potential factors for the retirement of 

measures. 


TKELLEY on DSK3SPTVN1PROD with NOTICES
69426 Federal Register / Vol. 78, No. 223 / Tuesday, November 19, 2013 / Notices 

As the Exchanges mature and 

enrollment in QHPs expands, we will 

consider reporting the QRS at more 

granular levels (that is, QHP metal 

levels as defined in section 1302(d)(1) of 

the Affordable Care Act). We will also 

consider the development of a quality 

rating system applicable to other 

Exchange offerings, such as stand-alone 

dental plans, catastrophic plans and 

health care saving accounts. 

VI. Collection of Information 

Requirements 

This document does not impose 

information collection and 

recordkeeping requirements. However, 

it does make reference to an information 

collection activity. The aforementioned 

Enrollee Satisfaction Survey is currently 

seeking OMB approval via notice and 

comment periods separate from this 

proposed notice. The 60-day Federal 

Register notice published on June 28, 

2013. Additionally, in future 

rulemaking, we will identify 

information collection requirements 

associated with the QRS and solicit 

public comment at that time. 

Dated: November 6, 2013. 

Marilyn Tavenner, 

Administrator, Centers for Medicare & 

Medicaid Services. 

[FR Doc. 2013–27649 Filed 11–14–13; 4:15 pm] 

BILLING CODE 4120–01–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

National Institutes of Health 

Submission for OMB Review; 30-Day 

Comment Request: NIH NCI Central 

Institutional Review Board (CIRB) 

Initiative (NCI) 

SUMMARY: Under the provisions of 

Section 3507(a)(1)(D) of the Paperwork 

Reduction Act of 1995, the National 

Institutes of Health (NIH), has submitted 

to the Office of Management and Budget 

(OMB) a request for review and 

approval of the information collection 

listed below. This proposed information 

collection was previously published in 

the Federal Register on August 22, 

2013, Vol. 78, P. 52204 and allowed 60- 

days for public comment. There were no 

public comments received. The purpose 

of this notice is to allow an additional 

30 days for public comment. The 

National Cancer Institute (NCI), 

National Institutes of Health, may not 

conduct or sponsor, and the respondent 

is not required to respond to, an 

information collection that has been 

extended, revised, or implemented on or 

after October 1, 1995, unless it displays 

a currently valid OMB control number. 

Direct Comments to OMB: Written 

comments and/or suggestions regarding 

the item(s) contained in this notice, 

especially regarding the estimated 

public burden and associated response 

time, should be directed to the: Office 

of Management and Budget, Office of 

Regulatory Affairs, OIRA_submission@

omb.eop.gov or by fax to 202–395–6974, 

Attention: NIH Desk Officer. 

Comment Due Date: Comments 

regarding this information collection are 

best assured of having their full effect if 

received within 30-days of the date of 

this publication. 

FOR FURTHER INFORMATION: To obtain a 

copy of the data collection plans and 

instruments or request more information 

on the proposed project contact: CAPT 

Michael Montello, Pharm. D., MBA, 

Cancer Therapy Evaluation Program, 

Operations and Informatics Branch, 

9609 Medical Center Drive, Rockville, 

MD 20850 or call non-toll-free number 

240–276–6080 or Email your request, 

including your address to: 

mike.montello@nih.gov. Formal requests 

for additional plans and instruments 

must be requested in writing. 

Proposed Collection: NIH NCI Central 

Institutional Review Board (CIRB) 

Initiative (NCI), 0925–0625, Expiration 

Date 1/31/2014, Revision, National 

Cancer Institute (NCI), National 

Institutes of Health (NIH). 

Need and Use of Information 

Collection: The National Cancer 

Institute (NCI) Central Institutional 

Review Board (CIRB) provides a 

centralized approach to human subject 

protection and provides a cost efficient 

approach avoiding duplication of effort 

at each institution. The CIRB provides 

the services of a fully constituted IRB 

and provides a comprehensive and 

efficient mechanism to meet regulatory 

requirements pertaining to human 

subject protections including: initial 

reviews, continuing reviews, review of 

amendments, and adverse events. The 

Initiative consists of three central IRBs: 

Adult CIRB—late phase emphasis, 

Adult CIRB—early phase emphasis, and 

Pediatric CIRB. CIRB membership 

includes oncology physicians, surgeons, 

nurses, patient advocates, ethicists, 

statisticians, pharmacists, attorneys and 

other health professionals. The benefits 

of the CIRB Initiative reaches research 

participants, investigators and research 

staff, Institutional Review Boards (IRB), 

and Institutions. Benefits include: study 

participants having dedicated review of 

NCI-sponsored trials for participant 

protections, access to more trials more 

quickly and access to trials for rare 

diseases, accrual to trials begin more 

rapidly, ease of opening trials, 

elimination of need to submit study 

materials to local IRBs, and elimination 

of the need for a full board review. The 

benefits to the National Clinical Trials 

Network and Experimental TherapyClinical Trials Network include a cost 

efficient approach that avoids 

duplication of efforts at each institution. 

A variety of information collection tools 

are needed to support NCI’s CIRB 

activities which include: worksheets, 

forms and a survey that is provided to 

all customers contacting the CIRB 

helpdesk. 

OMB approval is requested for 3 

years. There are no costs to respondents 

other than their time. The total 

estimated annualized burden hours are 

2,199. 

ESTIMATES OF ANNUAL BURDEN HOURS

Form name Type of respondents Number of 

respondents 

Frequency of 

responses per 

respondent 

Average 

burden per 

response 

(in hours) 

Total annual 

burden hours 

CIRB Customer Satisfaction Survey ........................ Participants/Board 

Members.

1,500 1 10/60 250 

Request for 30 Day Website Access Form ............. Participants ................ 25 1 10/60 4 

Authorization Agreement and Division of Responsibilities between the NCI CIRB and Signatory 

Institution.

Participants ................ 340 1 30/60 170 

NCI CIRB Signatory Enrollment Form ..................... Participants ................ 40 1 4 160 

IRB Staff at Signatory Institution’s IRB .................... Participants ................ 25 1 10/60 4 

Investigator at Signatory Institution ......................... Participants ................ 65 1 10/60 11 

Research Staff at Signatory Institution .................... Participants ................ 65 1 10/60 11 


TKELLEY on DSK3SPTVN1PROD with NOTICES