NURSING HOME 

QUALITY

CMS Should Improve 

Efforts to Monitor 

Implementation of the 

Quality Indicator 

Survey

Report to Congressional Requesters

February 2012

GAO-12-214

United States Government Accountability Office

GAO

This report was revised on March 9, 2012, to include omitted 

page 2 of the agency comment letter from the Department of 

Health and Human Services in appendix I, page 23.
United States Government Accountability Office

Highlights of GAO-12-214, a report to 

congressional requesters

February 2012

NURSING HOME QUALITY

CMS Should Improve Efforts to Monitor 

Implementation of the Quality Indicator Survey

Why GAO Did This Study

The Centers for Medicare & Medicaid 

Services (CMS) is responsible for 

ensuring that nursing homes receiving 

federal funding meet federal quality 

standards. CMS contracts with state 

survey agencies to conduct periodic 

inspections of nursing homes using a 

CMS-developed survey process. 

Weaknesses in the survey process led 

CMS to develop and begin 

implementing the Quality Indicator 

Survey (QIS), a new electronic survey. 

CMS developed the QIS to achieve 

several objectives, including improving 

the efficiency, accuracy, and 

consistency of the survey process. 

CMS expects the QIS to be 

implemented nationally by 2018. We 

examined the extent to which CMS 

(1) evaluates whether progress is 

being made in meeting the objectives 

of the QIS and (2) monitors and 

facilitates states’ implementation of the 

QIS. To do this, GAO reviewed 

relevant CMS documents, including

CMS-commissioned studies of the 

QIS, policies, and guidance. GAO also 

interviewed officials from CMS’s 

Central and all 10 Regional Offices as 

well as officials from a judgmental 

sample of nine state survey agencies.

What GAO Recommends

GAO recommends that CMS develop a 

means to routinely monitor the extent 

to which progress is being made in 

meeting the objectives of the QIS and 

systematic methods for monitoring and 

facilitating states’ efforts to implement 

the QIS. The Department of Health and 

Human Services fully concurred with 

all of the recommendations.

What GAO Found

CMS has commissioned three studies to evaluate the QIS-based survey process 

but does not routinely monitor the extent to which the objectives established for 

the QIS are being met. The first two studies, completed in 2006 and 2007, were 

commissioned to determine whether the new survey process could be used in 

real-world settings and whether the QIS was helping the agency meet several of 

the objectives established for the process. The studies’ findings suggested that 

surveyors could use the survey process and recommended changes intended to 

help improve the QIS. CMS officials reported taking steps to address the studies’ 

findings and recommendations. In 2009, CMS commissioned a third study that 

was completed in 2011 and identified aspects of the QIS process that could 

affect the consistency with which surveyors identify quality problems. For 

example, the study found that during resident interviews, surveyors did not 

consistently probe for further information when provided with incomplete 

responses to interview questions. However, CMS does not have the means to 

routinely monitor the extent to which the QIS is helping improve the survey 

process as intended. Such routine, ongoing monitoring would be consistent with 

federal internal control standards and could include the use of performance goals 

and measures. CMS does have access to some data, such as the amount of 

time surveyors have spent inspecting facilities, that could be used to help 

develop performance goals and measures.

CMS has taken some steps to monitor and facilitate states’ implementation of the 

QIS-based routine survey, but CMS’s efforts are not systematic. As part of the 

agency’s efforts to monitor states’ implementation, CMS primarily uses quarterly 

teleconferences with state survey agency officials to obtain information on the 

extent to which each state has completed training all its surveyors to use the 

QIS. However, states may not always participate in the teleconferences, and 

those that do may not provide complete information on their progress. As a 

result, the information CMS obtains through its monitoring of states’ progress 

may be incomplete. For example, CMS was unable to provide GAO with 

information on training completion dates of all state surveyors in states that 

completed training. Although CMS reported it plans to develop a more systematic 

process for monitoring states’ implementation progress, it has not established a 

time line for doing this. To help facilitate states’ implementation of the QIS, CMS 

provides states with guidance, gives presentations, and offers states 

opportunities to share their implementation experiences through quarterly 

teleconferences. However, CMS does not have a systematic method for 

obtaining, compiling and sharing information on state experiences, especially 

information on approaches states have taken to help facilitate implementation of 

the QIS. Systematically sharing such information—for example, through CMS’s 

annual conference in which all state survey agencies participate—could help the 

agency facilitate implementation in states that have not begun QIS 

implementation.

View GAO-12-214. For more information, 

contact Linda Kohn at (202) 512-7114 or

kohnl@gao.gov.
Page i GAO-12-214 Nursing Home Quality Assessment

Letter 1 

Background 4

CMS Commissioned Studies to Evaluate the QIS, but Does Not 

Routinely Monitor Progress toward Meeting the Objectives of 

the QIS 6

CMS Has Taken Steps to Monitor and Facilitate States’ 

Implementation of the QIS, but CMS’s Efforts Are Not 

Systematic 12

Conclusions 17

Recommendations for Executive Action 18

Agency Comments and Our Evaluation 18

Appendix I Comments from the Department of Health and Human Services 21

Abbreviations

CMS Centers for Medicare & Medicaid Services 

HHS Department of Health and Human Services

QIS Quality Indicator Survey

RTI Research Triangle Institute International

SSA Social Security Act

Contents

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-12-214 Nursing Home Quality Assessment

United States Government Accountability Office

Washington, DC 20548

February 1, 2012

The Honorable Herb Kohl

Chairman

Special Committee on Aging

United States Senate

The Honorable Charles E. Grassley

Ranking Member

Committee on the Judiciary

United States Senate

Nursing homes in the United States provide skilled nursing, rehabilitation, 

and custodial care to approximately 1.5 million elderly and disabled 

individuals. Federal and state governments share responsibility for 

ensuring that the approximately 15,700 nursing homes in the nation 

receiving payment from Medicare and/or Medicaid programs provide 

quality care in a safe environment for their highly vulnerable residents. 

Specifically, the Centers for Medicare & Medicaid Services (CMS), an 

agency within the Department of Health and Human Services (HHS), is 

responsible for ensuring that nursing homes receiving payment from 

Medicare and Medicaid programs meet federal quality standards 

established pursuant to the Omnibus Budget Reconciliation Act of 1987.1

To meet this responsibility, CMS, among other activities, contracts with 

state survey agencies to conduct on-site nursing home inspections, which 

consist of routine surveys and complaint investigations, using a process 

developed by CMS.2

In 1998, we found weaknesses in the traditional process used to conduct 

routine surveys, including poor measurement of serious care problems in 

 

1

Pub. L. No. 100-203, Title IV, Subtitle C, §§ 4201 et seq., 101 Stat. 1330, 1330-160, et 

seq. (amending Titles XVIII and XIX of the Social Security Act). According to CMS, 

combined Medicare and Medicaid payments for nursing home care in 2010 were about 

$91 billion.

2

Under contract with CMS, state survey agencies conduct inspections of health care 

facilities—such as nursing homes, home health agencies, intermediate care facilities for 

the mentally retarded, and hospitals—that participate in Medicare and Medicaid. 

 
 

Page 2 GAO-12-214 Nursing Home Quality Assessment

nursing homes.3

CMS began implementation of the QIS-based routine survey process in 

2005, by launching demonstrations of the QIS in five states. To 

implement the QIS, CMS relies on the state survey agencies, as these 

agencies are largely responsible for the administration of nursing home 

inspections.

For example, we reported that the resident sample used 

in the traditional process was inadequate and limited a surveyors’ ability 

to identify care problems and their prevalence within a facility. To address 

the problems we reported, and in response to criticism from others, CMS 

revised the routine survey process by developing the Quality Indicator 

Survey (QIS). According to CMS, the QIS was developed to help the 

agency meet several objectives, including improving surveyors’ 

documentation of quality concerns; improving the efficiency of the survey 

process by focusing resources on facilities, and on areas within facilities, 

with the greatest quality concerns; and improving the accuracy and 

consistency with which surveyors identify deficiencies.

4 To help assist state survey agencies in implementing the 

QIS, CMS provides policies, guidance and some training for state 

surveyors. CMS officials told us that as of September 2011, 26 states had 

trained or started training surveyors to use the QIS and 27 states had not 

yet started the training. CMS expects to complete implementation of the 

QIS by 2018 in all 50 states, the District of Columbia, Puerto Rico, and 

the U.S. Virgin Islands.5

You have raised questions about CMS’s efforts to implement the QIS. We 

previously provided you with information on the status of the agency’s 

efforts to implement the QIS nationally.6

 

3

See GAO, California Nursing Homes: Care Problems Persist Despite Federal and State 

Oversight, 

This report focuses on certain 

aspects of CMS’s implementation of the QIS and will examine the extent 

to which CMS (1) evaluates whether progress is being made in meeting 

GAO/HEHS-98-202 (Washington, D.C.: July 27, 1998).

4

State survey agencies administer and have discretion over many survey activities and 

policies, including hiring and retaining a surveyor workforce, training surveyors, and 

conducting supervisory reviews of surveys. For example, state survey agencies are 

largely responsible for training all surveyors in their states to use the QIS; in 2011, there 

were roughly 4,700 surveyors nationwide. 

5

In this report, we use the term “states” to refer to this group of states and territories. 

6

See GAO, Nursing Home Quality: Implementation of the Quality Indicator Survey, 

GAO-11-403R (Washington, D.C.: Apr. 6, 2011).
 

Page 3 GAO-12-214 Nursing Home Quality Assessment

the objectives of the QIS and (2) monitors and facilitates states’ 

implementation of the QIS.

To examine the steps CMS has taken to evaluate the QIS, we reviewed 

relevant CMS documents, including the agency’s plan for evaluation and 

improvement of the QIS and independent studies of the QIS 

commissioned by the agency. We also interviewed CMS officials from 

CMS’s Central Office. We asked these officials for information on CMS’s 

efforts to evaluate the QIS, including any efforts to routinely monitor the 

QIS, a practice consistent with federal internal control standards.7

To examine the extent to which CMS monitors and facilitates states’ 

implementation of the QIS, we reviewed CMS policies on the QIS as well 

as guidance the agency provided to state survey agencies that described 

their respective roles in implementing the QIS. We reviewed CMS’s plan 

for implementing the QIS nationally. We also interviewed officials from 

CMS’s Central Office and all 10 Regional Offices to learn more about the 

agency’s efforts to monitor and facilitate states’ implementation of the 

QIS.

Additionally, we interviewed the CMS contractors—Abt Associates Inc. 

and Research Triangle Institute International (RTI)—that conducted the 

independent studies of the QIS.

8 We also analyzed this information to determine whether CMS’s 

monitoring and facilitation efforts are consistent with federal internal 

control standards.9

 

7

See GAO, Standards for Internal Control in the Federal Government, 

Additionally, we conducted structured interviews with 

state survey agency officials from a judgmental sample of nine states 

(Arizona, Colorado, Connecticut, Florida, Maryland, Minnesota, New 

York, Ohio, and West Virginia), to obtain information about these states’ 

experience with implementing the QIS, including the extent to which CMS 

monitored their progress with implementation and helped to facilitate 

implementation in their state. We selected our sample from the 21 states 

that had begun implementation as of February 2011. These states were 

selected based on criteria such as variation in the states’ QIS 

implementation status, the percentage of nursing home inspections 

conducted using the QIS-based process within a state, and state size. 

GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). 

8

CMS’s Regional Offices are located in Atlanta, Boston, Chicago, Dallas, Denver, Kansas 

City, New York, Philadelphia, Seattle, and San Francisco. 

9

See GAO/AIMD-00-21.3.1.
 

Page 4 GAO-12-214 Nursing Home Quality Assessment

The findings from our structured interviews are limited to the nine states 

in our sample and are not representative; therefore, we will not be able to 

generalize these findings to all states.

We conducted this performance audit from June 2011 through December 

2011 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

Oversight of nursing homes is a shared federal-state responsibility. Based 

on statutory requirements, CMS defines standards that nursing homes 

must meet to participate in the Medicare and Medicaid programs and 

contracts with state survey agencies to assess whether homes meet 

these standards. A range of statutorily defined sanctions is available to 

CMS and the states to help ensure that homes maintain compliance with 

federal quality requirements. CMS also is responsible for monitoring the 

adequacy of state survey activities.

Every nursing home that receives payment from Medicare and/or 

Medicaid must undergo a nursing home inspection. These inspections 

consist of routine surveys that are conducted on average every 

12 months and complaint investigations that are conducted in response to 

allegations of quality problems.10

 

10Every nursing home is required to receive an inspection not less than once every 15 

months and the statewide average interval for these surveys must not exceed 12 months. 

42 C.F.R. § 488.308 (2010).

During a routine survey, teams of state 

surveyors—generally consisting of registered nurses, social workers, 

dieticians, or other specialists—evaluate a nursing home’s compliance 

with federal nursing home quality standards. There are approximately 200 

different standards that focus on the delivery of care, resident outcomes 

and facility conditions. The standards are grouped into 15 categories, 

such as Quality of Life, Resident Assessment, Quality of Care, and 

Administration. For example, there are 23 standards within the Quality of 

Care category ranging from the prevention of pressure sore development 

to keeping the resident environment as free of accident hazards as is 

Background 

Nursing Home Inspections
 

Page 5 GAO-12-214 Nursing Home Quality Assessment

possible. During nursing home inspections, state surveyors may cite 

deficiencies—areas in which facilities fail to meet federal quality 

standards—and nursing homes must prepare a correction plan to address 

most deficiencies.11

To ensure the reliability of state survey agencies’ inspections of nursing 

homes, federal survey teams composed of CMS staff from its 10 Regional 

offices conduct what are known as federal monitoring surveys, which 

include comparative and observational surveys. In a comparative survey, 

a federal survey team conducts an independent inspection of a home 

recently surveyed by a state survey agency in order to compare and 

contrast its findings with those of the state survey team. In an 

observational survey, federal surveyors accompany a state survey team 

to a nursing home to evaluate the team’s on-site survey performance and 

ability to document deficiencies. The federal monitoring surveys are to be 

conducted annually in at least 5 percent of state-surveyed Medicare and 

Medicaid nursing homes in each state.

State surveyors may also conduct what are known as 

revisit surveys to ensure that the homes corrected or took steps to correct 

identified deficiencies.

12 CMS has developed the QISbased tool for routine surveys and plans to develop QIS-based tools for 

all other types of nursing home inspections: revisit surveys, complaint 

investigations and federal monitoring surveys.13

The QIS process is similar to the traditional survey method.14

 

11For serious deficiencies, CMS generally imposes sanctions, which may result in the 

termination of a nursing home’s participation in the Medicare and Medicaid programs.

For 

example, both the QIS and the traditional survey process involve 

analyzing information available about the nursing home facilities and their 

12Under the Social Security Act (SSA), HHS is required to conduct a certain number of 

federal validation surveys, also referred to as federal monitoring surveys, each year. In no 

case is HHS to survey less than a total of five nursing homes in a state per year. SSA 

§1819(g)(3)(B)(Medicare) (codified at 42 U.S.C. §1395i-3(g)(3)(B)); SSA § 

1919(g)(3)(B)(Medicaid) (codified at 42 U.S.C. § 1396r(g)(3)(B)).

13Revisit surveys ensure that nursing homes remediate certain deficiencies. A complaint 

investigation generally focuses on a specific allegation regarding a resident’s care or 

safety. Federal monitoring surveys are done to ensure the reliability of state survey 

agencies’ routine surveys.

14See GAO-11-403R for more details on similarities and differences between the QIS and 

the traditional survey processes. 

QIS Process
 

Page 6 GAO-12-214 Nursing Home Quality Assessment

residents before entering the nursing home in order to identify areas of 

care that have been historically problematic and residents who may be at 

higher risk for poor care; interviewing residents, family members, and 

staff; observing staff and nursing home practices; reviewing medical 

records; and citing deficiencies when necessary. However, the QIS differs 

from the traditional survey in two key areas:

• The QIS provides an electronic means of conducting nursing home 

inspections. The collection, recording and analysis of information and 

documentation of findings are done electronically using a tablet 

computer; the traditional method is paper-based.

• The QIS software draws an expanded random sample of up to 

70 residents for surveyors to interview, observe, and review their 

medical records during the inspection. In contrast, the traditional 

survey method allows surveyors to review various reports, data sets, 

and guidance and use their judgment to select a sample of residents, 

consisting of about 20 percent of the current resident census.

CMS has commissioned three studies to evaluate the QIS, an electronic 

tool that CMS developed to assess the quality of nursing home care, and 

has taken some steps to improve the QIS in response to the findings of 

the studies. However, the agency does not routinely monitor the extent to 

which progress is being made in meeting the objectives established for 

the QIS.15

The first CMS-commissioned study was completed by Abt Associates Inc. 

in 2006 and tested the QIS under real-world conditions. This study 

analyzed survey data available from three states in which CMS had 

begun to conduct demonstrations of the QIS: Connecticut, Kansas, and 

 

15According to CMS, the objectives of the QIS include improving surveyors’ 

documentation of deficiencies; improving the efficiency of the routine survey process by 

focusing survey resources on facilities and areas within facilities where the greatest quality 

concerns exist; and improving the accuracy and consistency with which surveyors identify 

deficiencies.

CMS Commissioned 

Studies to Evaluate 

the QIS, but Does Not 

Routinely Monitor 

Progress toward 

Meeting the 

Objectives of the QIS

CMS Has Commissioned 

Studies to Evaluate 

Various Aspects of the QIS
 

Page 7 GAO-12-214 Nursing Home Quality Assessment

Ohio. The study was intended to examine surveyors’ experiences using 

the QIS to conduct actual nursing home inspections, and identify ways to 

improve the QIS. In order to meet this general purpose, the study 

assessed several areas, including whether surveyors were able to use an 

electronic tool to conduct routine surveys in a real-world setting, whether 

surveyors could complete the routine survey using the QIS in the same 

amount of time that surveyors needed to complete routine surveys using 

the traditional methodology, and whether any changes were needed to 

improve the QIS tool. Among other results, the study found that state 

surveyors were able to use the QIS-based process to conduct routine 

surveys and generally perceived the process to be an improvement over 

the traditional survey method. However, the study also found that 

surveyors generally needed more time to complete routine surveys using 

the QIS tool rather than the traditional survey method.16

In 2007, Abt Associates Inc. completed the second CMS-commissioned 

study that evaluated the use of the QIS-based routine survey process in 

the field. This study included survey data from all five demonstration 

states (California, Connecticut, Kansas, Louisiana, and Ohio) and was 

intended to evaluate, among other things, whether the QIS achieved 

certain of the agency’s objectives for the process.

The study cited 

several potential reasons for this time difference, ranging from problems 

with the QIS software and hardware to forms within the QIS survey 

process that could be made less time-consuming. These findings led the 

researchers to recommend ways to streamline and otherwise improve the 

QIS tool in order to reduce the amount of time surveyors needed to 

complete the QIS-based routine survey. CMS revised the QIS survey tool 

and process based on many of these recommendations. These revisions 

included providing more guidance on certain aspects of the QIS process 

during QIS training; eliminating certain duplicative requirements for 

collecting information; and altering various survey questions, for example, 

by adjusting the wording of survey questions to make them more 

understandable to nursing home residents and reducing the time required 

for surveyors to explain them.

17

 

16According to the report, the researchers were unable to draw firm conclusions regarding 

this comparison due to data limitations such as the limited number of QIS surveys 

conducted at the time.

These objectives 

17Although California participated in the five-state study, CMS allowed the state to 

postpone further implementation of QIS due to budgetary concerns within the state. 
 

Page 8 GAO-12-214 Nursing Home Quality Assessment

included providing a routine survey tool that improves surveyor 

accuracy—that is, helps surveyors better identify deficiencies; improves 

surveyor efficiency—that is, helps surveyors focus their time on facilities 

with the greatest number of quality concerns; and improves surveyors’ 

documentation of deficiencies. In terms of whether the QIS led to 

improved accuracy or improved documentation of deficiencies, the report 

found that the QIS-based routine survey and the traditional survey 

approach generally led to comparable results. The researchers, therefore, 

concluded that the QIS did not significantly enhance or diminish 

surveyors’ ability to accurately identify or document deficiencies. 

Similarly, the study suggested that in general, surveyors using the QIS 

tool completed routine surveys as efficiently, but not more efficiently, as 

surveyors using the traditional survey methodology.18 The study made 

several recommendations to CMS for improving the QIS process, 

including that CMS further evaluate how well the QIS protocols use 

preliminary information gathered during the survey to identify problem 

areas in the facility for more in-depth investigation. Further, the study 

recommended that CMS clarify the agency’s investigative protocols, 

which CMS publishes and makes available within the QIS software to 

help surveyors assess nursing home compliance with federal quality 

standards.19 Following completion of this study, CMS decided to 

implement the QIS nationally and CMS officials told us they worked to 

improve aspects of the QIS process by taking steps to address the 

study’s findings and recommendations.20

In 2009, CMS commissioned a third study to evaluate the QIS-based 

routine survey process and make recommendations for improvement. 

This study, conducted by RTI and completed in 2011, was intended to 

 

18The researchers noted that the study’s findings were limited by factors such as small 

sample size and an inability to control for certain external factors, such as surveyor skills, 

that may have affected results of the study. 

19These protocols, called critical element pathways, assist surveyors in completing a 

systematic review of problematic areas in facilities that are selected for in-depth 

investigation during the survey. For example, the critical element pathway for the federal 

standard regarding the prevention and treatment of pressure ulcers in nursing facilities 

provides a checklist of items for the surveyor to assess as well as questions, such as 

whether the facility comprehensively assessed the resident’s condition, for the surveyor to 

answer during this portion of the survey. 

20According to CMS officials, the agency considered the results of the second study, 

among other factors, when making the decision to continue implementation of the QIS on 

a national scale. 
 

Page 9 GAO-12-214 Nursing Home Quality Assessment

identify aspects of the QIS process that could affect how consistently 

surveyors identify quality problems and to suggest ways to improve these 

aspects of the process. This study included survey data from 19 states 

that had begun to implement the QIS.21 The study found that various 

aspects of the QIS process, including the way resident interviews were 

conducted and the surveyor initiation process (in which surveyors identify 

additional problem areas, not identified by the QIS software, for further 

investigation during surveys), might affect the consistency with which 

surveyors identify quality problems during nursing home inspections.22

For example, the study found that surveyors did not consistently probe for 

further information when residents gave incomplete answers to interview 

questions, and that some surveyors were likely to initiate problem areas 

during surveys on their own, while others relied mainly on the QIS 

software to identify areas for investigation. The study also identified 

potential methods for addressing these issues and improving the 

consistency of the QIS process, including providing additional guidance to 

surveyors on interviewing techniques and developing additional 

standardized training for new surveyors to ensure they are familiar with 

federal regulations. CMS officials told us that they intend to consider the 

recommendations made in this study in their efforts to continue improving 

the QIS.

Although CMS has commissioned past studies to evaluate the QIS, the 

agency does not monitor the QIS on an ongoing basis. Such ongoing 

monitoring of the QIS could allow CMS to collect data to routinely 

evaluate the extent to which progress is being made in meeting all of the 

objectives established for the QIS to improve the survey process as 

intended. Monitoring, a key component of federal internal control 

standards, helps provide management with reasonable assurance that an 

agency or its programs and initiatives are achieving effectiveness and 

efficiency of operations and compliance with applicable laws and 

 

21The study included both qualitative and quantitative analyses. Some analyses in the 

study used data from fewer than 19 states. 

22According to CMS, the QIS software uses preliminary survey data to select problematic 

areas in facilities for further investigation during surveys. However, surveyors have the 

option to initiate the selection of areas for investigation outside of those selected by the 

QIS software. Surveyors may rely on their professional experience and judgment to make 

these selections.

CMS Has Not Established 

the Means to Routinely 

Monitor Progress toward 

Meeting the Objectives of 

the QIS, Such as 

Developing Performance 

Goals and Measures
 

Page 10 GAO-12-214 Nursing Home Quality Assessment

regulations, among other factors.23 One way agencies can monitor 

program operations on an ongoing basis to ensure that the programs are 

meeting their objectives is through the use of performance goals and 

measures. Performance measurement involves identifying performance 

goals and measures, establishing performance baselines by tracking 

performance over time, identifying targets for improving performance, and 

measuring progress against those targets.24

CMS currently collects some data that would allow it to begin monitoring 

aspects of the performance of the QIS. For example, the agency currently 

collects information from routine surveys—such as data that describes 

the amount of time surveyors have spent inspecting facilities, as well as 

the deficiencies that surveyors cited during each inspection—that could 

be used to help measure the efficiency of the QIS process.

In the case of the QIS-based 

routine survey process, information collected through performance goals 

and measures could help CMS routinely monitor the extent to which the 

objectives of the QIS are being achieved. This information could also 

inform future efforts by CMS to modify and improve the QIS process as 

needed. CMS officials acknowledged the need for certain performance 

measures—specifically with regard to examining the effect of the QIS on 

surveyor consistency–but noted that they did not have performance goals 

or measures in place as of December 2011. This lack of performance 

goals and measures can affect CMS’s ability to effectively monitor the 

QIS.

25

 

23The five components of internal control are Control Environment, Risk Assessment, 

Control Activities, Information and Communications, and Monitoring. See also GAO, 

Standards for Internal Control in the Federal Government,

CMS could 

use these data to conduct analyses of surveyor efficiency, similar to the 

analysis conducted by Abt Associates Inc., and derive efficiency scores 

GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999). 

24See Pub. L. No. 103-62, § 4, 107 Stat. 285, 287 (generally codified, as amended, to 

31 U.S.C. §1115-1117).

25More specifically, CMS data that could be used to measure efficiency include the 

number, scope, and severity of deficiencies cited during QIS-based surveys, as well as 

the resources, in terms of the amount of time or number of staff, required to complete 

these surveys.
 

Page 11 GAO-12-214 Nursing Home Quality Assessment

that could be used for monitoring.26

Additionally, CMS could use data from desk audit reports to help measure 

progress towards the objective of improving surveyor consistency. 

According to CMS officials, desk audit reports are standardized reports 

that are generated and sent at established intervals to Regional Offices 

and state survey agencies.

These efficiency scores could also be 

used to measure changes in surveyor efficiency over time, with the goal 

of attaining an agency-established target efficiency score.

27 These reports contain data related to 

surveys, including data intended to describe surveyor performance during 

each QIS-based routine survey conducted. Specifically, desk audit 

reports contain data on approximately 30 different aspects of QIS 

surveys, such as the problem areas that were identified for further 

investigation during a survey and the number of residents for whom a 

surveyor did not document complete information during a survey. The 

reports also indicate how these data compare to rates at the state, 

regional, or national level.28 CMS officials told us that these reports are 

intended to serve as a tool to help officials from Regional Offices and 

state survey agencies identify and respond to inconsistencies in QIS 

surveyor performance. Since data in the reports can illustrate variation in 

how surveyors use the QIS, they could also be used as a starting point for 

CMS to create performance goals and measures for consistency and help 

the agency routinely evaluate whether progress is being made toward the 

objective of improved consistency.

 

26Abt Associates Inc. indicated that a lack of QIS data limited their ability to draw firm 

conclusions about the effect of the QIS on surveyor efficiency. This study was conducted 

when CMS first began to implement the QIS and included data from five states. As of 

September 2011, CMS reported that 26 states have begun QIS implementation. 

Therefore, more QIS data should be currently available for analysis.

27The reports are generated by a CMS contractor and sent to Regional Offices on a 

quarterly basis and to state survey agencies generally after every 10 QIS-based routine 

surveys conducted in the state.

28CMS officials noted that a version of the report for the CMS Central Office is under 

development; however, the agency has not yet established a time line for completing this 

work.
 

Page 12 GAO-12-214 Nursing Home Quality Assessment

CMS has taken some steps to monitor states’ progress implementing the 

QIS, but the agency’s monitoring of states’ efforts is not systematic. As a 

result, CMS has incomplete information on states’ implementation 

progress. CMS has also taken steps to facilitate states’ implementation of 

the QIS, such as holding quarterly teleconferences for all states 

implementing the QIS, but the agency lacks a systematic method for 

obtaining, compiling, and sharing information on states’ experiences 

implementing the QIS.

CMS has taken some steps to monitor states’ progress implementing the 

QIS-based routine survey, but these efforts are not systematic. As part of 

its national implementation plan, CMS established three milestones to 

guide states’ implementation of the QIS-based routine survey.29

• Monitoring of state training start dates—CMS officials said that 

they track start dates for the initial training of state surveyors to use 

the QIS-based routine survey; however, the information that CMS 

obtains only pertains to a small group of surveyors in each state. 

Specifically, CMS arranges and pays for a contractor to train a small 

group of surveyors in states to (1) conduct routine surveys using the 

QIS-based process and (2) become registered QIS trainers who will 

These 

milestones include (1) start dates by which each state is to begin training 

state surveyors to use the QIS, (2) completion of training for all surveyors 

within three years of the training start date, and (3) use of the QIS for all 

routine surveys by state surveyors after all surveyors in the state have 

completed training. CMS policy requires that all routine surveys in a state 

should be conducted using the QIS after all surveyors in a state have 

been trained. CMS officials told us that the agency uses various methods 

to periodically monitor states’ progress on the three milestones.

 

29CMS has established a phased approach for implementing the QIS nationally. The 

agency has grouped states into six cohorts. States within a cohort are to meet the 

milestones for beginning and completing surveyor training and using the QIS for all routine 

surveys within approximately the same general timeframe. According to CMS officials, the 

agency has rolled out the QIS in three of the six cohorts as of September 2011. 

CMS Has Taken Steps 

to Monitor and 

Facilitate States’ 

Implementation of the 

QIS, but CMS’s Efforts 

Are Not Systematic

CMS Has Taken Some 

Steps to Monitor States’ 

Progress Implementing the 

QIS but Does Not Have a 

Systematic Method of 

Monitoring 
 

Page 13 GAO-12-214 Nursing Home Quality Assessment

be responsible for training the rest of surveyors in the state.30 CMS 

officials told us the training contractors are responsible for providing 

CMS with information on the start dates of states’ training of this small 

group of surveyors.

• Monitoring of state surveyor training completion dates—CMS 

officials told us that they periodically monitor the extent to which each 

state completes training of all its surveyors to use the QIS-based 

process, but CMS’s primary method of obtaining this information is not 

systematic and may not allow CMS to collect complete information. 

CMS officials were unable to provide us with information on the 

training completion dates of all state surveyors in those states that the 

agency considered to have completed training. To monitor training 

completion, CMS officials told us that they review information on the 

number of state surveyors trained to use the QIS and compare the 

time between states’ training start dates and when states inform CMS 

that they have trained all of their surveyors. CMS officials told us they 

obtain this training completion information primarily through the 

agency’s quarterly teleconference with state survey agency officials 

implementing the QIS, a forum where states provide information on 

their implementation progress, including status of state surveyor 

training completion. However, we found that the information CMS 

obtains through this method may be incomplete because (1) states 

may not participate in these teleconferences and (2) some states that

participate may not provide complete information on their 

implementation progress. For example, we observed a quarterly 

teleconference in July 2011, during which at least 7 out of the 

24 states that participated in the call did not provide the agency with 

an update on state surveyor training completion during the 

teleconference.

• Monitoring the use of the QIS—CMS officials said that they review 

routine survey data to track the number of routine surveys performed 

using the QIS-based process; however, CMS may not have complete 

information to help determine the extent to which states are using the 

QIS. Specifically, agency officials said they periodically review 

whether state surveyors are conducting routine surveys using the 

 

30To begin training of state surveyors, states select an initial core group of eight 

surveyors—consisting of two survey teams of four surveyors each—to be trained by the 

CMS contractor to use the QIS. It is up to state survey agencies to train all remaining 

surveyors within CMS’s 3 year implementation time frame.
 

Page 14 GAO-12-214 Nursing Home Quality Assessment

QIS-based process exclusively after states complete training of all 

state surveyors to use the QIS.31 To do this, CMS officials pull 

information on the number of surveys completed using the QIS tool 

from their nursing home survey database. However, because CMS 

may not have complete information on the training completion dates 

for all state surveyors, agency officials may not be able to accurately 

determine the date after which states should have begun conducting 

routine surveys exclusively using the QIS-based process.32

CMS acknowledges that the agency does not have a systematic process 

in place that would allow the agency to obtain comprehensive information 

on states’ progress implementing the QIS. The ability to perform effective, 

ongoing monitoring in the course of normal operations is an integral 

element of managing a program, according to federal internal control 

standards.33 CMS officials told us that while they continue to periodically 

monitor states’ progress on the QIS milestones using various means, the 

agency plans to develop a more systematic process for monitoring states’ 

progress toward completion of state surveyor training and use of the QISbased process for routine surveys. According to CMS, this method will 

provide a way for the agency to centrally collect information directly from 

the states, including information on states’ completion of surveyor 

training. The agency plans to use this information to help monitor states’ 

overall progress toward implementing the QIS nationally. However, 

officials said the agency has not yet established a time line for the 

development of this method.

 

31Under CMS policy, once all of a state’s surveyors are trained in QIS that state is 

expected to conduct all routine surveys using QIS. CMS officials acknowledged that not all 

states reporting to have completed training of all surveyors may be using the QIS 

exclusively.

32State survey agencies may continue to use the traditional methodology to conduct 

nursing home inspections until all surveyors in that state have been trained to use the 

QIS. Additionally, those states that have not yet begun QIS implementation continue to 

use the traditional methodology. 

33See GAO/AIMD-00-21.3.1. 
 

Page 15 GAO-12-214 Nursing Home Quality Assessment

CMS has taken steps to facilitate states’ implementation of the QIS. The 

agency has issued periodic guidance to state survey agencies and 

conducted presentations on the QIS process at various venues, including 

CMS’s annual conference with state survey agency officials. In addition, 

as stipulated by its guidance, CMS provides opportunities for state survey 

agencies to discuss implementation issues and concerns with the agency 

and other states. CMS achieves this in part through its quarterly 

teleconferences for all states currently implementing the QIS. During 

these teleconferences, state survey agency officials have opportunities to 

share their experiences implementing the QIS, including how they 

address any challenges they encounter. CMS also provides updates and 

clarification of agency policies that are to guide state survey agencies as 

they implement the QIS during the teleconferences. For example, during 

the July 2011 CMS quarterly teleconference with state survey agencies, 

agency officials updated state survey agency officials on potential 

changes that the agency was considering regarding its policy on the 

number of surveyors assigned to conduct routine surveys using the QIS 

process.34

Despite the agency’s efforts, CMS does not have a systematic method for 

obtaining, compiling, and sharing information with state survey agencies 

on states’ experiences with implementing the QIS. According to federal 

internal control standards, an agency should establish internal controls 

that help ensure adequate communication and information sharing with 

key external stakeholders that may significantly affect the agency’s ability 

to meet its goals.

Further, the CMS Central Office and CMS Regional Office staff 

provides information and assistance on an as-needed basis to individual 

states. Through these efforts, CMS officials stated they learn of common 

challenges states face during implementation, such as problems with 

using QIS software. The agency has provided some assistance to states 

to resolve these challenges, including providing guidance and 

implementing QIS software upgrades.

35

 

34According to CMS, agency policy previously required a minimum of four state surveyors 

per team to conduct routine surveys using the QIS process, regardless of the size of the 

nursing home facility. As of August 2011, CMS revised this policy to allow states to 

determine the number of surveyors per team for facilities with fewer than 40 beds.

CMS obtains and shares information with states 

primarily through the quarterly teleconferences or direct assistance 

provided upon request to state survey agencies—the agency’s key 

35See GAO/AIMD-00-21.3.1. 

CMS Has Taken Some 

Steps to Facilitate States’ 

Implementation of the QIS, 

but Efforts to Obtain and 

Share Information on 

States’ Experiences Are 

Not Systematic
 

Page 16 GAO-12-214 Nursing Home Quality Assessment

external stakeholders. While states share some information on their 

implementation experiences during CMS’s quarterly teleconferences, 

such disclosures are voluntary. Additionally, participation in the 

teleconferences is limited to only states that are implementing the QIS. 

Further, states may not always participate in these teleconferences, so 

information communicated during these teleconferences may not reach 

all states. While CMS officials summarize the teleconference proceedings 

in the form of meeting minutes and share these minutes with meeting 

participants, the agency does not share this information with state survey 

agencies that have not yet begun implementation of the QIS.

We learned that some states have independently developed various 

approaches to help their state facilitate implementation of the QIS. While 

some states have shared these approaches with CMS, the agency is not 

systematically analyzing, compiling, and sharing these efforts with all 

other states.

• Officials in some states helped facilitate implementation by taking the 

initiative to complete state surveyor QIS training in less than 3 years. 

For example, some state survey agency officials told us they worked 

to efficiently train their surveyors to use the QIS-based process so 

that their survey workload would not be affected.36

• One state survey agency facilitated state surveyor training regionally 

by training surveyors in areas of the state with high concentrations of 

large nursing home facilities first. Officials said that they took this 

approach because it allowed them to efficiently balance their survey 

workload.

• All nine state survey agencies we spoke to reported technologyrelated issues that state surveyors encountered when using the QIS 

software and tablet computers during nursing home inspections.37

 

36Some state survey agency officials told us that QIS training can affect state survey 

workloads. Because surveyors’ completion of QIS training required them to be offline for 

at least 4 weeks, fewer state surveyors were then available to conduct surveys for this 

period during the implementation process.

State survey agency officials from two states reported inviting state 

survey agency information technology staff to participate in QIS 

37These technological issues included software malfunctions that caused surveyors’ tablet 

computers to crash or erase survey data collected during nursing home inspections. 
 

Page 17 GAO-12-214 Nursing Home Quality Assessment

surveyor training in order to enhance the staff’s ability to resolve the 

technological issues surveyors expected to encounter when using the 

QIS software and tablet computers.

• Some state survey agency officials we spoke to reported that some 

surveyors in their state experienced difficulties with using QIS 

software and tablet computers due to a learning curve or a lack of 

computer skills needed to operate these tools. For example, officials 

from four states reported developing additional training materials 

related to basic computer skills or general information on the QIS in 

order to improve state surveyors’ proficiency with using these QIS 

tools.

Providing a systematic method—such as using CMS’s annual conference 

in which all state survey agencies participate—to share information on 

state approaches to facilitate implementation and resolve implementation 

challenges could help the agency facilitate implementation in states that 

have begun implementation or will implement the QIS in the future.

CMS developed the QIS in order to improve nursing home inspections, 

which are used to help ensure the quality of nursing home care delivered 

each year to a vulnerable population of more than one million people. 

After conducting demonstrations and evaluating the results of studies, 

CMS determined that the QIS should be implemented nationwide by 

2018, using a phased implementation. To date, CMS has made progress 

implementing the QIS in 26 states. As this implementation continues over 

the next 6 years, it is important that CMS know the extent to which the 

implementation of the QIS is improving the survey process and whether 

states are making progress toward and meeting CMS’s implementation 

milestones.

Although CMS commissioned several studies of the QIS and has shown a 

willingness to adapt the QIS process based on some study findings, the 

agency has not established the means—such as performance goals and 

measures—that would allow for routine, ongoing monitoring of the extent 

to which the QIS is helping CMS improve the survey process as intended. 

Such performance goals and measures could be developed in part using 

data CMS currently collects. Routine monitoring that includes 

performance goals and measures for the QIS could also help CMS 

continue to identify areas for improvement as CMS officials expect to 

continue to make changes intended to improve the QIS throughout its 

phased implementation. While studies commissioned at key points can be 

Conclusions 
 

Page 18 GAO-12-214 Nursing Home Quality Assessment

valuable, the information CMS would obtain on the QIS through routine 

monitoring is needed for this ongoing effort.

CMS also does not have reasonable assurance that the phased 

implementation across the states is making sufficient progress in order to 

meet the 2018 goal. Although CMS uses various means to monitor and 

facilitate states’ efforts to implement the QIS, these efforts are not 

systematic. As a result, CMS lacks complete information on the extent to 

which states have met their implementation milestones, including the 

extent to which states are using the QIS tool once all surveyors in the 

state have completed training. CMS recognizes that it needs a more 

systematic methodology for monitoring states’ progress, but has not yet 

implemented it. To help facilitate implementation of the QIS, CMS 

provides the opportunity for all states implementing the QIS to discuss 

their implementation challenges; however, CMS could better leverage the 

experience of the states by systematically obtaining, compiling, and 

sharing strategies states have used to address their challenges 

implementing the QIS, for example sharing such information through 

CMS’s annual conference in which all state survey agencies participate. 

Such information could help the remaining states that have not yet begun 

implementation to learn from the experience of others.

As CMS continues to implement the QIS, we recommend the 

Administrator of CMS take the following three actions:

• develop a means—such as performance goals and measures—to 

routinely monitor the extent to which CMS is making progress in 

meeting the objectives established for the QIS;

• develop and implement a systematic methodology to track state 

survey agencies’ progress with implementation activities; and

• develop and implement a systematic method for obtaining, compiling, 

and sharing information from state survey agencies about their 

implementation experiences.

On behalf of CMS, HHS reviewed a draft of this report and provided 

written comments, which appear in appendix I. In its written comments, 

HHS fully concurred with all three of our recommendations and said that it 

would work to implement them expeditiously. Specifically, HHS said that 

CMS intends to: 

Recommendations for 

Executive Action

Agency Comments 

and Our Evaluation
 

Page 19 GAO-12-214 Nursing Home Quality Assessment

• Establish new or modify existing measures or processes to more 

effectively monitor CMS’s progress towards meeting QIS goals and 

objectives. HHS informed us that CMS has begun to review available 

data related to the QIS—for example, data on survey workload, 

survey deficiencies, and the number of surveys performed—to more 

effectively monitor the QIS and guide improvement efforts.

• Formalize the data collection method used to track states’ progress 

with QIS implementation activities. According to HHS, the agency 

plans to institute an automated process to obtain updated information 

on states’ progress with training surveyors which, when combined 

with existing data, could provide a more accurate assessment of 

implementation activities in any state.

• Enhance existing and add new methods of sharing information on the 

QIS with states. HHS stated that CMS plans to expand existing 

information sharing opportunities—for example, quarterly calls and 

presentations at annual meetings—to be available to all states, not 

just those in the process of implementing the QIS. It will also give 

consideration to using web-based capability to facilitate sharing 

information on QIS implementation.

In addition to the comments related to our recommendations, HHS noted 

that CMS has reassessed the timetable for QIS expansion to additional 

states due to budgetary concerns and the QIS will not be introduced to 

additional states in fiscal year 2012 as planned. According to HHS, CMS 

will instead focus efforts on more effectively implementing the QIS in 

those states where training of state surveyors has begun or has been 

completed, addressing barriers to QIS implementation, and completing 

the remaining components of the QIS. This reassessment of the QIS 

timeline raises concerns that nationwide implementation of the QIS will 

extend beyond 2018, as currently planned. The longer it takes the agency 

to implement the QIS nationwide, the longer it will take to fully realize the 

intended benefits of the QIS—improving the nursing home inspection 

process and helping to ensure the quality of nursing home care delivered 

each year to more than one million people.

As agreed with your offices, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies of this report to the 

Secretary of HHS, the Administrator of CMS, and relevant congressional 
 

Page 20 GAO-12-214 Nursing Home Quality Assessment

committees. In addition, the report will be available at no charge on the 

GAO website at http://www.gao.gov. 

If you or your staffs have any questions regarding this report, please 

contact me at (202) 512-7114 or kohnl@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. Key contributors to this report were E. Anne 

Laffoon, Assistant Director; George Bogart; Krister Friday; Seta 

Hovagimian; Samantha Poppe; and Priyanka Sethi.

Linda T. Kohn

Director, Health Care
Appendix I: Comments from the Department of 

Health and Human Services

Page 21 GAO-12-214 Nursing Home Quality Assessment

Appendix I: Comments from the Department 

of Health and Human Services
Appendix I: Comments from the Department of 

Health and Human Services

Page 22 GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of 

Health and Human Services

Page 23 GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of 

Health and Human Services

Page 24 GAO-12-214 Nursing Home Quality Assessment
Appendix I: Comments from the Department of 

Health and Human Services

Page 25 GAO-12-214 Nursing Home Quality Assessment (290932)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (www.gao.gov). Each weekday afternoon, 

GAO posts on its website newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.