GAO 

United States Government Accountability Office

Report to Congressional Requesters

NURSING HOMES

Addressing the 

Factors Underlying 

Understatement of 

Serious Care 

Problems Requires 

Sustained CMS and 

State Commitment 

November 2009 

GAO-10-70 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

November 2009

NURSING HOMES

Addressing the Factors Underlying Understatement 

of Serious Care Problems Requires Sustained CMS

Highlights of GAO-10-70, a report to and State Commitment 

congressional requesters 

Under contract with the CMS, 

states conduct surveys at nursing 

homes to help ensure compliance 

with federal quality standards. Over 

the past decade, GAO has reported 

on inconsistencies in states’ 

assessment of nursing homes’ 

quality of care, including 

understatement—that is, when 

state surveys fail to cite serious 

deficiencies or cite them at too low 

a level. In 2008, GAO reported that 

9 states had high and 10 had low 

understatement based on CMS data 

for fiscal years 2002 through 2007. 

This report examines the effect on 

nursing home deficiency 

understatement of CMS’s survey 

process, workforce shortages and 

training, supervisory reviews of 

surveys, and state agency practices. 

GAO primarily collected data 

through two Web-based 

questionnaires sent to all eligible 

nursing home surveyors and state 

agency directors, achieving 61 and 

98 percent response rates, 

respectively. 

What GAO Recommends

GAO is making seven 

recommendations to the CMS 

Administrator to address state and 

surveyor issues about CMS’s survey 

methodology and guidance, 

workforce shortages and 

insufficient training, 

inconsistencies in the focus and 

frequency of the supervisory 

review of deficiencies, and external 

pressure from the nursing home 

industry. CMS concurred with five 

of GAO’s seven recommendations 

and indicated it would explore 

alternate solutions to the remaining 

two recommendations. 

A substantial percentage of both state surveyors and directors identified 

general weaknesses in the nursing home survey process, that is, the survey 

methodology and guidance on identifying deficiencies. On the questionnaires, 

46 percent of surveyors and 36 percent of directors reported that weaknesses 

in the traditional survey methodology, such as too many survey tasks, 

contributed to understatement. Limited experience with a new data-driven 

survey methodology indicated possible improvements in consistency; 

however, an independent evaluation led CMS to conclude that other tools, 

such as survey guidance clarification and surveyor training and supervision, 

would help improve survey accuracy. 

According to questionnaire responses, workforce shortages and greater use of 

surveyors with less than 2 years’ experience sometimes contributed to 

understatement. Nearly three-quarters of directors reported that they always 

or frequently experienced a workforce shortage, while nearly two-thirds 

reported that surveyor inexperience always, frequently, or sometimes led to 

understatement. Substantial percentages of directors and surveyors indicated 

that inadequate training may compromise survey accuracy and lead to 

understatement. According to about 29 percent of surveyors in 9 high 

understatement states compared to 16 percent of surveyors in 10 low 

understatement states, initial surveyor training was not sufficient to cite 

appropriate scope and severity—a skill critical in preventing understatement. 

Furthermore, over half of directors identified the need for ongoing training for 

experienced surveyors on both this skill and on documenting deficiencies, a 

critical skill to substantiate citations. 

CMS provides little guidance to states on supervisory review processes. In 

general, directors reported on our questionnaire that supervisory reviews 

occurred more often on surveys with higher-level rather than on those with 

lower-level deficiencies, which were the most frequently understated. 

Surveyors who reported that survey teams had too many new surveyors also 

reported frequent changes to or removal of deficiencies, indicating heavier 

reliance on supervisory reviews by states with inexperienced surveyors. 

Surveyors and directors in a few states informed us that, in isolated cases, 

state agency practices or external pressure from stakeholders, such as the 

nursing home industry, may have led to understatement. Forty percent of 

surveyors in five states and four directors reported that their state had at least 

one practice not to cite certain deficiencies. Additionally, over 40 percent of 

surveyors in four states reported that their states’ informal dispute resolution 

processes favored concerns of nursing home operators over resident welfare. 

Furthermore, directors from seven states reported that pressure from the 

industry or legislators may have compromised the nursing home survey 

process, and two directors reported that CMS’s support is needed to deal with 

such pressure. If surveyors perceive that certain deficiencies may not be 

consistently upheld or enforced, they may choose not to cite them. 

View GAO-10-70 or key components.

For more information, contact John E. Dicken 

at (202) 512-7114 or dickenj@gao.gov. Also 

see GAO-10-74SP for summary data from the 

questionnaires. 
Page i GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 

Contents 

Letter 1

Background 5 

Weaknesses in CMS Survey Process Contributed to 

Understatement, but Long-Term Effect of New Survey 

Methodology Is Not Yet Known 15 

Workforce Shortages and Training Inadequacies May Contribute to 

Understatement 22 

State Supervisory Reviews Often Are Not Designed to Identify 

Understatement 30 

State Agency Practices and External Pressure May Compromise 

Survey Accuracy and Lead to Understatement in a Few States 36 

Conclusions 45 

Recommendations for Executive Action 48 

Agency and AHFSA Comments and Our Evaluation 49 

Appendix I Scope and Methodology 54 

Appendix II Comments from the Department of Health & 

Human Services 63 

Appendix III GAO Contact and Staff Acknowledgments 70 

Related GAO Products 71 

Tables 

Table 1: Scope and Severity of Deficiencies Identified during 

Nursing Home Surveys 7 

Table 2: Surveyors’ and State Agency Directors’ Responses to 

Questions on CMS’s Survey Process 16 

Table 3: Percentage of Surveyors Reporting That Guidance for 

Certain Federal Quality Standards Was Not Sufficient to 

Identify Deficiencies 18 

Table 4: State Agency Directors’ Responses to Questions about 

Surveyor Workforce Issues 23 
Table 5: State Survey Agency Vacancy Rates and Percentage of 

State Surveyors with Less Than 2 Years’ Experience 24 

Table 6: Surveyors’ and State Agency Directors’ Responses to 

Questions on Workforce Issues 26 

Table 7: Responses from Surveyors and State Agency Directors to 

Key Questions on Training 28 

Table 8: Percentage of Surveyors Reporting Changes in Deficiency 

Citations during Supervisory Review 35 

Table 9: Response Rates to GAO’s Questionnaire of Nursing Home 

Surveyors, 2008 56 

Figures 

Figure 1: Zero-, Low-, and High-Understatement States, Fiscal 

Years 2002-2007 13 

Figure 2: Eight State Agency Director Responses on Five Questions 

Related to the QIS 20 

Figure 3: Number of State Supervisory Reviews at the Potential for 

More than Minimal Harm (D-F) and Immediate Jeopardy 

Levels (J-L) 32 

Figure 4: Percentage of Surveyors in Each State Reporting at Least 

One Noncitation Practice 38 

Figure 5: Percentage of Surveyors in Each State Reporting the IDR 

Process Favored Concerns of Nursing Home Operators 

over Resident Welfare 44 

Page ii GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Abbreviations 

AHFSA Association of Health Facility Survey Agencies 

CMS Centers for Medicare & Medicaid Services 

HHS Department of Health & Human Services 

IDR Informal Dispute Resolution 

OSCAR On-Line Survey, Certification, and Reporting system 

RN registered nurse 

SMQT Surveyor Minimum Qualifications Test 

SOM State Operations Manual 

QIS Quality Indicator Survey 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page iii GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Page 1 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 

 

United States Government Accountability Office

Washington, DC 20548 

 

November 24, 2009 

The Honorable Herb Kohl 

Chairman 

Special Committee on Aging 

United States Senate 

The Honorable Charles E. Grassley 

Ranking Member 

Committee on Finance 

United States Senate 

Federal and state governments share responsibility for ensuring that 

nursing homes provide quality care in a safe environment for the nation’s 

1.5 million residents dependent on such care. The federal government is 

responsible for setting quality requirements that nursing homes must meet 

to participate in the Medicare and Medicaid programs.1

 The Centers for 

Medicare & Medicaid Services (CMS), within the Department of Health & 

Human Services (HHS), contracts with state survey agencies to conduct 

periodic inspections, known as surveys, and complaint investigations, both 

of which assess whether homes meet federal standards.2

 State survey 

agencies are required to follow federal regulations for surveying facilities; 

however, several survey activities and policies are left largely to the 

discretion of state survey agencies, including hiring and retaining a 

surveyor workforce, training surveyors, reviewing deficiency citations, 

and managing regulatory interactions with the industry and public. 

In response to congressional requests over the last decade, we have 

reported significant weaknesses in federal and state activities designed to 

detect and correct quality and safety problems in nursing homes and the 

persistence of serious deficiencies, which are those deficiencies that harm 

residents or place them at risk of death or serious injury.3

 In the course of 

1

Medicare is the federal health care program for elderly and certain disabled individuals. 

Medicaid is a joint federal-state health care financing program for certain categories of lowincome individuals. 

2

In addition to the oversight of nursing homes, CMS and state survey agencies are 

responsible for oversight of other Medicare and Medicaid providers, such as home health 

agencies, intermediate care facilities for the mentally retarded, and hospitals. 

3

See a list of related GAO products at the end of this report. 
our work, we regularly found significant variation across states in their 

citations of serious deficiencies—indicating inconsistencies in states’ 

assessment of quality of care. We also found evidence of substantial 

understatement—that is, state inspections that failed to cite serious 

deficiencies or that cited deficiencies at too low a level. 

In this report, we complete our response to your request to examine the 

understatement of serious deficiencies in nursing homes by state 

surveyors nationwide and the factors that contribute to understatement. 

Our first report, issued in May 2008, identified the extent of nursing home 

understatement nationwide.4

 It found that 15 percent of federal nursing 

home surveys nationwide and 25 percent of these surveys in nine states 

identified state surveys that failed to cite serious deficiencies. This report 

examines how the following factors affect the understatement of nursing 

home deficiencies: (1) the CMS survey process, (2) workforce shortages 

and training, (3) supervisory reviews, and (4) state agency practices. 

To do this work, we analyzed data collected from two GAO-administered 

Web-based questionnaires, one to nursing home surveyors and the other to 

state agency directors; analyzed federal and state nursing home survey 

results; interviewed CMS officials from the Survey and Certification Group 

and selected Regional Offices; reviewed federal regulations and guidance, 

and our prior work; and conducted follow-up interviews with state agency 

directors, as needed, to clarify and better understand their unique state 

circumstances.5

Our prior work documented the prevalence of understatement 

nationwide and described several factors that may contribute to survey 

inconsistency and the understatement of deficiencies by state survey 

teams: (1) weaknesses in CMS’s survey methodology, including poor 

 

4

See GAO, Nursing Homes: Federal Monitoring Surveys Demonstrate Continued 

Understatement of Serious Care Problems and CMS Oversight Weaknesses, GAO-08-517 

(Washington, D.C.: May 9, 2008). 

5

CMS’s Survey and Certification Group is responsible for ensuring the effectiveness of state 

survey activities and managing the federal monitoring survey program. 

Page 2 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
documentation of deficiencies,6

 (2) confusion among surveyors about the 

definition of actual harm,7

 (3) predictability of surveys, which allows 

homes to conceal problems if they so desire,8

 (4) inadequate quality 

assurance processes at the state level to help detect understatement in the 

scope and severity of deficiencies,9

 and (5) inexperienced state surveyors 

as a result of retention problems.10 We relied on this information and 

feedback from pretests with six surveyors from a local state and five 

current or former state agency directors to develop our questionnaires on 

the nursing home survey process and factors that contribute to the 

understatement of deficiencies. 

Our Web-based questionnaires of nursing home surveyors and state 

agency directors achieved response rates of 61 percent and 98 percent, 

respectively. The first questionnaire collected responses from 2,340 of the 

total 3,819 eligible nursing home surveyors in 49 states and the District of 

Columbia.11 The resulting sample of surveyors who responded to our 

questionnaire between May and July 2008 was representative of surveyors 

 

6

See GAO, Nursing Home Quality: Prevalence of Serious Problems, While Declining, 

Reinforces Importance of Enhanced Oversight, GAO-03-561 (Washington, D.C.: July 15, 

2003) and GAO, Nursing Home Reform: Continued Attention Is Needed to Improve 

Quality of Care in Small but Significant Share of Homes, GAO-07-794T (Washington, 

D.C.: May 2, 2007). In response to our recommendation to finalize the development, testing, 

and implementation of a more rigorous survey methodology, CMS evaluated and is 

currently implementing a revised survey methodology. 

7

See GAO, Nursing Homes: Despite Increased Oversight, Challenges Remain in Ensuring 

High-Quality Care and Resident Safety, GAO-06-117 (Washington, D.C.: Dec. 28, 2005). 

8

See GAO-03-561. Our analysis of survey predictability considered surveys to be predictable 

if (1) homes were surveyed within 15 days of the 1-year anniversary of the prior survey or 

(2) homes were surveyed within 1 month of the maximum 15-month interval between 

standard surveys. We used this rationale because homes know the maximum allowable 

interval between surveys, and those whose prior surveys were conducted 14 or 15 months 

earlier are aware that they are likely to be surveyed soon. 

9

See GAO-03-561. 

10See GAO-03-561. 

11Eligible surveyors are those who had conducted at least one health survey of a nursing 

home in fiscal year 2006 or 2007 and for whom we could obtain an e-mail or other address 

from their state survey agency. 

Page 3 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
nationally, with the exception of Pennsylvania.12 Fifty state agency 

directors responded to the second questionnaire from September to 

November 2008.13 Many questions on our questionnaires asked 

respondents to identify the frequency that an event occurred using the 

following scale—always, frequently, sometimes, infrequently, or never; 

however, for reporting purposes, we grouped responses into three 

categories—always/frequently, sometimes, and infrequently/never. In 

addition, our questionnaire to state agency directors asked them to rank 

the degree to which several factors, derived from our previous work, 

contributed to understatement.14 Summary results from the GAO 

questionnaires are available as an e-supplement to this report. See 

Nursing Homes: Responses from Two Web-Based Questionnaires to 

Nursing Home Surveyors and State Agency Directors (GAO-10-74SP), an 

e-supplement to GAO-10-70. 

We analyzed the data collected from these questionnaires as stand-alone 

datasets and in relationship to state performance on federal comparative 

and observational surveys as captured in the federal monitoring survey 

database, which we reported on in 2008.15 In addition, to inform our 

understanding of the extent to which each factor contributed to 

understatement, we examined relationships among the responses to both 

questionnaires and the results of the federal comparative and 

observational surveys for fiscal years 2002 through 2007. We used the 

results of the federal comparative surveys for these years to identify states 

with high and low percentages of serious missed deficiencies. We report 

results for tests of association and differences between group averages. 

 

12We excluded Pennsylvania from our analysis because Pennsylvania’s Deputy Secretary 

for Quality Assurance instructed the state’s surveyors not to respond to our survey. Two 

other states had response rates below 40 percent—Connecticut (28 percent), and Illinois 

(20 percent). Illinois’ response rate probably reflected that surveyors’ access to their e-mail 

accounts and our Web-based survey was limited to only 1 day per month. 

13The District of Columbia agency director did not respond to our questionnaire. 

14We did not ask nursing home surveyors a similar question because survey agency 

directors, as a result of their positions, were a more consistent source of knowledge about 

the influence of these factors on understatement. 

15See GAO-08-517. This database captures the results of two types of federal monitoring 

surveys. Federal comparative surveys are conducted independently by federal surveyors to 

evaluate state surveys. Federal surveyors resurvey a home that was recently inspected by 

state surveyors and compare the deficiencies identified during the two surveys. When 

federal surveyors accompany state surveyors to directly observe them during a nursing 

home survey it is considered a federal observational survey. 

Page 4 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
We also interviewed directors and other state agency officials in eight 

states to better understand unusual or interesting circumstances related to 

surveyor workforce and training, supervisory review, or state policies and 

practices. We selected these eight state agencies based on our analysis of 

questionnaire responses from the state agency directors and nursing home 

surveyors. 

To compare average facility citations on state survey records with the 

average citations on federal survey records, we collected information from 

the On-Line Survey, Certification, and Reporting (OSCAR) system for 

those facilities where federal teams assessed state surveyor performance 

for fiscal years 2002 through 2007.16 Except where otherwise noted, we 

used data from fiscal year 2007 because they were the most recently 

available data at the time of our analysis (see appendix I for more on our 

scope and methodology). 

We conducted this performance audit from April 2008 through November 

2009 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. 

Oversight of nursing homes is a shared federal-state responsibility. 

As part of this responsibility, CMS (1) sets federal quality standards, 

(2) establishes state responsibilities for ensuring federal quality standards 

are met, (3) issues guidance on determining compliance with these 

standards, and (4) performs oversight of state survey activities. It 

communicates these federal standards and state responsibilities in the 

State Operations Manual (SOM) and through special communications such 

as program memorandums and survey and certification letters. CMS 

provides less guidance on how states should manage the administration of 

their survey programs. CMS uses staff in its 10 regional offices to oversee 

states’ performance on surveys that ensure that facilities participating in 

Medicare and Medicaid provide high-quality care in a safe environment. 

Yet, the persistent understatement of serious nursing home deficiencies 

Background 

 

16We use the term survey record to refer to CMS’s Form 2567, which is the official 

statement of deficiencies with respect to federal quality standards. 

Page 5 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
that we have reported and survey quality weaknesses that we and the HHS 

Office of Inspector General identified serve as indicators of weaknesses in 

the federal, state, or shared components of oversight. 

Survey Process Every nursing home receiving Medicare or Medicaid payment must 

undergo a standard state survey not less than once every 15 months, 

and the statewide average interval for these surveys must not exceed 

12 months. During a standard survey, teams of state surveyors—generally 

consisting of registered nurses, social workers, dieticians, or other 

specialists—evaluate compliance with federal quality standards. The 

survey team determines whether the care and services provided meet the 

assessed needs of the residents and measure resident outcomes, such as 

the incidence of preventable pressure sores, weight loss, and accidents. In 

contrast to a standard survey, a complaint investigation generally focuses 

on a specific allegation regarding a resident’s care or safety and provides 

an opportunity for state surveyors to intervene promptly if problems arise 

between standard surveys. 

Surveyors assess facilities using federal nursing home quality standards 

that focus on the delivery of care, resident outcomes, and facility 

conditions. These standards total approximately 200 and are grouped into 

15 categories, such as Quality of Life, Resident Assessment, Quality of 

Care, and Administration.17 For example, there are 23 standards (known as 

F-tags) within the Quality of Care category ranging from the prevention of 

pressure sore development (F-314) to keeping the resident environment as 

free of accident hazards (F-323) as is possible. 

Surveyors categorize deficient practices identified on standard surveys 

and complaint investigations—facilities’ failures to meet federal 

standards—according to scope (i.e., the number of residents potentially or 

actually affected) and severity (i.e., the degree of relative harm 

involved)—using a scope and severity grid (see table 1). Homes with 

deficiencies at the A through C levels are considered to be in substantial 

compliance, while those with deficiencies at the D through L levels are 

considered out of compliance. Throughout this report, we refer to 

 

17Other areas include Admission, Transfer and Discharge Rights, Resident Rights, Resident 

Behavior and Facility Practices, Nursing Services, Pharmacy Services, Dietary Services, 

Physician Services, Specialized Rehabilitative Services, Dental Services, Infection Control, 

and Physical Environment. Surveys also examine compliance with federal fire safety 

requirements. 

Page 6 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
deficiencies at the actual harm and immediate jeopardy levels—G through 

L—as serious deficiencies. CMS guidance requires state survey teams to 

revisit a home to verify that serious deficiencies have actually been 

corrected.18

Table 1: Scope and Severity of Deficiencies Identified during Nursing Home 

Surveys 

Scope 

Severity Isolated Pattern Widespread 

Immediate jeopardyaJ K L 

Actual harm G H I 

Potential for more than minimal harm D E F 

Potential for minimal harmbA B C 

Source: CMS. 

a

Actual or potential for death / serious injury. 

b

Nursing home is considered to be in “substantial compliance.” 

In addition, when serious deficiencies are identified, sanctions can be 

imposed to encourage facilities to correct the deficiencies and enforce 

federal quality standards. Sanctions include fines known as civil money 

penalties, denial of payment for new Medicare or Medicaid admissions, or 

termination from the Medicare and Medicaid programs. For example, 

facilities that receive at least one G through L level deficiency on 

successive standard surveys or complaint investigations must be referred 

for immediate sanctions. Facilities may appeal cited deficiencies and if the 

appeal is successful, the severity of the sanction could be reduced or the 

sanction could be rescinded. Facilities have several avenues of appeal, 

including informal dispute resolution (IDR) at the state survey agency 

level.19 The IDR gives providers one opportunity to informally refute cited 

deficiencies after any survey. While CMS requires that states have an IDR 

policy in place, it does not specify how IDR processes should be 

structured. 

 

18Revisits are not required for most deficiencies cited below the actual-harm level—that is, 

A through F. 

19Nursing homes can also appeal deficiency citations, which result in hearings before an 

administrative law judge; nursing homes may also request HHS’s Departmental Appeals 

Board to review. 

Page 7 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
To conduct nursing home surveys, CMS has traditionally used a 

methodology that requires surveyors to select a sample of residents and 

(1) review data derived from the residents’ assessments and medical 

records; (2) interview nursing home staff, residents, and family members; 

and (3) observe care provided to residents during the course of the survey. 

When conducting a survey, surveyors have discretion in: selecting a 

sample of residents to evaluate; allocating survey time and emphasis 

within a framework prescribed by CMS; investigating potentially deficient 

practices observed during the survey; and determining what evidence is 

needed to identify a deficient practice. CMS has developed detailed 

investigative protocols to assist state survey agencies in determining 

whether nursing homes are in compliance with federal quality standards. 

These protocols are intended to ensure the thoroughness and consistency 

of state surveys and complaint investigations. 

Survey Methodology 

In 1998, CMS awarded a contract to revise the survey methodology. The 

new Quality Indicator Survey (QIS) was developed to improve the 

consistency and efficiency of state surveys and provide a more reliable 

assessment of quality. The QIS uses an expanded sample of residents and 

structured interviews with residents and family members in a two-stage 

process. Surveyors are guided through the QIS process using customized 

software on tablet personal computers. In stage 1, a large resident sample 

is drawn and relevant data from on- and off-site sources is analyzed to 

develop a set of quality-of-care indicators, which will be compared to 

national benchmarks.20 Stage 2 systematically investigates potential 

quality-of-care concerns identified in stage 1. Because of delays in 

implementing the QIS, we recommended in 2003 that CMS finalize the 

development, testing, and implementation of a more rigorous survey 

methodology, including investigative protocols that provide guidance to 

surveyors in documenting deficiencies at the appropriate scope and 

severity level.21 CMS concluded a five-state demonstration process of the 

QIS in 2007 and is currently expanding the implementation of the QIS. As 

of 2008, only Connecticut had implemented the QIS statewide, and CMS 

projected that the QIS would not be fully implemented in every state until 

2014. 

 

20On-site sources include observations, interviews, and records review. 

21See GAO-03-561. 

Page 8 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
States are largely responsible for the administration of the survey 

program. State survey agencies administer and have discretion over many 

survey activities and policies, including hiring and retaining a surveyor 

workforce, training surveyors, conducting supervisory reviews of surveys, 

and other activities. 

State Administration 

• Hiring and Retaining a Surveyor Workforce: State survey agencies hire the 

staff to conduct surveys of nursing homes and determine the salaries of 

these personnel according to the workforce practices and restrictions of 

the state. Salaries, particularly surveyor salaries, are the most significant 

cost component of state survey activities, which are supported through a 

combination of Medicare, Medicaid, and non-Medicaid state funds.22 CMS 

has some requirements for the make-up of nursing home survey teams, 

including the involvement of at least one registered nurse (RN) in each 

nursing home survey. In February 2009, we reported that officials from the 

Association of Health Facility Survey Agencies (AHFSA) and other state 

officials told us they have had difficulty recruiting and retaining the survey 

workforce for several years. In our report, we recommended that CMS 

undertake a broad-based reexamination to ensure, among other aspects, 

an adequate survey workforce with sufficient compensation to attract and 

retain qualified staff.23

• Training: States are responsible for training new surveyors through 

participating in actual surveys under direct supervision. Within their first 

year of employment, surveyors must complete two CMS online training 

courses—the Basic Health Facility Surveyor Course and Principles of 

Documentation—and a week-long CMS-led Basic Long-Term Care Health 

Facility Surveyor Training Course; at the conclusion of the course 

surveyors must pass the Surveyor Minimum Qualifications Test (SMQT) to 

survey independently. In addition, state survey agencies are required to 

have their own programs for staff development that respond to the need 

for continuing development and education of both new and experienced 

employees. Such staff development programs must include training for 

surveyors on all regulatory requirements and the skills necessary to 

 

22The federal government funds state surveys through the Medicare and Medicaid 

programs. States contribute a share of Medicaid and non-Medicaid funds to support survey 

activities. State non-Medicaid contributions are to reflect the benefit states derive from 

health care facilities that meet federal quality standards as well as the cost of assessing 

compliance with state licensing requirements. See GAO, Medicare and Medicaid 

Participating Facilities: CMS Needs to Reexamine Its Approach for Funding State 

Oversight of Health Care Facilities, GAO-09-64 (Washington, D.C.: Feb. 13, 2009). 

23See GAO-09-64. 

Page 9 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
conduct surveys. To assist in continuing education, CMS develops a 

limited number of courses for ongoing training and provides other training 

materials. 

• Supervisory Reviews: States may design a supervisory review process for 

deficiencies cited during surveys, although CMS does not require them to 

do so. In July 2003, we recommended that CMS require states to have a 

minimum quality-assurance process that includes a review of a sample of 

survey reports below the level of actual harm to assess the 

appropriateness of scope and severity levels cited and help reduce 

instances of understated quality-of-care problems.24 CMS did not 

implement this recommendation.25

• State Agency Practices and Policies: State survey agencies’ practices, 

including those on citing deficiencies and addressing pressure from the 

industry or others, are largely left to the discretion of state survey 

agencies. In the past, we reported that in one state, CMS officials had 

found surveyors were not citing all deficiencies.26 If a state agency fails to 

cite all deficiencies associated with noncompliance, nursing home 

deficiencies are understated on the survey record. CMS can identify or 

monitor states for systematic noncitation practices through reviews of 

citation patterns, informal feedback from state surveyors, state 

performance reviews, and federal monitoring surveys (discussed below).27

CMS also gives states latitude in defining their IDR process. 

 

24See GAO-03-561. 

25CMS commented on the importance of quality-assurance processes and noted it had 

already incorporated such reviews into CMS regional offices’ reviews of the state 

performance standards. However, the agency did not require states to initiate an ongoing 

process that would evaluate the appropriateness of the scope and severity of documented 

deficiencies, as we recommended. See GAO-03-561. 

26See GAO-03-561. 

27In addition to the federal monitoring surveys, CMS established annual state performance 

reviews in fiscal year 2001 to measure a state’s compliance with specific standards. These 

standards generally focus on the timeliness and quality of surveys, complaint 

investigations, and enforcement actions. CMS’s state performance reviews include (1) an 

examination of the quality of state survey agency investigations and decision making and 

(2) the timeliness and quality of complaint investigations. 

Page 10 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Federal law requires federal surveyors to conduct federal monitoring 

surveys in at least 5 percent of state-surveyed Medicare and Medicaid 

nursing homes in each state each year. CMS indicates it meets the 

statutory requirement by conducting a mix of on-site reviews: comparative 

and observational surveys.28

Federal Monitoring 

Surveys and Evidence of 

Understatement 

Comparative surveys. A federal survey team conducts an independent 

survey of a home recently surveyed by a state survey agency in order to 

compare and contrast its findings with those of the state survey team. This 

comparison takes place after completion of the federal survey. When 

federal surveyors identify a deficiency not cited by state surveyors, they 

assess whether the deficiency existed at the time of the state survey and 

should have been cited.29 This assessment is critical in determining 

whether understatement occurred, because some deficiencies cited by 

federal surveyors may not have existed at the time of the state survey. 

Our May 2008 report stated that comparative surveys found problems at 

the most serious levels of noncompliance—the actual harm and immediate 

jeopardy levels (G through L).30 About 15 percent of federal comparative 

surveys nationwide identified at least one deficiency at the G through L 

level that state surveyors failed to cite. While this proportion is small, CMS 

maintains that any missed serious deficiencies are unacceptable. Further, 

state surveys with understated deficiencies may allow the surveyed 

facilities to escape sanctions intended to discourage repeated 

noncompliance. 

In our May 2008 report we found that for nine states federal surveyors 

identified missed serious deficiencies in 25 percent or more comparative 

surveys for fiscal years 2002 through 2007; we defined these states as highunderstatement states (see fig. 1). Zero-understatement states were states 

 

28In 1998, the Health Care Financing Administration, the HHS agency now known as CMS, 

acknowledged the need to perform a greater number of comparative surveys and have 

done so. Between October 1998 and July 1999 only about 9 percent (64) of federal 

monitoring surveys were comparative. However, in our May 2008 report, we found that for 

the period of fiscal years 2002 through 2007 about 20 percent (976) of federal monitoring 

surveys were comparative surveys and the remaining 80 percent were observational 

surveys. By statute, comparative surveys must be conducted within 2 months of the 

completion of the state survey. 

29CMS began requiring regional offices to make this determination in fiscal year, 2002 and it 

is captured by a yes/no validation question. 

30See GAO-08-517. 

Page 11 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
that had no federal comparative surveys identifying missed deficiencies at 

the actual harm or immediate jeopardy levels; and low-understatement 

states were the 10 states with the lowest percentage of missed serious 

deficiencies (less than 6 percent), including all 7 zero-understatement 

states. 

Page 12 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Figure 1: Zero-, Low-, and High-Understatement States, Fiscal Years 2002-2007 

Source: GAO analysis of CMS data. Map: Copyright © Corel Corp. All rights reserved.

Zero

Low

Mid-range

High

OR

ID

ND ME

VT

WV

AK

WY

AZ NM

SD

NE

OK

MO

AR

TN

AL

SC

OH

Note: Zero-understatement states were those that had no missed serious deficiencies on federal 

comparative surveys. Low-understatement states were the 10 states with the lowest percentage of 

missed serious deficiencies on federal comparative surveys (less than 6 percent), including all zerounderstatement states. High-understatement states were the 9 states with the highest percentage of 

serious missed deficiencies (25 percent or more) on federal comparative surveys. 

Page 13 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Our May 2008 report also found that missed deficiencies at the potential 

for more than minimal harm level (D through F) were considerably more 

widespread than those at the G through L level on comparative surveys, 

with approximately 70 percent of comparative surveys nationwide 

identifying at least one missed deficiency at this level. Undetected care 

problems at this level are of concern because they could become more 

serious over time if nursing homes are not required to take corrective 

actions.31

Observational surveys. Federal surveyors accompany a state survey 

team to evaluate the team’s performance and ability to document survey 

deficiencies. State teams are evaluated in six areas, including two—

General Investigation and Deficiency Determination—that affect the 

appropriate identification and citation of deficiencies. The General 

Investigation segment assesses the effectiveness of state survey team 

actions such as collection of information, discussion of survey 

observations, interviews with nursing home residents, and implementation 

of CMS investigative protocols. The Deficiency Determination segment 

evaluates the skill with which the state survey teams (1) analyze and 

integrate all information collected, (2) use the guidance for surveyors, and 

(3) assess compliance with regulatory requirements. Federal observational 

surveys are not independent evaluations of the state survey because state 

surveyors may perform their survey tasks more attentively than they 

would if federal surveyors were not present; however, they provide more 

immediate feedback to state surveyors and may help identify state 

surveyor training needs. 

We previously reported that state survey teams’ poor performance on 

federal observational surveys in the areas of General Investigation and 

Deficiency Determination may contribute to the understatement of 

deficiencies.32 Further, poor state performance in these two areas 

supported the finding of understatement as identified through the federal 

comparative surveys. We found that about 8 percent of state survey teams 

observed by federal surveyors nationwide received below-satisfactory 

ratings on General Investigation and Deficiency Determination from fiscal 

 

31In May 2008, we found that understatement also occurred when state survey teams cited 

deficiencies at too low a level of scope and severity. At that time, CMS did not require 

federal surveyors to evaluate scope and severity differences between state and federal 

comparative surveys. However, as of October 2008, CMS began requiring such assessments. 

32See GAO-08-517. 

Page 14 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
years 2002 through 2007. However, surveyors in high-understatement 

states performed worse in these two areas of the federal observational 

surveys than surveyors in the low-understatement states. For example, an 

average of 12 and 17 percent of state survey teams observed by federal 

surveyors in high-understatement states received below satisfactory 

ratings for these two areas, respectively. In contrast, an average of 4 

percent of survey teams in low-understatement states received the same 

below-satisfactory scores for both deficiency determination and 

investigative skills. 

Nationwide, one-third of nursing homes had a greater average number of 

serious deficiencies on federal observational surveys than on state 

standard surveys during fiscal years 2002 through 2007, but in eight states, 

it was more than half of homes. Of the one-third of homes nationwide, 

state standard surveys cited 83 percent fewer serious deficiencies than 

federal surveys during this same time period. 

Over a third of both surveyors and state agency directors responding to 

our questionnaire identified weaknesses in the federal government’s 

nursing home survey process that contributed to the understatement of 

deficiencies.33 The weaknesses included problems with the current survey 

methodology; written guidance that is too long or complex; and to a lesser 

extent, survey predictability or other advance notice of inspections, which 

may allow nursing homes to conceal deficiencies. At the time our 

questionnaires were fielded, eight states had started implementing CMS’s 

new survey methodology. The limited experience among these states 

suggests that the new methodology may improve consistency of surveys, 

but information is limited, and the long-term ability of the new 

methodology to reduce understatement is not yet known. 

Weaknesses in CMS 

Survey Process 

Contributed to 

Understatement, but 

Long-Term Effect of 

New Survey 

Methodology Is Not 

Yet Known 

Weaknesses in CMS’s 

Survey Process 

Contributed to 

Understatement 

Both surveyors and state agency directors reported weaknesses in the 

survey process, and on our questionnaire linked these weaknesses to 

understatement of deficiencies. Nationally, 46 percent of nursing home 

surveyors responded that weaknesses in the current survey methodology 

resulted in missed or incorrectly identified deficiencies, with this number 

 

33For purposes of this report, we defined the federal nursing home survey process as both 

the traditional methodology used to evaluate compliance of nursing homes with federal 

requirements and the written guidance provided by CMS to help state agencies carry out 

survey activities. 

Page 15 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
ranging by state from 0 to 74 percent (see table 2).34 Thirty-six percent of 

state agency directors responded that weaknesses in the current survey 

methodology at least sometimes contributed to understatement of 

deficiencies in their states. One such weakness identified by both 

surveyors and directors was the number of survey tasks that need to be 

completed. 

Table 2: Surveyors’ and State Agency Directors’ Responses to Questions on CMS’s 

Survey Process 

Questions related to CMS’s survey process 

Percentage 

of surveyors’

responses

Percentage 

of directors’ 

responses

Weaknesses in the current survey methodology at least 

sometimes result in missed or incorrectly identified 

deficiencies at the facility 46 36

Additional training is needed to apply CMS guidance 40 58

Source: GAO. 

According to surveyors and agency directors responding to our 

questionnaire, another weakness with the federal survey process involved 

CMS’s written guidance to help state agencies follow federal regulations 

for surveying long-term care facilities.35 Both surveyors and state agency 

directors mentioned concerns about the length, complexity, and 

subjectivity of the written guidance. One state agency director we 

interviewed told us that the size of the SOM made it difficult for surveyors 

to carry the guidance and consult it during surveys. Although the SOM is 

available in an electronic format, surveyors in this state did not use 

laptops. In addition, a small percentage of surveyors commented on our 

questionnaire that CMS guidance was inconsistently applied in the field. A 

common complaint from these surveyors was that different supervisors 

required different levels of evidence in order to cite a deficiency at the 

actual harm or immediate jeopardy level. Forty percent of surveyors and 

Surveyor Quotation about CMS Written 

Guidance

“Appreciate the guidances and protocols. 

However, making Appendix PP [guidance for 

investigating federal quality standards] into a

tome is not helping us out in the field. They 

are too cumbersome and voluminous. Please 

find a way to be more concise in these 

guidances.”

 

34Survey methodology is defined as the traditional approach used to evaluate nursing home 

compliance with federal regulations as outlined by CMS in Appendix P of the SOM. 

35For purposes of this report, we defined CMS written guidance as the information in the 

SOM on the long-term care survey process, including the survey protocol for long-term care 

facilities in Appendix P and the guidance on federal quality standards in Appendix PP as 

well as any additional materials provided by CMS to assist surveyors, such as Survey and 

Certification letters. 

Page 16 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
58 percent of state agency directors reported that additional training on 

how to apply CMS guidance was needed. 

A specific concern raised about the current survey guidance was 

determining the severity level for an observed deficiency. Forty-four 

percent of state agency directors reported on our questionnaire that 

confusion about CMS’s definition of the actual–harm level severity 

requirements at least sometimes contributed to understatement in their 

states. CMS’s guidance for determining actual harm states, “this does not 

include a deficient practice that only could or has caused limited 

consequence to the resident.”36 State agency directors from several states 

found this language confusing, including one director who said it is 

unclear whether conditions like dehydration that are reversed in the 

hospital should be cited as actual harm. As we reported in 2003, CMS 

officials acknowledged that the language linking actual harm to practices 

that have “limited consequences” for a resident has created confusion; 

however, the agency has not changed or revised this language.37

State agency directors and surveyors indicated that CMS’s written 

guidance for certain federal nursing home quality standards could be 

improved and that revised investigative protocols were helpful.38

Specifically, 11 state agency directors reported that CMS guidance on 

quality standards related to abuse could be improved. State agency 

directors commented that the guidance for certain quality standards was 

too long, with the guidance for two standards being over 50 pages long. 

One state agency director also noted that overly complex guidance will 

lead to an unmanageable survey process. Surveyors’ concerns about the 

sufficiency of CMS’s guidance varied for different quality standards (see 

table 3). For instance, 21 percent of surveyors nationwide reported that 

CMS guidance on pain management was not sufficient to identify 

deficiencies, whereas only 5 percent reported that guidance on pressure 

ulcers was not sufficient. Our analysis found that fewer surveyors had 

concerns with the guidance on quality standards revised through CMS’s 

 

36Guidance for determining actual–harm level deficiencies is provided in Section IV, 

Appendix P of the SOM. 

37See GAO-03-561. 

38Federal nursing home quality standards detail requirements for the delivery of care, 

resident outcomes, and facility conditions. State survey teams use these federal quality 

standards to assess compliance during state nursing home surveys. 

Page 17 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
guidance update initiative.39 For example, the guidance on pressure ulcers 

was revised in 2004 and the guidance on accidents was revised in 2007; 

these topics ranked last among the areas of concern.40 Furthermore, state 

agency directors from several states commented on the usefulness of 

CMS’s revised investigative protocols for federal quality standards. 

Table 3: Percentage of Surveyors Reporting That Guidance for Certain Federal 

Quality Standards Was Not Sufficient to Identify Deficiencies 

Federal quality standard (number) 

Percentage reporting 

guidance on quality 

standard was not 

sufficient

Pain Management (multiple F-tags)a, b 21

Quality of Care / Provide Necessary Care and Services for 

Highest Practicable Well-Being (F-309)c 20

Range of Motion Mobility Treatment (F-318) 14

Accuracy of Resident Assessment (F-278) 13

Comprehensive Care Plans (F-279) 12

Sanitary Conditions for Food (F-371)b12

Abuse (F-223 through F-226)d11

Maintains Body Weight (F-325)b11

Physical Restraints (F-221) 11

Unnecessary Drugs (F-329)e11

Resident Participation in Planning Care and Treatment 

(F-280) 10

Accidents (F-323)e8

Pressure Ulcers (F-314)e 5

Source: GAO. 

a

CMS consolidated guidance on pain management into F-309 on March 31, 2009. 

b

CMS revised guidance after our questionnaire of nursing home surveyors was administered in May 

2008. 

 

39In October 2000, CMS began revising investigative protocols for assessing specific 

deficiencies. The intent of this initiative is to enable surveyors to better (1) identify specific 

deficiencies, (2) investigate whether a deficiency is the result of poor care, and 

(3) document the level of harm resulting from a home’s identified deficient care practices. 

See GAO-03-561. 

40Our questionnaire included 13 topics of the approximately 200 federal quality standards. 

Seven of these were taken from the Quality of Care category of federal quality standards, 

the others originated from different categories such as Resident Assessment and Dietary 

Services. See GAO-08-517. 

Page 18 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
c

CMS added guidance to F-309 for residents receiving hospice or dialysis services on April 10, 2009. 

d

CMS plans to begin revising guidance in Fall 2009. 

e

CMS revised guidance before our questionnaire of nursing home surveyors was administered in May 

2008. 

Another weakness associated with the federal survey process was the 

potential for surveys to be predictable based solely on their timing.41

Eighteen percent of state agency directors reported that survey 

predictability or other advance notice of inspections at least sometimes 

contributed to understatement in their states. We analyzed state agencies’ 

most-recent nursing home surveys and found that 29 percent of these 

surveys could be considered predictable due to their timing. We previously 

reported that survey predictability could contribute to understatement 

because it gives nursing homes the opportunity to conceal deficiencies if 

they choose to do so.42 CMS officials previously stated that reducing 

survey predictability could require increased funding because more 

surveys would need to be conducted within 9 months of the previous 

survey.43 However, CMS noted that state agencies are not funded to 

conduct any surveys within 9 months of the last standard survey. 

 

New Survey Methodology’s 

Effect on Understatement 

Inconclusive 

There was no consensus among the eight state agency directors who had 

started implementing the QIS as of November 2008 about how the new 

survey methodology would affect understatement.44 Three directors 

41Beginning January 1, 1999, CMS directed states to avoid scheduling surveys for the same 

month of the year as a nursing home’s previous survey. However, surveys can also be 

considered predictable if occurring at a time other than near the 1-year anniversary or 

15-month maximum date. For example, nursing home operators could be alerted when the 

state agency is surveying a facility in a nearby area if all the facilities in that area were 

surveyed at about the same time. 

42In 2003, we found that 34 percent of nursing home surveys were predictable. See 

GAO-03-561. 

43According to CMS, states consider 9 months to 15 months from the last standard survey 

as the window for completing standard surveys because it yields a 12-month average. Thus, 

to maintain an average survey interval of 12 months, given that some facilities are not 

surveyed until near or after 15 months, more surveys would need to occur within 9 months 

of the last standard survey. See GAO-06-117. 

44At the time of our survey, the QIS methodology was being implemented in eight state 

agencies: Connecticut, Florida, Kansas, Louisiana, Minnesota, New Mexico, North Carolina, 

and Ohio. According to a CMS official, Connecticut is the only one of these states that has 

implemented the QIS statewide. As of May 2008, CMS projected that the QIS would be fully 

implemented in all states in 2014. 

Page 19 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
reported that the QIS was likely to reduce understatement; three directors 

reported that it was not likely to reduce understatement; and two directors 

were unsure or had no opinion (see fig. 2). However, all eight directors 

reported that the new QIS methodology was likely to improve survey 

consistency both within and across states. In addition, five of these 

directors reported that the new QIS methodology was likely to improve 

survey quality. Five of the eight directors also indicated that the QIS 

required more time than the traditional survey methodology. 

Figure 2: Eight State Agency Director Responses on Five Questions Related to the 

QIS 

0

1

2

3

4

5

6

7

8

Not sure / No opinion

No

Yes

Reduce 

understatement

Require

more

time to

complete

Improve

quality

of nursing

home

surveys

Improve

consistency

across

states

Improve

consistency

within

this state

8 8

2

1

5

3

5

2

3

3

Number of responses

Source: GAO.

Improve

consistency 

within 

this state

Improve 

consistency 

across

states

Improve 

quality of 

nursing

home 

surveys

Require 

more 

time to 

complete

Reduce 

understatement

Page 20 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
CMS funded an independent evaluation of the QIS, which was completed 

by a contractor in December 2007.45 The evaluation assessed the 

effectiveness of the new methodology by studying (1) its effect on 

accuracy of surveys, (2) documentation of deficiencies, (3) time required 

to complete survey activities, (4) number of deficiencies cited, and 

(5) surveyor efficiency. The evaluation did not draw a firm conclusion 

about the overall effectiveness of the QIS as measured through these five 

areas. For instance, the QIS methodology was associated with an increase 

in the total number of deficiencies cited, including an increase in the 

number of G-level deficiencies and the number of quality standard areas 

cited. However, the evaluation did not find that the QIS methodology 

increased survey accuracy, noting that QIS and traditional survey samples 

were comparable in overall quality and in the frequency of standards cited 

for deficiencies with either a pattern or widespread scope.46 The results 

suggested that more deficiencies with higher scope could have been cited 

for both the QIS and traditional surveys. Similarly, there was no evidence 

that the QIS resulted in higher-quality documentation or improved 

surveyor efficiency. Although five state agency directors reported that the 

QIS required more time to complete than the traditional methodology, the 

evaluation found some evidence of a learning curve, suggesting that 

surveyors were able to complete surveys faster as they became familiar 

with the new process. The evaluation generated a number of 

recommendations for improving the QIS that are consistent with reducing 

understatement, such as improving the specificity and usability of 

investigative protocols and evaluating how well the new methodology 

accurately identifies the areas in which there are potential quality 

problems. Since the evaluation did not find improved accuracy, CMS 

concluded that non-QIS factors, including survey guidance clarification 

and surveyor training and supervision, would help improve survey 

accuracy. Additionally, CMS concluded that future QIS development 

efforts should concentrate on improving survey consistency and giving 

supervisors more tools to assess the performance of surveyor teams. 

 

45The QIS evaluation was conducted to answer questions about accuracy, documentation, 

changes in the number and types of deficiencies, and whether the QIS process is more 

efficient. HHS, CMS, Evaluation of the Quality Indicator Survey (QIS), Final Report 

(December 2007), http//:www.cms.hhs.gov/CertificationandComplianc/Downloads/ 

QISExecSummary.pdf (accessed July 17, 2009). 

46Scope refers to the number of residents potentially or actually affected and has three 

levels—isolated, pattern, and widespread. A pattern scope refers to deficiencies at the B, E, 

H, and K levels and a widespread scope refers to deficiencies at the C, F, I, and L levels. 

Page 21 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Ten state agency directors that had not yet started implementing the QIS 

responded to our questionnaire with concerns about the cost associated 

with implementing the new methodology, including the resources required 

to train staff and obtain new equipment.47 Of these 10 directors, 3 also 

expressed concerns that allotting staff time for QIS implementation would 

prevent the agency from completing mandatory survey activities. 

Workforce shortages and training inadequacies affected states’ ability to 

complete thorough surveys, contributing to understatement of nursing 

home deficiencies. Responses to our questionnaires indicated that states 

experienced workforce shortages or were attempting to accomplish their 

workload with a high percentage of inexperienced surveyors. In states 

with fewer staff to do the work, time frames were compressed. The 

increased workload burden may have had an effect on the thoroughness of 

surveys in those states and surveyors’ ability to attend training. The 

frequent hiring of new surveyors to address workforce shortages also 

burdened states’ surveyor training programs. Surveyors, state agency 

directors, and state performance on federal observational surveys 

indicated that inadequacies in initial and ongoing training may have 

compromised survey accuracy in high-understatement states. 

Workforce Shortages 

and Training 

Inadequacies May 

Contribute to 

Understatement 

Workforce Shortages 

Sometimes Contributed to 

Understatement 

Although a small percentage of state agency directors reported that 

workforce shortages always or frequently contributed to the 

understatement of nursing home deficiencies in their states, 36 percent 

indicated that workforce shortages sometimes contributed to 

understatement (see table 4). In many states, workforce shortages resulted 

in a greater reliance on inexperienced surveyors. According to state 

agency directors and surveyors, this collateral effect—inexperienced 

surveyors—also may have contributed to understatement. States also 

expressed concern about completing their workload, which appeared to 

be, in part, an outgrowth of workforce shortages and use of inexperienced 

surveyors. 

 

47We asked all 42 directors who had not participated in the QIS to provide their opinions on 

the new methodology; we received comments from 18 of the 42 directors. 

Page 22 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Table 4: State Agency Directors’ Responses to Questions about Surveyor 

Workforce Issues 

Percentage of state agency directors’ responses 

How frequently do the 

following issues contribute to 

understatement in this state 

survey agency? Always/frequently Sometimes Infrequently/never

Inadequate number of staff to 

complete thorough surveys 6 36 58

Inadequate time to complete 

thorough surveys 8 38 54

Reluctance to cite serious 

deficiencies because of workload 

burden 8 10 86

Inexperienced surveyors not yet 

comfortable with job 

responsibilities 16 48 34

Source: GAO. 

Workforce Shortages. Since 2003, we have reported that states have 

experienced pervasive workforce shortages, and responses to our 

questionnaires indicate that shortages continue to affect states.48 Seventytwo percent of state agency directors reported that they always or 

frequently had a surveyor workforce shortage, and another 16 said it 

occurred sometimes. The average vacancy rate for surveyors was 

14 percent, and one-fourth of states had a vacancy rate of higher than 

19 percent (see table 5).49 Among the 49 reporting states, the vacancy rate 

ranged from a maximum of 72 percent in Alabama to 0 percent in Nevada, 

Rhode Island, Vermont, and Utah. The workforce shortages have stemmed 

mostly from the preference to employ RNs as surveyors in state survey 

agencies, with half of reporting states employing RNs as more than 

75 percent of their surveyor workforce.50 In the past, states have claimed 

that they had difficulty matching RN salaries offered by the private sector, 

and this hampered the hiring and retention of RNs. The Virginia state 

agency director commented during an interview that the nursing home 

industry values individuals who have passed CMS’s SMQT and hires its 

surveyors after they are trained and certified by CMS. Virginia and others 

 

48See GAO-03-561. 

49Virginia did not provide the information needed to compute a vacancy rate. 

50Michigan and Illinois did not provide this information. 

Page 23 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
also identified the stress of the job—regular travel, time pressures to 

complete the workload, and the regulatory environment—as a challenge to 

retaining staff. Previously, we reported that workforce instability arising 

from noncompetitive RN surveyor salaries and hiring freezes affected 

states’ abilities to complete their survey workload or resulted in the hiring 

of less-qualified staff.51 Most recently, the poor economy has further 

constrained state budgets for surveyors. For example, to address its 

budget shortfall in 2009, California will furlough its state employees 

including surveyors for 2 days every month from February 2009 through 

June 2010.52 An additional 11 states also reported furloughs for 2009, and 

13 are considering furloughs, salary reductions, or layoffs or will employ 

such measures in the future. 

Table 5: State Survey Agency Vacancy Rates and Percentage of State Surveyors 

with Less Than 2 Years’ Experience 

Percentage of state agency directors’ responses 

All states

Lowunderstatement 

states

Highunderstatement 

states

Vacancy ratea14 12 24

Surveyors with less than 2 years’ 

experienceb

30 25 38

Source: GAO. 

a

Virginia did not provide the information needed to compute a vacancy rate and it was not a high- or 

low-understatement state. 

b

Seven states did not report the number of surveyors with less than 2 years of experience. Among the 

high- and low-understatement states, only West Virginia, a low-understatement state, did not report 

this information. 

Inexperienced Surveyors. Many states are attempting to accomplish 

their workload with a larger share of inexperienced surveyors, and state 

agency directors sometimes linked this reliance on inexperienced staff to 

the understatement of nursing home deficiencies. On average, 30 percent 

of surveyors had less than 2 years’ experience (see table 5); however the 

percentage of inexperienced surveyors ranged from 10 to 82 percent 

Surveyor Quotation about Inexperienced 

Staff

“I have been in this department for just over 

3 years, and I still do not feel comfortable 

with the process. I could personally use a

mentor to ensure a thorough understanding 

of the process. I don’t feel as if I can 

accurately identify deficiencies with the 

short amount of time given the survey teams

to conduct surveys. I feel as if I overlook 

things due to trying to meet survey length 

time frames.”

 

51See GAO-09-64. 

52This information was last updated in June 2009 before the governor of California signed 

the state’s budget revisions. 

Page 24 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
across states who reported this information.53 Among state agency 

directors, 16 percent indicated that inexperienced surveyors always or 

frequently contributed to understatement, while another 48 percent 

indicated that surveyor inexperience sometimes contributed to 

understatement in their states. In response to our questionnaires, 

26 percent of surveyors indicated that survey teams always or frequently 

had too many inexperienced surveyors and another 33 percent indicated 

that sometimes survey teams had too many inexperienced surveyors (see 

table 6). Half or more of all surveyors in six states—Alabama, Alaska, 

Arizona, Idaho, New Mexico, and Utah—reported that there were always 

or frequently too many new surveyors who were not yet comfortable with 

their job responsibilities. For example, 79 percent of surveyors in Arizona 

reported that too many new surveyors were not comfortable with their job 

responsibilities, and the state agency director was among the 34 percent 

who reported that survey teams sometimes had an insufficient number of 

experienced surveyors. Overall, 26 percent of state agency directors 

indicated that the skill level of surveyors has decreased in the last 5 years. 

 

53Seven states did not report the number of surveyors with less than 2 years of 

experience—Illinois, Michigan, Minnesota, Texas, Washington, West Virginia, and 

Wisconsin. 

Page 25 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Table 6: Surveyors’ and State Agency Directors’ Responses to Questions on Workforce Issues 

Percentage of responses 

Respondents Question Always/frequently Sometimes Infrequently/never

Surveyors How frequently have you observed the 

following problems on the nursing home 

surveys that you have worked on? 

Too many new surveyors not yet comfortable 

with job responsibilities 26 33 35

Survey team too small to conduct a thorough 

survey 21 33 42

Survey team not given sufficient time to 

conduct a thorough survey 25 29 42

State Agency 

Directors 

In this state survey agency, how frequently 

do the following occur? 

Survey teams have a sufficient number of 

experienced surveyors 62 34 4

Survey teams are sufficient size to conduct 

thorough surveys 74 18 8

 Survey teams are given sufficient time to 

conduct thorough surveys 78 16 6

Source: GAO. 

In interviews, six state agency directors commented that inexperienced 

surveyors possessed different skills or needed more time than experienced 

surveyors to complete surveys and that workforce shortages resulted in 

constant recruiting, over-burdened experienced surveyors, or the need for 

additional supervision and training resources. Four states—Kentucky, 

Nevada, New Mexico, and Virginia—reported not having enough dedicated 

training staff to handle the initial training for new surveyors. 

Workload. State inability to complete workload was, in part, an 

outgrowth of the workforce shortages and reliance on inexperienced 

surveyors. More than two-thirds of state agency directors reported on our 

questionnaire that staffing posed a problem for completing complaint 

surveys, and more than half reported that staffing posed a problem for 

completing standard or revisit surveys.54 In addition, 46 percent of state 

agency directors reported that time pressures always, frequently, or 

sometimes contributed to understatement in their states. In response to 

Surveyor Quotation about Insufficient 

Time to Complete Surveys

“Frequently G level or I/J level are not cited 

[due to a] lack [of] staff time.”

 

54Revisit surveys are generally conducted in facilities when a G-level or higher deficiency is 

cited by a survey team, to verify that serious deficiencies have been corrected by the home. 

Page 26 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
our questionnaire, 16 percent of surveyors nationwide reported that 

workload burden influenced the citation of deficiencies—including 14 

states with 20 percent or more surveyors reporting the same. More than 50 

percent of surveyors identified insufficient team size or time pressures as 

having an effect on the thoroughness of surveys. Surveyors’ comments 

reiterated these concerns—over 15 percent of surveyors who wrote 

comments complained about the amount of time allotted to complete 

surveys or survey paperwork, and 11 percent indicated that staffing was 

insufficient to complete surveys.55 One state agency director suggested to 

us that CMS establish a national team of surveyors to augment states’ 

when they fell behind on their workload or had staffing shortages. He 

thought the availability of national surveyors could assist states 

experiencing workforce shortages and help ensure state workloads were 

completed. This state had experience with a similar arrangement when it 

hired a national contractor to complete its surveys of Intermediate Care 

Facilities for the Mentally Retarded. 

Training Inadequacies May 

Compromise Survey 

Accuracy 

Surveyors, state agency directors, and state performance on federal 

observational surveys indicated that inadequacies in initial or ongoing 

training may compromise the accuracy of nursing home surveys and lead 

to the understatement of deficiencies. In addition, workload affected 

surveyors’ ability to attend training. 

Initial Surveyor Training. As noted earlier, even though CMS has 

established specific training requirements, including coursework and the 

SMQT certification test, states are responsible for preparing their new 

surveyors for the SMQT. According to CMS, 94 percent of new surveyors 

nationally passed the SMQT test in 2008 and, on average, surveyors 

answered about 77 percent of the questions correctly. These results seem 

to support the state agency directors’ assertions that initial training was 

insufficient and suggest that the bar for passing the test may be set too 

low. Even though we cannot be certain whether the inadequacies are with 

the federal or state components of the training, reported differences 

among states in satisfaction with the initial surveyor training also could 

reflect gaps in state training programs. About 29 percent of surveyors in 

high-understatement states reported that initial training was not sufficient 

 

55In an open-ended question at the end of the questionnaire, 842 surveyors commented on a 

wide range of topics related to surveys. These comments represented about 36 percent of 

all nursing home surveyor respondents; 15 percent of the comments represented about 

6 percent of all respondents. 

Page 27 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
to cite appropriate scope and severity levels, compared with 16 percent of 

surveyors in low-understatement states (see table 7). Similarly, 28 percent 

of surveyors in high-understatement states, compared with 20 percent of 

those in low-understatement states, indicated that initial training was not 

sufficient to identify deficiencies for nursing homes. Further, 18 percent of 

state agency directors linked the occurrence of understatement always, 

frequently, or sometimes with insufficient initial training. From 16 to 

20 percent of state agency directors indicated that initial training was 

insufficient to (1) enable surveyors to identify deficiencies and (2) assign 

the appropriate level of scope and severity. 

Table 7: Responses from Surveyors and State Agency Directors to Key Questions on Training 

Percentage of surveyors’ responses 

All states

Lowunderstatement 

states 

Highunderstatement 

states

Percentage 

of directors’ 

responses

Initial training is not sufficient: 

To ensure surveyors are able to cite appropriate scope and 

severity levels 24 16 29 20

To enable surveyors to identify deficiencies 26 20 28 16

Additional training is needed to: 

Interview nursing home residents 13 9 16 36a

Identify scope and severity levels 26 16 34 56b

Document deficiencies 32 27 35 62

Source: GAO. 

a

Two state agency directors did not respond to this question. 

b

One state agency director did not respond to this question. 

Ongoing Training. Ongoing training programs are the purview of state 

agencies; therefore, differences between states about the sufficiency of 

this training also may point to gaps in the state training programs. On our 

questionnaire, about 34 percent of surveyors in high-understatement states 

indicated a need for additional training on (1) identifying appropriate 

scope and severity levels and (2) documenting deficiencies. This was 

significantly more than those from low-understatement states, which 

indicated less of a need for additional training in these areas—16 and 

27 percent, respectively. Among state agency directors, 10 percent 

attributed understatement always or frequently to insufficient ongoing 

training, while 14 percent indicated that insufficient ongoing training 

sometimes gave rise to understatement. Although 74 percent of state 

agency directors indicated that the state had ongoing annual training 

Page 28 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
requirements, the required number of hours and the type of training varied 

widely by state in 2007. Among the 33 states that provided the required 

amount of annual state training, these hours ranged from 0 to 120 hours 

per year. Meanwhile, 37 states reported one or more type of required 

training: 32 states required surveyors to attend periodic training, 22 

required on-the-job training, 10 required online computerized training, and 

13 states required some other type of training. 

State agency directors indicated that they relied on CMS materials for 

ongoing training of experienced surveyors, yet many reported additional 

training needs and suggested that use of electronic media could make 

continuing education and new guidance more accessible. While 98 percent 

of states indicated that the CMS written guidance materials and resources 

were useful, over 50 percent of all state agency directors identified 

additional training needs in documenting deficiencies, citing deficiencies 

at the appropriate scope and severity level, and applying CMS guidance. 

On federal observational surveys, an average of 17 to 12 percent of survey 

teams in high-understatement states received below-satisfactory ratings 

for Deficiency Determination and General Investigation, respectively—two 

skills critical for preventing understatement. In contrast, an average of 

4 percent of survey teams in low-understatement states received the same 

below-satisfactory scores for both deficiency determination and 

investigative skills. Furthermore, of the 476 surveyors who commented 

about training needs, one-quarter indicated a need for training support 

from either CMS or state agencies; and between 12 to 7 percent of those 

who commented on training needs identified topics such as: documenting 

deficiencies, identifying scope and severity, CMS guidance, and medical 

knowledge.56

Inability to Attend Training. States’ workload requirements and 

workforce shortages affected the surveyors’ ability to attend initial and 

ongoing training. Seven of the eight state agency directors we interviewed 

linked workforce shortages and resource constraints to their state’s ability 

to complete the survey workload or allow staff to participate in training 

courses. One director stated that workload demands compromised 

comprehensive training for new staff, and another reported difficulty 

placing new staff in CMS’s initial training programs. Due to workload 

demands, a third state agency director stated that she could not allow 

 

56The 476 surveyors who responded to this open-ended question about training needs 

constituted about 20 percent of all respondents. 

Page 29 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
experienced staff time away from surveying to attend training courses 

even when staff paid their own way. Five of the seven state agency 

directors suggested that it would be more efficient for training activities to 

be conducted more locally such as in their states or to be available 

through online, video, or other electronic media, and several emphasized 

the need to reduce or eliminate travel for training. Although four states 

also expressed a preference for interactive training opportunities, one 

state believed that technological solutions could allow for more accessible 

training that was also interactive. 

State supervisory reviews, which generally occurred more frequently on 

higher-level deficiencies, often are not designed to identify understated 

deficiencies. State agencies generally conducted more supervisory reviews 

on surveys with higher-level deficiencies, compared to surveys with 

deficiencies at the potential for more than minimal harm level (D through 

F)—the deficiencies most likely to be understated. While focus on higherlevel deficiencies enables states to be certain that such deficiencies are 

well documented, not reviewing surveys with deficiencies at lower levels 

represents a missed opportunity to ensure that all serious deficiencies are 

cited. State surveyors who reported having frequent changes made to their 

survey reports during supervisory reviews also more often reported they 

were burdened by other factors contributing to understatement, such as 

workforce shortages and survey methodology weaknesses. 

State Supervisory 

Reviews Often Are 

Not Designed to 

Identify 

Understatement 

Supervisory Reviews Often 

Focused on Higher-Level 

Deficiencies 

According to state agency directors’ responses to our questionnaire, states 

generally focused supervisory review on surveys with higher-level 

deficiencies, rather than on the surveys with deficiencies at the potential 

for more than minimal-harm level (D through F)—the deficiencies most 

likely to be understated. During supervisory reviews, either direct-line 

supervisors or central state agency staff may review draft survey records.57

On average, surveys at the D through F level underwent about two steps of 

review, while surveys with deficiencies at the immediate jeopardy level 

 

57For supervisory review processes, we defined direct-line supervisors as including survey 

team leaders, direct supervisors, and supervisors at district or regional offices. Central 

state agency staff was defined as including quality assurance teams, legal counsel, state 

training coordinators, and compliance specialists. 

Page 30 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
(J through L) went through three steps.58 For example, Washington 

reviews its surveys using either a two-step review that includes survey 

team and field manager reviews or a three-step process that includes both 

these reviews and an additional review by central state agency staff for 

serious deficiencies. As a result, central state agency staff in Washington 

do not review deficiencies below the level of actual harm. In addition we 

found that five states—Alaska, Hawaii, Illinois, Nebraska, and Nevada—

did not review all surveys with deficiencies at the D through F levels. In 

fact, Hawaii did not report supervisory review of deficiencies at any level 

(see fig. 3).59 It is difficult to know if additional supervisory reviews—the 

second, third, or fourth review—help make survey records more accurate 

and less likely to be understated, or if these reviews result in more 

frequent changes to deficiency citations. However, if deficiency citations 

with the potential for more than minimal-harm level (D through F) are not 

reviewed, states miss the opportunity to assess whether these deficiencies 

warrant a higher-level citation, for example, the level of actual harm or 

immediate jeopardy. 

 

58These review steps could be done at the direct-line supervisor level or by central state 

agency staff. The difference between supervisory review levels for surveys with J through L 

citations and those for surveys with D through F citations was significant at the 1 percent 

level. 

59Forty states review a sample of all draft surveys. Such reviews may include additional 

examination of surveys with deficiencies at either the D through F or J through L levels. 

Page 31 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Figure 3: Number of State Supervisory Reviews at the Potential for More than 

Minimal Harm (D-F) and Immediate Jeopardy Levels (J-L) 

Source: GAO. Map: Copyright © Corel Corp. All rights reserved.

0 reviews

1 review

2 reviews

3 reviews

4 reviews

5+ reviews

States with D-F supervisory reviews

PA: 4

OR: 2

ID: 1

MT: 2

WY: 1

UT: 1

AZ: 3 NM: 3

CO: 4

ND: 3

SD: 2

TX: 4

KS: 2

OK: 2

MN:

2

IA:

4

MO:

3

AR:

2

LA:

1

MS:

2

IN:

1

KY: 3

TN: 3

AL:

4

FL:

3

GA:

1

SC: 2

NC: 1

VA: 1

OH:

3

MI:

1

CA: 2

WA: 2

WI:

3

NY:

3

ME:

4

WV:

1

VT: 2

NH: 3

MA: 1

RI: 4

CT: 4

NJ: 2

DE: 2

MD: 2

States with J-L supervisory reviews

OR: 3

ID: 1

MT: 2

WY: 1

UT: 2

AZ: 3 NM: 2

CO: 4

ND: 3

SD: 2

KS: 3

OK: 2

IA:

4

MO:

3

AR:

3

LA:

3

IL:

4

MS:

2

KY: 4

AL:

4

GA:

1

SC: 3

NC: 4

VA: 1

OH:

4

MI:

3

CA: 3

WA: 2 ME:

3

WV:

1

AK: 1

VT: 2

NH: 3

MA: 2

RI: 4

CT: 4

NJ: 3

DE: 2

MD: 3

NV: 0 NE: 0

IL:

0

HI: 0

AK: 0

NV: 0 NE: 1

HI: 0

PA: 5

TX: 5

MN:

5

IN:

5

TN: 6

FL:

5

WI:

6

NY:

5

Note: Hawaii did not report conducting supervisory reviews. Forty states review a sample of all draft 

surveys. Such reviews may include additional examination of surveys with deficiencies at either the D 

through F or J through L levels. 

Page 32 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Because a majority of states are organized into geographically-based 

district or regional offices, review by central state agency staff, 

particularly quality assurance staff, is critical to help ensure consistency 

and detect understatement. However, 26 states reported that no central 

state agency staff reviews were conducted for surveys with deficiencies at 

the potential for more than minimal harm (D through F). These results are 

consistent with a finding from our 2003 report—that half of the 16 states 

we contacted for that report did not have a quality assurance process to 

help ensure that the scope and severity of less serious deficiencies were 

not understated.60

According to most of the eight state officials we interviewed, supervisory 

reviews commonly focused on documentation principles or evidentiary 

support, not on reducing understatement. For example, all eight states 

used supervisory reviews to assess the accuracy and strength of the 

evidence surveyors used to support deficiency citations, and three of these 

states reported that they emphasized reviewing survey records for 

documentation principles. Furthermore, seven out of eight states indicated 

that surveys with serious deficiencies—those that may be subject to 

enforcement proceedings—went through additional steps of review 

compared with surveys citing deficiencies with the potential for more than 

minimal harm (D through F). 

Reports of Changes to 

Deficiencies during 

Supervisory Reviews May 

Be Related to Other 

Factors That Contribute to 

Understatement 

Surveyor reports of changes to deficiency citations during supervisory 

reviews may be related to other factors the state is experiencing that also 

contribute to understatement, such as workforce shortages and survey 

methodology weaknesses. 

Changes to Deficiencies. Fifty-four percent of surveyors nationwide 

reported on our questionnaire that supervisors at least sometimes 

removed the deficiency that was cited, and 53 percent of surveyors noted 

that supervisors at least sometimes changed the scope and severity level 

of cited deficiencies. Of the surveyors, who reported that supervisors 

sometimes removed deficiencies, 13 percent reported that supervisors 

always or frequently removed deficiencies—including 12 states with 20 

Surveyor Quotation about Supervisory 

Review

“We have problems at times with nonclinical 

supervisors and district managers, [and with] 

a past branch chief not understanding clinical 

issues and thus not supporting surveyor 

findings. We’ve had deficiencies tossed out 

for surveys and IDR deficiencies deleted, not 

for lack of documentation, but for lack of 

understanding of the issues involved.”

 

60See GA0-03-561. 

Page 33 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
percent or more of their surveyors reporting that deficiencies were 

removed. 

Surveyor reports of changes in deficiency citations alone make it difficult 

to know whether the original deficiency citation or the supervisor’s 

revised citation was a more accurate reflection of a nursing home’s quality 

of care. Additionally, there are many reasons that survey records might be 

changed during supervisory review. When a surveyor fails to provide 

sufficient evidence for deficient practices, it may be difficult to tell 

whether the deficiency was not appropriately cited or if the surveyor did 

not collect all the available evidence. Kentucky’s state agency director 

offered one possible explanation—that changes to surveys often reflected 

a need for more support for the deficiencies cited, such as additional 

evidence from observations. Nevada’s state agency director stated that 

changes to survey records occurred when it was often too late to gather 

more evidence in support of deficiencies. 

Surveyors who reported that supervisors frequently changed deficiencies 

also more often reported experiencing other factors that contribute to 

understatement. We found associations between surveyor reports of 

changes to deficiencies and workforce shortages and survey methodology 

weaknesses. 

• Workforce shortages. Surveyors reporting workforce shortages, 

including survey teams with too many new surveyors and survey teams 

that were either too small or given insufficient time to conduct thorough 

surveys, more often also reported that supervisors frequently removed 

deficiencies or changed the scope and severity of deficiency citations 

during supervisory reviews. 

• Survey methodology weaknesses. Surveyors reporting weaknesses in 

the current survey methodology more often also reported that supervisors 

frequently removed deficiencies or changed the scope and severity of 

deficiency citations during supervisory reviews. 

Supervisory Reviews and Understatement. In certain cases, survey 

agency directors and state performance on federal comparative surveys 

linked supervisory reviews to understatement. Twenty-two percent of 

state agency directors attributed inadequate supervisory review processes 

Page 34 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
to understatement in their states at least sometimes.61 In addition, 

significant differences existed between zero-understatement states and all 

other states, including high-understatement states, in the percentage of 

surveyors reporting frequent changes to citations during supervisory 

reviews. Only about 4 percent of surveyors in zero-understatement states 

reported that citations were always or frequently removed or changed and 

that the scope and severity cited were changed, while about 12 percent of 

surveyors in all other states indicated the same (see table 8). To address 

concerns with supervisory reviews, Nevada recently reduced its process 

from two steps to a single step review by survey team supervisors to 

address surveyor complaints about changes made during supervisory 

reviews. 

Table 8: Percentage of Surveyors Reporting Changes in Deficiency Citations during 

Supervisory Review 

Percentage of surveyors’ responses

Zero-understatement 

states

All other 

states

Surveyors reporting that: 

Supervisors always or frequently remove or 

change deficiency cited 5 13

Supervisors always or frequently changed 

the scope and severity cited 4 12

Source: GAO. 

In addition, we observed a relationship between state practices to notify 

surveyors of changes made during supervisory reviews and surveyor 

reports of deficiency removal and explanation of changes. Specifically, 

compared to surveyors in states that require supervisors to notify 

surveyors of changes made during supervisory review, surveyors from 

states where no notification is required reported more often that 

supervisors removed deficiencies and less often that explanations for 

these changes, when given, were reasonable. 

Similarly, we found an association between the frequency of explained and 

reasonable changes and zero-understatement states, possibly 

demonstrating the positive effect of practices to notify surveyors of 

 

61Seventy-four percent of state agency directors indicated that inadequate supervisory 

review processes infrequently or never contributed to understatement in their state, and 

4 percent of state agency directors were unsure or had no opinion on this topic. 

Page 35 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
changes made during supervisory reviews. Nursing home surveyors from 

zero-understatement states more often reported that supervisors 

explained changes and that their explanations seemed reasonable 

compared to surveyors in all other states. State agency directors in 

Massachusetts and New Mexico stated that explanations of changes to the 

survey record provided opportunities for one-on-one feedback to 

surveyors and discussions about deficiencies being removed. 

Nursing home surveyors and state agency directors in a minority of states 

told us that in isolated cases issues such as a state agency practice of 

noncitation, external pressure from the nursing home industry, and an 

unbalanced IDR process may have led to the understatement of 

deficiencies. In a few states, surveyors more often identified problems 

with noncitation practices and IDR processes compared to state agency 

directors. Yet, a few state agency directors acknowledged either 

noncitation practices, external pressure, or an IDR process that favored 

nursing home operators over resident welfare. Although not all the issues 

raised by surveyors were corroborated by the state agency directors in 

their states, surveyor reports clustered in a few states gives credence to 

the notion that such conditions may lead to understatement. 

State Agency 

Practices and 

External Pressure 

May Compromise 

Survey Accuracy and 

Lead to 

Understatement in a 

Few States 

Surveyors Reported 

Noncitation Practices in a 

Small Number of States 

Approximately 20 percent of surveyors nationwide and over 40 percent of 

surveyors in five states reported that their state agency had at least one of 

the following noncitation practices: (1) not citing certain deficiencies, 

(2) not citing deficiencies above a certain scope and severity level, and 

(3) allowing nursing homes to correct deficiencies without receiving a 

citation (see fig. 4). Only four state agency directors acknowledged the 

existence of such practices in their states on our questionnaire and only 

one of these directors was from the five states most often identified by 

surveyors. One of these directors commented on our questionnaire that 

one of these practices occurs only in “rare individual cases.” Another 

director commented that a particular federal quality standard is not related 

to patient outcome and therefore should not be cited above a Level F. 

According to CMS protocols, when noncompliance with a federal 

requirement has been identified, the state agency should cite all 

Surveyor Quotation about State 

Noncitation Practices

“I have been criticized by my supervisor on 

more than one occasion for citing too many 

deficiencies at facilities [that] have an 

ongoing history of repeat tags from survey 

to survey and many complaints surveys

between annual surveys. My supervisor 

states that citing too many deficiencies

‘confuses’ the facility and creates a

‘hostile’ environment.”

Page 36 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
deficiencies associated with the noncompliance. CMS regional officials we 

interviewed were not aware of any current statewide noncitation 

practices.62

 

62CMS previously identified the existence of a potential noncitation practice in one state 

that had an unusually high number of homes with no deficiencies on their standard 

surveys. Contrary to federal guidance, surveyors in that state were not citing all identified 

deficiencies but rather brought them to the homes’ attention with the expectation that the 

deficiencies would be corrected. See GAO-03-561. 

Page 37 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Figure 4: Percentage of Surveyors in Each State Reporting at Least One Noncitation Practice 

Source: GAO. Map: Copyright © Corel Corp. All rights reserved.

WY

UT

AZ

CO

SD

TX

KS

MN

IA

MO

AR

LA

IL

MS

IN

KY

FL

SC

NC

VA

OH

MA

RI MI

CA

WA

WI NY

MT

OK

TN

AL GA

CT

NJ

DE

MD

DC

No data

0–10

11–20

21–30

31–40

41+

OR

ID

ND ME

WV

VT

HI

AK

PAa

NV

NM

NE

NH

a

Responses from Pennsylvania surveyors could not be included because the state agency directed 

nursing home surveyors not to respond to our questionnaire. 

Not citing certain deficiencies. Nationally, 9 percent of surveyors 

reported a state agency practice that surveyors not cite certain 

deficiencies. However, in four states over 30 percent of surveyors reported 

their state agency had this noncitation practice, including over 60 percent 

of New Mexico surveyors. In some cases, surveyors reported receiving 

Page 38 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
direct instructions from supervisors not to cite certain deficiencies. In 

other cases, surveyors’ reports of noncitation practices may have been 

based on their interpretation of certain management practices. For 

instance, surveyors commented that some state agency practices—such as 

providing inadequate time to observe and document deficiencies or 

frequently deleting deficiency citations during supervisory review—

seemed like implicit or indirect leadership from the agency to avoid citing 

deficiencies. One state agency director we interviewed agreed that 

surveyors may report the existence of noncitation practices when their 

citations are changed during supervisory review. This official told us that 

when surveyors’ deficiencies are deleted or downgraded, the surveyors 

may choose not to cite similar deficiencies in the future because they 

perceive being overruled as an implicit state directive not to cite those 

deficiencies. 

Not citing deficiencies above a certain scope and severity level. 

Although nationwide less than 8 percent of surveyors reported a state 

agency practice that surveyors not cite deficiencies above a certain scope 

and severity level, in two states over 25 percent of surveyors reported that 

their state agency used this type of noncitation practice. One reason state 

agencies might use this noncitation practice could be to help manage the 

agency’s workload. In particular, citing deficiencies at a lower scope and 

severity might help the agency avoid additional work associated with 

citing higher-level deficiencies, such as survey revisits or IDR.63 In one of 

the two states mentioned above, 54 percent of surveyors indicated that the 

workload burden influenced their citations. Additionally, as we described 

earlier, 16 percent of surveyors nationwide indicated that workload 

burden influenced the citation of deficiencies and more than half of state 

agency directors (including those from the two states mentioned above) 

responded that staffing was not sufficient to complete revisit surveys. 

While our questionnaire focused on not citing deficiencies above a certain 

scope and severity level, a few surveyors commented on being 

discouraged from citing lower-level deficiencies due to time pressures to 

complete surveys. Agency officials in two states told us that surveyors may 

miss some deficiencies due to limited survey time and resources. 

Allowing nursing homes to correct deficiencies without citing them 

on the survey record. Nationwide, approximately 12 percent of 

 

63CMS requires on-site revisits for any noncompliance identified at level F (with a finding of 

substandard quality of care) or any level higher than F. 

Page 39 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
surveyors reported this type of noncitation practice. However, in five 

states, at least 30 percent of surveyors reported their state agency allowed 

nursing homes to correct deficiencies without citing those deficiencies on 

the official survey record. Comments from surveyors suggest that state 

agencies may use this type of practice to avoid actions that nursing homes 

or the industry would dispute or interpret as excessive. Similarly, several 

surveyors commented that they were instructed to cite only one deficiency 

for a single type of negative outcome, even when more than one problem 

existed. However, CMS guidance requires state agencies to cite all 

problems that lead to a negative outcome. The decrease in G-level 

citations that occurred after CMS implemented the double G immediate 

sanctions policy in January 2000 also suggests that some states may have 

avoided citing deficiencies that would result in enforcement actions for 

the nursing home.64 The total number of G-level deficiency citations 

nationwide dropped from approximately 10,000 in 1999 to 7,700 in 2000.65

State Agency Directors in a 

Few States Reported That 

External Pressure 

Contributed to 

Understatement 

State agency directors from 12 states reported experiencing external 

pressure from at least one of the following stakeholder groups: (1) the 

nursing home surveyed, (2) the nursing home industry, or (3) state or 

federal legislators. Examples of such external pressure include pressure to 

reduce federal or state nursing home regulation or to delete specific 

deficiencies cited by the state agency. Of the 12 state agency directors, 

7 reported that external pressure at least sometimes contributed to the 

understatement of deficiencies in their states, while the other 5 indicated 

that it infrequently or never contributed to understatement. 

Surveyor Quotation about External 

Pressure

“The larger corporations often pressure our 

[state agency] Central Office to change and 

delete citations. Our Central Office changes

not only wording but content and intent of the 

citation, when they were not on site. There is

a great deal of political push and pull—the 

interference from State Senators and 

Representatives protecting their re-electability 

and not the rights of the residents (who don’t 

vote).”

Adversarial attitude toward nursing home surveys. Two states we 

interviewed—State A and State B—commented on the adversarial attitude 

that industry and legislative representatives had toward nursing home 

surveys at times.66 For instance, state agency officials from State A told us 

 

64Effective January 2000, CMS expanded its immediate sanctions policy, requiring referral 

of homes found to have harmed one or a small number of residents (G-level deficiencies) 

on successive routine surveys or intervening complaint investigations. 

65CMS officials previously acknowledged that the double G policy may have had an 

unintended negative effect on the rate of deficiency citations. See GAO, Nursing Homes: 

Efforts to Strengthen Federal Enforcement Have Not Deterred Some Homes from 

Repeatedly Harming Residents, GAO-07-241 (Washington, D.C.: Mar. 26, 2007). 

66In this section, we refer to two states we interviewed as State A and State B to maintain 

confidentiality for the officials from these state agencies. The corresponding regional 

offices are referred to as regional office responsible for State A or State B, respectively. 

Page 40 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
that the state nursing home association organized several forums to garner 

public and legislative support for curtailing state regulation of facilities. 

According to officials in this state, the influential industry groups 

threatened to request legislation to move the state agency to a different 

department and to deny the confirmation of the director’s gubernatorial 

appointment if the citations of G level or higher deficiencies increased. 

CMS regional office officials responsible for State A told us that the state 

may be experiencing more intense external pressure this year given the 

current economy, because providers have greater concerns about the 

possible financial implications of deficiency citations—fines or increased 

insurance rates. 

Similarly, officials from State B told us that when facilities are close to 

termination, the state agency receives phone calls from state delegates 

questioning the agency’s survey results. Officials from State B also told us 

that the Governor’s office instructed the state agency not to recommend 

facilities for enforcement actions. Officials from the CMS regional office 

responsible for State B told us that this situation was not problematic 

because CMS was ultimately responsible for determining enforcement 

actions based on deficiency citations. However, this regional office’s 

statement is inconsistent with (1) language in the SOM that calls for 

states to recommend enforcement actions to the regional office, and 

(2) assertions from the regional office responsible for State A that it 

infrequently disagrees with state recommendations for sanctions. A third 

state agency director commented that the agency had been called before 

state legislative committees in 2007-2008 to defend deficiency citations 

that led to the termination of facilities. A fourth state agency director also 

commented on our questionnaire that legislators had pressured the state 

agency on behalf of nursing homes to get citations reduced or eliminated 

and prevent enforcement actions for the facilities. In addition, a few 

surveyors commented that at times when nursing homes were unhappy 

with their survey results the homes or their state legislators would ask 

state agency management to remove the citations from the survey record, 

resulting in the deletion or downgrading of deficiencies. Further, 

comments from a few surveyors indicated that they may steer clear of 

citing deficiencies when they perceive the citation might cause a home to 

complain or exert pressure for changes in the survey record. 

Interference in the survey process. In a few cases, external pressure 

appeared to directly interfere with the nursing home survey process. State 

agency officials from two states—State A and an additional fifth state—

reported that state legislators or industry representatives had appeared onsite during nursing home surveys. Although in some cases the legislators 

Page 41 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
just observed the survey process, officials from these two states explained 

that third parties also have interfered with the process by questioning or 

intimidating surveyors. The state agency director from the fifth state 

commented on our questionnaire that the nursing home industry sent legal 

staff on-site during surveys to interfere with the survey process. Similarly, 

officials from State A told us that during one survey, a home’s lawyer was 

on-site reviewing nursing home documentation before surveyors were 

given access to these documents. Officials from State A also told us that 

state legislators have attended surveys to question surveyors about their 

work and whether state agency executives were coercing them to find 

deficiencies. We discussed this issue with the CMS regional officials 

responsible for State A, who acknowledged that this type of interference 

had occurred. 

States’ need for support from CMS. In the face of significant external 

pressure, officials from States A and B suggested that they need support 

from CMS; however, CMS regional office officials did not always 

acknowledge external pressure reported by the states. This year, State A 

terminated a survey due to significant external pressure from a nursing 

home and requested that the CMS regional office complete the revisit 

survey for them. Six weeks later, the federal team completed the survey 

and found many of the same problems that this state team had previously 

identified before it stopped the survey. Officials from State A suggested 

the need for other support as well, such as creating a federal law that 

would require state agencies to report external pressure and ensure 

whistleblower protections for state officials who report pressure and 

allowing sanctions for inappropriate conduct. CMS officials from the 

regional office responsible for State A stated that external pressure might 

indirectly contribute to understatement by increasing surveyor mistakes 

from the additional stress, workload, focus on documentation, and 

supervisory reviews. Conversely, CMS regional officials did not 

acknowledge that State B experienced external pressure and officials from 

State B thought that CMS should be more consistent in its requirements 

and enforcement actions. 

Unbalanced IDR Processes 

Might Have Contributed to 

Understatement in a Few 

States 

States with unbalanced IDR processes may experience more 

understatement. IDR processes vary across states in structure, volume of 

proceedings, and resulting changes. According to state agency directors’ 

responses to our questionnaire, 16 IDRs were requested per 100 homes in 

Page 42 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
fiscal year 2007, with this number ranging among states from 0 to 57 per 

100 homes.67 For IDRs occurring in fiscal year 2007, 20 percent of disputed 

deficiencies were deleted and 7 percent were downgraded in scope or 

severity, but in four states, at least 40 percent of disputed deficiencies 

were deleted through this process.68 CMS does not provide protocols on 

how states should operate their IDR processes, leaving IDR operations to 

state survey agencies’ discretion. For example, states may choose to 

conduct IDR meetings in writing, by telephone, or through face-to-face 

conferences. State agencies also have the option to involve outside 

entities, including legal representation, in their IDR operations. 

On the basis of responses from surveyors and state agency directors 

clustered in a few states, problems with the IDR processes—such as 

frequent hearings, deficiencies that are frequently deleted or downgraded 

through the IDR process, or outcomes that favor nursing home operators 

over resident welfare—may have contributed to the understatement of 

deficiencies in those states. Although reports of such problems were not 

common—only 16 percent of surveyors nationwide reported on our 

questionnaire that their state’s IDR process favored nursing home 

operators—in four states over 40 percent of surveyors reported that their 

IDR process favored nursing home operators (see fig. 5), including one 

state where a substantial percentage of surveyors identified at least one 

noncitation practice. While only one state agency director reported that 

the IDR process favored nursing home operators, three other directors 

acknowledged that frequent IDR hearings at least sometimes contributed 

to the understatement of deficiencies. For example, in some states 

surveyors may hesitate to cite deficiencies that they believe will be 

disputed by the nursing home. 

Surveyor Quotation about the IDR Process

“The IDR process is inconsistent. Over the 

years we have been on all ends of the 

spectrum—between having involved panels

who have an understanding of the survey 

process versus people who know nothing 

about the process and have no idea how to 

apply the federal regulations. (The latter is

the current make-up.) When we have a panel 

made up of the latter, the word spreads

throughout the state and there is a very large 

increase in requests for IDR. The reason is

that this type of panel tends to delete most 

everything for ‘insufficient evidence’ but 

cannot coherently explain how they came 

[to] that decision.”

 

67Alaska, Hawaii, Illinois, and West Virginia did not report information on the number of 

IDRs. 

68The following states did not report the number of deficiencies deleted or downgraded 

through the IDR process: Alaska, Hawaii, Illinois, Kentucky, Nebraska, New Jersey, New 

Mexico, Vermont, or West Virginia. Maine, Oklahoma, Utah, Washington, and Wyoming 

provided the number of deficiencies deleted but not the number that were downgraded. 

Page 43 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Figure 5: Percentage of Surveyors in Each State Reporting the IDR Process Favored Concerns of Nursing Home Operators 

over Resident Welfare 

Source: GAO. Map: Copyright © Corel Corp. All rights reserved.

VT

NH

CT

No data

0–10

11–20

21–30

31–40

41+

MA

RI

NJ

DE

MD

DC

NV

WY

UT

AZ NM

CO

SD

NE

TX

KS

MN

IA

MO

AR

IL

MS

IN

KY

FL

SC

NC

OH

MI

CA

WA

WI NY

MT

OK

TN

AL GA

HI

AK

OR

ID

ND ME

WV

PAa

LA

VA

a

Responses from Pennsylvania surveyors could not be included because the state agency directed 

nursing home surveyors not to respond to our questionnaire. 

Page 44 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
In isolated cases, a lack of balance with the IDR process appeared to be a 

result of external pressure. In one state, the state agency director reported 

that the nursing home industry sent association representatives to the IDR, 

which increased the contentiousness of the process. In another state, 

officials told us that a large nursing home chain worked with the state 

legislature to set up an alternative to the state IDR process, which has 

been used only by facilities in this chain. Through this alternative appeals 

process, both the state agency and the nursing home have legal 

representation, and compliance decisions are made by an adjudicator. 

According to agency officials in this state, the adjudicators for this 

alternative appeals process do not always have health care backgrounds. 

While CMS gives states the option to allow outside entities to conduct the 

IDR, the states should maintain ultimate responsibility for IDR decisions.69

CMS regional officials stated it would not consider the outcome of this 

alternative appeals process when assessing deficiencies or determining 

enforcement actions. Regardless, these actions may have affected 

surveyors’ perceptions of the balance of the states’ IDRs, because over 

twice the national average of surveyors in this state reported that their IDR 

process favored nursing home operators. 

Reducing understatement is critical to protecting the health and safety of 

vulnerable nursing home residents and ensuring the credibility of the 

survey process. Federal and state efforts will require a sustained, longterm commitment because understatement arises from weaknesses in 

several interrelated areas—including CMS’s survey process, surveyor 

workforce and training, supervisory review processes, and state agency 

practices and external pressure. 

Conclusions 

• Concerns about CMS’s Survey Process. Survey methodology and 

guidance are integral to reliable and consistent state nursing home 

surveys, and we found that weaknesses in these areas were linked to 

understatement by both surveyors and state agency directors. Both groups 

reported struggling to interpret existing guidance, and differences in 

interpretation were linked to understatement, especially in determining 

what constitutes actual harm. Surveyors noted that the current survey 

guidance was too lengthy, complex, and subjective. Additionally, they had 

fewer concerns about care areas for which CMS has issued revised 

 

69According to CMS guidance, if an outside entity conducts the IDR, the results of the 

process may serve only as a recommendation to the state survey agency of noncompliance 

or compliance with the federal requirements for nursing homes. 

Page 45 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
interpretive protocols. In its development of the QIS, CMS has taken steps 

to revise the nursing home survey methodology. However, development 

and implementation of the QIS in a small group of states has taken 

approximately 10 years, and full implementation of the new methodology 

is not expected to be completed until 2014. The experience of the QIS was 

mixed regarding improvement in the quality of surveys, and the 

independent evaluation generated a number of recommendations for 

improving the QIS. CMS concluded that it needed to focus future QIS 

development efforts on improving survey consistency and giving 

supervisors more tools to assess performance of surveyor teams. 

• Ongoing Workforce and Surveyor Training Challenges. Workforce 

shortages in state survey agencies increase the need for high-quality initial 

and ongoing training for surveyors. Currently, high vacancy rates can 

place pressure on state surveyors to complete surveys under difficult 

circumstances, including compressed time frames, inadequately staffed 

survey teams, and too many inexperienced surveyors. States are 

responsible for hiring and retaining surveyors and have grappled with 

pervasive and intractable workforce shortages. State agency directors 

struggling with these workforce issues reported the need for more readily 

accessible training for both their new and experienced surveyors that did 

not involve travel to a central location. Nearly 30 percent of surveyors in 

high-understatement states stated that initial surveyor training, which is 

primarily a state activity that incorporates two CMS on-line computer 

courses and a 1-week federal basic training course culminating in the 

SMQT, was not adequate to identify deficiencies and cite them at the 

appropriate scope and severity level. State agency directors reported that 

workforce shortages also impede states’ ability to provide ongoing training 

opportunities for experienced staff and that additional CMS online training 

and electronic training media would help states maintain an experienced, 

well-informed workforce. They noted that any such support should be 

cognizant of states’ current resource constraints, including limited funding 

of travel for training. 

• Supervisory Review Limitations. Currently, CMS provides little 

guidance on how states should structure supervisory review processes, 

leaving the scope of this important quality-assurance tool exclusively to 

the states and resulting in considerable variation throughout the nation in 

how these processes are structured. We believe that state quality 

assurance processes are a more effective preventive measure against 

understatement because they have the potential to be more immediate and 

cover more surveys than the limited number of federal comparative 

surveys conducted in each state. However, compared to reviews of serious 

deficiencies, states conducted relatively fewer reviews of deficiencies at 

Page 46 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
the D through F level, those that were most frequently understated 

throughout the nation, to assess whether or not such deficiencies were 

cited at too low a scope and severity level. In addition, we found that 

frequent changes to survey results made during supervisory review were 

symptomatic of workforce shortages and survey methodology 

weaknesses. For example, surveyors who reported that survey teams had 

too many new surveyors, more often also reported either frequent changes 

to or removals of deficiencies during supervisory reviews—indicating that 

states with inexperienced workforces may rely more heavily on 

supervisory reviews. In addition, variation existed in the type of feedback 

surveyors receive when deficiencies are changed or removed during 

supervisory reviews, providing surveyors with inconsistent access to 

valuable feedback and training. CMS did not implement our previous 

recommendation to require states to have a quality assurance process that 

includes, at a minimum, a review of a sample of survey reports below the 

actual harm level to assess the appropriateness of the scope and severity 

cited and help reduce understatement. 

• State Agency Practices and External Pressure. In a few states, 

noncitation practices, challenging relationships with the industry or 

legislators, or unbalanced IDR processes—those that surveyors regard as 

favoring nursing home operators over resident welfare—may have had a 

negative effect on survey quality and resulted in the citation of fewer 

nursing home deficiencies than was warranted. In one state, both the state 

agency director and over 40 percent of surveyors acknowledged the 

existence of a noncitation practice such as allowing a home to correct a 

deficiency without receiving a citation. Forty percent of surveyors in four 

other states also responded on our questionnaire that noncitation 

practices existed. Currently, CMS does not explicitly address such 

practices in its guidance to states, and its oversight is limited to reviews of 

citation patterns, feedback from state surveyors, state performance 

reviews, and federal monitoring surveys to determine if such practices 

exist. Twelve state agency directors reported on our questionnaire 

experiencing some kind of external pressure. For example, in one state a 

legislator attended a survey and questioned surveyors as to whether state 

agency executives were coercing them to find deficiencies. Under such 

circumstances, it is difficult to know if the affected surveyors are 

consistently enforcing federal standards and reporting all deficiencies at 

the appropriate scope and severity levels. States’ differing experiences 

regarding the enforcement of federal standards and collaboration with 

their CMS regional offices in the face of significant external pressure also 

may confuse or undermine a thorough and independent survey process. If 

surveyors believe that CMS does not fully or consistently support the 

enforcement of federal standards, these surveyors may choose to avoid 

Page 47 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
citing deficiencies that they perceive may trigger a reaction from external 

stakeholders. In addition, deficiency determinations may be influenced 

when IDR processes are perceived to favor nursing home operators over 

resident welfare. 

Because many aspects of federal and state operations contribute to the 

understatement of deficiencies on nursing home surveys, mitigating this 

problem will require the concerted effort of both entities. The interrelated 

nature of these challenges suggests a need for increased CMS attention on 

the areas noted above and additional federal support for states’ efforts to 

enforce federal nursing home quality standards. 

To address concerns about weaknesses in CMS survey methodology and 

guidance, we recommend that the Administrator of CMS take the 

following two actions: 

Recommendations for 

Executive Action 

• make sure that action is taken to address concerns identified with the new 

QIS methodology, such as ensuring that it accurately identifies potential 

quality problems; and 

• clarify and revise existing CMS written guidance to make it more concise, 

simplify its application in the field, and reduce confusion, particularly on 

the definition of actual harm. 

To address surveyor workforce shortages and insufficient training, we 

recommend that the Administrator of CMS take the following two actions: 

• consider establishing a pool of additional national surveyors that could 

augment state survey teams or identify other approaches to help states 

experiencing workforce shortages; 

• evaluate the current training programs and division of responsibility 

between federal and state components to determine the most costeffective approach to: (1) providing initial surveyor training to new 

surveyors, and (2) supporting the continuing education of experienced 

surveyors. 

To address inconsistencies in state supervisory reviews, we recommend 

that the Administrator of CMS take the following action: 

• set an expectation through guidance that states have a supervisory review 

program as a part of their quality-assurance processes that includes 

Page 48 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
routine reviews of deficiencies at the level of potential for more than 

minimal harm (D-F) and that provides feedback to surveyors regarding 

changes made to citations. 

To address state agency practices and external pressure that may 

compromise survey accuracy, we recommend that the Administrator of 

CMS take the following two actions: 

• reestablish expectations through guidance to state survey agencies that 

noncitation practices—official or unofficial—are inappropriate, and 

systematically monitor trends in states’ citations; and 

• establish expectations through guidance to state survey agencies to 

communicate and collaborate with their CMS regional offices when they 

experience significant pressure from legislators or the nursing home 

industry that may affect the survey process or surveyors’ perceptions. 

We provided a draft of this report to HHS and AHFSA for comment. In 

response, the Acting Administrator of CMS provided written comments. 

CMS noted that the report adds value to important public policy 

discussions regarding the survey process and contributes ideas for 

solutions on the underlying potential causes of understatement. CMS fully 

endorsed five of our seven recommendations and indicated it would 

explore alternate solutions to our remaining two recommendations, one of 

which the agency did not plan to implement on a national scale. (CMS’s 

comments are reprinted in appendix II.) AHFSA’s comments noted that 

several states agreed with one of our recommendations, but did not 

directly express agreement or disagreement with the other 

recommendations. AHFSA made several other comments on our findings 

and recommendations as summarized below. 

Agency and AHFSA 

Comments and Our 

Evaluation 

CMS CMS agreed with five of our recommendations that called for: 

(1) addressing issues identified with the new QIS methodology, 

(2) evaluating current training programs, (3) setting expectations that 

states have a supervisory review program, (4) reestablishing expectations 

that noncitation practices are inappropriate, and (5) establishing 

expectations that states communicate with their CMS regional office when 

they experience significant pressure from legislators or the nursing home 

industry. In its comments, the agency cited several ongoing efforts as 

mechanisms for addressing some of our recommendations. While we 

acknowledge the importance of these ongoing efforts, in some areas we 

Page 49 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
believe more progress and investigation are likely needed to fully address 

our findings and recommendations. For example, we recommended that 

CMS ensure that measures are taken to address issues identified with the 

new QIS methodology, such as ensuring that it accurately identifies 

potential quality problems; CMS’s response cited Desk Audit Reports that 

enable supervisors to provide improved feedback to surveyors and 

quarterly meetings of a user group as evidence of efforts under way to 

continuously improve the QIS and to increase survey consistency. 

However, we noted that a 2007 evaluation of the QIS did not find improved 

survey accuracy compared to the traditional survey process and 

recommended that CMS evaluate how well the QIS accurately identifies 

areas in which there were potential quality problems. While improving the 

consistency of the survey process is important, CMS must also focus on 

addressing the accuracy of QIS surveys. 

For the remaining two recommendations, CMS described alternative 

solutions that it indicated the agency would explore: 

• Guidance. The agency agreed in principle with our recommendation to 

clarify and revise existing written guidance to make it more concise, 

simplify its application in the field, and reduce confusion. However, CMS 

disagreed with shortening the guidance as the preferred method for 

achieving such clarification. Instead, the agency suggested an alternative—

the creation of some short reference documents for use in the field that 

contain cross-links back to the full guidance—that we believe would fulfill 

the intent of our recommendation. 

• National surveyor pool. CMS indicated it did not plan to implement our 

recommendation to consider establishing a pool of additional national 

surveyors that could augment state survey teams experiencing workforce 

shortages, at least not on a national scale. The agency stated that the 

establishment of national survey teams was problematic for several 

reasons, including that it (1) began to blur the line between state 

accountability for meeting performance expectations and compensating 

states for problematic performance due to state management decisions, 

and (2) was improper for CMS to tell states how to make personnel 

decisions While the agency noted that it used national contractors to 

perform surveys for other types of facilities such as organ transplant 

centers, it expressed concern about their use to compensate for state 

performance issues because of the more frequent nursing home surveys. 

Page 50 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
We believe that state workforce shortages are a separate issue from state 

performance on surveys. Since 2003, we have reported pervasive state 

workforce shortages and this report confirms that such shortages 

continue.70 For example, we reported that one-fourth of states had vacancy 

rates higher than 19 percent and that one state reported a 72 percent 

vacancy rate. We also believe that addressing workforce shortages is 

critical to creating an effective system of oversight for nursing homes and 

reducing understatement throughout the nation. 

However, CMS noted that it would explore this issue with a state-federal 

work group in order to identify any circumstances in which a national pool 

may be advisable and to identify any additional solutions. Reflecting this 

comment from CMS, we have revised our original recommendation to 

include other potential solutions as well as a national pool of surveyors. 

One suggestion in AHFSA comments may be worth exploring in this 

regard—providing funds to state survey agencies for recruitment and 

retention activities. 

AHFSA AHFSA commented that vigorous oversight and enforcement are essential 

to improving the quality of life and quality of care for health care 

consumers and are critical if improvements already achieved are to be 

maintained. The association noted that several states agreed with our 

recommendation on the need for CMS to revise existing written guidance 

to make it more concise. While the association did not directly express 

agreement or disagreement with our other recommendations, it did note 

that most states would need additional funding to meet any new staffing 

requirements associated with our recommendation that CMS set an 

expectation for states to have a supervisory review program. 

However, AHFSA noted what it considered to be conflicting assertions 

within the report. For example, it noted that we cited inexperienced staff 

as a factor that contributes to understatement but also appeared to take 

issue with the practice of supervisors changing reports prepared by 

inexperienced staff. While our report identifies a wide variety of factors 

that may contribute to understatement, we did not and could not 

meaningfully prioritize among these factors based on the responses of 

nursing home surveyors and state agency directors. We did find that many 

states were attempting to accomplish their survey workload with a large 

 

70See GAO-03-561. 

Page 51 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
share of inexperienced surveyors and that state agency directors 

sometimes linked this reliance on inexperienced staff to the 

understatement of nursing home deficiencies. In addition, we found that 

frequent changes made during supervisory review were symptomatic of 

workforce shortages and survey methodology weaknesses. For example, 

surveyors who reported that survey teams had too many new surveyors, 

more often also reported either frequent changes to or removals of 

deficiencies during supervisory reviews. We believe that state quality 

assurance processes have the potential to play an important role in 

preventing understatement, which may result in states with inexperienced 

workforces relying more heavily on supervisory reviews. 

AHFSA also stated that our report did not address limitations of federal 

monitoring surveys, specifically the potential inconsistency among CMS 

regional offices in how these surveys are conducted. Assessing CMS’s 

performance on federal monitoring surveys was beyond the scope of this 

report. However, our May 2008 report noted several improvements CMS 

had made since fiscal years 2002 and 2003 in federal comparative surveys 

intended to make them more comparable to the state surveys they are 

assessing; these improvement include; (1) reducing the time between the 

state and federal surveys to ensure that they more accurately capture the 

conditions at the time of the state survey, (2) including at least half of the 

residents from state survey investigative samples to allow for a more clearcut determination of whether the state survey should have cited a 

deficiency, and (3) using the same number of federal surveyors as the 

corresponding state survey, again to more closely mirror the conditions 

under which the state survey was conducted.71

Finally, AHFSA questioned whether the information that we received from 

surveyors about the IDR process was universally valid because their input 

about quality assurance reviews might be biased. Our methodology did not 

rely solely on surveyor responses to our questionnaire but used a separate 

questionnaire sent to state survey agency directors to help corroborate 

their responses. Thus we reported both that (1) over 40 percent of 

surveyors in four states indicated that their IDR process favored nursing 

home operators and (2) one state survey agency director agreed and three 

others acknowledged that frequent IDR hearings sometimes contributed to 

the understatement of deficiencies. We also collected and reported data on 

 

71See GAO-08-517. 

Page 52 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
the number of deficiencies modified or overturned, which AHFSA said was 

a more accurate measure of the effect of IDRs. 

We also incorporated technical comments from AHFSA as appropriate. 

As arranged with your offices, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

issue date. At that time, we will send copies to the Administrator of the 

Centers for Medicare & Medicaid Services and appropriate congressional 

committees. In addition, the report will be available at no charge on GAO’s 

Web site at http://www.gao.gov. 

If you or your staffs have any questions about this report, please contact 

me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff who made major contributions to this report are 

John E. Dicken 

listed in appendix II. 

Director, Health Care 

Page 53 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

Page 54 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 

Appendix I: Scope and Methodology 

This appendix describes the data and methods we used to identify the 

factors that contribute to the understatement of serious deficiencies on 

nursing home surveys.1

 This report relies largely on the data collected 

through (1) two GAO-administered Web-based questionnaires to nursing 

home surveyors and state agency directors and (2) analysis of federal and 

state nursing home survey results as reported in the federal monitoring 

survey database and the On-Line Survey, Certification, and Reporting 

(OSCAR) system. Summary results from the GAO questionnaires are 

available as an e-supplement to this report. See Nursing Homes: 

Responses from Two Web-Based Questionnaires to Nursing Home 

Surveyors and State Agency Directors (GAO-10-74SP), an E-supplement 

to GAO-10-70. To augment our quantitative analysis, we also interviewed 

officials at the Centers for Medicare & Medicaid (CMS) Survey and 

Certification Group and select regional offices;2

 reviewed federal 

regulations, guidance, and our prior work; and conducted follow-up 

interviews with eight state agency directors and a select group of 

surveyors. Except where otherwise noted, we used data from fiscal year 

2007 because they were the most recently available data at the time of our 

analysis. 

We developed two Web-based questionnaires—one for the nursing home 

surveyors and one for the state agency directors. Development of 

Questionnaires and 

Analysis of Responses 

Development of the 

Questionnaires 

The questionnaires were developed and the data collection and analysis 

conducted to (1) minimize errors arising from differences in how a 

particular question might be interpreted and in the sources of information 

available to respondents and (2) reduce variability in responses that 

should be qualitatively the same. GAO social science survey specialists 

aided in the design and development of both questionnaires. We pretested 

the two questionnaires with six surveyors from a local state and five 

 

1

This report follows and expands on our May 2008 report, which examined (1) the 

information contained in federal monitoring surveys about understatement nationwide, 

and (2) CMS management and oversight of the federal monitoring survey program, see 

GAO-08-517. 

2

CMS’s Survey and Certification Group is responsible for ensuring the effectiveness of state 

survey activities and managing the federal monitoring survey program. 
Appendix I: Scope and Methodology 

former or current state agency directors, respectively. Based on feedback 

from these pretests, the questionnaires were revised to improve clarity and 

the precision of responses, and ensure that all questions were fair and 

unbiased. Most questions were closed-ended, which limited the 

respondent to answers such as yes or no, or to identifying the frequency 

that an event occurred using a scale—always, frequently, sometimes, 

infrequently, or never. For reporting purposes, we grouped the scaled 

responses into three categories—always/frequently, sometimes, and 

infrequently/never. Both questionnaires included some open-ended 

questions to allow respondents to identify specific training needs or other 

concerns. 

With few exceptions, respondents entered their responses directly into the 

Web-based questionnaire databases.3

 These questionnaires were sent to 

the eligible population of nursing home surveyors and all state agency 

directors. We performed computer analyses to identify illogical or 

inconsistent responses and other indications of possible error. We also 

conducted follow-up interviews with select respondents to clarify and gain 

a contextual understanding of their responses.4

Questionnaire for Nursing 

Home Surveyors 

This questionnaire was designed to gather information from nursing home 

surveyors nationwide about the process for identifying and citing nursing 

home deficiencies. It included questions about various aspects of the 

survey process identified by our prior work that may contribute to survey 

inconsistency and the understatement of deficiencies. Such aspects 

included survey methodology and guidance, deficiency determination, 

surveyor training, supervisory review of draft surveys, and state agency 

policies and procedures.5

We fielded the questionnaire from May through July 2008 to 3,819 eligible 

nursing home surveyors. To identify the eligible population, we 

 

3

We mailed paper copies of the questionnaire to 15 surveyors in Arkansas, who did not 

have a state-issued e-mail address; on request, an additional copy was faxed to a surveyor. 

Seven out of the 16 paper copies were completed and returned to GAO. 

4

Although nursing home surveyors’ responses were anonymous to preserve their 

confidentiality, a few surveyors voluntarily provided their contact information and agreed 

to be interviewed. 

5

Questions about CMS’s survey methodology directed surveyors to respond about the 

traditional survey methodology, not the new Quality Indicator Survey (QIS) methodology, 

which had been implemented in eight states. 

Page 55 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

downloaded a list of identification numbers for surveyors who had 

conducted at least one health survey of a nursing home in fiscal years 2006 

or 2007 from CMS’s OSCAR database and we obtained surveyors’ e-mail 

addresses from state survey agencies. We received complete responses 

from 2,340 state surveyors, for a 61 percent response rate.6

 The state-level 

response rates were above 40 percent for all but three states—

Connecticut, Illinois, and Pennsylvania.7

 We excluded Pennsylvania from 

our analysis because Pennsylvania’s Deputy Secretary for Quality 

Assurance instructed the state’s surveyors not to respond to our survey 

and few responded. (For response rates by state, see table 9.) 

Table 9: Response Rates to GAO’s Questionnaire of Nursing Home Surveyors, 2008

State 

Number of 

respondents

Number of 

eligible surveyors 

Response rate 

(in percent)

Alabama 36 52 69

Alaska 4 6 67

Arizona 19 28 68

Arkansas 28 54 52

California 306 544 56

Colorado 16 38 42

Connecticut 17 61 28

Delaware 13 16 81

District of Columbia 6 10 60

Florida 128 226 57

Georgia 47 54 87

Hawaii 4 7 57

Idaho 6 11 55

Illinois 34 171 20

Indiana 92 101 91

Iowa 37 59 63

Kansas 34 59 58

Kentucky 44 86 51

 

6

When respondents indicated that they did not conduct health safety surveys of nursing 

homes and therefore should have been excluded from the population of eligible nursing 

home surveyors, these surveyors and their responses were excluded. 

7

The Illinois response rate likely reflects that surveyors’ access to their e-mail accounts, 

and our Web-based survey, was limited to only 1 day per month. 

Page 56 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

State 

Number of 

respondents

Number of 

eligible surveyors 

Response rate 

(in percent)

Louisiana 79 134 59

Maine 24 28 86

Maryland 29 47 62

Massachusetts 39 88 44

Michigan 50 80 63

Minnesota 58 85 68

Mississippi 21 35 60

Missouri 175 192 91

Montana 20 21 95

Nebraska 26 33 79

Nevada 21 25 84

New Hampshire 9 15 60

New Jersey 35 77 45

New Mexico 18 25 72

New York 108 250 43

North Carolina 53 84 63

North Dakota 12 16 75

Ohio 77 145 53

Oklahoma 63 92 68

Oregon 31 45 69

Rhode Island 20 27 74

South Carolina 15 27 56

South Dakota 20 22 91

Tennessee 52 79 66

Texas 201 281 72

Utah 16 25 64

Vermont 11 16 69

Virginia 36 42 86

Washington 60 85 71

West Virginia 14 22 64

Wisconsin 66 83 80

Wyoming 10 10 100.0

Total 2,340 3,819 61

Source: GAO. 

Page 57 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

The questionnaire for state agency directors was designed to gather 

information on the nursing home survey process in each state. Directors 

were asked many of the same questions as the surveyors, but the survey 

agency directors’ questionnaire contained additional questions on the 

overall organization of the survey agency, resource and staffing issues, 

CMS’s Quality Indicator Survey (QIS), and experience with CMS’s federal 

monitoring surveys.8

 In addition, the questionnaire for state agency 

directors asked them to rank the degree to which several factors, derived 

from our previous work, contributed to understatement.9

 This 

questionnaire was fielded from September to November 2008 to all 50 

state survey agency directors and the survey agency director for the 

District of Columbia. We received completed responses from 50 of 51 

survey agency directors, for a 98 percent response rate. The District of 

Columbia survey agency director did not respond. 

Questionnaire for State 

Agency Directors 

Analysis of Responses To analyze results from the survey questions among groups, we used 

standard descriptive statistics. In addition, we looked for associations 

between questions through correlations and tests of the differences in 

means for groups. For certain open-ended questions, we used a standard 

content review method to identify topics that respondents mentioned such 

as “applying CMS guidance,” “on-the-job training,” “time to complete 

survey onsite,” or “time to complete the survey paperwork.” Our coding 

process involved one independent coder and an independent analyst who 

verified a random sample of the coded comments. For open-ended 

questions that enabled respondents to provide additional general 

information, we used similar standard content review methods, including 

independent coding by two raters who resolved all disagreements through 

discussion. 

 

8

All state agency directors were asked about CMS’s traditional survey methodology, which 

all states used in 2008. However eight state agency directors, who indicated that the QIS 

has been implemented in at least part of their states, were asked additional questions 

specifically about the QIS. 

9

We did not ask nursing home surveyors a similar question because survey agency 

directors, as a result of their positions, were a more consistent source of knowledge about 

the influence of these factors on understatement. 

Page 58 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

In addition to the precautions taken during the development of the 

questionnaires, we performed automated checks on these data to identify 

inappropriate answers. We also reviewed the data for missing or 

ambiguous responses.10 Where comments on open-ended questions 

provided more detail or contradicted responses to categorical questions, 

the latter were corrected. On the basis of the strength of our systematic 

survey processes and follow-up procedures, we determined that the 

questionnaire responses were representative of the experience and 

perceptions of nursing home surveyors and state agency directors 

nationally and at the state level, with the exception of Pennsylvania 

surveyors and the survey agency director of the District of Columbia. On 

the basis of the response rates and these activities, we determined that the 

data were sufficiently reliable for our purposes. 

We also interviewed directors and other state agency officials in eight 

states to better understand unusual or interesting circumstances related to 

surveyor workforce and training, supervisory review, or state policies and 

practices. We selected these eight states based on our analysis of 

questionnaire responses from state agency directors and nursing home 

surveyors. 

We used information from our May 2008 report on federal comparative 

surveys nationwide for fiscal years 2002 through 2007 to categorize states 

into groups.11 We used these results to identify states with high and low 

percentages of serious missed deficiencies.12 We classified nine states as 

high-understatement states—those that had 25 percent or more federal 

comparative surveys identifying at least one missed deficiency at the 

actual harm or immediate jeopardy levels across all years. These states 

were Alabama, Arizona, Missouri, New Mexico, Oklahoma, South Carolina, 

South Dakota, Tennessee, and Wyoming. Zero-understatement states were 

those that had no federal comparative surveys identifying missed 

deficiencies at the actual harm or immediate jeopardy levels. These seven 

Validity and Reliability of 

Data 

Analysis of Federal 

Comparative and 

Observational Surveys 

 

10Where responses to particular questions were fewer than the overall number of responses 

for the questionnaire, this limitation is indicated in the text. 

11During this period, fiscal year 2002 was the first year that the database contained all the 

information needed to assess the results of federal comparative surveys. (See GAO-08-517.) 

12Federal comparative surveys are done on a small group of facilities that are not randomly 

selected, and the understatement of deficiencies identified through comparative surveys 

are not representative of all nursing home surveys or survey teams within each state. 

Page 59 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

states were Alaska, Idaho, Maine, North Dakota, Oregon, Vermont, and 

West Virginia. Low-understatement states were the 10 with the lowest 

percentage of missed serious deficiencies (less than 6 percent)—Arkansas, 

Nebraska, Ohio, and all seven zero-understatement states. 

Response rates among the high-, low-, and zero-understatement states—

approximately 77, 62, and 71 percent, respectively—supported statistical 

testing of associations and differences among these state groupings. 

Therefore, in addition to descriptive statistics, we used correlations and 

tests of the differences in means for groups to identify questionnaire 

responses that were associated with differences in understatement.13 We 

reported the statistically significant results for tests of association and 

differences between group averages at the 5 percent level, unless 

otherwise noted. 

In a previous report, we found a possible relationship between the 

understatement of nursing home deficiencies on the federal comparative 

surveys and surveyor performance in General Investigation and Deficiency 

Determination on federal observational surveys—that is, highunderstatement states more often had below-satisfactory ratings in 

General Investigation and Deficiency Determination than lowunderstatement states.14 For this report, we applied the same statistical 

analysis to identify when responses to our questionnaires were associated 

with satisfactory performance on General Investigative and Deficiency 

Determination skills on the federal observational surveys. We interpreted 

such relationships as an indication of additional training needs. 

 

13For our descriptive statistics, we computed means, the minimum and maximum 

responses, responses at the 25th, 50th, and 75th percentiles, frequencies among categories 

of respondents, such as those from high- and low-understatement states, as well as 

frequencies across two or more categories of respondents. Correlations were computed as 

Pearson’s correlations of association. T-tests were done to identify when the mean 

response from two different categories of respondents, such as high- and lowunderstatement states, were significantly different from each other. 

14GAO-08-517. The General Investigation segment assesses the effectiveness of state survey 

team actions such as collection of information, discussion of survey observations, 

interviews with nursing home residents, and implementation of CMS investigative 

protocols. The Deficiency Determination segment evaluates the skill with which the state 

survey teams (1) integrate and analyze all information collected and (2) use the guidance 

for surveyors and identify deviations from regulatory requirements. 

Page 60 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

We used information from OSCAR and the federal monitoring survey 

databases to (1) compare the deficiencies cited by state and federal 

surveyors, (2) analyze the timing of nursing home surveys, and (3) assess 

trends in deficiency citations. OSCAR is a comprehensive database that 

contains information on the results of state nursing home surveys. CMS 

reviews these data and uses them to compute nursing home facility and 

state performance measures. When we analyzed these data, we included 

automated checks of data fields to ensure that they contain complete 

information. For these reasons, we determined that the OSCAR data were 

sufficiently reliable for our purposes. 

Analysis of OSCAR 

• We used OSCAR and the federal monitoring survey database to compare 

average facility citations on state survey records with the average citations 

on federal observational survey records for the same facilities during fiscal 

years 2002 through 2007.15 We computed the average number of serious 

deficiencies cited on federal observational surveys between fiscal years 

2002 through 2007, and for the same facilities and time period, calculated 

the average number of serious deficiencies cited on state surveys. Next, 

we determined which facilities had greater average serious deficiency 

citations on federal observational surveys compared to state standard 

surveys between fiscal years 2002 through 2007. For these facilities, we 

computed the percentage difference between the average number of 

serious deficiencies cited on federal observational surveys and those cited 

on state surveys. 

• We used OSCAR to determine the percentage of the most recent state 

surveys that were predictable because of their timing. Our analysis of 

survey predictability compared the time between state agencies’ current 

and prior standard nursing home surveys as of June 2008. According to 

CMS, states consider 9 months to 15 months from the last standard survey 

as the window for completing standard surveys because it yields a 12-

month average. We considered surveys to be predictable if (1) homes were 

surveyed within 15 days of the 1-year anniversary of their prior survey or 

(2) homes were surveyed within 1 month of the maximum 15-month 

interval between standard surveys. 

 

15We use the term survey record to refer to CMS’s Form 2567, which is the official 

statement of deficiencies with respect to federal quality standards. 

Page 61 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix I: Scope and Methodology 

• We calculated the number of serious deficiencies on state surveys in 

OSCAR from calendar year 1999 through 2007. We examined the trend in 

G-level and higher deficiencies to assess whether CMS’s expanded 

enforcement policy appeared to affect citation rates. Effective January 

2000, CMS completed the implementation of its immediate-sanctions 

policy, requiring the referral of homes that caused actual harm or 

immediate jeopardy on successive standard surveys or intervening 

complaint investigations. 

Page 62 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department Appendix II: Comments from the Department of Health & Human Services 

of Health & Human Services 

Page 63 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 64 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 65 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 66 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 67 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 68 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix II: Comments from the Department 

of Health & Human Services 

Page 69 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Appendix III: GAO

A

 Contact and Staff 

cknowledgments 

Page 70 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 

Appendix III: GAO Contact and Staff 

Acknowledgments 

GAO Contact John E. Dicken, (202) 512-7114 or dickenj@gao.gov 

In addition to the contact named above, Walter Ochinko, Assistant 

Director; Stefanie Bzdusek; Leslie V. Gordon; Martha R. W. Kelly; 

Katherine Nicole Laubacher; Dan Lee; Elizabeth T. Morrison; Dan Ries; 

Steve Robblee; Karin Wallestad; Rachael Wojnowicz; and Suzanne Worth 

made key contributions to this report. 

Staff 

Acknowledgments 
Related GAO Products Related GAO Products 

Nursing Homes: Opportunities Exist to Facilitate the Use of the 

Temporary Management Sanction. GAO-10-37R. Washington, D.C.: 

November 20, 2009. 

Nursing Homes: CMS’s Special Focus Facility Methodology Should Better 

Target the Most Poorly Performing Homes, Which Tended to Be Chain 

Affiliated and For-Profit. GAO-09-689. Washington, D.C.: August 28, 2009. 

Medicare and Medicaid Participating Facilities: CMS Needs to 

Reexamine Its Approach for Funding State Oversight of Health Care 

Facilities. GAO-09-64. Washington, D.C.: February 13, 2009. 

Nursing Homes: Federal Monitoring Surveys Demonstrate Continued 

Understatement of Serious Care Problems and CMS Oversight 

Weaknesses. GAO-08-517. Washington, D.C.: May 9, 2008. 

Nursing Home Reform: Continued Attention Is Needed to Improve 

Quality of Care in Small but Significant Share of Homes. GAO-07-794T. 

Washington, D.C.: May 2, 2007. 

Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not 

Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241. 

Washington, D.C.: March 26, 2007. 

Nursing Homes: Despite Increased Oversight, Challenges Remain in 

Ensuring High-Quality Care and Resident Safety. GAO-06-117. 

Washington, D.C.: December 28, 2005. 

Nursing Home Quality: Prevalence of Serious Problems, While 

Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561. 

Washington, D.C.: July 15, 2003. 

Nursing Homes: Quality of Care More Related to Staffing than Spending. 

GAO-02-431R. Washington, D.C.: June 13, 2002. 

Nursing Homes: Sustained Efforts Are Essential to Realize Potential of 

the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.: 

September 28, 2000. 

Nursing Home Care: Enhanced HCFA Oversight of State Programs 

Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.: 

November 4, 1999. 

Page 71 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
Related GAO Products 

Nursing Homes: Proposal to Enhance Oversight of Poorly Performing 

Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999. 

Nursing Homes: Additional Steps Needed to Strengthen Enforcement of 

Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.: 

March 18, 1999. 

California Nursing Homes: Care Problems Persist Despite Federal and 

State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998. 

(290612) Page 72 GAO-10-70 Understatement of Nursing Homes’ Serious Care Problems 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper