Nursing Homes: Federal Monitoring Surveys Demonstrate Continued Understatement of Serious Care Problems and CMS Oversight Weaknesses
Published: 2008-05-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reports since 1998 have demonstrated that state surveyors, who evaluate the quality of nursing home care on behalf of CMS, sometimes understate the extent of serious care problems in homes because they miss deficiencies. CMS oversees the effectiveness of state surveys through the federal monitoring survey program. In this program, federal surveyors in CMS's regional offices either independently evaluate state surveys by resurveying a home (comparative surveys) or directly observe state surveyors during a routine nursing home survey (observational surveys). GAO was asked to evaluate the information federal monitoring surveys provide on understatement and the effectiveness of CMS management and oversight of the survey program. To do this, GAO analyzed the results of federal monitoring surveys for fiscal years 2002 through 2007, reviewed CMS guidance for the survey program, and interviewed headquarters and regional office officials.
section_title: What GAO Found
paragraphs: A substantial proportion of federal comparative surveys identify missed deficiencies at the potential for more than minimal harm level or above. During fiscal years 2002 through 2007, about 15 percent of federal comparative surveys nationwide identified state surveys that failed to cite at least one deficiency at the most serious levels of noncompliance--actual harm and immediate jeopardy. Overall, nine states missed serious deficiencies on 25 percent or more of comparative surveys; in seven states federal surveyors identified no such missed deficiencies. During the same period, missed deficiencies at the lowest level of noncompliance--the potential for more than minimal harm--were more widespread: nationwide, approximately 70 percent of federal comparative surveys identified state surveys missing at least one deficiency at the lowest level of noncompliance, and in all but five states the number of state surveys with such missed deficiencies was greater than 40 percent. Undetected care problems at this level are a concern because they could become more serious if nursing homes are not required to take corrective action. The most frequently missed type of deficiency on comparative surveys, at the potential for more than minimal harm level and above, was poor quality of care, such as ensuring proper nutrition and hydration and preventing pressure sores. Federal observational surveys highlighted two factors that may contribute to understatement of deficiencies: weaknesses in state surveyors' (1) investigative skills and (2) ability to integrate and analyze information collected to make an appropriate deficiency determination. These factors may contribute to understatement because they directly affect the appropriate identification and citation of deficiencies. CMS has taken steps to improve the federal monitoring survey program, but weaknesses remain in program management and oversight. For example, CMS has improved processes to ensure that comparative surveys more accurately reflect conditions at the time of the state survey, such as requiring that comparative surveys occur within 30 working days of the state survey rather than within the 2 months set in statute. Despite these improvements, the management and oversight potential of the program has not been fully realized. For example, CMS has only begun to explore options for identifying understatement that occurs in cases where state surveys cite deficiencies at too low a level, for possible implementation in fiscal year 2009. In addition, CMS is not effectively managing the federal monitoring survey database to ensure that the regional offices are entering data accurately and reliably--CMS was unaware, for example, that a considerable number of comparative surveys had not been entered. Furthermore, CMS is not using the database to oversee consistent implementation of the program by the regional offices--for example, the agency is not using the database to identify inconsistencies between comparative and observational survey results.

section_title: Letter
paragraphs: Since 1998, Congress has focused considerable attention on the need to  improve the quality of care for the nation’s 1.5 million nursing home  residents, a highly vulnerable population of elderly and disabled  individuals for whom remaining at home is no longer feasible. Poor quality  of care—worsening pressure sores or untreated weight loss—in a small  but unacceptably high number of nursing homes continues to harm  residents or place them in immediate jeopardy, that is, at risk of death or  serious injury. About 1 in 5 homes nationwide were cited for such serious  deficiencies on state inspections, known as surveys, in fiscal year 2007  (see app. I). Our previous work, however, demonstrated that state surveys  sometimes understated the extent of serious care problems and that  federal oversight of state survey activities had weaknesses.  Understatement can occur when a state surveyor fails to cite a deficiency  altogether or cites a deficiency at too low a level.
The Centers for Medicare & Medicaid Services (CMS) is the federal agency  responsible for ensuring the effectiveness of state surveys. Through  CMS’s federal monitoring survey program, federal surveyors either   (1) independently evaluate state surveys by resurveying a home recently  inspected by state surveyors and comparing the deficiencies identified  during the two surveys, known as a comparative survey, or (2) directly  observe state surveyors during a routine nursing home survey, known as  an observational survey. Results from both federal comparative and  observational surveys—which are recorded in the federal monitoring  survey database—allow CMS to gauge states’ abilities to accurately assess  nursing home quality.
Recently, we reported that federal comparative surveys in five large states  identified the continuing understatement of serious care problems by state  surveyors. You asked us to look at the understatement of serious  deficiencies by state surveyors nationwide. In this report we address two  questions: (1) what information do federal monitoring surveys provide  about understatement nationwide, and (2) how effective are CMS  management and oversight of the federal monitoring survey program?
To answer the first question, we analyzed the results of comparative and  observational surveys nationwide for fiscal years 2002 through 2007 using  the federal monitoring survey database. During this period, federal  surveyors conducted 976 comparative surveys and 4,023 observational  surveys. To identify understatement on comparative surveys, we focused  our analysis on cases where federal surveyors determined that state  surveyors should have cited a deficiency but failed to do so or where state  surveyors cited a deficiency at too low a level. We analyzed the results of  observational surveys in order to better understand why understatement  might occur. Deficiencies identified during nursing home surveys are  categorized according to their scope (i.e., the number of residents  potentially or actually affected) and severity (i.e., the degree of relative  harm involved). Homes with deficiencies at the A though C levels are  considered to be in substantial compliance, while those with deficiencies  at the D through L levels are considered out of compliance. (See table 1.)  Throughout this report, we refer to deficiencies at the actual harm and  immediate jeopardy levels as serious deficiencies.
To ensure reliability of the federal monitoring survey database, we  discussed data entry procedures with all 10 CMS regional offices, whose  staff enter information into the database. In addition, we conducted  several data reliability tests, including (1) automated checks of data fields  to ensure that they contained complete information and (2) manual  reviews of a random sample of all deficiencies cited by federal but not  state surveyors to ensure that federal surveyors had used the data fields  appropriately. We also eliminated a small number of deficiencies that did  not correspond to a defined severity level or contained illogical survey  dates—such as a comparative survey that began prior to the state survey.  Based on these activities, we determined that the information was  sufficiently reliable for our purposes. Data on comparative surveys,  however, cannot be used to project the extent of understatement across all  state surveys because the state surveys selected for federal monitoring  surveys are not representative of all nursing home surveys or survey teams  within each state.
To answer the second question, we reviewed CMS guidance for the federal  monitoring survey program and interviewed officials in CMS headquarters  and all 10 regional offices. Our work focused on CMS’s (1) efforts to  improve the use of comparative surveys as an oversight tool; (2) ability to  track the understatement of deficiencies; and (3) management of the  federal monitoring survey database, including the use of the database to  oversee regional office implementation of the federal monitoring survey  program. We also analyzed (1) the comments entered into the federal  monitoring survey database by federal surveyors for certain discrepancies  between federal and state survey findings and (2) the consistency between  comparative and observational survey results within states and CMS  regional offices. We performed our work from July 2007 through May 2008  in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Oversight of nursing homes is a shared federal-state responsibility. Based  on statutory requirements, CMS (1) defines quality standards that nursing  homes must meet to participate in the Medicare and Medicaid programs  and (2) contracts with state survey agencies to assess whether homes  meet those standards through annual surveys and complaint  investigations. Although CMS has issued extensive guidance to states on  determining compliance with federal quality requirements, we have found  that some state surveys understate quality problems at nursing homes.
subsections: section_title: Federal Quality Standards
paragraphs: Federal nursing home quality standards focus on the delivery of care,  resident outcomes, and facility conditions. These standards, totaling  approximately 200, are grouped into 15 categories, such as Resident  Rights, Quality of Life, Resident Assessment, Quality of Care, Pharmacy  Services, and Administration. For example, there are 23 standards within  the Quality of Care category ranging from “promote the prevention of  pressure  development” to “the resident environment remains as free  of accident hazards as is possible.” CMS has also developed detailed  investigative protocols to assist state survey agencies in determining  whether nursing homes are in compliance with federal quality standards.  This guidance is intended to ensure the thoroughness and consistency of  state surveys and complaint investigations.
section_title: Standard Surveys and Complaint Investigations
paragraphs: Every nursing home receiving Medicare or Medicaid payment must  undergo a standard state survey not less than once every 15 months, and  the statewide average interval for these surveys must not exceed   12 months. During a standard survey, teams of state surveyors—generally  consisting of registered nurses, social workers, dieticians, or other  specialists—evaluate compliance with federal quality standards. Based on  the care provided to a sample of residents, the survey team (1) determines  whether the care and services provided meet the assessed needs of the  residents and (2) measures resident outcomes, such as the incidence of  preventable pressure sores, weight loss, and accidents. In contrast to a  standard survey, a complaint investigation generally focuses on a specific  allegation regarding a resident’s care or safety and provides an  opportunity for state surveyors to intervene promptly if problems arise  between standard surveys. Surveyors generally follow state procedures  when investigating complaints, but must comply with certain federal  guidelines and time frames.
section_title: Enforcement
paragraphs: When deficiencies are identified, federal sanctions can be imposed to help  encourage homes to correct them. Sanctions are generally reserved for  serious deficiencies—those at the G through L levels—which constitute  actual harm and immediate jeopardy. Sanctions for such serious quality  problems can affect a home’s revenues and provide financial incentives to  return to and maintain compliance. Such sanctions include fines known as  civil money penalties, denial of payment for new Medicare or Medicaid  admissions, or termination from the Medicare and Medicaid programs.
State surveys that miss serious deficiencies or cite deficiencies at too low  a scope and severity level have enforcement implications because a  nursing home may escape sanctions intended to discourage repeated  noncompliance. For example, facilities that receive at least one G through  L level deficiency on successive standard surveys or complaint  investigations must be referred for immediate sanctions. In addition, CMS  guidance calls for higher fines when a home has a poor compliance history  and requires that state survey teams revisit a home to verify that serious  deficiencies have actually been corrected (such revisits are not required  for most deficiencies cited below the actual harm level—A through F).
section_title: CMS Oversight of State Surveys
paragraphs: Statutorily required federal monitoring surveys, which are conducted  annually in at least 5 percent of state-surveyed Medicare and Medicaid  nursing homes in each state, are a key CMS oversight tool in ensuring the  adequacy of state surveys. CMS headquarters—specifically, CMS’s Survey  and Certification Group—is responsible for the management of the federal  monitoring survey database and for oversight of the 10 CMS regional  offices’ implementation of the federal monitoring survey program.  Federal surveyors located in regional offices conduct federal monitoring  surveys. The surveys can be either comparative or observational, with  each offering unique advantages and disadvantages as an oversight tool.  For example, an advantage of comparative surveys is that they are an  independent evaluation of a nursing home recently surveyed by a state  survey agency team. A disadvantage is that the time lag between the two  surveys can make analysis of differences difficult.
Comparative survey. A federal survey team conducts an independent  survey of a home recently surveyed by a state survey agency in order to  compare and contrast the findings. This comparison takes place after  completion of the federal survey. When federal surveyors identify a  deficiency not cited by state surveyors, they assess whether the deficiency  existed at the time of the state survey and should have been cited by  entering either yes or no to the question, “Based on the evidence available  to the , should the  team have cited this ?”  This assessment is critical in determining whether understatement  occurred because some deficiencies cited by federal surveyors may not  have existed at the time of the state survey. For example, a deficiency  identified during a federal survey could involve a resident who was not in  the nursing home at the time of the earlier state survey. By statute,  comparative surveys must be conducted within 2 months of the  completion of the state survey. However, differences in timing, resident  sample selection, and staffing can make analysis of differences between  the state and federal comparative surveys difficult. On the basis of our  prior recommendations, CMS has taken several steps to ensure that  comparative surveys more accurately capture conditions at the time of the  state survey. For example, CMS now calls for the length of time between  the state and federal surveys to be between 10 and 30 working days and  requires federal surveyors conducting a comparative survey in a nursing  home to include at least half of the state survey’s sample of residents from  that nursing home in the comparative survey sample, making it easier to  determine whether state surveyors missed a deficiency. Furthermore,  federal comparative survey teams are expected to mimic the number of  staff assigned to the state survey. CMS also issued guidance in October  2002 defining the criteria for federal surveyors to consider when selecting  facilities for comparative surveys. These selection criteria can generally  be categorized as state survey team performance and facility  characteristics. Regional offices were given latitude in their use of these  criteria and may supplement them with other selection factors unique to  their regions. For example, some regions use statistics on the prevalence  of pressure sores in a nursing home’s resident population as a comparative  survey selection factor.
Observational survey. Federal surveyors accompany a state survey team  to a nursing home to evaluate the team’s on-site survey performance and  ability to document survey deficiencies. State teams are evaluated in six  areas—Concern Identification, Sample Selection, General Investigation,  Food-Borne Illness Investigation, Medication Investigations, and  Deficiency Determination—and are rated in one of five categories for each  of the six measures. The rating categories—from highest to lowest—are  extremely effective, very effective, satisfactory, less than satisfactory, and  much less than satisfactory. CMS annual state performance reviews  require that state survey teams achieve an average rating of satisfactory.  Observational surveys allow federal surveyors to provide more immediate  feedback to state surveyors and to identify state surveyor training needs.  However, observational surveys are not independent evaluations of the  state survey. Because state surveyors may perform their survey tasks more  attentively than they would if federal surveyors were not present,  observational surveys may not provide an accurate picture of state  surveyors’ typical performance. Since 2001, CMS has also taken steps to  strengthen observational surveys. For example, the agency issued written  guidance defining a standard process for resolving disagreements and a  new manual to increase consistency across observational surveys.
The 976 federal comparative surveys conducted from fiscal year 2002  through 2007 ranged from as few as 10 in Vermont, which has about   40 facilities, to as many as 49 in California, which has about 1,300 facilities.  Of the 4,023 federal observational surveys conducted during the same  period, the number ranged from 16 in New Hampshire to 346 in California.
The results of federal monitoring surveys, including information on the  corresponding state surveys, are entered in the federal monitoring survey  database. In fiscal year 2002, CMS began including information on  comparative surveys in the database, and the agency began requiring  federal surveyors to determine whether a deficiency cited by federal but  not state surveyors had been missed by determining whether state  surveyors should have cited the deficiency.
section_title: Understatement of Deficiency Scope and Severity Level
paragraphs: Although comparative surveys and the wide variability across states in the  proportion of homes with deficiencies at the actual harm and immediate  jeopardy levels indicate that state surveyors miss some serious  deficiencies, our prior work has also indicated that state surveyors  sometimes understate the scope and severity of a deficiency. In 2003, we  found widespread understatement of actual harm deficiencies in a sample  of surveys from homes with a history of harming residents. Overall,   39 percent of the 76 state surveys we reviewed had documented problems  that should have been classified as actual harm instead of as lower-level  deficiencies.
section_title: Substantial Proportion of Federal Comparative Surveys Identify Missed Deficiencies
paragraphs: A substantial proportion of federal comparative surveys identify missed  deficiencies at the potential for more than minimal harm level or above.  From fiscal year 2002 through 2007, about 15 percent of federal  comparative surveys nationwide identified state surveys that failed to cite  at least one deficiency at the most serious levels of noncompliance—the  actual harm and immediate jeopardy levels (G through L). There was wide  variation across states in the proportion of comparative surveys that found  at least one missed serious deficiency, from more than 25 percent in nine  states to none in seven others. In contrast to missed serious deficiencies,  missed deficiencies at the potential for more than minimal harm level   (D through F) were considerably more widespread, with such missed  deficiencies greater than 40 percent in all but five states. Every state had at  least one comparative survey with missed D through F level deficiencies.  At both levels of noncompliance, the most frequently missed deficiencies  involved Quality of Care standards. Federal observational survey results  and prior GAO reports have highlighted several factors that may  contribute to the understatement of deficiencies.
subsections: section_title: Small but Unacceptably High Proportion of Federal Comparative Surveys Found That State Survey Teams Missed Serious Deficiencies
paragraphs: About 15 percent (142) of the 976 comparative surveys conducted from  fiscal year 2002 through 2007 identified state surveys that missed at least  one deficiency at the actual harm or immediate jeopardy level (G through  L), the most serious levels of noncompliance. This proportion fluctuated  from a high of 17.5 percent in fiscal year 2003 to a low of 11.1 percent in  fiscal year 2004, but it has remained relatively constant at about 15 percent  for the last several fiscal years (see fig. 1). This proportion is small, but  CMS maintains that any missed serious deficiencies are unacceptable.
From fiscal year 2002 through 2007, federal surveyors identified missed  serious deficiencies in 25 percent or more of their comparative surveys in  nine states. The proportion of missed serious deficiencies in these nine  states ranged from 26.3 percent in Tennessee to 33.3 percent in New  Mexico, South Carolina, South Dakota, and Wyoming (see table 2). The  total number of missed deficiencies at the G through L levels also varied  across these nine states, from a low of 4 in South Dakota to a high of 19 in  South Carolina. Federal surveyors identified no missed serious  deficiencies in seven states (see app. II for complete state results).
section_title: Missed Deficiencies at the Potential for More Than Minimal Harm Level Were Widespread on Federal Comparative Surveys
paragraphs: In contrast to missed serious deficiencies, missed deficiencies at the  potential for more than minimal harm level (D through F) were  considerably more widespread on comparative surveys conducted during  fiscal years 2002 through 2007. Approximately 70 percent of comparative  surveys conducted nationwide identified state surveys that missed at least  one deficiency at the potential for more than minimal harm level (D  through F), with such missed deficiencies identified on greater than   40 percent of comparative surveys in all but five states—Alaska, Ohio,  Vermont, West Virginia, and Wisconsin. On average, state surveys  selected for comparative surveys failed to identify 2.5 D through F level  deficiencies per survey. Undetected care problems at the D through F level  are of concern because they could become more serious over time if  nursing homes are not required to take corrective actions. Missed  deficiencies at the potential for more than minimal harm level were not  isolated to a single year during the 6 fiscal years we examined and  continued to be a problem for states in fiscal year 2007. Nationally, the  proportion of comparative surveys identifying at least one missed D  through F level deficiency in fiscal year 2007 was about 74 percent (see   fig. 2). For results by state, see appendix III.
section_title: Most Frequently Missed Deficiencies Involved Quality of Care
paragraphs: Our analysis found that the most frequently missed deficiencies at both the  potential for more than minimal harm (D through F) and the actual harm  or immediate jeopardy (G through L) levels occurred in quality standards  under CMS’s Quality of Care category. Missed deficiencies in this category  involved residents’ receipt of the necessary care and services to attain and  maintain the highest practicable physical, mental, and psychosocial well- being—such as prevention of pressure sores, nutrition and hydration,  accident prevention, and assistance with bathing and grooming.
From fiscal year 2002 through 2007, 11.9 percent of federal comparative  surveys (116) cited at least one Quality of Care deficiency at the actual  harm or immediate jeopardy level that state survey teams failed to cite.  These 116 surveys contained a total of 143 missed serious Quality of Care  deficiencies. The category with the next highest frequency of missed  serious deficiencies was Resident Behavior and Facility Practices, with  only 2.2 percent of total federal comparative surveys. At the potential for  more than minimal harm level, Quality of Care was one of two categories  with the highest frequency of missed deficiencies—31.7 percent. For the  percentage of missed deficiencies in each of the CMS quality standard  categories, see appendix IV.
section_title: Federal Observational Surveys and Prior GAO Reports Identified Factors That May Contribute to Deficiency Understatement by State Survey Teams
paragraphs: Both federal observational surveys and our prior reports have identified  factors that may contribute to the understatement of deficiencies by state  survey teams. From fiscal year 2002 through 2007, 80 percent of the 4,999  federal monitoring surveys were observational. Our review of  observational survey data—which are collected during direct observation  of state survey teams—found that some of the lowest state survey team  ratings nationwide were in the General Investigation and Deficiency  Determination areas. Together, these two areas directly affect the  appropriate identification and citation of deficiencies.
The General Investigation segment of an observational survey evaluates  the effectiveness with which the state survey team collected information  to determine how the facility’s environment and care of residents affect  residents’ quality of life, health and safety, and ability to reach their  highest practicable physical, mental, and psychosocial well-being. This  segment includes observations of state survey team actions such as  collection of information, discussion of survey observations, interviews  with facility residents, and implementation of CMS investigative protocols.
The Deficiency Determination segment of an observational survey  evaluates the skill with which the state survey teams (1) integrate and  analyze all information collected and (2) use the guidance to surveyors  and regulatory requirements to make accurate compliance determinations.  This segment includes observations of state survey team actions such as  reviews of regulatory requirements, team participation in deficiency  discussions, presentation of complete information, accurate decision  making, and accurate citation of deficiencies.
Nationwide, 7.7 percent of the state survey teams observed by federal  surveyors received below satisfactory ratings on the General Investigation  measure from fiscal year 2002 through 2007. During the same 6 fiscal  years, 9.2 percent, or about 1 in 11, of the state survey teams observed by  federal surveyors received below satisfactory ratings on the Deficiency  Determination measure. Our analysis found variation across states in  survey team performance in General Investigation and Deficiency  Determination. Sixteen states had more teams than the national average  receive below satisfactory ratings for both measures, while 28 states had  fewer teams than the national average receive below satisfactory ratings  (see app. V).
Poor performance on these observational survey measures may be a  contributing factor to the understatement of deficiencies by state survey  teams. For example, of the nine states in table 2 with the highest  percentage of missed serious deficiencies on comparative surveys, six had  more teams than the national average receive below satisfactory ratings  for both General Investigation and Deficiency Determination (see   table 3).
Our prior reports have described some other factors that may contribute  to survey inconsistency and the understatement of deficiencies by state  survey teams: (1) weaknesses in CMS’s survey methodology, such as poor  documentation of deficiencies; (2) confusion about the definition of  actual harm; (3) predictability of surveys, which allows homes to conceal  problems if they so desire; (4) inadequate quality assurance processes at  the state level to help detect understatement in the scope and severity of  deficiencies; and (5) inexperienced state surveyors as a result of  retention problems. In ongoing work, we are investigating the factors that  contribute to understatement.
section_title: CMS Has Taken Steps to Improve the Federal Monitoring Survey Program, but Weaknesses in Management and Oversight Remain
paragraphs: CMS has taken steps to improve the federal monitoring survey program,  but weaknesses remain in program management and oversight. For  example, CMS has improved processes to ensure that comparative surveys  more accurately reflect conditions at the time of the state survey, has  switched control of the federal monitoring survey database to the office  responsible for ensuring the effectiveness of state surveys, and has begun  examining how to use monitoring survey data to improve oversight.  Despite this progress, the management and oversight potential of the  program has not been fully realized. In particular, CMS (1) has only begun  exploring options for identifying understatement that occurs in cases  where state surveys cite deficiencies at too low a level, for possible  implementation in fiscal year 2009, and (2) is not effectively managing the  federal monitoring survey database or using the database to oversee  consistent implementation of the federal monitoring survey program by its  regional offices.
subsections: section_title: CMS Policy Changes Have Improved Federal Monitoring Surveys
paragraphs: CMS has taken steps in three areas—time between surveys, resident  sample, and survey resources—to ensure that comparative surveys more  accurately capture the conditions at the time of the state survey.
Time between surveys. In fiscal year 2002, CMS initiated a policy that  shortened the length of time between state and comparative surveys from  2 months to 1 month. CMS relaxed the 1 month standard by changing the  requirement to 30 working days in fiscal year 2003. As a result of  shortening the time between the two surveys, the conditions at the time of  the comparative survey are more likely to reflect those at the time of the  state survey; for example, the same residents are still likely to be in the  nursing home. Comparative surveys during fiscal year 2007 took place on  average 21.4 working days (30.9 calendar days) after state surveys.
Resident sample. Beginning in fiscal year 2003, CMS policy required that  comparative surveys include at least half of the residents from state survey  investigative samples. Officials from several regional offices said that  examining the same resident allows for more clear-cut determinations of  whether the state should have cited a deficiency. Since the policy change,  about 78 percent of comparative surveys from fiscal year 2003 through  2007 included at least half the residents from state surveys’ investigative  samples. By comparison, only 13 percent of comparative surveys met that  50 percent threshold in fiscal year 2002, the year before the policy went  into effect.
Survey resources. Beginning in fiscal year 2003, CMS initiated a policy  that each comparative survey should have the same number of federal  surveyors as its corresponding state survey, again to more closely mirror  the conditions under which the state survey was conducted. We found  that in fiscal year 2007, the average state survey team (3.4 surveyors) was  larger than the average federal survey team (3.0 surveyors). However, on  average, federal surveyors remained on-site longer than state surveyors— 4.3 days for federal surveyors compared with 3.7 days for state surveyors.  When the number of surveyors and time on-site are taken together, state  surveys averaged 12.6 surveyor-days and federal comparative surveys  averaged 12.9 surveyor-days.
Given these improvements, we asked the regional offices how receptive  state survey teams were to feedback that they had missed deficiencies.  Most regional office officials told us that in general the feedback session  with state surveyors on missed deficiencies was not contentious and that  state surveyors generally accepted the feedback provided. However, CMS  established a formal dispute resolution process for comparative surveys in  October 2007. The process is similar to the process already in place for  resolving disagreements about observational survey results.
section_title: Federal Monitoring Survey Database Has Not Been Used to Capture Understatement of Scope and Severity Levels
paragraphs: While CMS requires federal surveyors to determine whether a deficiency  cited on a comparative but not a state survey was missed by state  surveyors, there is no comparable requirement for deficiencies that are  cited at different scope and severity levels. As a result, comparative  surveys do not effectively capture the extent of the understatement of  serious deficiencies by state surveyors. As with missed deficiencies, a  discrepancy between federal and state survey results does not  automatically indicate understatement. For example, the deficiency could  have worsened by the time of the federal survey.
Although CMS does not require federal surveyors to evaluate scope and  severity differences between the two sets of surveys, we found that some  regional offices used the validation question for missed deficiencies— ”based on the evidence available to the , should the   team have cited this ?”—to make such a determination. Using  the validation question to make these determinations is contrary to CMS  guidance issued in October 2003, which instructed comparative survey  teams to only answer this question when the state failed to cite the  deficiency altogether.
To assess whether differences in scope and severity levels were actually  understated—rather than deficiencies that worsened between the state  and federal surveys—we first identified all 71 deficiencies on comparative  surveys conducted from fiscal year 2002 through 2007 where federal  survey teams cited actual harm or immediate jeopardy deficiencies that  state survey teams cited at a lower scope and severity level. We then  examined the comment fields in the federal monitoring survey database  associated with those deficiencies. Our analysis identified 27 deficiencies  (38 percent) in which federal survey teams determined that a state’s scope  and severity citation was too low. For another 22 deficiencies (31 percent),  federal survey teams found that the state’s lower scope and severity   determination was appropriate, given the circumstances at the time of the  state survey. The remaining 22 deficiencies (31 percent) did not have  comments or contained remarks that were inconclusive about whether the  state deficiency citation was too low. When the confirmed scope and  severity understatement was included with understatement caused by  missed deficiencies, the total percentage of comparative surveys with  understatement of serious deficiencies increased by an average of about   1 percentage point over the 6 fiscal years we analyzed (see fig. 3).
While CMS headquarters does not require federal surveyors to determine  whether a deficiency cited by state survey teams was cited at too low a  scope and severity level, some regional offices have developed their own  procedures to track this information and use it to provide feedback to  state survey agencies. For example, in one regional office an individual  reviews all comment fields for a year’s worth of comparative surveys,  makes a hand count of scope and severity differences that states should  have cited, and then shares this with the state survey agencies during their  annual performance reviews. Because the federal monitoring survey  database does not automatically collect data on scope and severity  determinations, CMS headquarters does not have access to the data  analyses the regions have independently conducted. Some of the regional  offices told us that they would like to have a specific way that the federal  monitoring survey database could track scope and severity  understatement that is similar to how deficiencies missed by state  surveyors are tracked.
In January 2008, CMS officials told us that they had initiated a pilot  program in October 2007 to test the collection of data on understatement  of scope and severity differences. According to CMS, the pilot, which will  run through 2008 for possible fiscal year 2009 implementation, is  necessary because the agency needs to determine which scope and  severity understatement differences should be captured. For example,  CMS is uncertain whether regions should only focus on differences that  would raise the scope and severity level to actual harm or immediate  jeopardy and not assess differences for understatement that occurs at  lower scope and severity levels.
section_title: CMS Could More Effectively Manage the Federal Monitoring Survey Database and Use It for Regional Office Oversight
paragraphs: Our analysis found that CMS headquarters was not effectively managing  the federal monitoring survey database or using the database to oversee  consistent implementation of the federal monitoring survey program by  regional offices. While CMS uses data from comparative and  observational surveys to provide feedback to state survey agencies during  state performance reviews, CMS officials told us that they recognized the  need to improve their management and use of the database for better  oversight of the agency’s 10 regional offices.
We identified two problems in CMS’s management of the federal  monitoring survey database. CMS was not aware that (1) the results of a  considerable number of comparative surveys were missing from the  database and (2) the validation question for missed deficiencies was being  used by some regional offices to identify scope and severity differences,  contrary to CMS guidance.
Missing data. In October 2007, we identified missing comparative surveys  for two regional offices dating back to 2005 and asked CMS to follow up  with officials in those regions. At least one of the regions had completed  the surveys but had failed to upload them into the national database. We  also found that CMS had not included data in the federal monitoring  survey database from 162 contractor-led comparative surveys conducted  between fiscal years 2004 and 2007.
Use of validation question contrary to CMS guidance. Some regional  offices were using the missed deficiency validation question to make  determinations about whether scope and severity differences constituted  understatement, making it difficult to distinguish between missed  deficiencies and scope and severity understatement. In addition, we found  that the regional office answer to the validation question was not always  consistent with the information recorded in the comment box.
Similarly, we identified weaknesses in CMS’s use of the database for  regional office oversight. For example, CMS was not (1) examining  comparative survey data to ensure that regional offices comply with CMS  guidance intended to ensure that comparative surveys more accurately  capture the conditions at the time of the state survey and (2) using the  database to identify inconsistencies between comparative and  observational survey results.
Ensuring regional office compliance. While CMS has provided  guidance to its regional offices to help ensure that comparative surveys  more accurately capture the conditions at the time of the state survey, the  agency is not fully using available data to ensure that the regional offices  implement the agency’s guidance. For example, we found that the length  of time between state and comparative surveys varied broadly by CMS  region. In 2007, the average time gap ranged from a low of 15.4 working  days (22.5 calendar days) in the Boston region to a high of 38.5 working  days (54.4 calendar days) in the New York region. Furthermore, while   78 percent of comparative surveys from fiscal year 2003 through 2007  followed CMS’s guidance to include at least half of the residents from state  surveys’ investigative samples, 22 percent of comparative surveys did not  meet this threshold. Finally, when we contacted officials in CMS  headquarters to ask clarifying questions about the data variables needed to  conduct these analyses, the headquarters officials were not familiar with a  number of the variables and referred us to a CMS staff person in one of the  regional offices. Together, these three examples suggest that CMS is not  effectively using the data to hold regional offices accountable for  implementing guidance.
Identify inconsistencies between comparative and observational  results. CMS officials told us that they have begun to explore regional  office differences in less than satisfactory ratings for state survey teams on  observational surveys. However, CMS officials told us that they do not  plan to use the database to identify inconsistencies between comparative  and observational surveys that may warrant follow-up to ensure that  regional offices are adhering to CMS guidance and consistently assessing  state surveyor performance. For example, some states that performed  below the national average in identifying serious deficiencies on  comparative surveys received above-average marks on observational  survey measures for Deficiency Determination and General Investigation.  Wyoming’s 33.3 percent rate for surveys with missed serious deficiencies  was more than double the national average of about 15 percent for surveys  conducted during fiscal years 2002 and 2007. Yet Wyoming never received  a below satisfactory rating on its General Investigation or Deficiency  Determination measures during 18 observational surveys over that same   6-year period. We found similar inconsistencies in the results of federal  monitoring surveys for South Dakota and a few other states. Although  inconsistencies between comparative and observational surveys may not  necessarily indicate a problem, they may warrant investigation. For  example, in a small state like Wyoming it is likely that comparative and  observational surveys have evaluated the same group of state surveyors.  Further, Wyoming and South Dakota are two of six states whose federal  monitoring surveys are conducted by CMS’s Denver regional office. Of the  140 observational surveys conducted from fiscal year 2002 through 2007,  federal surveyors from the Denver regional office gave one below  satisfactory rating on the Deficiency Determination measure. That   0.7 percent rate of below satisfactory performance was more than four  times lower than the regional office with the next-lowest percentage—the  Chicago regional office—which awarded below satisfactory ratings to   3.3 percent of state survey teams it observed.
section_title: Conclusions
paragraphs: With about 1 in 6 comparative surveys concluding that state survey teams  had missed a serious deficiency or understated its scope and severity level,  it is evident that state survey agency performance limits the federal  government’s ability to obtain an accurate picture of how often nursing  home residents face actual harm or are at risk of serious injury or death.  These missed serious deficiencies most frequently involved Quality of  Care, reflecting shortcomings in fundamental provider responsibilities  such as ensuring proper nutrition and hydration, accident prevention, and  preventing pressure sores. Observational survey results also underscore  problems state surveyors may face in identifying facility deficiencies;  about 1 in 11 state survey teams nationwide were rated as below  satisfactory by CMS surveyors on the Deficiency Determination measure.
We found that comparative survey data may mask the true extent of  understatement because CMS’s current protocol does not require regional  offices to track in the federal monitoring survey database when state  surveyors cite lower-than-appropriate scope and severity levels. As we  conducted our work, CMS officials recognized this problem and in  October 2007 began to experiment with a pilot program to measure  understated scope and severity. However, at the conclusion of the pilot,  scheduled for fiscal year 2008, CMS may decide not to implement a  validation question for all scope and severity differences. We believe it is  important to assess differences for understatement that occurs at the   D through L levels—potential for more than minimal harm, actual harm,  and immediate jeopardy.
We also found that CMS was not effectively managing the federal  monitoring survey database to ensure that regional offices were entering  data in a timely and consistent fashion. Lack of accurate and reliable data  hinders effective oversight. For example, we found that the database was  missing a considerable number of comparative surveys. Further, CMS has  not used the federal monitoring survey database to its full potential as an  oversight tool. For example, CMS is not fully using data on comparative  surveys to ensure that regional offices are implementing guidance  intended to improve federal monitoring surveys. Although CMS’s Survey  and Certification Group assumed control of the database in January 2007,  headquarters staff often referred us to CMS regional office staff to answer  specific database questions, suggesting a lack of familiarity with the  organization and content of the database. In addition, agency officials told  us that they do not plan to follow up on inconsistencies between  comparative and observational survey results that could indicate  weaknesses in how regional offices evaluate state surveyors’ performance.  Identifying and following up on such inconsistencies could help ensure  database reliability and hold regional office officials accountable for their  implementation of the federal monitoring survey program, a program  required by statute.
section_title: Recommendations for Executive Action
paragraphs: To address weaknesses in CMS’s management of the federal monitoring  survey database that also affect the agency’s ability to effectively track  understatement, we recommend that the Administrator of CMS take the  following two actions:    Require regional offices to determine if there was understatement when  state surveyors cite a deficiency at a lower scope and severity level than  federal surveyors do and to track this information in the federal  monitoring survey database.
Establish quality controls to improve the accuracy and reliability of  information entered into the federal monitoring survey database.
To address weaknesses that affect CMS’s ability to oversee regional office  implementation of the federal monitoring survey program, we recommend  that the Administrator of CMS take the following two actions:    Routinely examine comparative survey data and hold regional offices  accountable for implementing CMS guidance that is intended to ensure  that comparative surveys more accurately capture the conditions at the  time of the state survey.
Regularly analyze and compare federal comparative and observational  survey results.
section_title: Agency Comments
paragraphs: In written comments on our draft report, HHS indicated that it fully  endorsed and would implement our four recommendations intended to  strengthen management and oversight of the federal monitoring survey  program. The comments generally outlined CMS’s implementation plan  through 2009 and indicated that some steps, such as improved  management of the federal monitoring survey database, are already under  way. HHS’s comments are reproduced in appendix VI.
The majority of HHS’s comments focused on its strategic approach to  improving oversight: (1) ensuring that all nursing homes are surveyed at  least once every 15 months, (2) improving surveyor understanding of  federal quality requirements through improved guidance and training,   (3) increasing the consistency of state surveys through the introduction of  a new nursing home survey methodology, and (4) improving the use of  data generated by federal monitoring surveys. Many of these strategies aim  to address the underlying causes of understatement, the topic of a  forthcoming GAO report. HHS also noted that limitations in the Medicare  survey and certification budget underscore the agency’s need to target  resources effectively to maximize results. For example, HHS indicated that  the implementation of the new survey methodology will be dependent on  the level of funding in the overall survey and certification budget through  fiscal year 2014. Survey and Certification funding is the subject of another  forthcoming GAO report.
Two of HHS’s observations merit further discussion. First, HHS noted that  understatement that arises from a lack of understanding or confusion  about federal requirements would generally not be detected through  federal monitoring surveys because both federal and state surveyors  would be affected by the same limitation. We believe that the consistency  with which federal surveys have identified serious deficiencies missed by  state surveyors from fiscal year 2002 through 2007—about 15 percent, on  average—suggests that federal surveyors have a better understanding of  CMS quality requirements than do state surveyors. We have previously  reported that the limited experience level of state surveyors because of the  high turnover rate was a contributing factor to deficiency  understatement.
Second, HHS questioned our use of “one missed deficiency per survey” as  a measure of understatement. We believe that this standard is appropriate  for serious deficiencies that result in harm or immediate jeopardy   (G through L level) because the goal of state surveys should be to identify  and require nursing homes to address all such deficiencies. CMS itself uses  this standard during annual state performance reviews. We also used this  standard to describe the proportion of comparative surveys that identified  missed deficiencies at the potential for more than minimal harm level   (D through F). Identifying and requiring nursing homes to correct such  deficiencies is important because if uncorrected they have the potential to  become more serious. Compared to missed serious deficiencies, we found  that understatement of potential for more than minimal harm deficiencies  was more widespread—about 70 percent of comparative surveys identified  at least one state survey with such missed deficiencies. The number of  state surveys with missed deficiencies at the D through F level was greater  than 40 percent in all but five states, and state surveys selected for  comparative surveys failed to identify an average of 2.5 deficiencies in this  range per survey. In short, the magnitude of understatement at the  potential for more than minimal harm level should be a cause for concern.
HHS also provided technical comments, which we incorporated as  appropriate.
As arranged with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies to the Administrator of the  Centers for Medicare & Medicaid Services and appropriate congressional  committees. We will also make copies available to others upon request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix VII.
section_title: Appendix I: Percentage of Nursing Homes Cited for Actual Harm or Immediate Jeopardy during Standard Surveys
paragraphs: In order to identify trends in the percentage of nursing homes cited with  actual harm or immediate jeopardy deficiencies, we analyzed data from  the Centers for Medicare & Medicaid Service’s (CMS) On-Line Survey,  Certification, and Reporting system (OSCAR) database for fiscal years  2002 through 2007 (see table 4). Because homes must be surveyed at least  every 15 months, with a required 12-month statewide average, it is possible  that a home was surveyed more than once in any fiscal year. To avoid  double counting homes, we included only a home’s most recent survey  from each fiscal year. Because CMS conducts a relatively small number of  comparative surveys, it is not possible to compare the results of  comparative surveys to the results of all state surveys.
section_title: Appendix II: Percentage of Comparative Surveys Identifying Missed Deficiencies at Actual Harm or Immediate Jeopardy Level
section_title: Appendix III: Percentage of Comparative Surveys Identifying Missed Deficiencies with Potential for More Than Minimal Harm
section_title: Appendix IV: Percentage of Comparative Surveys with at Least One Missed Deficiency, by Federal Quality Standard Category
section_title: Appendix V: Percentage of Below Satisfactory State Survey Ratings for General Investigation and Deficiency Determination
section_title: Appendix VI: Comments from the Department of Health & Human Services
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Walter Ochinko, Assistant  Director; Katherine Nicole Laubacher; Dan Lee; Elizabeth T. Morrison;   Steve Robblee; Karin Wallestad; and Rachael Wojnowicz made key  contributions to this report.
section_title: Related GAO Products
paragraphs: Nursing Home Reform: Continued Attention Is Needed to Improve  Quality of Care in Small but Significant Share of Homes. GAO-07-794T.  Washington, D.C.: May 2, 2007.
Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not  Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241.  Washington, D.C.: March 26, 2007.
Nursing Homes: Despite Increased Oversight, Challenges Remain in  Ensuring High-Quality Care and Resident Safety. GAO-06-117.  Washington, D.C.: December 28, 2005.
Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses  in State and Federal Oversight of Quality of Care. GAO-05-78.  Washington, D.C.: November 12, 2004.
Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in  Federal Standards and Oversight. GAO-04-660. Washington D.C.: July 16,  2004.
Nursing Home Quality: Prevalence of Serious Problems, While  Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561.  Washington, D.C.: July 15, 2003.
Nursing Homes: Public Reporting of Quality Indicators Has Merit, but  National Implementation Is Premature. GAO-03-187. Washington, D.C.:  October 31, 2002.
Nursing Homes: Quality of Care More Related to Staffing than Spending.  GAO-02-431R. Washington, D.C.: June 13, 2002.
Nursing Homes: More Can Be Done to Protect Residents from Abuse.  GAO-02-312. Washington, D.C.: March 1, 2002.
Nursing Homes: Federal Efforts to Monitor Resident Assessment Data  Should Complement State Activities. GAO-02-279. Washington, D.C.:  February 15, 2002.
Nursing Homes: Sustained Efforts Are Essential to Realize Potential of  the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.:   September 28, 2000.
Nursing Home Care: Enhanced HCFA Oversight of State Programs  Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.:  November 4, 1999.
Nursing Home Oversight: Industry Examples Do Not Demonstrate That  Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington,  D.C.: August 13, 1999.
Nursing Homes: Proposal to Enhance Oversight of Poorly Performing  Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999.
Nursing Homes: Complaint Investigation Processes Often Inadequate to  Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999.
Nursing Homes: Additional Steps Needed to Strengthen Enforcement of  Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.:   March 18, 1999.
California Nursing Homes: Care Problems Persist Despite Federal and  State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.