Nursing Homes: Addressing the Factors Underlying Understatement of Serious Care Problems Requires Sustained CMS and State Commitment
Published: 2009-11-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Under contract with the Centers for Medicare and Medicaid Services (CMS), states conduct surveys at nursing homes to help ensure compliance with federal quality standards. Over the past decade, the Government Accountability Office (GAO) has reported on inconsistencies in states' assessment of nursing homes' quality of care, including understatement--that is, when state surveys fail to cite serious deficiencies or cite them at too low a level. In 2008, GAO reported that 9 states had high and 10 had low understatement based on CMS data for fiscal years 2002 through 2007. This report examines the effect on nursing home deficiency understatement of CMS's survey process, workforce shortages and training, supervisory reviews of surveys, and state agency practices. GAO primarily collected data through two Web-based questionnaires sent to all eligible nursing home surveyors and state agency directors, achieving 61 and 98 percent response rates, respectively.
section_title: What GAO Found
paragraphs: A substantial percentage of both state surveyors and directors identified general weaknesses in the nursing home survey process, that is, the survey methodology and guidance on identifying deficiencies. On the questionnaires, 46 percent of surveyors and 36 percent of directors reported that weaknesses in the traditional survey methodology, such as too many survey tasks, contributed to understatement. Limited experience with a new data-driven survey methodology indicated possible improvements in consistency; however, an independent evaluation led CMS to conclude that other tools, such as survey guidance clarification and surveyor training and supervision, would help improve survey accuracy. According to questionnaire responses, workforce shortages and greater use of surveyors with less than 2 years' experience sometimes contributed to understatement. Nearly three-quarters of directors reported that they always or frequently experienced a workforce shortage, while nearly two-thirds reported that surveyor inexperience always, frequently, or sometimes led to understatement. Substantial percentages of directors and surveyors indicated that inadequate training may compromise survey accuracy and lead to understatement. According to about 29 percent of surveyors in 9 high understatement states compared to 16 percent of surveyors in 10 low understatement states, initial surveyor training was not sufficient to cite appropriate scope and severity--a skill critical in preventing understatement. Furthermore, over half of directors identified the need for ongoing training for experienced surveyors on both this skill and on documenting deficiencies, a critical skill to substantiate citations. CMS provides little guidance to states on supervisory review processes. In general, directors reported on our questionnaire that supervisory reviews occurred more often on surveys with higher-level rather than on those with lower-level deficiencies, which were the most frequently understated. Surveyors who reported that survey teams had too many new surveyors also reported frequent changes to or removal of deficiencies, indicating heavier reliance on supervisory reviews by states with inexperienced surveyors. Surveyors and directors in a few states informed us that, in isolated cases, state agency practices or external pressure from stakeholders, such as the nursing home industry, may have led to understatement. Forty percent of surveyors in five states and four directors reported that their state had at least one practice not to cite certain deficiencies. Additionally, over 40 percent of surveyors in four states reported that their states' informal dispute resolution processes favored concerns of nursing home operators over resident welfare. Furthermore, directors from seven states reported that pressure from the industry or legislators may have compromised the nursing home survey process, and two directors reported that CMS's support is needed to deal with such pressure. If surveyors perceive that certain deficiencies may not be consistently upheld or enforced, they may choose not to cite them.

section_title: Letter
paragraphs: Federal and state governments share responsibility for ensuring that  nursing homes provide quality care in a safe environment for the nation’s  1.5 million residents dependent on such care. The federal government is  responsible for setting quality requirements that nursing homes must meet  to participate in the Medicare and Medicaid programs. The Centers for  Medicare & Medicaid Services (CMS), within the Department of Health &  Human Services (HHS), contracts with state survey agencies to conduct  periodic inspections, known as surveys, and complaint investigations, both  of which assess whether homes meet federal standards. State survey  agencies are required to follow federal regulations for surveying facilities;  however, several survey activities and policies are left largely to the  discretion of state survey agencies, including hiring and retaining a  surveyor workforce, training surveyors, reviewing deficiency citations,  and managing regulatory interactions with the industry and public.
In response to congressional requests over the last decade, we have  reported significant weaknesses in federal and state activities designed to  detect and correct quality and safety problems in nursing homes and the  persistence of serious deficiencies, which are those deficiencies that harm  residents or place them at risk of death or serious injury. In the course of  our work, we regularly found significant variation across states in their  citations of serious deficiencies—indicating inconsistencies in states’  assessment of quality of care. We also found evidence of substantial  understatement—that is, state inspections that failed to cite serious  deficiencies or that cited deficiencies at too low a level.
In this report, we complete our response to your request to examine the  understatement of serious deficiencies in nursing homes by state  surveyors nationwide and the factors that contribute to understatement.  Our first report, issued in May 2008, identified the extent of nursing home  understatement nationwide. It found that 15 percent of federal nursing  home surveys nationwide and 25 percent of these surveys in nine states  identified state surveys that failed to cite serious deficiencies. This report  examines how the following factors affect the understatement of nursing  home deficiencies: (1) the CMS survey process, (2) workforce shortages  and training, (3) supervisory reviews, and (4) state agency practices.
To do this work, we analyzed data collected from two GAO-administered  Web-based questionnaires, one to nursing home surveyors and the other to  state agency directors; analyzed federal and state nursing home survey  results; interviewed CMS officials from the Survey and Certification Group  and selected Regional Offices; reviewed federal regulations and guidance,  and our prior work; and conducted follow-up interviews with state agency  directors, as needed, to clarify and better understand their unique state  circumstances.
Our prior work documented the prevalence of understatement   nationwide and described several factors that may contribute to survey  inconsistency and the understatement of deficiencies by state survey  teams: (1) weaknesses in CMS’s survey methodology, including poor  documentation of deficiencies, (2) confusion among surveyors about the  definition of actual harm, (3) predictability of surveys, which allows  homes to conceal problems if they so desire, (4) inadequate quality  assurance processes at the state level to help detect understatement in the  scope and severity of deficiencies, and (5) inexperienced state surveyors  as a result of retention problems. We relied on this information and  feedback from pretests with six surveyors from a local state and five  current or former state agency directors to develop our questionnaires on  the nursing home survey process and factors that contribute to the  understatement of deficiencies.
Our Web-based questionnaires of nursing home surveyors and state  agency directors achieved response rates of 61 percent and 98 percent,  respectively. The first questionnaire collected responses from 2,340 of the  total 3,819 eligible nursing home surveyors in 49 states and the District of  Columbia. The resulting sample of surveyors who responded to our  questionnaire between May and July 2008 was representative of surveyors  nationally, with the exception of Pennsylvania. Fifty state agency  directors responded to the second questionnaire from September to  November 2008. Many questions on our questionnaires asked  respondents to identify the frequency that an event occurred using the  following scale—always, frequently, sometimes, infrequently, or never;  however, for reporting purposes, we grouped responses into three  categories—always/frequently, sometimes, and infrequently/never. In  addition, our questionnaire to state agency directors asked them to rank  the degree to which several factors, derived from our previous work,  contributed to understatement. Summary results from the GAO  questionnaires are available as an e-supplement to this report. See  Nursing Homes: Responses from Two Web-Based Questionnaires to  Nursing Home Surveyors and State Agency Directors (GAO-10-74SP), an  e-supplement to GAO-10-70.
We analyzed the data collected from these questionnaires as stand-alone  datasets and in relationship to state performance on federal comparative  and observational surveys as captured in the federal monitoring survey  database, which we reported on in 2008. In addition, to inform our  understanding of the extent to which each factor contributed to  understatement, we examined relationships among the responses to both  questionnaires and the results of the federal comparative and  observational surveys for fiscal years 2002 through 2007. We used the  results of the federal comparative surveys for these years to identify states  with high and low percentages of serious missed deficiencies. We report  results for tests of association and differences between group averages.
We also interviewed directors and other state agency officials in eight  states to better understand unusual or interesting circumstances related to  surveyor workforce and training, supervisory review, or state policies and  practices. We selected these eight state agencies based on our analysis of  questionnaire responses from the state agency directors and nursing home  surveyors.
To compare average facility citations on state survey records with the  average citations on federal survey records, we collected information from  the On-Line Survey, Certification, and Reporting (OSCAR) system for  those facilities where federal teams assessed state surveyor performance  for fiscal years 2002 through 2007. Except where otherwise noted, we  used data from fiscal year 2007 because they were the most recently  available data at the time of our analysis (see appendix I for more on our  scope and methodology).
We conducted this performance audit from April 2008 through November  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Oversight of nursing homes is a shared federal-state responsibility.   As part of this responsibility, CMS (1) sets federal quality standards,   (2) establishes state responsibilities for ensuring federal quality standards  are met, (3) issues guidance on determining compliance with these  standards, and (4) performs oversight of state survey activities. It  communicates these federal standards and state responsibilities in the  State Operations Manual (SOM) and through special communications such  as program memorandums and survey and certification letters. CMS  provides less guidance on how states should manage the administration of  their survey programs. CMS uses staff in its 10 regional offices to oversee  states’ performance on surveys that ensure that facilities participating in  Medicare and Medicaid provide high-quality care in a safe environment.  Yet, the persistent understatement of serious nursing home deficiencies  that we have reported and survey quality weaknesses that we and the HHS  Office of Inspector General identified serve as indicators of weaknesses in  the federal, state, or shared components of oversight.
subsections: section_title: Survey Process
paragraphs: Every nursing home receiving Medicare or Medicaid payment must  undergo a standard state survey not less than once every 15 months,   and the statewide average interval for these surveys must not exceed   12 months. During a standard survey, teams of state surveyors—generally  consisting of registered nurses, social workers, dieticians, or other  specialists—evaluate compliance with federal quality standards. The  survey team determines whether the care and services provided meet the  assessed needs of the residents and measure resident outcomes, such as  the incidence of preventable pressure sores, weight loss, and accidents. In  contrast to a standard survey, a complaint investigation generally focuses  on a specific allegation regarding a resident’s care or safety and provides  an opportunity for state surveyors to intervene promptly if problems arise  between standard surveys.
Surveyors assess facilities using federal nursing home quality standards  that focus on the delivery of care, resident outcomes, and facility  conditions. These standards total approximately 200 and are grouped into  15 categories, such as Quality of Life, Resident Assessment, Quality of  Care, and Administration. For example, there are 23 standards (known as  F-tags) within the Quality of Care category ranging from the prevention of  pressure sore development (F-314) to keeping the resident environment as  free of accident hazards (F-323) as is possible.
Surveyors categorize deficient practices identified on standard surveys  and complaint investigations—facilities’ failures to meet federal  standards—according to scope (i.e., the number of residents potentially or  actually affected) and severity (i.e., the degree of relative harm  involved)—using a scope and severity grid (see table 1). Homes with  deficiencies at the A through C levels are considered to be in substantial  compliance, while those with deficiencies at the D through L levels are  considered out of compliance. Throughout this report, we refer to  deficiencies at the actual harm and immediate jeopardy levels—G through  L—as serious deficiencies. CMS guidance requires state survey teams to  revisit a home to verify that serious deficiencies have actually been  corrected.
In addition, when serious deficiencies are identified, sanctions can be  imposed to encourage facilities to correct the deficiencies and enforce  federal quality standards. Sanctions include fines known as civil money  penalties, denial of payment for new Medicare or Medicaid admissions, or  termination from the Medicare and Medicaid programs. For example,  facilities that receive at least one G through L level deficiency on  successive standard surveys or complaint investigations must be referred  for immediate sanctions. Facilities may appeal cited deficiencies and if the  appeal is successful, the severity of the sanction could be reduced or the  sanction could be rescinded. Facilities have several avenues of appeal,  including informal dispute resolution (IDR) at the state survey agency  level. The IDR gives providers one opportunity to informally refute cited  deficiencies after any survey. While CMS requires that states have an IDR  policy in place, it does not specify how IDR processes should be  structured.
section_title: Survey Methodology
paragraphs: To conduct nursing home surveys, CMS has traditionally used a  methodology that requires surveyors to select a sample of residents and  (1) review data derived from the residents’ assessments and medical  records; (2) interview nursing home staff, residents, and family members;  and (3) observe care provided to residents during the course of the survey.  When conducting a survey, surveyors have discretion in: selecting a  sample of residents to evaluate; allocating survey time and emphasis  within a framework prescribed by CMS; investigating potentially deficient  practices observed during the survey; and determining what evidence is  needed to identify a deficient practice. CMS has developed detailed  investigative protocols to assist state survey agencies in determining  whether nursing homes are in compliance with federal quality standards.  These protocols are intended to ensure the thoroughness and consistency  of state surveys and complaint investigations.
In 1998, CMS awarded a contract to revise the survey methodology. The  new Quality Indicator Survey (QIS) was developed to improve the  consistency and efficiency of state surveys and provide a more reliable  assessment of quality. The QIS uses an expanded sample of residents and  structured interviews with residents and family members in a two-stage  process. Surveyors are guided through the QIS process using customized  software on tablet personal computers. In stage 1, a large resident sample  is drawn and relevant data from on- and off-site sources is analyzed to  develop a set of quality-of-care indicators, which will be compared to  national benchmarks. Stage 2 systematically investigates potential  quality-of-care concerns identified in stage 1. Because of delays in  implementing the QIS, we recommended in 2003 that CMS finalize the  development, testing, and implementation of a more rigorous survey  methodology, including investigative protocols that provide guidance to  surveyors in documenting deficiencies at the appropriate scope and  severity level. CMS concluded a five-state demonstration process of the  QIS in 2007 and is currently expanding the implementation of the QIS. As  of 2008, only Connecticut had implemented the QIS statewide, and CMS  projected that the QIS would not be fully implemented in every state until  2014.
section_title: State Administration
paragraphs: States are largely responsible for the administration of the survey  program. State survey agencies administer and have discretion over many  survey activities and policies, including hiring and retaining a surveyor  workforce, training surveyors, conducting supervisory reviews of surveys,  and other activities.
Hiring and Retaining a Surveyor Workforce: State survey agencies hire the  staff to conduct surveys of nursing homes and determine the salaries of  these personnel according to the workforce practices and restrictions of  the state. Salaries, particularly surveyor salaries, are the most significant  cost component of state survey activities, which are supported through a  combination of Medicare, Medicaid, and non-Medicaid state funds. CMS  has some requirements for the make-up of nursing home survey teams,  including the involvement of at least one registered nurse (RN) in each  nursing home survey. In February 2009, we reported that officials from the  Association of Health Facility Survey Agencies (AHFSA) and other state  officials told us they have had difficulty recruiting and retaining the survey  workforce for several years. In our report, we recommended that CMS  undertake a broad-based reexamination to ensure, among other aspects,  an adequate survey workforce with sufficient compensation to attract and  retain qualified staff.
Training: States are responsible for training new surveyors through  participating in actual surveys under direct supervision. Within their first  year of employment, surveyors must complete two CMS online training  courses—the Basic Health Facility Surveyor Course and Principles of  Documentation—and a week-long CMS-led Basic Long-Term Care Health  Facility Surveyor Training Course; at the conclusion of the course  surveyors must pass the Surveyor Minimum Qualifications Test (SMQT) to  survey independently. In addition, state survey agencies are required to  have their own programs for staff development that respond to the need  for continuing development and education of both new and experienced  employees. Such staff development programs must include training for  surveyors on all regulatory requirements and the skills necessary to  conduct surveys. To assist in continuing education, CMS develops a  limited number of courses for ongoing training and provides other training  materials.
Supervisory Reviews: States may design a supervisory review process for  deficiencies cited during surveys, although CMS does not require them to  do so. In July 2003, we recommended that CMS require states to have a  minimum quality-assurance process that includes a review of a sample of  survey reports below the level of actual harm to assess the  appropriateness of scope and severity levels cited and help reduce  instances of understated quality-of-care problems. CMS did not  implement this recommendation.
State Agency Practices and Policies: State survey agencies’ practices,  including those on citing deficiencies and addressing pressure from the  industry or others, are largely left to the discretion of state survey  agencies. In the past, we reported that in one state, CMS officials had  found surveyors were not citing all deficiencies. If a state agency fails to  cite all deficiencies associated with noncompliance, nursing home  deficiencies are understated on the survey record. CMS can identify or  monitor states for systematic noncitation practices through reviews of  citation patterns, informal feedback from state surveyors, state  performance reviews, and federal monitoring surveys (discussed below).  CMS also gives states latitude in defining their IDR process.
section_title: Federal Monitoring Surveys and Evidence of Understatement
paragraphs: Federal law requires federal surveyors to conduct federal monitoring  surveys in at least 5 percent of state-surveyed Medicare and Medicaid  nursing homes in each state each year. CMS indicates it meets the  statutory requirement by conducting a mix of on-site reviews: comparative  and observational surveys.
Comparative surveys. A federal survey team conducts an independent  survey of a home recently surveyed by a state survey agency in order to  compare and contrast its findings with those of the state survey team. This  comparison takes place after completion of the federal survey. When  federal surveyors identify a deficiency not cited by state surveyors, they  assess whether the deficiency existed at the time of the state survey and  should have been cited. This assessment is critical in determining  whether understatement occurred, because some deficiencies cited by  federal surveyors may not have existed at the time of the state survey.
Our May 2008 report stated that comparative surveys found problems at  the most serious levels of noncompliance—the actual harm and immediate  jeopardy levels (G through L). About 15 percent of federal comparative  surveys nationwide identified at least one deficiency at the G through L  level that state surveyors failed to cite. While this proportion is small, CMS  maintains that any missed serious deficiencies are unacceptable. Further,  state surveys with understated deficiencies may allow the surveyed  facilities to escape sanctions intended to discourage repeated  noncompliance.
In our May 2008 report we found that for nine states federal surveyors  identified missed serious deficiencies in 25 percent or more comparative  surveys for fiscal years 2002 through 2007; we defined these states as high- understatement states (see fig. 1). Zero-understatement states were states  that had no federal comparative surveys identifying missed deficiencies at  the actual harm or immediate jeopardy levels; and low-understatement  states were the 10 states with the lowest percentage of missed serious  deficiencies (less than 6 percent), including all 7 zero-understatement  states.
Our May 2008 report also found that missed deficiencies at the potential  for more than minimal harm level (D through F) were considerably more  widespread than those at the G through L level on comparative surveys,  with approximately 70 percent of comparative surveys nationwide  identifying at least one missed deficiency at this level. Undetected care  problems at this level are of concern because they could become more  serious over time if nursing homes are not required to take corrective  actions.
Observational surveys. Federal surveyors accompany a state survey  team to evaluate the team’s performance and ability to document survey  deficiencies. State teams are evaluated in six areas, including two— General Investigation and Deficiency Determination—that affect the  appropriate identification and citation of deficiencies. The General  Investigation segment assesses the effectiveness of state survey team  actions such as collection of information, discussion of survey  observations, interviews with nursing home residents, and implementation  of CMS investigative protocols. The Deficiency Determination segment  evaluates the skill with which the state survey teams (1) analyze and  integrate all information collected, (2) use the guidance for surveyors, and  (3) assess compliance with regulatory requirements. Federal observational  surveys are not independent evaluations of the state survey because state  surveyors may perform their survey tasks more attentively than they  would if federal surveyors were not present; however, they provide more  immediate feedback to state surveyors and may help identify state  surveyor training needs.
We previously reported that state survey teams’ poor performance on  federal observational surveys in the areas of General Investigation and  Deficiency Determination may contribute to the understatement of  deficiencies. Further, poor state performance in these two areas  supported the finding of understatement as identified through the federal  comparative surveys. We found that about 8 percent of state survey teams  observed by federal surveyors nationwide received below-satisfactory  ratings on General Investigation and Deficiency Determination from fiscal  years 2002 through 2007. However, surveyors in high-understatement  states performed worse in these two areas of the federal observational  surveys than surveyors in the low-understatement states. For example, an  average of 12 and 17 percent of state survey teams observed by federal  surveyors in high-understatement states received below satisfactory  ratings for these two areas, respectively. In contrast, an average of 4  percent of survey teams in low-understatement states received the same  below-satisfactory scores for both deficiency determination and  investigative skills.
Nationwide, one-third of nursing homes had a greater average number of  serious deficiencies on federal observational surveys than on state  standard surveys during fiscal years 2002 through 2007, but in eight states,  it was more than half of homes. Of the one-third of homes nationwide,  state standard surveys cited 83 percent fewer serious deficiencies than  federal surveys during this same time period.
section_title: Weaknesses in CMS Survey Process Contributed to Understatement, but Long-Term Effect of New Survey Methodology Is Not Yet Known
paragraphs: Over a third of both surveyors and state agency directors responding to  our questionnaire identified weaknesses in the federal government’s  nursing home survey process that contributed to the understatement of  deficiencies. The weaknesses included problems with the current survey  methodology; written guidance that is too long or complex; and to a lesser  extent, survey predictability or other advance notice of inspections, which  may allow nursing homes to conceal deficiencies. At the time our  questionnaires were fielded, eight states had started implementing CMS’s  new survey methodology. The limited experience among these states  suggests that the new methodology may improve consistency of surveys,  but information is limited, and the long-term ability of the new  methodology to reduce understatement is not yet known.
subsections: section_title: Weaknesses in CMS’s Survey Process Contributed to Understatement
paragraphs: Both surveyors and state agency directors reported weaknesses in the  survey process, and on our questionnaire linked these weaknesses to  understatement of deficiencies. Nationally, 46 percent of nursing home  surveyors responded that weaknesses in the current survey methodology  resulted in missed or incorrectly identified deficiencies, with this number  ranging by state from 0 to 74 percent (see table 2). Thirty-six percent of  state agency directors responded that weaknesses in the current survey  methodology at least sometimes contributed to understatement of  deficiencies in their states. One such weakness identified by both  surveyors and directors was the number of survey tasks that need to be  completed.
According to surveyors and agency directors responding to our  questionnaire, another weakness with the federal survey process involved  CMS’s written guidance to help state agencies follow federal regulations  for surveying long-term care facilities. Both surveyors and state agency  directors mentioned concerns about the length, complexity, and  subjectivity of the written guidance. One state agency director we  interviewed told us that the size of the SOM made it difficult for surveyors  to carry the guidance and consult it during surveys. Although the SOM is  available in an electronic format, surveyors in this state did not use  laptops. In addition, a small percentage of surveyors commented on our  questionnaire that CMS guidance was inconsistently applied in the field. A  common complaint from these surveyors was that different supervisors  required different levels of evidence in order to cite a deficiency at the  actual harm or immediate jeopardy level. Forty percent of surveyors and  58 percent of state agency directors reported that additional training on  how to apply CMS guidance was needed.
A specific concern raised about the current survey guidance was  determining the severity level for an observed deficiency. Forty-four  percent of state agency directors reported on our questionnaire that  confusion about CMS’s definition of the actual–harm level severity  requirements at least sometimes contributed to understatement in their  states. CMS’s guidance for determining actual harm states, “this does not  include a deficient practice that only could or has caused limited  consequence to the resident.” State agency directors from several states  found this language confusing, including one director who said it is  unclear whether conditions like dehydration that are reversed in the  hospital should be cited as actual harm. As we reported in 2003, CMS  officials acknowledged that the language linking actual harm to practices  that have “limited consequences” for a resident has created confusion;  however, the agency has not changed or revised this language.
State agency directors and surveyors indicated that CMS’s written  guidance for certain federal nursing home quality standards could be  improved and that revised investigative protocols were helpful.  Specifically, 11 state agency directors reported that CMS guidance on  quality standards related to abuse could be improved. State agency  directors commented that the guidance for certain quality standards was  too long, with the guidance for two standards being over 50 pages long.  One state agency director also noted that overly complex guidance will  lead to an unmanageable survey process. Surveyors’ concerns about the  sufficiency of CMS’s guidance varied for different quality standards (see  table 3). For instance, 21 percent of surveyors nationwide reported that  CMS guidance on pain management was not sufficient to identify  deficiencies, whereas only 5 percent reported that guidance on pressure  ulcers was not sufficient. Our analysis found that fewer surveyors had  concerns with the guidance on quality standards revised through CMS’s  guidance update initiative. For example, the guidance on pressure ulcers  was revised in 2004 and the guidance on accidents was revised in 2007;  these topics ranked last among the areas of concern. Furthermore, state  agency directors from several states commented on the usefulness of  CMS’s revised investigative protocols for federal quality standards.
Another weakness associated with the federal survey process was the  potential for surveys to be predictable based solely on their timing.  Eighteen percent of state agency directors reported that survey  predictability or other advance notice of inspections at least sometimes  contributed to understatement in their states. We analyzed state agencies’  most-recent nursing home surveys and found that 29 percent of these  surveys could be considered predictable due to their timing. We previously  reported that survey predictability could contribute to understatement  because it gives nursing homes the opportunity to conceal deficiencies if  they choose to do so. CMS officials previously stated that reducing  survey predictability could require increased funding because more  surveys would need to be conducted within 9 months of the previous  survey. However, CMS noted that state agencies are not funded to  conduct any surveys within 9 months of the last standard survey.
section_title: New Survey Methodology’s Effect on Understatement Inconclusive
paragraphs: There was no consensus among the eight state agency directors who had  started implementing the QIS as of November 2008 about how the new  survey methodology would affect understatement. Three directors  reported that the QIS was likely to reduce understatement; three directors  reported that it was not likely to reduce understatement; and two directors  were unsure or had no opinion (see fig. 2). However, all eight directors  reported that the new QIS methodology was likely to improve survey  consistency both within and across states. In addition, five of these  directors reported that the new QIS methodology was likely to improve  survey quality. Five of the eight directors also indicated that the QIS  required more time than the traditional survey methodology.
CMS funded an independent evaluation of the QIS, which was completed  by a contractor in December 2007. The evaluation assessed the  effectiveness of the new methodology by studying (1) its effect on  accuracy of surveys, (2) documentation of deficiencies, (3) time required  to complete survey activities, (4) number of deficiencies cited, and   (5) surveyor efficiency. The evaluation did not draw a firm conclusion  about the overall effectiveness of the QIS as measured through these five  areas. For instance, the QIS methodology was associated with an increase  in the total number of deficiencies cited, including an increase in the  number of G-level deficiencies and the number of quality standard areas  cited. However, the evaluation did not find that the QIS methodology  increased survey accuracy, noting that QIS and traditional survey samples  were comparable in overall quality and in the frequency of standards cited  for deficiencies with either a pattern or widespread scope. The results  suggested that more deficiencies with higher scope could have been cited  for both the QIS and traditional surveys. Similarly, there was no evidence  that the QIS resulted in higher-quality documentation or improved  surveyor efficiency. Although five state agency directors reported that the  QIS required more time to complete than the traditional methodology, the  evaluation found some evidence of a learning curve, suggesting that  surveyors were able to complete surveys faster as they became familiar  with the new process. The evaluation generated a number of  recommendations for improving the QIS that are consistent with reducing  understatement, such as improving the specificity and usability of  investigative protocols and evaluating how well the new methodology  accurately identifies the areas in which there are potential quality  problems. Since the evaluation did not find improved accuracy, CMS  concluded that non-QIS factors, including survey guidance clarification  and surveyor training and supervision, would help improve survey  accuracy. Additionally, CMS concluded that future QIS development  efforts should concentrate on improving survey consistency and giving  supervisors more tools to assess the performance of surveyor teams.
Ten state agency directors that had not yet started implementing the QIS  responded to our questionnaire with concerns about the cost associated  with implementing the new methodology, including the resources required  to train staff and obtain new equipment. Of these 10 directors, 3 also  expressed concerns that allotting staff time for QIS implementation would  prevent the agency from completing mandatory survey activities.
section_title: Workforce Shortages and Training Inadequacies May Contribute to Understatement
paragraphs: Workforce shortages and training inadequacies affected states’ ability to  complete thorough surveys, contributing to understatement of nursing  home deficiencies. Responses to our questionnaires indicated that states  experienced workforce shortages or were attempting to accomplish their  workload with a high percentage of inexperienced surveyors. In states  with fewer staff to do the work, time frames were compressed. The  increased workload burden may have had an effect on the thoroughness of  surveys in those states and surveyors’ ability to attend training. The  frequent hiring of new surveyors to address workforce shortages also  burdened states’ surveyor training programs. Surveyors, state agency  directors, and state performance on federal observational surveys  indicated that inadequacies in initial and ongoing training may have  compromised survey accuracy in high-understatement states.
subsections: section_title: Workforce Shortages Sometimes Contributed to Understatement
paragraphs: Although a small percentage of state agency directors reported that  workforce shortages always or frequently contributed to the  understatement of nursing home deficiencies in their states, 36 percent  indicated that workforce shortages sometimes contributed to  understatement (see table 4). In many states, workforce shortages resulted  in a greater reliance on inexperienced surveyors. According to state  agency directors and surveyors, this collateral effect—inexperienced  surveyors—also may have contributed to understatement. States also  expressed concern about completing their workload, which appeared to  be, in part, an outgrowth of workforce shortages and use of inexperienced  surveyors.
Workforce Shortages. Since 2003, we have reported that states have  experienced pervasive workforce shortages, and responses to our  questionnaires indicate that shortages continue to affect states. Seventy- two percent of state agency directors reported that they always or  frequently had a surveyor workforce shortage, and another 16 said it  occurred sometimes. The average vacancy rate for surveyors was   14 percent, and one-fourth of states had a vacancy rate of higher than   19 percent (see table 5). Among the 49 reporting states, the vacancy rate  ranged from a maximum of 72 percent in Alabama to 0 percent in Nevada,  Rhode Island, Vermont, and Utah. The workforce shortages have stemmed  mostly from the preference to employ RNs as surveyors in state survey  agencies, with half of reporting states employing RNs as more than   75 percent of their surveyor workforce. In the past, states have claimed  that they had difficulty matching RN salaries offered by the private sector,  and this hampered the hiring and retention of RNs. The Virginia state  agency director commented during an interview that the nursing home  industry values individuals who have passed CMS’s SMQT and hires its  surveyors after they are trained and certified by CMS. Virginia and others  also identified the stress of the job—regular travel, time pressures to  complete the workload, and the regulatory environment—as a challenge to  retaining staff. Previously, we reported that workforce instability arising  from noncompetitive RN surveyor salaries and hiring freezes affected  states’ abilities to complete their survey workload or resulted in the hiring  of less-qualified staff. Most recently, the poor economy has further  constrained state budgets for surveyors. For example, to address its  budget shortfall in 2009, California will furlough its state employees  including surveyors for 2 days every month from February 2009 through  June 2010. An additional 11 states also reported furloughs for 2009, and  13 are considering furloughs, salary reductions, or layoffs or will employ  such measures in the future.
Inexperienced Surveyors. Many states are attempting to accomplish  their workload with a larger share of inexperienced surveyors, and state  agency directors sometimes linked this reliance on inexperienced staff to  the understatement of nursing home deficiencies. On average, 30 percent  of surveyors had less than 2 years’ experience (see table 5); however the  percentage of inexperienced surveyors ranged from 10 to 82 percent  across states who reported this information. Among state agency  directors, 16 percent indicated that inexperienced surveyors always or  frequently contributed to understatement, while another 48 percent  indicated that surveyor inexperience sometimes contributed to  understatement in their states. In response to our questionnaires,   26 percent of surveyors indicated that survey teams always or frequently  had too many inexperienced surveyors and another 33 percent indicated  that sometimes survey teams had too many inexperienced surveyors (see  table 6). Half or more of all surveyors in six states—Alabama, Alaska,  Arizona, Idaho, New Mexico, and Utah—reported that there were always  or frequently too many new surveyors who were not yet comfortable with  their job responsibilities. For example, 79 percent of surveyors in Arizona  reported that too many new surveyors were not comfortable with their job  responsibilities, and the state agency director was among the 34 percent  who reported that survey teams sometimes had an insufficient number of  experienced surveyors. Overall, 26 percent of state agency directors  indicated that the skill level of surveyors has decreased in the last 5 years.
In interviews, six state agency directors commented that inexperienced  surveyors possessed different skills or needed more time than experienced  surveyors to complete surveys and that workforce shortages resulted in  constant recruiting, over-burdened experienced surveyors, or the need for  additional supervision and training resources. Four states—Kentucky,  Nevada, New Mexico, and Virginia—reported not having enough dedicated  training staff to handle the initial training for new surveyors.
Workload. State inability to complete workload was, in part, an  outgrowth of the workforce shortages and reliance on inexperienced  surveyors. More than two-thirds of state agency directors reported on our  questionnaire that staffing posed a problem for completing complaint  surveys, and more than half reported that staffing posed a problem for  completing standard or revisit surveys. In addition, 46 percent of state  agency directors reported that time pressures always, frequently, or  sometimes contributed to understatement in their states. In response to  our questionnaire, 16 percent of surveyors nationwide reported that  workload burden influenced the citation of deficiencies—including 14  states with 20 percent or more surveyors reporting the same. More than 50  percent of surveyors identified insufficient team size or time pressures as  having an effect on the thoroughness of surveys. Surveyors’ comments  reiterated these concerns—over 15 percent of surveyors who wrote  comments complained about the amount of time allotted to complete  surveys or survey paperwork, and 11 percent indicated that staffing was  insufficient to complete surveys. One state agency director suggested to  us that CMS establish a national team of surveyors to augment states’  when they fell behind on their workload or had staffing shortages. He  thought the availability of national surveyors could assist states  experiencing workforce shortages and help ensure state workloads were  completed. This state had experience with a similar arrangement when it  hired a national contractor to complete its surveys of Intermediate Care  Facilities for the Mentally Retarded.
section_title: Training Inadequacies May Compromise Survey Accuracy
paragraphs: Surveyors, state agency directors, and state performance on federal  observational surveys indicated that inadequacies in initial or ongoing  training may compromise the accuracy of nursing home surveys and lead  to the understatement of deficiencies. In addition, workload affected  surveyors’ ability to attend training.
Initial Surveyor Training. As noted earlier, even though CMS has  established specific training requirements, including coursework and the  SMQT certification test, states are responsible for preparing their new  surveyors for the SMQT. According to CMS, 94 percent of new surveyors  nationally passed the SMQT test in 2008 and, on average, surveyors  answered about 77 percent of the questions correctly. These results seem  to support the state agency directors’ assertions that initial training was  insufficient and suggest that the bar for passing the test may be set too  low. Even though we cannot be certain whether the inadequacies are with  the federal or state components of the training, reported differences  among states in satisfaction with the initial surveyor training also could  reflect gaps in state training programs. About 29 percent of surveyors in  high-understatement states reported that initial training was not sufficient  to cite appropriate scope and severity levels, compared with 16 percent of  surveyors in low-understatement states (see table 7). Similarly, 28 percent  of surveyors in high-understatement states, compared with 20 percent of  those in low-understatement states, indicated that initial training was not  sufficient to identify deficiencies for nursing homes. Further, 18 percent of  state agency directors linked the occurrence of understatement always,  frequently, or sometimes with insufficient initial training. From 16 to   20 percent of state agency directors indicated that initial training was  insufficient to (1) enable surveyors to identify deficiencies and (2) assign  the appropriate level of scope and severity.
Ongoing Training. Ongoing training programs are the purview of state  agencies; therefore, differences between states about the sufficiency of  this training also may point to gaps in the state training programs. On our  questionnaire, about 34 percent of surveyors in high-understatement states  indicated a need for additional training on (1) identifying appropriate  scope and severity levels and (2) documenting deficiencies. This was  significantly more than those from low-understatement states, which  indicated less of a need for additional training in these areas—16 and   27 percent, respectively. Among state agency directors, 10 percent  attributed understatement always or frequently to insufficient ongoing  training, while 14 percent indicated that insufficient ongoing training  sometimes gave rise to understatement. Although 74 percent of state  agency directors indicated that the state had ongoing annual training  requirements, the required number of hours and the type of training varied  widely by state in 2007. Among the 33 states that provided the required  amount of annual state training, these hours ranged from 0 to 120 hours  per year. Meanwhile, 37 states reported one or more type of required  training: 32 states required surveyors to attend periodic training, 22  required on-the-job training, 10 required online computerized training, and  13 states required some other type of training.
State agency directors indicated that they relied on CMS materials for  ongoing training of experienced surveyors, yet many reported additional  training needs and suggested that use of electronic media could make  continuing education and new guidance more accessible. While 98 percent  of states indicated that the CMS written guidance materials and resources  were useful, over 50 percent of all state agency directors identified  additional training needs in documenting deficiencies, citing deficiencies  at the appropriate scope and severity level, and applying CMS guidance.  On federal observational surveys, an average of 17 to 12 percent of survey  teams in high-understatement states received below-satisfactory ratings  for Deficiency Determination and General Investigation, respectively—two  skills critical for preventing understatement. In contrast, an average of   4 percent of survey teams in low-understatement states received the same  below-satisfactory scores for both deficiency determination and  investigative skills. Furthermore, of the 476 surveyors who commented  about training needs, one-quarter indicated a need for training support  from either CMS or state agencies; and between 12 to 7 percent of those  who commented on training needs identified topics such as: documenting  deficiencies, identifying scope and severity, CMS guidance, and medical  knowledge.
Inability to Attend Training. States’ workload requirements and  workforce shortages affected the surveyors’ ability to attend initial and  ongoing training. Seven of the eight state agency directors we interviewed  linked workforce shortages and resource constraints to their state’s ability  to complete the survey workload or allow staff to participate in training  courses. One director stated that workload demands compromised  comprehensive training for new staff, and another reported difficulty  placing new staff in CMS’s initial training programs. Due to workload  demands, a third state agency director stated that she could not allow  experienced staff time away from surveying to attend training courses  even when staff paid their own way. Five of the seven state agency  directors suggested that it would be more efficient for training activities to  be conducted more locally such as in their states or to be available  through online, video, or other electronic media, and several emphasized  the need to reduce or eliminate travel for training. Although four states  also expressed a preference for interactive training opportunities, one  state believed that technological solutions could allow for more accessible  training that was also interactive.
section_title: State Supervisory Reviews Often Are Not Designed to Identify Understatement
paragraphs: State supervisory reviews, which generally occurred more frequently on  higher-level deficiencies, often are not designed to identify understated  deficiencies. State agencies generally conducted more supervisory reviews  on surveys with higher-level deficiencies, compared to surveys with  deficiencies at the potential for more than minimal harm level (D through  F)—the deficiencies most likely to be understated. While focus on higher- level deficiencies enables states to be certain that such deficiencies are  well documented, not reviewing surveys with deficiencies at lower levels  represents a missed opportunity to ensure that all serious deficiencies are  cited. State surveyors who reported having frequent changes made to their  survey reports during supervisory reviews also more often reported they  were burdened by other factors contributing to understatement, such as  workforce shortages and survey methodology weaknesses.
subsections: section_title: Supervisory Reviews Often Focused on Higher-Level Deficiencies
paragraphs: According to state agency directors’ responses to our questionnaire, states  generally focused supervisory review on surveys with higher-level  deficiencies, rather than on the surveys with deficiencies at the potential  for more than minimal-harm level (D through F)—the deficiencies most  likely to be understated. During supervisory reviews, either direct-line  supervisors or central state agency staff may review draft survey records.  On average, surveys at the D through F level underwent about two steps of  review, while surveys with deficiencies at the immediate jeopardy level   (J through L) went through three steps. For example, Washington  reviews its surveys using either a two-step review that includes survey  team and field manager reviews or a three-step process that includes both  these reviews and an additional review by central state agency staff for  serious deficiencies. As a result, central state agency staff in Washington  do not review deficiencies below the level of actual harm. In addition we  found that five states—Alaska, Hawaii, Illinois, Nebraska, and Nevada— did not review all surveys with deficiencies at the D through F levels. In  fact, Hawaii did not report supervisory review of deficiencies at any level  (see fig. 3). It is difficult to know if additional supervisory reviews—the  second, third, or fourth review—help make survey records more accurate  and less likely to be understated, or if these reviews result in more  frequent changes to deficiency citations. However, if deficiency citations  with the potential for more than minimal-harm level (D through F) are not  reviewed, states miss the opportunity to assess whether these deficiencies  warrant a higher-level citation, for example, the level of actual harm or  immediate jeopardy.
Because a majority of states are organized into geographically-based  district or regional offices, review by central state agency staff,  particularly quality assurance staff, is critical to help ensure consistency  and detect understatement. However, 26 states reported that no central  state agency staff reviews were conducted for surveys with deficiencies at  the potential for more than minimal harm (D through F). These results are  consistent with a finding from our 2003 report—that half of the 16 states  we contacted for that report did not have a quality assurance process to  help ensure that the scope and severity of less serious deficiencies were  not understated.
According to most of the eight state officials we interviewed, supervisory  reviews commonly focused on documentation principles or evidentiary  support, not on reducing understatement. For example, all eight states  used supervisory reviews to assess the accuracy and strength of the  evidence surveyors used to support deficiency citations, and three of these  states reported that they emphasized reviewing survey records for  documentation principles. Furthermore, seven out of eight states indicated  that surveys with serious deficiencies—those that may be subject to  enforcement proceedings—went through additional steps of review  compared with surveys citing deficiencies with the potential for more than  minimal harm (D through F).
section_title: Reports of Changes to Deficiencies during Supervisory Reviews May Be Related to Other Factors That Contribute to Understatement
paragraphs: Surveyor reports of changes to deficiency citations during supervisory  reviews may be related to other factors the state is experiencing that also  contribute to understatement, such as workforce shortages and survey  methodology weaknesses.
Changes to Deficiencies. Fifty-four percent of surveyors nationwide  reported on our questionnaire that supervisors at least sometimes  removed the deficiency that was cited, and 53 percent of surveyors noted  that supervisors at least sometimes changed the scope and severity level  of cited deficiencies. Of the surveyors, who reported that supervisors  sometimes removed deficiencies, 13 percent reported that supervisors  always or frequently removed deficiencies—including 12 states with 20  percent or more of their surveyors reporting that deficiencies were  removed.
Surveyor reports of changes in deficiency citations alone make it difficult  to know whether the original deficiency citation or the supervisor’s  revised citation was a more accurate reflection of a nursing home’s quality  of care. Additionally, there are many reasons that survey records might be  changed during supervisory review. When a surveyor fails to provide  sufficient evidence for deficient practices, it may be difficult to tell  whether the deficiency was not appropriately cited or if the surveyor did  not collect all the available evidence. Kentucky’s state agency director  offered one possible explanation—that changes to surveys often reflected  a need for more support for the deficiencies cited, such as additional  evidence from observations. Nevada’s state agency director stated that  changes to survey records occurred when it was often too late to gather  more evidence in support of deficiencies.
Surveyors who reported that supervisors frequently changed deficiencies  also more often reported experiencing other factors that contribute to  understatement. We found associations between surveyor reports of  changes to deficiencies and workforce shortages and survey methodology  weaknesses.
Workforce shortages. Surveyors reporting workforce shortages,  including survey teams with too many new surveyors and survey teams  that were either too small or given insufficient time to conduct thorough  surveys, more often also reported that supervisors frequently removed  deficiencies or changed the scope and severity of deficiency citations  during supervisory reviews.
Survey methodology weaknesses. Surveyors reporting weaknesses in  the current survey methodology more often also reported that supervisors  frequently removed deficiencies or changed the scope and severity of  deficiency citations during supervisory reviews.
Supervisory Reviews and Understatement. In certain cases, survey  agency directors and state performance on federal comparative surveys  linked supervisory reviews to understatement. Twenty-two percent of  state agency directors attributed inadequate supervisory review processes  to understatement in their states at least sometimes. In addition,  significant differences existed between zero-understatement states and all  other states, including high-understatement states, in the percentage of  surveyors reporting frequent changes to citations during supervisory  reviews. Only about 4 percent of surveyors in zero-understatement states  reported that citations were always or frequently removed or changed and  that the scope and severity cited were changed, while about 12 percent of  surveyors in all other states indicated the same (see table 8). To address  concerns with supervisory reviews, Nevada recently reduced its process  from two steps to a single step review by survey team supervisors to  address surveyor complaints about changes made during supervisory  reviews.
In addition, we observed a relationship between state practices to notify  surveyors of changes made during supervisory reviews and surveyor  reports of deficiency removal and explanation of changes. Specifically,  compared to surveyors in states that require supervisors to notify  surveyors of changes made during supervisory review, surveyors from  states where no notification is required reported more often that  supervisors removed deficiencies and less often that explanations for  these changes, when given, were reasonable.
Similarly, we found an association between the frequency of explained and  reasonable changes and zero-understatement states, possibly  demonstrating the positive effect of practices to notify surveyors of  changes made during supervisory reviews. Nursing home surveyors from  zero-understatement states more often reported that supervisors  explained changes and that their explanations seemed reasonable  compared to surveyors in all other states. State agency directors in  Massachusetts and New Mexico stated that explanations of changes to the  survey record provided opportunities for one-on-one feedback to  surveyors and discussions about deficiencies being removed.
section_title: State Agency Practices and External Pressure May Compromise Survey Accuracy and Lead to Understatement in a Few States
paragraphs: Nursing home surveyors and state agency directors in a minority of states  told us that in isolated cases issues such as a state agency practice of  noncitation, external pressure from the nursing home industry, and an  unbalanced IDR process may have led to the understatement of  deficiencies. In a few states, surveyors more often identified problems  with noncitation practices and IDR processes compared to state agency  directors. Yet, a few state agency directors acknowledged either  noncitation practices, external pressure, or an IDR process that favored  nursing home operators over resident welfare. Although not all the issues  raised by surveyors were corroborated by the state agency directors in  their states, surveyor reports clustered in a few states gives credence to  the notion that such conditions may lead to understatement.
subsections: section_title: Surveyors Reported Noncitation Practices in a Small Number of States
paragraphs: Approximately 20 percent of surveyors nationwide and over 40 percent of  surveyors in five states reported that their state agency had at least one of  the following noncitation practices: (1) not citing certain deficiencies,   (2) not citing deficiencies above a certain scope and severity level, and   (3) allowing nursing homes to correct deficiencies without receiving a  citation (see fig. 4). Only four state agency directors acknowledged the  existence of such practices in their states on our questionnaire and only  one of these directors was from the five states most often identified by  surveyors. One of these directors commented on our questionnaire that  one of these practices occurs only in “rare individual cases.” Another  director commented that a particular federal quality standard is not related  to patient outcome and therefore should not be cited above a Level F.  According to CMS protocols, when noncompliance with a federal  requirement has been identified, the state agency should cite all   deficiencies associated with the noncompliance. CMS regional officials we  interviewed were not aware of any current statewide noncitation  practices.
Not citing certain deficiencies. Nationally, 9 percent of surveyors  reported a state agency practice that surveyors not cite certain  deficiencies. However, in four states over 30 percent of surveyors reported  their state agency had this noncitation practice, including over 60 percent  of New Mexico surveyors. In some cases, surveyors reported receiving  direct instructions from supervisors not to cite certain deficiencies. In  other cases, surveyors’ reports of noncitation practices may have been  based on their interpretation of certain management practices. For  instance, surveyors commented that some state agency practices—such as  providing inadequate time to observe and document deficiencies or  frequently deleting deficiency citations during supervisory review— seemed like implicit or indirect leadership from the agency to avoid citing  deficiencies. One state agency director we interviewed agreed that  surveyors may report the existence of noncitation practices when their  citations are changed during supervisory review. This official told us that  when surveyors’ deficiencies are deleted or downgraded, the surveyors  may choose not to cite similar deficiencies in the future because they  perceive being overruled as an implicit state directive not to cite those  deficiencies.
Not citing deficiencies above a certain scope and severity level.  Although nationwide less than 8 percent of surveyors reported a state  agency practice that surveyors not cite deficiencies above a certain scope  and severity level, in two states over 25 percent of surveyors reported that  their state agency used this type of noncitation practice. One reason state  agencies might use this noncitation practice could be to help manage the  agency’s workload. In particular, citing deficiencies at a lower scope and  severity might help the agency avoid additional work associated with  citing higher-level deficiencies, such as survey revisits or IDR. In one of  the two states mentioned above, 54 percent of surveyors indicated that the  workload burden influenced their citations. Additionally, as we described  earlier, 16 percent of surveyors nationwide indicated that workload  burden influenced the citation of deficiencies and more than half of state  agency directors (including those from the two states mentioned above)  responded that staffing was not sufficient to complete revisit surveys.  While our questionnaire focused on not citing deficiencies above a certain  scope and severity level, a few surveyors commented on being  discouraged from citing lower-level deficiencies due to time pressures to  complete surveys. Agency officials in two states told us that surveyors may  miss some deficiencies due to limited survey time and resources.
Allowing nursing homes to correct deficiencies without citing them  on the survey record. Nationwide, approximately 12 percent of  surveyors reported this type of noncitation practice. However, in five  states, at least 30 percent of surveyors reported their state agency allowed  nursing homes to correct deficiencies without citing those deficiencies on  the official survey record. Comments from surveyors suggest that state  agencies may use this type of practice to avoid actions that nursing homes  or the industry would dispute or interpret as excessive. Similarly, several  surveyors commented that they were instructed to cite only one deficiency  for a single type of negative outcome, even when more than one problem  existed. However, CMS guidance requires state agencies to cite all  problems that lead to a negative outcome. The decrease in G-level  citations that occurred after CMS implemented the double G immediate  sanctions policy in January 2000 also suggests that some states may have  avoided citing deficiencies that would result in enforcement actions for  the nursing home. The total number of G-level deficiency citations  nationwide dropped from approximately 10,000 in 1999 to 7,700 in 2000.
section_title: State Agency Directors in a Few States Reported That External Pressure Contributed to Understatement
paragraphs: State agency directors from 12 states reported experiencing external  pressure from at least one of the following stakeholder groups: (1) the  nursing home surveyed, (2) the nursing home industry, or (3) state or  federal legislators. Examples of such external pressure include pressure to  reduce federal or state nursing home regulation or to delete specific  deficiencies cited by the state agency. Of the 12 state agency directors,   7 reported that external pressure at least sometimes contributed to the  understatement of deficiencies in their states, while the other 5 indicated  that it infrequently or never contributed to understatement.
Adversarial attitude toward nursing home surveys. Two states we  interviewed—State A and State B—commented on the adversarial attitude  that industry and legislative representatives had toward nursing home  surveys at times. For instance, state agency officials from State A told us  that the state nursing home association organized several forums to garner  public and legislative support for curtailing state regulation of facilities.  According to officials in this state, the influential industry groups  threatened to request legislation to move the state agency to a different  department and to deny the confirmation of the director’s gubernatorial  appointment if the citations of G level or higher deficiencies increased.  CMS regional office officials responsible for State A told us that the state  may be experiencing more intense external pressure this year given the  current economy, because providers have greater concerns about the  possible financial implications of deficiency citations—fines or increased  insurance rates.
Similarly, officials from State B told us that when facilities are close to  termination, the state agency receives phone calls from state delegates  questioning the agency’s survey results. Officials from State B also told us  that the Governor’s office instructed the state agency not to recommend  facilities for enforcement actions. Officials from the CMS regional office  responsible for State B told us that this situation was not problematic  because CMS was ultimately responsible for determining enforcement  actions based on deficiency citations. However, this regional office’s  statement is inconsistent with (1) language in the SOM that calls for   states to recommend enforcement actions to the regional office, and   (2) assertions from the regional office responsible for State A that it  infrequently disagrees with state recommendations for sanctions. A third  state agency director commented that the agency had been called before  state legislative committees in 2007-2008 to defend deficiency citations  that led to the termination of facilities. A fourth state agency director also  commented on our questionnaire that legislators had pressured the state  agency on behalf of nursing homes to get citations reduced or eliminated  and prevent enforcement actions for the facilities. In addition, a few  surveyors commented that at times when nursing homes were unhappy  with their survey results the homes or their state legislators would ask  state agency management to remove the citations from the survey record,  resulting in the deletion or downgrading of deficiencies. Further,  comments from a few surveyors indicated that they may steer clear of  citing deficiencies when they perceive the citation might cause a home to  complain or exert pressure for changes in the survey record.
Interference in the survey process. In a few cases, external pressure  appeared to directly interfere with the nursing home survey process. State  agency officials from two states—State A and an additional fifth state— reported that state legislators or industry representatives had appeared on- site during nursing home surveys. Although in some cases the legislators  just observed the survey process, officials from these two states explained  that third parties also have interfered with the process by questioning or  intimidating surveyors. The state agency director from the fifth state  commented on our questionnaire that the nursing home industry sent legal  staff on-site during surveys to interfere with the survey process. Similarly,  officials from State A told us that during one survey, a home’s lawyer was  on-site reviewing nursing home documentation before surveyors were  given access to these documents. Officials from State A also told us that  state legislators have attended surveys to question surveyors about their  work and whether state agency executives were coercing them to find  deficiencies. We discussed this issue with the CMS regional officials  responsible for State A, who acknowledged that this type of interference  had occurred.
States’ need for support from CMS. In the face of significant external  pressure, officials from States A and B suggested that they need support  from CMS; however, CMS regional office officials did not always  acknowledge external pressure reported by the states. This year, State A  terminated a survey due to significant external pressure from a nursing  home and requested that the CMS regional office complete the revisit  survey for them. Six weeks later, the federal team completed the survey  and found many of the same problems that this state team had previously  identified before it stopped the survey. Officials from State A suggested  the need for other support as well, such as creating a federal law that  would require state agencies to report external pressure and ensure  whistleblower protections for state officials who report pressure and  allowing sanctions for inappropriate conduct. CMS officials from the  regional office responsible for State A stated that external pressure might  indirectly contribute to understatement by increasing surveyor mistakes  from the additional stress, workload, focus on documentation, and  supervisory reviews. Conversely, CMS regional officials did not  acknowledge that State B experienced external pressure and officials from  State B thought that CMS should be more consistent in its requirements  and enforcement actions.
section_title: Unbalanced IDR Processes Might Have Contributed to Understatement in a Few States
paragraphs: States with unbalanced IDR processes may experience more  understatement. IDR processes vary across states in structure, volume of  proceedings, and resulting changes. According to state agency directors’  responses to our questionnaire, 16 IDRs were requested per 100 homes in  fiscal year 2007, with this number ranging among states from 0 to 57 per  100 homes. For IDRs occurring in fiscal year 2007, 20 percent of disputed  deficiencies were deleted and 7 percent were downgraded in scope or  severity, but in four states, at least 40 percent of disputed deficiencies  were deleted through this process. CMS does not provide protocols on  how states should operate their IDR processes, leaving IDR operations to  state survey agencies’ discretion. For example, states may choose to  conduct IDR meetings in writing, by telephone, or through face-to-face  conferences. State agencies also have the option to involve outside  entities, including legal representation, in their IDR operations.
On the basis of responses from surveyors and state agency directors  clustered in a few states, problems with the IDR processes—such as  frequent hearings, deficiencies that are frequently deleted or downgraded  through the IDR process, or outcomes that favor nursing home operators  over resident welfare—may have contributed to the understatement of  deficiencies in those states. Although reports of such problems were not  common—only 16 percent of surveyors nationwide reported on our  questionnaire that their state’s IDR process favored nursing home  operators—in four states over 40 percent of surveyors reported that their  IDR process favored nursing home operators (see fig. 5), including one  state where a substantial percentage of surveyors identified at least one  noncitation practice. While only one state agency director reported that  the IDR process favored nursing home operators, three other directors  acknowledged that frequent IDR hearings at least sometimes contributed  to the understatement of deficiencies. For example, in some states  surveyors may hesitate to cite deficiencies that they believe will be  disputed by the nursing home.
In isolated cases, a lack of balance with the IDR process appeared to be a  result of external pressure. In one state, the state agency director reported  that the nursing home industry sent association representatives to the IDR,  which increased the contentiousness of the process. In another state,  officials told us that a large nursing home chain worked with the state  legislature to set up an alternative to the state IDR process, which has  been used only by facilities in this chain. Through this alternative appeals  process, both the state agency and the nursing home have legal  representation, and compliance decisions are made by an adjudicator.  According to agency officials in this state, the adjudicators for this  alternative appeals process do not always have health care backgrounds.  While CMS gives states the option to allow outside entities to conduct the  IDR, the states should maintain ultimate responsibility for IDR decisions.  CMS regional officials stated it would not consider the outcome of this  alternative appeals process when assessing deficiencies or determining  enforcement actions. Regardless, these actions may have affected  surveyors’ perceptions of the balance of the states’ IDRs, because over  twice the national average of surveyors in this state reported that their IDR  process favored nursing home operators.
section_title: Conclusions
paragraphs: Reducing understatement is critical to protecting the health and safety of  vulnerable nursing home residents and ensuring the credibility of the  survey process. Federal and state efforts will require a sustained, long- term commitment because understatement arises from weaknesses in  several interrelated areas—including CMS’s survey process, surveyor  workforce and training, supervisory review processes, and state agency  practices and external pressure.
Concerns about CMS’s Survey Process. Survey methodology and  guidance are integral to reliable and consistent state nursing home  surveys, and we found that weaknesses in these areas were linked to  understatement by both surveyors and state agency directors. Both groups  reported struggling to interpret existing guidance, and differences in  interpretation were linked to understatement, especially in determining  what constitutes actual harm. Surveyors noted that the current survey  guidance was too lengthy, complex, and subjective. Additionally, they had  fewer concerns about care areas for which CMS has issued revised  interpretive protocols. In its development of the QIS, CMS has taken steps  to revise the nursing home survey methodology. However, development  and implementation of the QIS in a small group of states has taken  approximately 10 years, and full implementation of the new methodology  is not expected to be completed until 2014. The experience of the QIS was  mixed regarding improvement in the quality of surveys, and the  independent evaluation generated a number of recommendations for  improving the QIS. CMS concluded that it needed to focus future QIS  development efforts on improving survey consistency and giving  supervisors more tools to assess performance of surveyor teams.
Ongoing Workforce and Surveyor Training Challenges. Workforce  shortages in state survey agencies increase the need for high-quality initial  and ongoing training for surveyors. Currently, high vacancy rates can  place pressure on state surveyors to complete surveys under difficult  circumstances, including compressed time frames, inadequately staffed  survey teams, and too many inexperienced surveyors. States are  responsible for hiring and retaining surveyors and have grappled with  pervasive and intractable workforce shortages. State agency directors  struggling with these workforce issues reported the need for more readily  accessible training for both their new and experienced surveyors that did  not involve travel to a central location. Nearly 30 percent of surveyors in  high-understatement states stated that initial surveyor training, which is  primarily a state activity that incorporates two CMS on-line computer  courses and a 1-week federal basic training course culminating in the  SMQT, was not adequate to identify deficiencies and cite them at the  appropriate scope and severity level. State agency directors reported that  workforce shortages also impede states’ ability to provide ongoing training  opportunities for experienced staff and that additional CMS online training  and electronic training media would help states maintain an experienced,  well-informed workforce. They noted that any such support should be  cognizant of states’ current resource constraints, including limited funding  of travel for training.
Supervisory Review Limitations. Currently, CMS provides little  guidance on how states should structure supervisory review processes,  leaving the scope of this important quality-assurance tool exclusively to  the states and resulting in considerable variation throughout the nation in  how these processes are structured. We believe that state quality  assurance processes are a more effective preventive measure against  understatement because they have the potential to be more immediate and  cover more surveys than the limited number of federal comparative  surveys conducted in each state. However, compared to reviews of serious  deficiencies, states conducted relatively fewer reviews of deficiencies at  the D through F level, those that were most frequently understated  throughout the nation, to assess whether or not such deficiencies were  cited at too low a scope and severity level. In addition, we found that  frequent changes to survey results made during supervisory review were  symptomatic of workforce shortages and survey methodology  weaknesses. For example, surveyors who reported that survey teams had  too many new surveyors, more often also reported either frequent changes  to or removals of deficiencies during supervisory reviews—indicating that  states with inexperienced workforces may rely more heavily on  supervisory reviews. In addition, variation existed in the type of feedback  surveyors receive when deficiencies are changed or removed during  supervisory reviews, providing surveyors with inconsistent access to  valuable feedback and training. CMS did not implement our previous  recommendation to require states to have a quality assurance process that  includes, at a minimum, a review of a sample of survey reports below the  actual harm level to assess the appropriateness of the scope and severity  cited and help reduce understatement.
State Agency Practices and External Pressure. In a few states,  noncitation practices, challenging relationships with the industry or  legislators, or unbalanced IDR processes—those that surveyors regard as  favoring nursing home operators over resident welfare—may have had a  negative effect on survey quality and resulted in the citation of fewer  nursing home deficiencies than was warranted. In one state, both the state  agency director and over 40 percent of surveyors acknowledged the  existence of a noncitation practice such as allowing a home to correct a  deficiency without receiving a citation. Forty percent of surveyors in four  other states also responded on our questionnaire that noncitation  practices existed. Currently, CMS does not explicitly address such  practices in its guidance to states, and its oversight is limited to reviews of  citation patterns, feedback from state surveyors, state performance  reviews, and federal monitoring surveys to determine if such practices  exist. Twelve state agency directors reported on our questionnaire  experiencing some kind of external pressure. For example, in one state a  legislator attended a survey and questioned surveyors as to whether state  agency executives were coercing them to find deficiencies. Under such  circumstances, it is difficult to know if the affected surveyors are  consistently enforcing federal standards and reporting all deficiencies at  the appropriate scope and severity levels. States’ differing experiences  regarding the enforcement of federal standards and collaboration with  their CMS regional offices in the face of significant external pressure also  may confuse or undermine a thorough and independent survey process. If  surveyors believe that CMS does not fully or consistently support the  enforcement of federal standards, these surveyors may choose to avoid  citing deficiencies that they perceive may trigger a reaction from external  stakeholders. In addition, deficiency determinations may be influenced  when IDR processes are perceived to favor nursing home operators over  resident welfare.
Because many aspects of federal and state operations contribute to the  understatement of deficiencies on nursing home surveys, mitigating this  problem will require the concerted effort of both entities. The interrelated  nature of these challenges suggests a need for increased CMS attention on  the areas noted above and additional federal support for states’ efforts to  enforce federal nursing home quality standards.
section_title: Recommendations for Executive Action
paragraphs: To address concerns about weaknesses in CMS survey methodology and  guidance, we recommend that the Administrator of CMS take the  following two actions:    make sure that action is taken to address concerns identified with the new  QIS methodology, such as ensuring that it accurately identifies potential  quality problems; and   clarify and revise existing CMS written guidance to make it more concise,  simplify its application in the field, and reduce confusion, particularly on  the definition of actual harm.
To address surveyor workforce shortages and insufficient training, we  recommend that the Administrator of CMS take the following two actions:    consider establishing a pool of additional national surveyors that could  augment state survey teams or identify other approaches to help states  experiencing workforce shortages;   evaluate the current training programs and division of responsibility  between federal and state components to determine the most cost- effective approach to: (1) providing initial surveyor training to new  surveyors, and (2) supporting the continuing education of experienced  surveyors.
To address inconsistencies in state supervisory reviews, we recommend  that the Administrator of CMS take the following action:    set an expectation through guidance that states have a supervisory review  program as a part of their quality-assurance processes that includes  routine reviews of deficiencies at the level of potential for more than  minimal harm (D-F) and that provides feedback to surveyors regarding  changes made to citations.
To address state agency practices and external pressure that may  compromise survey accuracy, we recommend that the Administrator of  CMS take the following two actions:  reestablish expectations through guidance to state survey agencies that  noncitation practices—official or unofficial—are inappropriate, and  systematically monitor trends in states’ citations; and   establish expectations through guidance to state survey agencies to  communicate and collaborate with their CMS regional offices when they  experience significant pressure from legislators or the nursing home  industry that may affect the survey process or surveyors’ perceptions.
section_title: Agency and AHFSA Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS and AHFSA for comment. In  response, the Acting Administrator of CMS provided written comments.  CMS noted that the report adds value to important public policy  discussions regarding the survey process and contributes ideas for  solutions on the underlying potential causes of understatement. CMS fully  endorsed five of our seven recommendations and indicated it would  explore alternate solutions to our remaining two recommendations, one of  which the agency did not plan to implement on a national scale. (CMS’s  comments are reprinted in appendix II.) AHFSA’s comments noted that  several states agreed with one of our recommendations, but did not  directly express agreement or disagreement with the other  recommendations. AHFSA made several other comments on our findings  and recommendations as summarized below.
CMS agreed with five of our recommendations that called for:   (1) addressing issues identified with the new QIS methodology,   (2) evaluating current training programs, (3) setting expectations that  states have a supervisory review program, (4) reestablishing expectations  that noncitation practices are inappropriate, and (5) establishing  expectations that states communicate with their CMS regional office when  they experience significant pressure from legislators or the nursing home  industry. In its comments, the agency cited several ongoing efforts as  mechanisms for addressing some of our recommendations. While we  acknowledge the importance of these ongoing efforts, in some areas we  believe more progress and investigation are likely needed to fully address  our findings and recommendations. For example, we recommended that  CMS ensure that measures are taken to address issues identified with the  new QIS methodology, such as ensuring that it accurately identifies  potential quality problems; CMS’s response cited Desk Audit Reports that  enable supervisors to provide improved feedback to surveyors and  quarterly meetings of a user group as evidence of efforts under way to  continuously improve the QIS and to increase survey consistency.  However, we noted that a 2007 evaluation of the QIS did not find improved  survey accuracy compared to the traditional survey process and  recommended that CMS evaluate how well the QIS accurately identifies  areas in which there were potential quality problems. While improving the  consistency of the survey process is important, CMS must also focus on  addressing the accuracy of QIS surveys.
For the remaining two recommendations, CMS described alternative  solutions that it indicated the agency would explore:    Guidance. The agency agreed in principle with our recommendation to  clarify and revise existing written guidance to make it more concise,  simplify its application in the field, and reduce confusion. However, CMS  disagreed with shortening the guidance as the preferred method for  achieving such clarification. Instead, the agency suggested an alternative— the creation of some short reference documents for use in the field that  contain cross-links back to the full guidance—that we believe would fulfill  the intent of our recommendation.
National surveyor pool. CMS indicated it did not plan to implement our  recommendation to consider establishing a pool of additional national  surveyors that could augment state survey teams experiencing workforce  shortages, at least not on a national scale. The agency stated that the  establishment of national survey teams was problematic for several  reasons, including that it (1) began to blur the line between state  accountability for meeting performance expectations and compensating  states for problematic performance due to state management decisions,  and (2) was improper for CMS to tell states how to make personnel  decisions While the agency noted that it used national contractors to  perform surveys for other types of facilities such as organ transplant  centers, it expressed concern about their use to compensate for state  performance issues because of the more frequent nursing home surveys.
We believe that state workforce shortages are a separate issue from state  performance on surveys. Since 2003, we have reported pervasive state  workforce shortages and this report confirms that such shortages  continue. For example, we reported that one-fourth of states had vacancy  rates higher than 19 percent and that one state reported a 72 percent  vacancy rate. We also believe that addressing workforce shortages is  critical to creating an effective system of oversight for nursing homes and  reducing understatement throughout the nation.
However, CMS noted that it would explore this issue with a state-federal  work group in order to identify any circumstances in which a national pool  may be advisable and to identify any additional solutions. Reflecting this  comment from CMS, we have revised our original recommendation to  include other potential solutions as well as a national pool of surveyors.  One suggestion in AHFSA comments may be worth exploring in this  regard—providing funds to state survey agencies for recruitment and  retention activities.
AHFSA commented that vigorous oversight and enforcement are essential  to improving the quality of life and quality of care for health care  consumers and are critical if improvements already achieved are to be  maintained. The association noted that several states agreed with our  recommendation on the need for CMS to revise existing written guidance  to make it more concise. While the association did not directly express  agreement or disagreement with our other recommendations, it did note  that most states would need additional funding to meet any new staffing  requirements associated with our recommendation that CMS set an  expectation for states to have a supervisory review program.
However, AHFSA noted what it considered to be conflicting assertions  within the report. For example, it noted that we cited inexperienced staff  as a factor that contributes to understatement but also appeared to take  issue with the practice of supervisors changing reports prepared by  inexperienced staff. While our report identifies a wide variety of factors  that may contribute to understatement, we did not and could not  meaningfully prioritize among these factors based on the responses of  nursing home surveyors and state agency directors. We did find that many  states were attempting to accomplish their survey workload with a large  share of inexperienced surveyors and that state agency directors  sometimes linked this reliance on inexperienced staff to the  understatement of nursing home deficiencies. In addition, we found that  frequent changes made during supervisory review were symptomatic of  workforce shortages and survey methodology weaknesses. For example,  surveyors who reported that survey teams had too many new surveyors,  more often also reported either frequent changes to or removals of  deficiencies during supervisory reviews. We believe that state quality  assurance processes have the potential to play an important role in  preventing understatement, which may result in states with inexperienced  workforces relying more heavily on supervisory reviews.
AHFSA also stated that our report did not address limitations of federal  monitoring surveys, specifically the potential inconsistency among CMS  regional offices in how these surveys are conducted. Assessing CMS’s  performance on federal monitoring surveys was beyond the scope of this  report. However, our May 2008 report noted several improvements CMS  had made since fiscal years 2002 and 2003 in federal comparative surveys  intended to make them more comparable to the state surveys they are  assessing; these improvement include; (1) reducing the time between the  state and federal surveys to ensure that they more accurately capture the  conditions at the time of the state survey, (2) including at least half of the  residents from state survey investigative samples to allow for a more clear- cut determination of whether the state survey should have cited a  deficiency, and (3) using the same number of federal surveyors as the  corresponding state survey, again to more closely mirror the conditions  under which the state survey was conducted.
Finally, AHFSA questioned whether the information that we received from  surveyors about the IDR process was universally valid because their input  about quality assurance reviews might be biased. Our methodology did not  rely solely on surveyor responses to our questionnaire but used a separate  questionnaire sent to state survey agency directors to help corroborate  their responses. Thus we reported both that (1) over 40 percent of  surveyors in four states indicated that their IDR process favored nursing  home operators and (2) one state survey agency director agreed and three  others acknowledged that frequent IDR hearings sometimes contributed to  the understatement of deficiencies. We also collected and reported data on  the number of deficiencies modified or overturned, which AHFSA said was  a more accurate measure of the effect of IDRs.
We also incorporated technical comments from AHFSA as appropriate.
As arranged with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies to the Administrator of the  Centers for Medicare & Medicaid Services and appropriate congressional  committees. In addition, the report will be available at no charge on GAO’s  Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix II.
section_title: Appendix I: Scope and Methodology
paragraphs: This appendix describes the data and methods we used to identify the  factors that contribute to the understatement of serious deficiencies on  nursing home surveys. This report relies largely on the data collected  through (1) two GAO-administered Web-based questionnaires to nursing  home surveyors and state agency directors and (2) analysis of federal and  state nursing home survey results as reported in the federal monitoring  survey database and the On-Line Survey, Certification, and Reporting  (OSCAR) system. Summary results from the GAO questionnaires are  available as an e-supplement to this report. See Nursing Homes:  Responses from Two Web-Based Questionnaires to Nursing Home  Surveyors and State Agency Directors (GAO-10-74SP), an E-supplement  to GAO-10-70. To augment our quantitative analysis, we also interviewed  officials at the Centers for Medicare & Medicaid (CMS) Survey and  Certification Group and select regional offices; reviewed federal  regulations, guidance, and our prior work; and conducted follow-up  interviews with eight state agency directors and a select group of  surveyors. Except where otherwise noted, we used data from fiscal year  2007 because they were the most recently available data at the time of our  analysis.
subsections: section_title: Development of Questionnaires and Analysis of Responses
paragraphs: We developed two Web-based questionnaires—one for the nursing home  surveyors and one for the state agency directors.
subsections: section_title: Development of the Questionnaires
paragraphs: The questionnaires were developed and the data collection and analysis  conducted to (1) minimize errors arising from differences in how a  particular question might be interpreted and in the sources of information  available to respondents and (2) reduce variability in responses that  should be qualitatively the same. GAO social science survey specialists  aided in the design and development of both questionnaires. We pretested  the two questionnaires with six surveyors from a local state and five  former or current state agency directors, respectively. Based on feedback  from these pretests, the questionnaires were revised to improve clarity and  the precision of responses, and ensure that all questions were fair and  unbiased. Most questions were closed-ended, which limited the  respondent to answers such as yes or no, or to identifying the frequency  that an event occurred using a scale—always, frequently, sometimes,  infrequently, or never. For reporting purposes, we grouped the scaled  responses into three categories—always/frequently, sometimes, and  infrequently/never. Both questionnaires included some open-ended  questions to allow respondents to identify specific training needs or other  concerns.
With few exceptions, respondents entered their responses directly into the  Web-based questionnaire databases. These questionnaires were sent to  the eligible population of nursing home surveyors and all state agency  directors. We performed computer analyses to identify illogical or  inconsistent responses and other indications of possible error. We also  conducted follow-up interviews with select respondents to clarify and gain  a contextual understanding of their responses.
section_title: Questionnaire for Nursing Home Surveyors
paragraphs: This questionnaire was designed to gather information from nursing home  surveyors nationwide about the process for identifying and citing nursing  home deficiencies. It included questions about various aspects of the  survey process identified by our prior work that may contribute to survey  inconsistency and the understatement of deficiencies. Such aspects  included survey methodology and guidance, deficiency determination,  surveyor training, supervisory review of draft surveys, and state agency  policies and procedures.
We fielded the questionnaire from May through July 2008 to 3,819 eligible  nursing home surveyors. To identify the eligible population, we  downloaded a list of identification numbers for surveyors who had  conducted at least one health survey of a nursing home in fiscal years 2006  or 2007 from CMS’s OSCAR database and we obtained surveyors’ e-mail  addresses from state survey agencies. We received complete responses  from 2,340 state surveyors, for a 61 percent response rate. The state-level  response rates were above 40 percent for all but three states— Connecticut, Illinois, and Pennsylvania. We excluded Pennsylvania from  our analysis because Pennsylvania’s Deputy Secretary for Quality  Assurance instructed the state’s surveyors not to respond to our survey  and few responded. (For response rates by state, see table 9.)
section_title: Questionnaire for State Agency Directors
paragraphs: The questionnaire for state agency directors was designed to gather  information on the nursing home survey process in each state. Directors  were asked many of the same questions as the surveyors, but the survey  agency directors’ questionnaire contained additional questions on the  overall organization of the survey agency, resource and staffing issues,  CMS’s Quality Indicator Survey (QIS), and experience with CMS’s federal  monitoring surveys. In addition, the questionnaire for state agency  directors asked them to rank the degree to which several factors, derived  from our previous work, contributed to understatement. This  questionnaire was fielded from September to November 2008 to all 50  state survey agency directors and the survey agency director for the  District of Columbia. We received completed responses from 50 of 51  survey agency directors, for a 98 percent response rate. The District of  Columbia survey agency director did not respond.
section_title: Analysis of Responses
paragraphs: To analyze results from the survey questions among groups, we used  standard descriptive statistics. In addition, we looked for associations  between questions through correlations and tests of the differences in  means for groups. For certain open-ended questions, we used a standard  content review method to identify topics that respondents mentioned such  as “applying CMS guidance,” “on-the-job training,” “time to complete  survey onsite,” or “time to complete the survey paperwork.” Our coding  process involved one independent coder and an independent analyst who  verified a random sample of the coded comments. For open-ended  questions that enabled respondents to provide additional general  information, we used similar standard content review methods, including  independent coding by two raters who resolved all disagreements through  discussion.
section_title: Validity and Reliability of Data
paragraphs: In addition to the precautions taken during the development of the  questionnaires, we performed automated checks on these data to identify  inappropriate answers. We also reviewed the data for missing or  ambiguous responses. Where comments on open-ended questions  provided more detail or contradicted responses to categorical questions,  the latter were corrected. On the basis of the strength of our systematic  survey processes and follow-up procedures, we determined that the  questionnaire responses were representative of the experience and  perceptions of nursing home surveyors and state agency directors  nationally and at the state level, with the exception of Pennsylvania  surveyors and the survey agency director of the District of Columbia. On  the basis of the response rates and these activities, we determined that the  data were sufficiently reliable for our purposes.
We also interviewed directors and other state agency officials in eight  states to better understand unusual or interesting circumstances related to  surveyor workforce and training, supervisory review, or state policies and  practices. We selected these eight states based on our analysis of  questionnaire responses from state agency directors and nursing home  surveyors.
section_title: Analysis of Federal Comparative and Observational Surveys
paragraphs: We used information from our May 2008 report on federal comparative  surveys nationwide for fiscal years 2002 through 2007 to categorize states  into groups. We used these results to identify states with high and low  percentages of serious missed deficiencies. We classified nine states as  high-understatement states—those that had 25 percent or more federal  comparative surveys identifying at least one missed deficiency at the  actual harm or immediate jeopardy levels across all years. These states  were Alabama, Arizona, Missouri, New Mexico, Oklahoma, South Carolina,  South Dakota, Tennessee, and Wyoming. Zero-understatement states were  those that had no federal comparative surveys identifying missed  deficiencies at the actual harm or immediate jeopardy levels. These seven  states were Alaska, Idaho, Maine, North Dakota, Oregon, Vermont, and  West Virginia. Low-understatement states were the 10 with the lowest  percentage of missed serious deficiencies (less than 6 percent)—Arkansas,  Nebraska, Ohio, and all seven zero-understatement states.
Response rates among the high-, low-, and zero-understatement states— approximately 77, 62, and 71 percent, respectively—supported statistical  testing of associations and differences among these state groupings.  Therefore, in addition to descriptive statistics, we used correlations and  tests of the differences in means for groups to identify questionnaire  responses that were associated with differences in understatement. We  reported the statistically significant results for tests of association and  differences between group averages at the 5 percent level, unless  otherwise noted.
In a previous report, we found a possible relationship between the  understatement of nursing home deficiencies on the federal comparative  surveys and surveyor performance in General Investigation and Deficiency  Determination on federal observational surveys—that is, high- understatement states more often had below-satisfactory ratings in  General Investigation and Deficiency Determination than low- understatement states. For this report, we applied the same statistical  analysis to identify when responses to our questionnaires were associated  with satisfactory performance on General Investigative and Deficiency  Determination skills on the federal observational surveys. We interpreted  such relationships as an indication of additional training needs.
section_title: Analysis of OSCAR
paragraphs: We used information from OSCAR and the federal monitoring survey  databases to (1) compare the deficiencies cited by state and federal  surveyors, (2) analyze the timing of nursing home surveys, and (3) assess  trends in deficiency citations. OSCAR is a comprehensive database that  contains information on the results of state nursing home surveys. CMS  reviews these data and uses them to compute nursing home facility and  state performance measures. When we analyzed these data, we included  automated checks of data fields to ensure that they contain complete  information. For these reasons, we determined that the OSCAR data were  sufficiently reliable for our purposes.
We used OSCAR and the federal monitoring survey database to compare  average facility citations on state survey records with the average citations  on federal observational survey records for the same facilities during fiscal  years 2002 through 2007. We computed the average number of serious  deficiencies cited on federal observational surveys between fiscal years  2002 through 2007, and for the same facilities and time period, calculated  the average number of serious deficiencies cited on state surveys. Next,  we determined which facilities had greater average serious deficiency  citations on federal observational surveys compared to state standard  surveys between fiscal years 2002 through 2007. For these facilities, we  computed the percentage difference between the average number of  serious deficiencies cited on federal observational surveys and those cited  on state surveys.
We used OSCAR to determine the percentage of the most recent state  surveys that were predictable because of their timing. Our analysis of  survey predictability compared the time between state agencies’ current  and prior standard nursing home surveys as of June 2008. According to  CMS, states consider 9 months to 15 months from the last standard survey  as the window for completing standard surveys because it yields a 12- month average. We considered surveys to be predictable if (1) homes were  surveyed within 15 days of the 1-year anniversary of their prior survey or  (2) homes were surveyed within 1 month of the maximum 15-month  interval between standard surveys.
We calculated the number of serious deficiencies on state surveys in  OSCAR from calendar year 1999 through 2007. We examined the trend in  G-level and higher deficiencies to assess whether CMS’s expanded  enforcement policy appeared to affect citation rates. Effective January  2000, CMS completed the implementation of its immediate-sanctions  policy, requiring the referral of homes that caused actual harm or  immediate jeopardy on successive standard surveys or intervening  complaint investigations.
section_title: Appendix II: Comments from the Department of Health & Human Services
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Walter Ochinko, Assistant  Director; Stefanie Bzdusek; Leslie V. Gordon; Martha R. W. Kelly;  Katherine Nicole Laubacher; Dan Lee; Elizabeth T. Morrison; Dan Ries;  Steve Robblee; Karin Wallestad; Rachael Wojnowicz; and Suzanne Worth  made key contributions to this report.
section_title: Related GAO Products
paragraphs: Nursing Homes: Opportunities Exist to Facilitate the Use of the  Temporary Management Sanction. GAO-10-37R. Washington, D.C.:  November 20, 2009.
Nursing Homes: CMS’s Special Focus Facility Methodology Should Better  Target the Most Poorly Performing Homes, Which Tended to Be Chain  Affiliated and For-Profit. GAO-09-689. Washington, D.C.: August 28, 2009.
Medicare and Medicaid Participating Facilities: CMS Needs to  Reexamine Its Approach for Funding State Oversight of Health Care  Facilities. GAO-09-64. Washington, D.C.: February 13, 2009.
Nursing Homes: Federal Monitoring Surveys Demonstrate Continued  Understatement of Serious Care Problems and CMS Oversight  Weaknesses. GAO-08-517. Washington, D.C.: May 9, 2008.
Nursing Home Reform: Continued Attention Is Needed to Improve  Quality of Care in Small but Significant Share of Homes. GAO-07-794T.  Washington, D.C.: May 2, 2007.
Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not  Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241.  Washington, D.C.: March 26, 2007.
Nursing Homes: Despite Increased Oversight, Challenges Remain in  Ensuring High-Quality Care and Resident Safety. GAO-06-117.  Washington, D.C.: December 28, 2005.
Nursing Home Quality: Prevalence of Serious Problems, While  Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561.  Washington, D.C.: July 15, 2003.
Nursing Homes: Quality of Care More Related to Staffing than Spending.  GAO-02-431R. Washington, D.C.: June 13, 2002.
Nursing Homes: Sustained Efforts Are Essential to Realize Potential of  the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.:   September 28, 2000.
Nursing Home Care: Enhanced HCFA Oversight of State Programs  Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.:  November 4, 1999.
Nursing Homes: Proposal to Enhance Oversight of Poorly Performing  Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999.
Nursing Homes: Additional Steps Needed to Strengthen Enforcement of  Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.:   March 18, 1999.
California Nursing Homes: Care Problems Persist Despite Federal and  State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.