Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should Complement State Activities
Published: 2002-02-15T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: Nursing homes that participate in Medicare and Medicaid must periodically assess the needs of residents in order to develop an appropriate plan of care. Such resident assessments are known as the minimum data set (MDS). According to officials in the 10 states with MDS accuracy review programs in operation as of January 2001, these programs were established to set Medicaid payments and identify quality of care problems. Nine of the 10 states conduct periodic on-site reviews in all or a significant portion of their nursing homes to assess the accuracy of the MDS data. These reviews sample a home's MDS assessments to determine whether the basis for the assessments is adequately documented in residents' medical records. These reviews often include interviews of nursing home personnel familiar with residents and observations of the residents themselves. States with separate MDS review programs identified various approaches to improve MDS accuracy. State officials highlighted the on-site review process itself and provider education activities as their primary approaches. State officials also reported such remedies as requiring nursing homes to prepare a corrective action plan or imposing financial penalties on nursing homes when serious or extensive errors in MDS data are found. Following the 1998 implementation of Medicare's MDS-based payment system, the Health Care Financing Administration began its own review program to ensure the accuracy of MDS data.

section_title: Letter
paragraphs: Nursing homes play an important role in the health care system of the United States. More than 40 percent of elderly Americans will use a nursing home at some time in their lives. Such facilities provide skilled nursing, therapy, or supportive care to older individuals who do not need the intensive medical care provided by hospitals, but for whom receiving care at home is not feasible. Under the Medicare and Medicaid programs, nursing homes were expected to receive $58 billion in 2001, with a federal share of approximately $38 billion. Nursing homes that participate in these programs are required to periodically assess the care needs of residents in order to develop an appropriate plan of care. Such resident assessment data are known as the minimum data set (MDS). The federal government contracts with states to periodically inspect or survey nursing homes, and state surveyors use MDS data to help assess the quality of resident care.Medicare and some state Medicaid programs also use MDS data to adjust nursing home payments to account for variation in resident care needs.
Thus, the accuracy of MDS data has implications for the identification of quality problems and the level of nursing home payments.
MDS accuracy is one of many areas that state surveyors are expected to examine during periodic nursing home surveys. Federal guidance for state surveyors regarding the accuracy of MDS assessments focuses on whether appropriate personnel completed or coordinated the assessments and whether there are any indications that the assessments were falsified. This guidance also instructs surveyors to conduct a check of specific MDS items to ensure that the resident’s condition is appropriately characterized.  Concerns exist, however, that state surveyors already have too many tasks and that, as a result, the survey process may not adequately address MDS accuracy. In addition, our prior work on nursing home quality issues has identified weaknesses in the survey process that raise questions about the thoroughness and consistency of state surveys.
In response to your request, we assessed (1) how states monitor the accuracy of MDS data compiled by nursing homes through review programs separate from their standard nursing home survey process, (2) how states attempt to improve the data’s accuracy where there are indications of problems, and (3) how the federal government ensures the accuracy of MDS data. We surveyed the 50 states and the District of Columbia to determine whether states had a separate MDS review program—distinct from any MDS oversight that might occur during the periodic nursing home surveys performed by all states. We then conducted structured interviews with officials in 10 of the 11 states that indicated they had separate MDS review programs. We also interviewed staff from the Centers for Medicare and Medicaid Services (CMS), an agency within the Department of Health and Human Services (HHS) that manages the Medicare and Medicaid programs, who were responsible for developing the agency’s MDS review program. In addition, we reviewed regulations, literature, and other documents relating to MDS data. We performed our work from December 2000 through January 2002 in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The nation’s 17,000 nursing homes play an essential role in our health care system, providing services to 1.6 million elderly and disabled persons who are temporarily or permanently unable to care for themselves but who do not require the level of care furnished in an acute care hospital. Depending on the identified needs of each resident, as determined through MDS assessments, nursing homes provide a variety of services, including nursing and custodial care, physical, occupational, and speech therapy, and medical social services. The majority of nursing home residents have their care paid for by Medicaid, a joint federal-state program for certain low-income individuals. Almost all nursing homes serve Medicaid residents, while more than 14,000 nursing homes are also Medicare- certified. Medicare, the federal health care program for elderly and disabled Americans, pays for posthospital nursing home stays if a beneficiary needs skilled nursing or rehabilitative services. Medicare- covered skilled nursing home days account for approximately 9 percent of total nursing home days. Medicare beneficiaries tend to have shorter nursing home stays and receive more rehabilitation services than individuals covered by Medicaid.
subsections: section_title: MDS Used to Assess Nursing Home Residents
paragraphs: Since 1991, nursing homes have been required to develop a plan of care for each resident based on the periodic collection of MDS data. The MDS contains individual assessment items covering 17 areas, such as mood and behavior, physical functioning, and skin conditions. MDS assessments of each resident are conducted in the first 14 days after admission and are used to develop a care plan. A range of professionals, including nurses, attending physicians, social workers, activities professionals, and occupational, speech, and physical therapists, complete designated parts of the MDS. Assessing a resident’s condition in certain areas requires observation, often over a period of days. For example, nursing staff must assess the degree of resident assistance needed during the previous 7 days—none, supervised, limited, extensive, or total dependence—to carry out the activities of daily living (ADL), such as using a toilet, eating, or dressing. To obtain this information, staff completing the MDS assessments are required to communicate with direct care staff, such as nursing assistants or activities aides, who have worked with the resident over different time periods. These staff have first-hand knowledge of the resident and will often be the primary and most reliable source of information regarding resident performance of different activities. While a registered nurse is required to verify that the MDS assessment is complete, each professional staff member who contributed to the assessment must sign and attest to the accuracy of his or her portion of the assessment.
section_title: MDS Used in Quality Oversight and as Basis for Payments
paragraphs: MDS data are also submitted by nursing homes to states and CMS for use in the nursing home survey process and to serve as the basis for adjusting payments. CMS contracts with states to periodically survey nursing homes to review the quality of care and assure that the services delivered meet the residents’ assessed needs. In fiscal year 2001, the federal government spent about $278 million on the nursing home survey process. Effective July 1999, the agency instructed states to begin using quality indicators derived from MDS data to review the care provided to a nursing home’s residents before state surveyors actually visit the home to conduct a survey. Quality indicators are essentially numeric warning signs of potential care problems, such as greater-than-expected instances of weight loss, dehydration, or pressure sores among a nursing home’s residents. They are used to rank a facility in 24 areas compared with other nursing homes in a state. In addition, by using the quality indicators before the on- site visit to select a preliminary sample of residents to review, surveyors should be better prepared to identify potential care problems.
In addition to quality oversight, some state Medicaid programs and Medicare use MDS data to adjust nursing home payments to reflect the expected resource needs of their residents. Such payment systems are commonly known as “case-mix” reimbursement systems. Because not all residents require the same amount of care, the rate paid for each resident is adjusted using a classification system that groups residents based on their expected costs of care. Facilities use MDS data to assign residents to case-mix categories or groups that are defined according to clinical condition, functional status, and expected use of services. In Medicare, these case-mix groups are known as resource utilization groups. Each case-mix group represents beneficiaries who have similar nursing and therapy needs. As of January 2001, 18 states had introduced such payment systems for their Medicaid programs. As directed by the Congress, HCFA in 1998 implemented a prospective payment system (PPS) for skilled nursing facilities (SNF)—nursing homes that are certified to serve Medicare beneficiaries. The SNF PPS also uses MDS data to adjust nursing home payments.
section_title: MDS Review Activities Can Be On-Site or Off-Site
paragraphs: States and CMS use the term “accuracy reviews” to describe efforts that help ensure MDS assessments accurately reflect residents’ conditions. Review activities can be performed on-site—that is, at the nursing home— or off-site. On-site reviews generally consist of documentation reviews to determine whether the resident’s medical record supports the MDS assessment completed by the facility. If the MDS assessment is recent, the review may also include direct observation of the resident and interviews with nursing home staff who have recently evaluated or treated the resident.
While documentation reviews may also be conducted outside of the nursing home, other off-site reviews of MDS data include examining trends across facilities. For example, off-site review activities could involve the examination of monthly reports showing the distribution of residents’ case-mix categories across different facilities in a state. Similarly, off-site reviews could also involve an examination of particular MDS elements, such as the distribution of ADLs within and across nursing homes to identify aberrant or inconsistent patterns that may indicate the need for further investigation. Off-site and on-site reviews may also be combined as a way of leveraging limited resources to conduct MDS accuracy activities.
section_title: Only Eleven States Conduct Separate On- Site or Off-Site Reviews of MDS Accuracy
paragraphs: Eleven states conduct separate MDS accuracy reviews apart from their standard nursing home survey process.  Ten of these states’ reviews were in operation as of January 2001. An additional 7 states reported that they intend to initiate similar accuracy reviews. All 18 of these states either currently use an MDS-based Medicaid payment system or plan to implement such a system. The remaining 33 states have no plans to implement separate MDS review programs and currently rely on their periodic nursing home surveys for MDS oversight. In all but one of the states with separate MDS review programs operating as of January 2001, accuracy reviews entail periodic on-site visits to nursing homes. The reviews focus on whether a sample of MDS assessments completed by the facility is supported by residents’ medical records. If the MDS assessments reviewed are recent enough that residents are still in the facility and their health status has not changed, the on-site review may also be supplemented with interviews of nursing home staff familiar with the residents, as well as observations of the residents themselves, to validate the record review. About half of these states also conduct off-site data analyses in which reviewers look for significant changes or outliers, such as facilities with unexplained large shifts in the distribution of residents across case-mix categories over a short period. Officials primarily attributed the errors found during their on-site reviews to differences in clinical interpretation and mistakes, such as a misunderstanding of MDS definitions. A few of these states have been able to show some recoupments of Medicaid payments since the implementation of their on- site review programs.
subsections: section_title: Most States Do Not Have Separate MDS Review Programs
paragraphs: Of the 50 states and the District of Columbia, only 11 conduct accuracy reviews of MDS data that are separate from the state’s nursing home survey process. (See table 1.) These 11 states provide care to approximately 22 percent of the nation’s nursing home residents and all but one have an MDS-based payment system (Virginia began conducting MDS accuracy reviews in April 2001 in anticipation of adopting such a payment system in 2002). Seven additional states plan to initiate separate MDS reviews—three currently have an MDS-based payment system and four are planning to implement such a payment system. Officials in the 10 states with separate, longer standing MDS review programs said that the primary reason for implementing reviews was to ensure the accuracy of the MDS data used in their payment systems. Several of these states also indicated that the use of MDS data in generating quality indicators was another important consideration. Vermont officials, in particular, emphasized the link to quality of care, noting that the state had created its own MDS-based quality indicators prior to HCFA’s requirement to use quality indicators in nursing home surveys. A state official told us it was critical that the MDS data be accurate because Vermont was making this information available to the public as well as using it internally as a normal part of the nursing home survey process.
To varying degrees, three major factors influenced the decision of 33 states not to establish separate MDS review programs. First, the majority—28 states—do not have MDS-based Medicaid payment systems. Second, some states cited the cost of conducting separate reviews. Kansas, for example, reported a lack of funding and staff resources as the reason for halting a brief period of on-site visits in 1996 as a follow-up to nursing home surveys. Arkansas as well reported insufficient staff for conducting a separate review of MDS data. Finally, officials in about one- third of the states without separate MDS reviews volunteered that they had some assurance of the accuracy of MDS data either because of training programs for persons responsible for completing MDS assessments or because of the nursing home survey process. For example, Missouri operates a state funded quality improvement project in which nurses with MDS training visit facilities to assist staff with the MDS process and use of quality indicator reports. North Carolina also reported that its quarterly training sessions provide MDS training to approximately 800 providers a year. Regarding standard surveys, Connecticut and Maryland reported that their nursing home survey teams reviewed MDS assessments to determine if they were completed correctly and if the assessment data matched surveyor observations of the resident. In Connecticut, surveyors may also review a sample of facility MDS assessments for possible errors whenever they identify aberrant or questionable data on the quality indicator reports.
Officials in the 10 states with separate, longer standing MDS review programs generally said that the survey process itself does not detect MDS accuracy issues as effectively as separate MDS review programs. Some noted that nursing home surveyors do not have time to thoroughly review MDS accuracy and often review a smaller sample size than MDS reviewers. The surveyors’ primary focus, they indicated, was on quality of care and resident outcomes—not accuracy of MDS data. For example, surveyors would look at whether the resident needed therapy and whether it was provided. In contrast, the MDS reviewer would calculate the total number of occupational, speech, and physical therapy minutes to ensure that the resident was placed in the appropriate case-mix category. Officials in Iowa similarly noted that surveyors do not usually cite MDS accuracy as a specific concern unless there are egregious MDS errors, again, because the focus of the survey process is on quality of care.
section_title: States with Separate MDS Review Programs Emphasize On-Site Oversight, but Also Conduct Off-Site Monitoring
paragraphs: Nine of the 10 states with separate, longer standing MDS accuracy review programs use on-site reviews to test the accuracy of MDS data, generally visiting all or a significant portion of facilities in the state at least annually, if not more frequently. (See app. I for a summary of state on-site review programs.) Due to a lack of staff, one state—West Virginia—limits its MDS reviews to off-site analysis of facility-specific monthly data. Most of these states have been operating their MDS review programs for 7 years or longer and developed them within a year of implementing an MDS-based payment system. Three of the nine states arrive at the facility unannounced while the other six provide advanced notice ranging from 48 hours to 2 weeks.
The sample of facility MDS assessments reviewed by each state varies considerably. Assessment sample sizes generally range from 10 to 40 percent of a nursing home’s total residents but some states select a specific number of residents, not a percentage, and a few specifically target residents in particular case-mix categories. For example, Indiana selects a sample of 40 percent—or no less than 25 residents—across all major case-mix categories, while Ohio’s sample can be based on a particular case-mix category, such as residents classified as “clinically complex.” Iowa officials told us that its reviewers select at least 25 percent of a facility’s residents, with a minimum of 5 residents, while Pennsylvania chooses 15 residents from each facility, regardless of case- mix category or facility size. Some states expand the resident sample when differences between the MDS assessment and supporting documentation reach a certain threshold. For example, if the on-site review for the initial sample in Iowa finds that 25 percent or more of the MDS assessments have errors, a supplemental random sample is selected for review. While a few states limit their sample to Medicaid residents only, most select assessments to review from the entire nursing home’s population.
On-site reviews generally involve a comparison of the documentation in the resident’s medical record to the MDS assessment prepared by the facility. Generally, the on-site process also allows reviewers to interview nursing home staff and to directly observe residents, permitting a better understanding of the documentation in a resident’s medical record and clarifying any discrepancies that may exist. Staff interviews and resident observations can enhance the reviewer’s understanding of the resident’s condition and allow a more thorough MDS review than one relying primarily on documentation. However, as the interval between the facility’s MDS assessment and the on-site review increases, staff interviews and resident observations become less reliable and more difficult to conduct. For example, staff knowledge of a particular patient may fade over time, the patient’s health status may change, or the patient may be discharged from the facility. Pennsylvania officials, who reported reviewing assessments that were 6 to 12 months old, told us that the state’s MDS reviews tended to identify whether the nursing home had adequate documentation. Reviewing such old assessments tends to focus the review process on the adequacy of the documentation rather than on whether the MDS assessment was accurate. Four of the nine states review assessments between 30 and 90 days old, a process that likely increases the value of interviews and observation. The combination of interviews and observations can be valuable, but limiting reviews to only recent MDS assessments and providing homes advance notice may undermine the effectiveness of on-site reviews. Under such circumstances, facilities have an opportunity to focus on the accuracy of their recent assessments, particularly if the nursing home knows when their reviews will occur, instead of adopting facility-wide practices that increase the accuracy of all MDS assessments.
Based on their on-site reviews, officials in the nine states identified seven areas as having a high potential for MDS errors, with two areas most often identified as being among the highest potential for error: (1) mood and behavior and (2) nursing rehabilitation and restorative care. (See fig. 1.) Assessments of resident mood and behavior are used to calculate quality indicators and, along with nursing rehabilitation and restorative care, are often important in determining nursing home payments. CMS indicated that several of the MDS elements cited in figure 1 were also identified by a CMS contractor as areas of concern. Officials in most states with separate on-site review programs told us that errors discovered during their on-site reviews often resulted from differences in clinical interpretation or mistakes, such as a misunderstanding of MDS definitions by those responsible for completing MDS assessments. Officials in only four of the nine states were able to tell us whether the errors identified in their MDS reviews on average resulted in a case-mix category that was too high or too low. Two of these states reported roughly equal numbers of MDS errors that inappropriately placed a resident in either a higher or lower case-mix category; a third indicated that errors more often resulted in higher payments; and a fourth found that errors typically resulted in payments that were too low. None of the nine states track whether quality indicator data were affected by MDS errors.
Two of the 10 states with MDS review programs were able to tell us the amount of Medicaid recoupments resulting from inaccurate MDS assessments. From state fiscal years 1994 through 1997, South Dakota officials reported that the state had recouped about $360,000 as a result of recalculating nursing home payments after MDS reviews. West Virginia received $1 million in 1999 related to MDS errors for physical therapy discovered during a 1995 on-site review at a nursing home. Officials in five additional states told us that they recalculate nursing home payments when MDS errors are found, but could not provide the amount recovered.
Of the 10 states with longer standing MDS review programs, four use off- site analyses to supplement their on-site reviews, while one state relies on off-site analyses exclusively. Both Maine and Washington examine MDS data off-site to monitor changes by facility in the mix of residents across case-mix categories. Such changes may help identify aberrant or inconsistent patterns that may indicate the need for further investigation. Ohio, a state with approximately 1,000 facilities—more than any other state that conducts MDS reviews—analyzes data off-site to identify facilities with increased Medicaid payments and changes in case-mix categories to select the approximately 20 percent of facilities visited each year. West Virginia has eliminated its on-site reviews and now focuses solely on analyzing monthly reports for its 141 facilities—for example, significant changes in case-mix categories or ADLs across consecutive MDS assessments. In addition to informally sharing results of off-site reviews with the state nursing home surveyors, West Virginia is trying to formalize a process in which off-site reviews could trigger additional on- site or off-site documentation reviews.
section_title: States Attempt to Improve MDS Data Accuracy through On- Site Reviews, Training, and Other Remedies
paragraphs: Officials in the nine states with on-site review programs consistently cited three features of their review programs that strengthened the ability of nursing home staff to complete accurate MDS assessments and thus decrease errors: (1) the actual presence of reviewers, (2) provider education, and (3) remedies that include corrective action plans and financial penalties. On-site reviews, for example, underscore the state’s interest in MDS accuracy and provide an opportunity to train and coach those who are responsible for completing MDS assessments. Similarly, the errors discovered during on-site reviews guide the development of more formal training sessions that are offered by the state outside of the nursing home. Requiring nursing homes to prepare corrective action plans and imposing financial penalties signal the importance of MDS accuracy to facilities and are tools to improve the accuracy of the MDS data. As a result of these efforts, some states have been able to show a notable decrease in their overall error rates.
Most of the nine states view on-site visits and training as interrelated elements that form the foundation of their MDS review programs. State officials said that nursing homes pay more attention to properly documenting and completing the MDS assessments because reviewers visit the facilities regularly. On-site visits also allow reviewers to discuss MDS documentation issues or requirements with staff, providing an opportunity for informal MDS training. For example, Indiana officials told us that 2 to 3 hours of education are a routine part of each facility’s MDS review. Noting the high staff turnover rates in nursing homes, many states reported that frequent training for the staff responsible for completing MDS assessments is critical. Officials in seven of the nine states with on- site reviews told us that high staff turnover was one of the top three factors contributing to MDS errors in their states. In addition, many of the reasons cited for MDS errors—such as a misunderstanding of MDS definitions and other mistakes—reinforce the need for training.
States with on-site reviews use the process to guide provider education activities—both on-site and off-site. For example, during Pennsylvania’s annual MDS reviews of all nursing homes, state reviewers determine the types of training needed. According to state officials, the state uses the results of these reviews to shape and provide facility-specific training, if it is needed, within a month of the review and subsequently conducts a follow-up visit to see if the facility is improving in these areas. They indicated that all 685 homes visited during 2000, the first year of this approach, were provided with some type of training. To improve MDS accuracy, several states also provide voluntary training opportunities outside of the nursing home. Maine, Iowa, Indiana, and South Dakota, for example, provide MDS training regularly throughout the state, rotating the location of the training by region so that it is accessible to staff from all facilities.
While states generally emphasized on-site reviews and training as the primary ways to improve the accuracy of the MDS data, some reported that they have also instituted certain remedies, such as corrective action plans and financial penalties. Indiana and Pennsylvania, for example, require facilities to submit a corrective action plan detailing how the facility will address errors identified during an on-site review. Two states—Maine and Indiana—impose financial penalties. Maine has instituted financial penalties for recurring serious errors, collecting approximately $390,000 since late 1995. Maine also requires facilities with any MDS errors that result in a case-mix category change to complete and submit a corrected MDS assessment for the resident. While Indiana imposes financial penalties, it does not view them as the primary tool for improving MDS accuracy. Rather, officials attributed a decrease in MDS errors to the education of providers and the on-site presence of reviewers. Other remedies cited by states include conducting more frequent on-site MDS reviews and referring suspected cases of fraud to their state’s Medicaid Fraud Control Unit.
Five of the nine states that conduct on-site MDS reviews told us that their efforts have resulted in a notable decrease in MDS errors across all facilities since the implementation of their review programs. (See table 2.) South Dakota officials, for example, reported that the percentage of assessments with MDS errors across facilities had decreased from approximately 85 percent to 10 percent since the implementation of the state’s MDS review program in 1993. Similarly, Indiana reported a decrease in the statewide average error rate from 75 percent to 30 percent of assessments in 1 year’s time. Four states could not provide these data. In calculating these decreases, three of the five states—Indiana, Maine, and South Dakota—define MDS errors as an unsupported MDS assessment that caused the case-mix category to be inaccurate. Iowa’s definition, however, includes MDS elements that are not supported by medical record documentation, observation, or interviews, regardless of whether the MDS error changed the case-mix category. Similarly, while Pennsylvania does not limit errors to those that changed the case-mix category, the state defines errors as a subset of MDS elements that are not supported by the medical record.
section_title: CMS’ MDS Review Program Could Better Leverage Existing State and Federal Accuracy Activities
paragraphs: Following implementation of Medicare’s MDS-based payment system in 1998, HCFA began building the foundation for its own separate review program—distinct from state efforts—to help ensure the accuracy of MDS data. In the course of developing and testing accuracy review approaches, its contractor found widespread MDS errors that resulted in a change in Medicare payment categories for 67 percent of the resident assessments sampled. In September 2001, CMS awarded a new contract to implement a nationwide MDS review program over a 2- to 3-year period. Despite the benefits of on-site reviews, as demonstrated by states with separate review programs, the current plan involves conducting on-site reviews in fewer than 200 of the nation’s 17,000 nursing homes each year. In addition, the contractor’s combined on-site and off-site reviews to evaluate MDS accuracy will involve only about 1 percent of the approximately 14.7 million MDS assessments expected to be prepared in 2001. In contrast, states that conduct separate on-site MDS reviews typically visit all or a significant portion of their nursing homes and generally examine from 10 to 40 percent of assessments. While CMS’ approach may yield some broad sense of the accuracy of MDS assessments on an aggregate level, it may be insufficient to help ensure the accuracy of MDS assessments in most of the nation’s nursing homes. At present, it does not appear that CMS plans to leverage the considerable resources already devoted to state nursing home surveys and states’ separate MDS review programs that together entail a routine on-site presence in all nursing homes nationwide. Nor does it plan to more systematically evaluate the performance of state survey agencies regarding MDS accuracy through its own federal comparative surveys. Finally, CMS is not requiring nursing homes to provide documentation for the full MDS assessment, which could undermine the efficacy of its MDS reviews.
subsections: section_title: Testing of MDS Accuracy Approaches Identified Widespread Accuracy Problems
paragraphs: In September 1998, HCFA contracted with Abt Associates to develop and test various on-site and off-site approaches for verifying and improving the accuracy of MDS data. Two of the approaches resembled state on-site MDS reviews and the off-site documentation reviews performed by CMS contractors that review Medicare claims. Another approach used off-site data analysis to target facilities for on-site review. To determine the effectiveness of the approaches tested in identifying MDS inaccuracies, Abt compared the errors found under each approach to those found in its “reference standard”—independent assessments performed by MDS- trained nurses hired by Abt for approximately 600 residents in 30 facilities in three states. Abt found errors in every facility, with little variation in the percentage of assessments with errors across facilities. On average, the errors found affected case-mix categories in 67 percent of the sampled Medicare assessments. Abt concluded that the errors did not result in systematic overpayments or underpayments to facilities even though there were more errors that placed residents in too high as opposed to too low a case-mix category. Abt did not determine, however, the extent to which errors affected quality indicators.
Due to the prevalence of errors, Abt recommended a review program that included periodically visiting all facilities during the program’s first several years. Recognizing the expense of visiting every facility, however, Abt also recommended eventually transitioning to the use of off-site mechanisms to target facilities and specific assessments for on-site review. Abt also made recommendations to address the underlying causes of MDS errors: simplifying the MDS assessment tool, clarifying certain MDS definitions (particularly for ADLs), and improving MDS training for facilities.
section_title: The Federal MDS Review Program Is Too Limited to Evaluate State-Level Accuracy Assurance Efforts
paragraphs: Building on the work of Abt Associates, in the summer of 2000, the agency began formulating its own distinct nationwide review program to address long-term MDS monitoring needs. The agency developed a request for proposal for MDS data assessment and verification activities and sought proposals from its 12 program safeguard contractors. On September 28, 2001, CMS awarded a 3-year contract for approximately $26 million to Computer Sciences Corporation. The contract calls for the initiation of on- site and off-site reviews by late spring 2002, but the full scope of MDS review activities will not be underway until the second year of the contract. (See table 3.)
Despite this broad approach, the contractor is not specifically tasked with assessing the adequacy of each state’s MDS reviews. Instead, it is required to develop a strategy for coordinating its review activities with other state and federal oversight, such as the selection of facilities and the timing of visits, to avoid unnecessary overlap with routine nursing home surveys or states’ separate MDS review programs. This approach does not appear to build on the benefits of on-site visits that are already occurring as part of state review activities. Rather, the contract specifies independent federal on-site and off-site reviews of roughly 1 percent of the approximately 14.7 million MDS assessments expected to be prepared in 2001—80,000 during the first contract year and 130,000 per year thereafter. The contractor, however, tentatively recommended that the majority of reviews, about 90 percent, be conducted off-site. According to CMS, these off-site reviews could include a range of activities, such as the off-site targeting approaches developed by Abt or medical record reviews similar to those conducted by CMS contractors for purposes of reviewing Medicare claims. In addition, the contractor is expected to conduct a range of off-site data analyses that could include a large number of MDS assessments. The remaining 10 percent of MDS assessments— representing fewer than 200 of the nation’s 17,000 nursing homes—would be reviewed on-site each year. This limited on-site presence is inconsistent with Abt’s earlier recommendation regarding the benefits of on-site reviews in detecting accuracy problems, and with the view of almost all of the states with separate MDS review programs that an on-site presence at a significant number of their nursing homes is central to their review efforts.
While CMS’ approach may yield some broad sense of the accuracy of MDS assessments on an aggregate level, it appears to be insufficient to provide confidence about the accuracy of MDS assessments in the vast bulk of nursing homes nationwide. Given the substantial resources invested in on- site nursing home visits associated with standard surveys or states’ separate MDS review programs, CMS’ MDS review program could view states’ routine presence as the cornerstone of its program and instead focus its efforts on ensuring the adequacy of state reviews. CMS could build on its established federal monitoring survey process for nursing home oversight. The agency is required by statute to annually resurvey at least 5 percent of all nursing homes that participate in Medicare and Medicaid. One of the ways CMS accomplishes this requirement is by conducting nursing home comparative surveys to independently assess the states’ performance in their nursing home survey process. During a comparative survey, a federal team independently surveys a nursing home recently inspected by a state in order to compare and contrast the results. These federal comparative surveys have been found to be most effective when completed in close proximity to the state survey and involve the same sample of nursing home residents to the maximum extent possible. Abt also attempted to review recently completed MDS assessments.
Finally, a potential issue that could undermine the efficacy of the federal MDS accuracy reviews involves the level of documentation required to support an MDS assessment. CMS requires specific documentation for some MDS elements, but officials said that the MDS itself—which can simply consist of checking off boxes or selecting multiple choice answers on the assessment form—generally constitutes support for the assessment without any additional documentation. CMS officials consider the MDS assessment form to have equal weight with the other components of the medical record, such as physician notes and documentation of services provided. As a result, CMS asserts that the assessment must be consistent with, but need not duplicate, the medical record. In contrast, most of the nine states with separate on-site review programs require that support for each MDS element that they review be independently documented in the medical record. State officials told us that certain MDS elements, such as ADLs, are important to thoroughly document because they require observation of many activities by different nursing home staff over several days. As a result, some of these states require the use of separate flow charts or tables to better document ADLs. Similarly, some states require documentation for short-term memory loss rather than accepting a nursing home’s assertion that a resident has this condition. CMS’ training manual describes several appropriate tests for identifying memory loss, such as having a resident describe a recent event.  In one of its December 2000 reports, the HHS OIG recommended that nursing homes be required to establish an “audit trail” to support certain MDS elements. HCFA disagreed, noting that it does not expect all information in the MDS to be duplicated elsewhere in the medical record. However, given the uses of MDS data, especially in adjusting nursing home payments and producing quality indicators, documenting the basis for the MDS assessments in the medical record is critical to assessing their accuracy.
section_title: Conclusions
paragraphs: In complying with federal nursing home participation and quality requirements, about 17,000 nursing homes were expected to produce almost 15 million MDS assessments during 2001 on behalf of their residents. This substantial investment of nursing home staff time contributes to multiple functions, including establishing patient care plans, assisting with quality oversight, and setting nursing home payments that account for variation in resident care needs. While some states, particularly those with MDS-based Medicaid payment systems, stated that ensuring MDS accuracy requires establishing a separate MDS review program, many others rely on standard nursing home surveys to assess the data’s accuracy. Flexibility in designing accuracy review programs that fit specific state needs, however, should not preclude achieving the important goal of ensuring accountability across state programs. It is CMS’ responsibility to consistently ensure that states are fulfilling statutory requirements to accurately assess and provide for the care needs of nursing home residents.
The level of federal financial support for state MDS accuracy activities is already substantial. The federal government pays up to 75 percent of the cost of separate state MDS review activities and in fiscal year 2001 contributed $278 million toward the cost of the state nursing home survey process, which is intended in part to review MDS accuracy. Instead of establishing a distinct but limited federal review program, reorienting the thrust of its review program in order to complement ongoing state MDS accuracy efforts could prove to be a more efficient and effective means to achieve CMS’ stated goals. Such a shift in focus should include (1) taking full advantage of the periodic on-site visits already conducted at every nursing home nationwide through the routine state survey process, (2) ensuring that the federal MDS review process is designed and sufficient to consistently assess the performance of all states’ reviews for MDS accuracy, and (3) providing additional guidance, training, and other technical guidance to states as needed to facilitate their efforts. With its established federal monitoring system for nursing home surveys— especially the comparative survey process—that helps assess state performance in conducting the nursing home survey process, CMS has a ready mechanism in place that it can use to systematically assess state performance for this important task. Finally, to help improve the effectiveness of MDS review activities, CMS should take steps to ensure that each MDS assessment is adequately supported in the medical record.
section_title: Recommendations for Executive Action
paragraphs: With the goal of complementing and leveraging the considerable federal and state resources already devoted to nursing home surveys and to separate MDS accuracy review programs, we recommend that the administrator of CMS review the adequacy of current state efforts to ensure the accuracy of MDS data, and provide, where necessary, additional guidance, training, and technical assistance;   monitor the adequacy of state MDS accuracy activities on an ongoing basis, such as through the use of the established federal comparative survey process; and   provide guidance to state agencies and nursing homes that sufficient evidentiary documentation to support the full MDS assessment be included in residents’ medical records.
section_title: Agency and State Comments and Our Evaluation
paragraphs: We provided a draft of this report to CMS and the 10 states with separate MDS accuracy programs for their review and comment. (See app. II for CMS’ comments.) CMS agreed with the importance of assessing and monitoring the adequacy of state MDS accuracy efforts. CMS also recognized that the MDS affects reimbursement and care planning and that it is essential that the assessment data reflect the resident’s health status so that the resident may receive the appropriate quality care and that providers are appropriately reimbursed. However, CMS’ comments did not indicate that it planned to implement our recommendations and reorient its MDS review program. Rather, CMS’ comments suggested that its current efforts provide adequate oversight of state activities and complement state efforts.
While CMS stated that it currently evaluates, assesses, and monitors the accuracy of the MDS through the nursing home survey process, it also acknowledged the wide variation in the adequacy of current state accuracy review efforts. Our work in the 10 states with separate MDS review programs raised serious questions about the thoroughness and adequacy of the nursing home survey process for reviewing MDS accuracy. Officials in many of these states said that the survey process itself does not detect MDS accuracy issues as effectively as separate MDS review programs. Surveyors, we were told, do not have time to thoroughly review MDS accuracy and their focus is on quality of care and resident outcomes, not accuracy of MDS data.
In response to our recommendations on assessing and monitoring the adequacy of each state’s MDS reviews, CMS commented that it would consider adding a new standard to the state performance expectations that the agency initiated in October 2000. CMS indicated that the state agency performance review program would result in a more comprehensive assessment of state activities related to MDS accuracy than could be obtained through the comparative survey process. CMS also outlined planned analytic activities—such as a review of existing state and private sector MDS review methodologies and instruments, ongoing communications with states to share the knowledge gained, and comprehensive analyses of MDS data to identify systemic accuracy problems within states as well as across states—that it believes will help to evaluate state performance.
We agree that some of CMS’ proposed analytic activities could provide useful feedback to states on problem areas at the provider, state, region, and national levels. Similarly, the addition of MDS accuracy activities to its state performance standards for nursing home surveys, which CMS is considering, has merit. While CMS plans to consider adding a new standard to its state agency performance review program, the agency has a mechanism in place—the comparative survey process—that it could readily use to systematically assess state performance. However, CMS apparently does not intend to do so. Based on our discussions with agency officials, it does not appear that CMS’ approach will yield a consistent evaluation of each state’s performance. We continue to believe that assessment and routine monitoring of each state’s efforts should be the cornerstone of CMS’ review program. As we previously noted, the agency’s proposed on-site and off-site reviews of MDS assessments are too limited to systematically assess MDS accuracy in each state and would consume resources that could be devoted to complementing and overseeing ongoing state activities. A comprehensive review of the adequacy of state MDS accuracy activities, particularly in those states without a separate review program, is essential to establish a baseline and to allow CMS to more efficiently target additional guidance, training, or technical assistance that it acknowledged is necessary.
CMS did not agree with our recommendation that it should provide guidance to states regarding adequate documentation in the medical record for each MDS assessment. CMS stated that requiring documentation of all MDS items places an unnecessary burden on facilities. Skilled reviewers, it stated, should be able to assess the accuracy of completed MDS assessments through a combination of medical record review, observation, and interviews. CMS further stated that requiring duplicative documentation might result in documentation that is manufactured and of questionable accuracy. Of course, the potential for manufactured data could also be an issue with the MDS, when supporting documentation is absent or limited. Without adequate documentation, it is unclear whether the nursing home staff sufficiently observed the resident to determine his or her care needs or merely checked off a box on the assessment form. We continue to believe, as do most of the states with separate MDS review programs, that requiring documentation for the full MDS assessment is necessary to ensure the accuracy of MDS data. In our view, however, this documentation need not be duplicative of that which is already in the medical record but rather demonstrative of the basis for the higher-level summary judgments about a resident’s condition.  Some states have already developed tools to accomplish this and in commenting on a draft of this report, two states said that CMS should establish documentation requirements for responses on the MDS. In addition, the discrepancies cited by the HHS OIG in its studies stemmed from inconsistencies between MDS assessments and documentation in residents’ medical records. The OIG acknowledged that the results of its analyses were limited by the information available in the medical record— for example, when a facility MDS assessment was based on resident observation, the facility may not have documented these observations in the medical record. The importance of adequate documentation is further reinforced by the fact that using interviews and observation to validate MDS assessments may often not be possible, particularly for residents who have been discharged from the nursing home before an MDS accuracy review. Given the importance of MDS data in adjusting nursing home payments and guiding resident care, documenting the basis for the MDS assessment—in a way that can be independently validated—is critical to achieving its intended purposes.
CMS provided additional clarifying information that we incorporated as appropriate.  In addition, the states that commented on the draft report generally concurred with our findings and provided technical comments that we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of this report earlier, we will not distribute it until 30 days after its date. At that time, we will send copies to the administrator of CMS; appropriate congressional committees; and other interested parties. We will also make copies available to others upon request.
If you or your staff have any questions, please call me at (202) 512-7114 or Walter Ochinko at (202) 512-7157. Major contributors to this report include Carol Carter, Laura Sutton Elsberg, Leslie Gordon, and Sandra Gove.
section_title: Appendix I: Summary of State On-Site MDS Reviews As of January 2001
subsections: section_title: Year state began: • MDS-based payment system • MDS reviews 2000 (payment) 2000 (reviews)
paragraphs: Average time lapse between facility MDS and state review 90 days 1998 (payment) 1998 (reviews)
State reviews most recent MDS assessment 1993 (payment) 1994 (reviews)
Minimum of 10 assessments per facility 1988 (payment) 1992 (reviews)
At least 20 percent of residents in facility 1993 (payment) 1994 (reviews)
Year state began:
MDS-based   payment system
MDS reviews 1996 (payment) 1994 (reviews)
Frequency of on-site reviews (all facilities unless otherwise noted)
Average time lapse between facility MDS and state review 6-12 months 1993 (payment) 1993 (reviews)
At least 25 percent of residents in facility 1992 (payment) 1992 (reviews)
MDS never older than 90 days 1998 (payment) 1998 (reviews)
Annually (Staff also conduct quarterly quality review audits)
Yes (effective 10/1/01)
The types of MDS errors that commonly reoccur relate to misapplication of MDS definitions, and may in large part be due to facility staff turnover. In commenting on a draft of this report, officials told us that these errors are consistent with those found in other states with MDS-based payment systems.
State plans to publish the results of MDS accuracy reviews on a Web page to prevent simple but recurring errors Virginia is not included because of the newness of its MDS review program (began operating in April 2001). We have included the nine other states with longer standing on-site review programs.
This column reflects the frequency of initial reviews for each facility. Some states conduct follow-up reviews more frequently for facilities where problems have been identified.
We asked states to select from the following categories: more important, equally important, and less important.
Indiana officials added the following language to characterize MDS errors: An error occurs when the audit findings are different from the facility’s transmitted MDS data and those differences result in a different case-mix category.
section_title: Appendix II: Comments from the Centers for Medicare and Medicaid Services