Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National Implementation Is Premature
Released: 2002-11-13T00:00:00

Letter

Background

Since 1998, the results from state surveys of nursing homes have been the principal source of public information on nursing home quality, which is posted and routinely updated on CMS’s Nursing Home Compare Web site. Under contract with CMS, states are required to conduct periodic surveys that focus on determining whether care and services meet the assessed needs of the residents and whether homes are in compliance with federal quality requirements, such as preventing avoidable pressure sores, weight loss, or accidents. During a survey, a team that includes registered nurses spends several days at a home reviewing the quality of care provided to a sample of residents. States are also required to investigate complaints lodged against nursing homes by residents, families, and others. In contrast to surveys, complaint investigations generally target a single area in response to a complaint filed against a home. Any deficiencies identified during routine surveys or complaint investigations are classified according to the number of residents potentially or actually affected (isolated, pattern, or widespread) and their severity (potential for minimal harm, potential for more than minimal harm, actual harm, and immediate jeopardy).

To improve the rigor of the survey process, HCFA contracted for the development of quality indicators and required their use by state surveyors beginning in 1999. Quality indicators are derived from data collected during nursing homes’ assessments of residents, known as the minimum data set (MDS). The MDS contains individual assessment items covering 17 areas, such as mood and behavior, physical functioning, and skin conditions. MDS assessments of each resident are conducted in the first 14 days after admission and periodically thereafter and are used to develop a resident’s plan of care. Facility-reported MDS data are used by state surveyors to help identify quality problems at nursing homes and by CMS to determine the level of nursing home payments for Medicare; some states also use MDS data to calculate Medicaid nursing home payments.

Because it also envisioned using indicators to communicate nursing home quality to consumers, HCFA recognized that any publicly reported indicators must pass a very rigorous standard for validity and reliability. Valid quality indicators that distinguish between good and poor care provided by nursing homes would be a useful adjunct to existing quality data. Such indicators must also be reliable—that is, they must consistently distinguish between good and bad care. HCFA contracted with Abt to review existing quality indicators and determine if they were suitable for public reporting. Abt catalogued and evaluated 143 existing quality indicators, including those used by state surveyors. It also identified the need for additional indicators both for individuals with chronic conditions who are long-term residents of a facility and for individuals who enter a nursing home for a short period, such as after a hospitalization (a postacute stay). According to Abt, a main concern about publicly reporting quality indicators was that the quality indicator scores might be influenced by other factors, such as residents’ health status. Abt concluded that the specification of appropriate risk adjustment models was a key requirement for the validity of any quality indicators. Risk adjustment is important because it provides consumers with an “apples-to-apples” comparison of nursing homes by taking into consideration the characteristics of individual residents and adjusting quality indicator scores accordingly. For example, a home with a disproportionate number of residents who are bedfast or who present a challenge for maintaining an adequate level of nutrition—factors that contribute to the development of pressures sores—may have a higher pressure sore score. Adjusting a home’s quality indicator score to fairly represent to what extent a home does—or does not—admit such residents is important for consumers who may wish to compare one home to another. After several years of work, Abt recommended 39 risk-adjusted quality indicators to CMS in October 2001. Twenty-two were based on existing indicators and the remaining 17 were newly developed by Abt, including 9 indicators for nursing home residents with chronic conditions and 8 indicators for individuals who enter a nursing home for a short period.

In September 2001, CMS contracted with the NQF to review Abt’s work with the objective of (1) recommending a set of quality indicators for use in its planned six-state pilot and (2) developing a core set of indicators for national implementation of the initiative scheduled for late 2002. NQF established a steering committee to accomplish these two tasks. The steering committee met in November 2001 and identified 11 indicators for use in the pilot, 9 of which were selected by CMS. The committee made its selection from among Abt’s list of 39 indicators but it did not recommend use of Abt’s risk-adjustment approach. Moreover, the steering committee indicated that it would not be limited to the same Abt list in developing its recommended core set of indicators for national implementation. In April 2002, NQF released a draft consensus report identifying the indicators it had distributed to its members and the public for comment on their potential inclusion in the national implementation. Under its contract, NQF was scheduled to make a final recommendation to CMS prior to the national reporting of quality indicators.

Appropriateness of Quality Indicators Proposed for Public Reporting Is Unresolved

CMS’s initiative to augment existing public data on nursing home quality has considerable merit but more time is needed to assure that the indicators proposed by CMS for public reporting are appropriate in terms of their validity and reliability. Based on work by Abt to validate the indicators it developed for CMS, the agency selected quality indicators for national reporting. The full Abt validation report—which is important for a thorough analysis of the appropriateness of the quality indicators--was still not available to us as of October 28, 2002. Our review of available portions of the Abt report, however, raised serious questions about whether testing and validation of the selected indicators has been sufficient to move forward with national reporting at this time. Moreover, CMS plans to initiate national reporting before it receives recommendations from NQF, its contractor, on appropriate quality indicators.

On August 9, 2002, CMS announced the 10 indicators selected for its nationwide reporting of quality indicators, which it plans to launch in mid- November 2002. CMS selected these indicators from those that Abt had validated in its August 2, 2002, validation report. Abt classified the indicators it studied as to the degree of validity—top, middle, or not valid. The indicators that CMS selected were in the top category with one exception—residents in physical restraints—which was in the middle category. The objective of Abt’s validation study was to confirm that the indicators reflect the actual quality of care that individual nursing facilities provide, after taking into account resident and facility-level characteristics. For example, a validation analysis could confirm that a low percentage of pressure sores among residents was linked to a facility’s use of procedures to prevent their development. Successful validation reduces the chance that publicly reported data could misrepresent a high-quality facility as a low-quality facility—or vice versa.

CMS’s decision to implement national reporting in November 2002 is troubling, given the issues raised by our review of the available portions of Abt’s validation report. Although we asked CMS for a copy of Abt’s 11 technical appendixes, as of October 28, 2002, they were still undergoing review and were not available to us. The technical appendixes are essential to adequately understand and evaluate Abt’s validation approach. Our review of the available portions of the Abt report raised serious questions about whether the effort to date has been sufficient to validate the indicators. The validation study is based on a sample that is drawn from six states; it is not representative of nursing homes nationwide and may not be representative of facilities in these six states. Selected facilities were allowed to decline participation and about 50 percent did so. For those facilities in the validation study, Abt deemed most of the indicators as valid—that is, better care processes were associated with higher quality indicator scores, taking into account resident and facility-level characteristics. However, we could not evaluate these findings because Abt provided little information on the specific care processes against which the indicators were validated. Unresolved questions also exist about the risk adjustment of the quality indicators. Risk adjustment is a particularly important element in determining certain quality indicators because it may change the ranking of individual facilities—a facility that is among the highest on a particular quality indicator without risk adjustment may fall to the middle or below after risk adjustment—and vice versa. Data released by CMS in March 2002 demonstrated that Abt’s risk adjustment approaches could either lower or raise facility scores by 40 percent or more. Although such changes in ranking may be appropriate, Abt did not provide detailed information on how its risk adjustment approaches changed facility rankings or a basis for assessing the appropriateness of the changes.

In addition to the questions raised by our review of the Abt validation report, CMS is not planning to wait for the expert advice it sought on quality indicators through its contract with the NQF. Under this contract, the NQF steering committee issued a consensus draft in April 2002 with a set of potential indicators for public reporting. The steering committee had planned to complete its review of these indicators using its consensus process by August 2002. In late June, however, CMS asked NQF to delay finalizing its recommendations until early 2003 to allow (1) consideration of Abt’s August 2002 report on the validity of its indicators and risk- adjustment methods—including the technical appendices, when they become available and (2) a review of the pilot evaluation results expected in October 2002. An NQF official told us that the organization agreed to the delay because the proposed rapid implementation timeline had been a concern since the initiative’s inception. CMS’s list of quality indicators for the November 2002 national rollout did not include six indicators under consideration by NQF—depression, incontinence, catheterization, bedfast residents, weight loss, and rehospitalization (see app. I). Instead, CMS intends to consider NQF’s recommendations and revise the indicators used in the mid-November national rollout sometime next year.

CMS is also moving forward without a consensus on risk adjustment of quality indicators. CMS is planning to report one indicator with facility- level adjustment based on a profile of residents’ status at admission, and two indicators both with and without this Abt-developed risk adjuster. However, both Abt and NQF have concluded that adjusting for the type of residents admitted to the nursing home required further research to determine its validity. We believe that reporting the same indicator with and without facility-level risk adjustment could serve to confuse rather than help consumers. Two of the three consultants hired by NQF specifically recommended against the use of facility-level adjustments in public reporting at this time. We also found that, as of October 1, 2002, CMS had not reached internal consensus on how to describe the risk- adjustment methods used in each of the 10 indicators it plans to begin reporting nationally in November 2002. Several agency officials agreed with our assessment that the descriptions on its Web site were inconsistent with Abt’s own descriptions of the risk adjustment associated with each indicator.

CMS Has Not Addressed Concerns About the Underlying Accuracy of MDS Data Used to Develop Quality Indicators

Two different Abt studies have presented CMS with conflicting messages about the accuracy of MDS data. Abt’s August 2002 quality indicator validation report suggested that the underlying data used to calculate most indicators were, in the aggregate, very reliable. However, our analysis of more detailed facility-level data in a February 2001 Abt report raised questions about the reliability of some of the same MDS data. Because MDS data are used by CMS and some states to determine the level of nursing home payments for Medicare and Medicaid and to calculate quality indicators, ensuring its accuracy at the facility level is critical both for determining appropriate payments and for public reporting of the quality indicators. Recognizing the importance of accurate MDS data, CMS is in the process of implementing a national MDS accuracy review program expected to become fully operational in 2003, after the nationwide reporting of quality indicators begins in November 2002. We recently reported that CMS’s review program is too limited in scope to provide adequate confidence in the accuracy of MDS assessments in the vast bulk of nursing homes nationwide.

Abt’s August 2, 2002, validation report concluded that the reliability of the underlying MDS data used to calculate 39 quality indicators ranged from acceptable to superior, with the data for only 1 indicator proving unacceptable. Abt’s findings were based on a comparison of assessments conducted by its own nurses to assessments performed by the nursing home staff in 209 sample facilities. For each quality indicator, Abt reported the overall reliability for all of the facilities in its sample. However, because quality indicators will be reported for each nursing home, overall reliability is not a sufficient assurance that the underlying MDS data are reliable for each nursing home. Although Abt did not provide information on MDS reliability for individual facilities, it noted that reliability varied considerably within and across states.

Earlier work by Abt and others calls into question the reliability of MDS data. Abt’s February 2001 report on MDS data accuracy identified significant variation in the rate of MDS errors across the 30 facilities sampled. Differences between assessments conducted by Abt’s nurses and the nursing home staff were classified as errors by Abt. Error rates for all MDS items averaged 11.7 percent but varied across facilities by a factor of almost two—from 7.8 percent to 14.5 percent. As shown in figure 1, the majority of error rates were higher than 10.5 percent. Furthermore, error rates for some of the individual MDS items used to calculate the quality indicators were much higher than the average error rate. According to Abt, the least accurate sections of the MDS included physical functioning and skin conditions. Abt also noted that there was a tendency for facilities to underreport residents with pain. MDS items from these portions of the assessment are used to calculate several quality indicators that CMS plans to report nationally in November 2002—activities of daily living, pressure sores, and pain management. Table 1 shows that the error rate across the residents sampled ranged from 18 percent for pressure sores to 42 percent for pain intensity. Abt’s February 2001 findings were consistent with areas that states have identified as having a high potential for error, including activities of daily living and skin conditions. Moreover, a study by the HHS Office of Inspector General (OIG), which identified differences between the MDS assessment and the medical record, found that activities of daily living was among the areas that provided the greatest source of differences. In addition, the OIG report noted that 40 percent of the nursing home MDS coordinators it surveyed identified the physical functioning section, used to calculate the quality indicator on activities of daily living, as the most difficult to complete. Some coordinators explained that facility staff view a resident’s capabilities differently and thus the assessments tend to be subjective.

As part of CMS’s efforts to improve MDS accuracy, its contractor is still field-testing the on-site aspect of its approach, which is not expected to be implemented until 2003. Although Abt’s February 2001 report found widespread MDS errors, CMS intends to review roughly 1 percent of the MDS assessments prepared over the course of a year, which numbered 14.7 million in 2001. Moreover, only 10 percent of the reviews will be conducted on-site at nursing homes. In contrast, our prior work on MDS found that 9 of the 10 states with MDS-based Medicaid payment systems that examine MDS data’s accuracy conduct periodic on-site reviews in all or a significant portion of their nursing homes, generally examining from 10 to 40 percent of assessments. On-site reviews heighten facility staff awareness of the importance of MDS data and can lead to the correction of practices that contribute to MDS errors. We reported earlier that CMS’s approach may yield some broad sense of the accuracy of MDS assessments on an aggregate level but is insufficient to provide confidence about the accuracy of MDS assessments in the vast bulk of nursing homes nationwide.

The Public May Be Confused by Quality Data and CMS Is Not Prepared to Respond to Consumers’ Questions

While CMS is strongly committed to making more information available to the public on nursing home quality and such an initiative has considerable merit, the agency had not demonstrated a readiness to assist the public in understanding and using those data. We found that CMS’s reporting of quality indicators in the six pilot states was neither consumer friendly nor reported in a format consistent with the data’s limitations, implying a greater degree of precision than is currently warranted. Our analysis of the data currently available in the six pilot states demonstrated the potential for public confusion over both the quality indicators themselves and inconsistencies with other available data on deficiencies identified during nursing home surveys—which, to date, are the primary source of public data on nursing home quality. Moreover, our phone calls to the Medicare and QIO toll-free numbers revealed that CMS was not adequately prepared to address consumers’ questions raised by discrepancies between conflicting sources of quality data.

Our review of the quality indicators on the CMS Web site found that the presentation of the data was not consumer friendly and that the reporting format implies a greater confidence in the data’s precision than may be warranted at this time. Quality indicators are reported as the percentage of residents in a facility having the particular characteristics measured by each indicator. The Web site explains that having a low percentage of residents with pressure sores or pain is better than having a high percentage. In the six-state pilot, the public can compare a nursing home’s score to the statewide and overall average for each quality indicator. We believe that equating a high score with poor performance is counterintuitive and could prove confusing to consumers. Despite the Web site’s explanation of how to interpret the scores, the public might well assume that a high score is a positive sign.

In addition, reporting actual quality indicator scores rather than the range of scores a home falls into for an indicator—a low, medium, or high score— can be confusing and implies a confidence in the precision of the results that is currently a goal rather than a reality. Consumers will find it difficult to assess a home with a score that is 5 to 10 percentage points from the state average. Such a home could be an outlier—one of the best or the worst on that indicator; alternatively, it could be that the home was close to the state average because the outliers involved much larger differences. Concerns about the validity of the indicators and the potential reliability of the data make comparisons of homes with similar scores questionable. Consumers may be misled if a difference of several percentage points between two homes is perceived as demonstrating that one is better or worse than the other. To partially address these types of concerns, Maryland has reported quality indicator data on its own Web site since August 2001 in ranges rather than individual values. Thus, it indicates if a facility falls into the bottom 10 percent, the middle 70 percent, or the top 20 percent of facilities in the state.

Consumers may also be confused about how to interpret missing information. Although the CMS Web site explains that quality indicator scores are not reported for nursing homes with too few residents, it does not acknowledge the extent of such missing data. We found that 6 percent of all nursing homes in the six pilot states have no score for any of the nine quality indicators and that, for individual indicators, from 9 percent to 40 percent of facilities have missing scores (see table 2). When data for homes of potential interest to consumers are not reported, consumers may need some assistance in how to incorporate such instances into their decisionmaking.

Consumer confusion may also occur when quality indicator scores send conflicting messages about the overall quality of care at a home. We found that the Web site data for a significant number of facilities contained such inconsistencies. Seventeen percent of nursing homes in the six pilot states had an equal number of highly positive and highly negative quality indicator scores. We defined highly positive scores as those indicating that a facility was among the 25 percent of homes with the lowest percentage of residents exhibiting poor outcomes, such as a decline in their ability to walk or use the toilet. In contrast, facilities with a highly negative score were among the top 25 percent of homes with poor outcomes. We also found that 37 percent of nursing homes with four or more highly positive quality indicator scores had two or more highly negative scores.

In addition, our comparison of survey deficiency data available on the Web site with quality indicator scores also revealed inconsistencies. For example, 17 percent of nursing homes with four or more highly positive quality indicator scores and no highly negative scores—seemingly “good” nursing homes—had at least one serious quality-of-care deficiency on a recent state survey. We have found that serious deficiencies cited by state nursing home surveyors were generally warranted and indeed reflected instances of documented actual harm to nursing home residents. Moreover, 73 percent of nursing homes with four or more highly negative quality indicator scores—seemingly “bad” facilities—had no serious quality-of-care deficiencies on a recent survey (see table 3). The latter situation is consistent with our past work that surveyors often miss serious quality-of-care problems. Nevertheless, consumers will generally lack such insights on the reliability of state surveys that would permit them to better assess the available data on quality of care.

With the apparent need for assistance to consumers in interpreting and using this information, the important role of the Medicare and QIO toll-free numbers is evident. We requested and reviewed copies of the Medicare hotline and QIO scripts and found that they did not address the issue of responding to questions about conflicting or confusing quality data. Furthermore, our calls to the Medicare hotline and to QIO toll-free numbers in the six pilot states demonstrated that the staff were not adequately prepared to handle basic questions about the quality data available under the pilot. CMS officials had told us that Medicare hotline callers with complicated questions would be seamlessly transferred to a QIO without having to hang up and call another number. Although we asked the Medicare hotline staff if another organization might be better able to respond to our questions, no one offered to refer us to QIOs, even when we specifically asked about them. In fact, one hotline staff member told us that a QIO would not be an appropriate referral. Consequently, we independently attempted to call the QIOs in the six pilot states. We found that it was difficult to reach a QIO staff member qualified to answer questions. Each QIO had a toll-free number but neither the automated recordings at four QIOs nor operators at the remaining two indicated that the caller had reached a QIO. In addition, the automated recordings did not contain a menu choice for questions about nursing home quality indicators. We were unable to contact one QIO because the hotline had neither an operator nor a voice mail capability. On other calls, after reaching a QIO staff person, it frequently took several referrals to identify an appropriate contact point. One QIO took 5 working days for a staff member to call us back. Four of the five QIOs we contacted explained that their primary role was to work with nursing homes to improve quality of care. In general, QIO staff were not prepared to respond to consumer questions.

Staff at the Medicare hotline and the QIOs varied greatly in their basic understanding of quality indicators and survey deficiencies. While two of the nine staff we contacted were generally knowledgeable about different types of quality data, others were unable to answer simple questions and the majority provided erroneous or misleading data. One QIO staff member told us that MDS data were not representative of all residents of a nursing home but only presented a “little picture” based on a few residents. However, assessments of all residents are taken into consideration in calculating quality indicators. When we expressed concern about a home identified on the Web site with a “level-3” deficiency, a Medicare hotline staff member incorrectly told us that it was not a serious deficiency because level 3 indicated potential harm. CMS designates actual harm deficiencies as “level-3” deficiencies. A QIO staff member incorrectly told us that actual harm pressure sore deficiencies had nothing to do with patient care and might be related to paperwork. Our review of survey reports has shown that actual harm deficiencies generally involved serious quality-of-care problems resulting in resident harm. Generally, hotline staff did not express a preference for using either nursing home surveys or quality indicators in choosing a nursing home. Two QIO staff, however, stated that the nursing home survey information gave a better picture of nursing home care than the quality indicators, which they judged to be imprecise and subject to variability.

Pilot Evaluation Is Limited and Will Not Be Completed Prior to National Reporting of Quality Indicators

CMS’s evaluation of the pilot is limited and will not be completed prior to national reporting of quality indicators because of the short period of time between the launch of the pilot and the planned November 2002 national implementation. According to CMS officials, the pilot evaluation was never intended to help decide whether the initiative should be implemented nationally or to measure the impact on nursing home quality. While CMS is interested in whether nursing home quality actually improves as a result of the initiative, it will be some time before such a determination can be made. Thus, CMS focused the pilot evaluation on identifying improvements that could be incorporated into the initiative’s design prior to the scheduled national implementation in November 2002. A CMS official told us that initial pilot evaluation results were expected by early October 2002, allowing just over a month to incorporate any lessons learned. In commenting on a draft of this report, CMS stated that it was using preliminary findings to steer national implementation. The final results of the pilot evaluation will not be completed until sometime in 2003.

CMS’s evaluation of the pilot is focused on identifying how to communicate more effectively with consumers about the initiative and how to improve QIO interaction with nursing homes. Specifically, CMS will assess whether (1) the target audiences were reached; (2) the initiative increased consumer use of nursing home quality information; (3) consumers used the new information to choose a nursing home; (4) QIO activities influenced nursing home quality improvement activities; (5) nursing homes found the assistance provided by QIOs useful; and (6) the initiative influenced those who might assist consumers in selecting a nursing home, such as hospital discharge planners and physicians. Information is being collected by conducting consumer focus groups, tracking Web site “hits” and toll-free telephone inquiries, administering a Web site satisfaction survey, and surveying nursing homes, hospital discharge planners, and physicians. As of late August 2002, CMS teams were also in the process of completing site visits to stakeholders in the six pilot states, including QIOs, nursing homes, ombudsmen, survey agencies, nursing home industry representatives, and consumer advocacy groups. The teams’ objective is to obtain a first-hand perspective of how the initiative is working with the goal of implementing necessary changes and better supporting the program in the future.

Conclusions

Although CMS’s initiative to publicly report nursing home quality indicators is a commendable and worthwhile goal, we believe that it is important for CMS to wait for and consider input from NQF and make necessary adjustments to the initiative based on its input. We believe several factors demonstrate that CMS’s planned national reporting of quality indicators in November 2002 is premature. Our review of the available portions of Abt’s validation report raised serious questions about whether the effort to date has been sufficient to validate the quality indicators. NQF was asked to delay recommending a set of indicators for national reporting until 2003, in part, to provide sufficient time for it to review Abt’s report. Although limited in scope, CMS’s planned MDS accuracy review program will not begin on-site accuracy reviews of the data underlying quality indicators until 2003. CMS’s own evaluation of the pilot, designed to help refine the initiative, was limited to fit CMS’s timetable for the initiative and the preliminary finding were not available until October 2002, leaving little time to incorporate the results into the planned national rollout. Other aspects of the evaluation will not be available until early 2003. We also have serious concerns about the potential for public confusion over quality data, highlighting the need for clear descriptions of the data’s limitations and easy access to informed experts at both the Medicare and QIO hotlines. CMS has not yet demonstrated its readiness to meet these consumer needs either directly or through the QIOs.

Recommendations for Executive Action

To ensure that publicly reported quality indicator data accurately reflect the status of quality in nursing homes and fairly compare homes to one another, we recommend that the Administrator of CMS delay the implementation of nationwide reporting of quality indicators until there is greater assurance that the quality indicators are appropriate for public reporting—including the validity of the indicators selected and the use of an appropriate risk-adjustment methodology—based on input from the NQF and other experts and, if necessary, additional analysis and testing; and a more thorough evaluation of the pilot is completed to help improve the initiative’s effectiveness, including an assessment of the presentation of information on the Web site and the resources needed to assist consumers’ use of the information.

Comments from CMS and the NQF and Our Evaluation

CMS and the NQF reviewed and provided comments on a draft of this report. (See app. II and app. III, respectively). CMS reiterated its commitment to continually improve the quality indicators and to work to resolve the issues discussed in our report. Although CMS stated it would use our report to help improve the initiative over time, it intends to move forward with national implementation in November 2002 as planned. It stated that “waiting for more reliability, more validity, more accuracy, and more usefulness will delay needed public accountability, and deprive consumers, clinicians, and providers of important information they can use now.” The NQF commented that it unequivocally supports CMS’s plans to publicly report quality indicators but indicated that the initiative would benefit from a short-term postponement of 3 to 4 months to achieve a consensus on a set of indicators and to provide additional time to prepare the public on how to use and interpret the data. We continue to support the concept of reporting quality indicators, but remain concerned that a flawed implementation could seriously undercut support for and the potential effectiveness of this very worthwhile initiative. CMS’s comments and our evaluation focused largely on two issues: (1) the selection and validity of quality indicators, and (2) lessons learned from CMS’s evaluation of the pilot initiative.

Selection and Validity of Quality Indicators

CMS asserts that the quality indicators it plans to report nationally are reliable, valid, accurate, and useful and that it has received input from a number of sources in selecting the indicators for this initiative. However, CMS provided no new evidence addressing our findings regarding the appropriateness of the quality indicators selected for public reporting and the accuracy of the underlying data. We continue to believe that, prior to nationwide implementation, CMS should resolve these open issues.

CMS intends to move forward with nationwide implementation without a requested NQF assessment of the full Abt validation report and without NQF’s final recommendations on quality indicators. CMS would not share the technical appendices to Abt’s validation report with us because they were undergoing review and revision. The technical appendices are critical to assessing Abt’s validation approach. CMS’s comments did not address our specific findings on the available portions of Abt’s validation report, including: (1) the validation results are not representative of nursing homes nationwide because of limitations in the selection of a sample of nursing homes to participate in the validation study, and (2) Abt provided little information on the specific care processes against which the indicators were validated or how its risk adjustment approaches changed facility rankings and the appropriateness of the changes. Although both Abt and the NQF concluded that Abt’s facility-level risk adjustment approach required further research to determine its validity, CMS plans to report two indicators with and without facility-level adjustments. CMS’s comments indicated that it has chosen to report these measures both ways in order to evaluate their usefulness and to allow facilities and consumers the additional information. We continue to believe that reporting data of uncertain validity is inappropriate and, as such, will likely not be useful to either facilities or consumers.

For quality indicators to be reliable, the underlying MDS data used to calculate the indicators must be accurate. CMS’s comments did not specifically address the conflicting findings on MDS accuracy from Abt’s August 2002 validation report and its February 2001 report to CMS. Abt’s August 2002 validation report concluded that, in aggregate, the underlying MDS data were very reliable but that the reliability varied considerably within and across states. Aggregate reliability, however, is insufficient because quality indicators are reported separately for each facility. In its February 2001 report to CMS, Abt identified widespread errors in the accuracy of facility-specific assessments used to calculate some of the quality indicators that CMS has selected for reporting in November. CMS indicated that its efforts since 1999 have improved MDS accuracy. But because CMS does not plan to begin limited on-site MDS accuracy reviews until 2003, there is little evidence to support this assertion.

Lessons Learned from CMS’s Evaluation of the Pilot Initiative

CMS commented that findings from a number of activities evaluating the six-state pilot were not available prior to the time we asked for comments on our draft report. While final reports are not yet available for some of these studies, CMS stated that the pilot allowed it to work through important issues and incorporate lessons learned before a national launch. We pointed out that the pilot evaluation was limited and incomplete—an additional reason to delay the initiative. CMS also did not evaluate a key implementation issue—the adequacy of assistance available to consumers through its toll-free telephone hotlines. Moreover, the lack of formal evaluation reports to help guide the development of a consensus about key issues, such as how quality indicators should be reported, is troubling.

In its comments, CMS stated that it was committed to working aggressively to help the public understand nursing home quality information using lessons learned from the pilot. However, CMS learned about the flaws in its hotline operations not from its pilot evaluation but from our attempts to use the Medicare and QIO toll-free phone numbers to obtain information on quality data. Acknowledging the weaknesses we identified, the agency laid out a series of actions intended to strengthen the hotlines’ ability to respond to public inquiries, such as providing additional training to customer service representatives prior to the national launch of the initiative. CMS outlined other steps it plans to take such as providing its customer service representatives with new scripts and questions and answers to the most frequently asked questions. At the outset of the pilot in April 2002, CMS described seamless transfers from the Medicare to the QIO hotlines for complicated consumer questions but now acknowledges that limitations in QIO telephone technology prevent such transfers. Instead of automatic transfers, CMS stated that, when referrals to QIOs are necessary, callers will be provided with a direct toll-free phone number. CMS also commented that consumers should be encouraged to consider multiple types of information on nursing home quality. While we agree, we believe it is critical that customer service representatives have a clear understanding of the strengths and limitations of different types of data to properly inform consumers when they inquire.

CMS commented that we offered no explanation of the analysis that led us to conclude that (1) consumers could be confused because scores on quality indicators can conflict with each other and the results of routine nursing home surveys, and (2) the public may confuse a high quality indicator score with a positive result. Our draft clearly states that our findings were based on our analysis of the quality indicator data and survey results available in the six pilot states—a database that CMS provided at our request. In its comments, CMS provided limited data to support its assertion that consumers are not confused by the quality indicators and are very satisfied with the current presentation on its Web site. According to CMS, over two-thirds of respondents to its August 2002 online satisfaction survey of randomly chosen users of Nursing Home Compare information said they were highly satisfied with the information, for example, it was clearly displayed, easy to understand, and valuable. It is not clear, however, that these responses were representative of all nursing home consumers accessing the Web site, as CMS implied. For example, CMS informed us that this survey was part of a larger survey of all Medicare Web site users, which had a low overall response rate of 29 percent. Moreover, of the 654 respondents to the Nursing Home Compare component of the survey, fewer than half (40 percent) were identified as Medicare beneficiaries, family members, or friends. NQF feedback to CMS on its Web site presentation was consistent with our findings. In commenting on our draft report, NQF noted that it had offered informal guidance to CMS, such as using positive or neutral wording to describe indicators, exploring alternative ways of presenting information about differences among facilities, and ensuring that the presentation of the data reflects meaningful differences in topics important to consumers. While justifying its current presentation of quality indicator data, CMS commented that it is seriously considering not reporting individual nursing home scores but rather grouping homes into ranges such as the bottom 10 percent, middle 70 percent, and top 20 percent of facilities in a state. Such a change, however, would not come before the national rollout. We agree with CMS that, when grouping homes into ranges, homes on the margin—close to the bottom 10 percent or top 20 percent—may not be significantly different from one another. However, the same is true of reporting individual facility scores. Moreover, reporting ranges more clearly identifies homes that are outliers for consumers.

Additional CMS Comments

CMS also commented on our characterization of the scope of the nursing home quality initiative. CMS stated that we had narrowly framed the initiative as one designed solely for consumers, ignoring the QIO’s quality improvement activities with individual nursing homes requesting assistance. Our report acknowledged and briefly outlined the quality improvement role of the QIOs. However, based on our requestors’ concerns about the relatively short pilot timeframe prior to national implementation of public reporting of quality indicators, we focused our work on that key aspect of the initiative. CMS cited its Interim Report on Evaluation Activities for the Nursing Home Quality Initiative to support its conclusion that the initiative was successful in promoting quality improvement activities among nursing homes. The improvements cited in the Interim Report were self-reported by facilities and CMS offered no insights on the nature of the quality improvement changes. The Interim Report was not available when we sent our draft report to CMS for comment.

CMS provided several technical comments which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the Administrator of CMS, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions, please call me at (202) 512-7118 or Walter Ochinko at (202) 512-7157. GAO staff acknowledgments are listed in appendix IV.

Comparison of Quality Indicators Proposed by NQF and CMS for National Rollout

Comments from the Centers for Medicare & Medicaid Services

Comments from the National Quality Forum

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

The following staff made important contributions to this report: Laura Sutton Elsberg, Patricia A. Jones, Dean Mohs, Dae Park, Jonathan Ratner, Peter Schmidt, Paul M. Thomas, and Phyllis Thorburn.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs