Nursing Homes: Consumers Could Benefit from Improvements to the Nursing Home Compare Website and Five-Star Quality Rating System
Released: 2016-12-06T00:00:00

Letter

Background

CMS’s goals for Nursing Home Compare and its Five-Star System are consistent with its strategy to improve the quality of health care by providing transparent information about the quality of health care services, including those delivered in nursing homes. According to the strategy, to achieve better care, patients must be given access to understandable information and decision support tools that help them manage their health and navigate the health care delivery system.

Nursing Home Compare

Since 1998, CMS has publicly reported information on nursing home quality on its Nursing Home Compare website and has increased the amount of information reported on the website over time. CMS initially reported information only about nursing home characteristics and nursing home health inspection results on Nursing Home Compare. Later, CMS began reporting additional information on the website, such as the ratio of nursing staff to residents, nursing homes’ performance on various quality measures, and the number of complaints registered against nursing homes. Additionally, CMS has updated the appearance and functionality of the Nursing Home Compare website over time, with the most significant change being the introduction of the Five-Star System in 2008.

Five-Star System

In December 2008, CMS made the Five-Star System available to the public on its Nursing Home Compare website in order to help consumers compare nursing homes more easily. The Five-Star System assigns each nursing home participating in the Medicare or Medicaid programs an overall “star” rating, ranging from one to five. Nursing homes with five stars are considered to have much above average quality, while nursing homes receiving one star are considered to have much below average quality. Calculation of the overall star rating is based on separate ratings that nursing homes receive for each of three components: health inspections, staffing, and quality measures.

Health inspection rating. CMS contracts with state survey agencies to conduct unannounced, on-site nursing home health inspections— known as standard surveys—to determine whether nursing homes meet federal quality standards. Every nursing home receiving Medicare or Medicaid payment must undergo a standard survey not less than once every 15 months, and the statewide average interval for these surveys must not exceed 12 months. State surveyors also conduct complaint investigations in response to allegations of quality problems. If nursing homes are found to be out of compliance with any requirements, state surveyors issue deficiency citations that reflect the scope (number of residents affected) and severity (level of harm to residents) of the deficiency. Surveyors conduct revisits to the nursing home to ensure that the deficiencies identified have been corrected. A nursing home’s health inspection rating is relative to other nursing homes’ health inspection ratings in their state. As such, health inspection ratings are assigned to generally achieve the following distribution within each state: the top 10 percent of nursing homes receive five stars, the bottom 20 percent receive one star, and the middle 70 percent of nursing homes receive two, three, or four stars.

Staffing rating. Nursing homes self-report staffing hours worked for a 2-week period at the time of the standard survey. CMS converts the reported point-in-time staffing hours for nursing staff—registered nurses, licensed practical nurses, and certified nursing assistants— into measures that indicate the number of registered nurse and total nursing hours per resident per day. CMS adjusts the staffing levels for differences in the level of complexity of nursing services required to care for residents across nursing homes—referred to as resident acuity. Each nursing home’s staffing rating is assigned based on how its total nursing and registered nurse staffing levels compare to the distribution of staffing levels for freestanding homes in the nation and staffing level thresholds identified by CMS.

Quality measure rating. Nursing homes regularly collect assessment information on all their residents, including information on the residents’ health, physical functioning, mental status, and general well-being. Nursing homes self-report this information to CMS. CMS uses some of the assessment information to measure the quality of certain aspects of nursing home care, such as the prevalence of pressure sores and changes in residents’ mobility. At the time of our analysis, CMS calculated this rating for each nursing home based on 11 of the 18 quality measures posted on Nursing Home Compare. Information on the remaining 7 quality measures is posted on the website but not used in the calculation of the rating. A nursing home’s quality measure rating is assigned based on national thresholds established by CMS.

The overall star rating is calculated using a process that combines the star ratings from the health inspection, staffing, and quality measure components—with the greatest weight given to the health inspection rating. The overall rating is assigned based on the following steps: 1. Start with the number of stars for the health inspection rating. 2. Add one star if the staffing rating is four or five stars and also greater than the health inspection rating. Subtract one star if the staffing rating is one star. The overall rating cannot go above five stars or below one star. 3. Add one star if the quality measure rating is five stars. Subtract one star if the quality measure rating is one star. The rating cannot go above five stars or below one star.

See figure 1 for an example of how a nursing home’s overall rating is calculated. CMS updates the ratings on a monthly basis; however, a particular home’s overall rating will only change if it had new data that affected any one of the component ratings. For example, when a home has a health inspection survey, either a standard survey or a complaint investigation, the deficiency data from the survey will become a part of the calculation for the health inspection rating and the overall rating will also be adjusted, if necessary.

CMS Collects Information on the Use of the Nursing Home Compare Website, but Lacks a Systematic Process for Prioritizing and Implementing Improvements

CMS Collects Information on Website Use

We found that CMS utilizes three standard mechanisms for collecting information on the use of the Nursing Home Compare website: website analytics, website user surveys, and website usability tests.

Website analytics. CMS utilizes website analytics to gauge the performance of Nursing Home Compare and improve the visibility of the website in search engine listings. Through this mechanism, CMS is able to track data such as how many users, sessions, bounce rates, and page views Nursing Home Compare has per year. For example, these data show that from 2013 to 2015, Nursing Home Compare averaged 1.5 million sessions per year and 914,000 users per year. The website analytics also track the average session duration and the average number of pages that are viewed per session. For this same time period, the average session duration was 5.8 minutes and the average number of pages that were viewed per session was 4.8.

Website user surveys. CMS utilizes website user surveys to collect information about Nursing Home Compare users, how they use the site, and their opinions about the site. According to CMS officials, these surveys, which CMS began using in 2013, randomly pop up in web browsers to 50 percent of the website’s visitors. The surveys ask the user to identify themselves (for example, a caregiver or a researcher), the primary purpose for visiting the site, and the user’s experience in using the site. These surveys provide the only way CMS determines the type of users who come to the website, according to CMS officials. In October 2015, website survey data showed that 59 percent of users of Nursing Home Compare identified themselves as consumers, and the majority of users report coming to the site to research or select nursing homes for themselves or a family member.

Website usability tests. CMS utilizes usability tests—in the form of one-on-one sessions with nine consumer participants—to assess how well consumers navigate the website. CMS has conducted four usability tests from 2011 through 2015. The tests focus on the navigability of the website; however, they also include a few background questions about consumer use. For example, the usability tests ask if participants were previously aware of Nursing Home Compare, what factors consumers find most important in searching for a nursing home, and what expectations consumers have for a nursing home comparison website such as this.

In addition to these three standard mechanisms, CMS officials also told us that they gain insight into the use of Nursing Home Compare by holding ad-hoc meetings with a variety of stakeholders that are familiar with Nursing Home Compare to discuss the website. CMS held three stakeholder meetings from 2010 through 2015. According to CMS documents, stakeholders have included groups that represent consumers, such as ombudsmen, consumer advocate groups, provider advocate groups, and others that are involved in nursing home services. Information exchanged during stakeholder meetings includes CMS presentations on pending changes, such as the development of new quality measures, and stakeholder feedback.

The mechanisms that CMS utilizes to collect information about the use of Nursing Home Compare provide the agency with valuable information. However, these mechanisms do not provide CMS with information on the usefulness of the website to a broader range of consumers. Specifically, the usability tests are not designed to assess the website’s usefulness to consumers, and the website analytics and user surveys only provide information about consumers who access the website. Therefore, the mechanisms do not provide CMS with information on nursing home consumers who have not used the website because they are unaware of it, or choose not to use it, as well as the reasons why. In stakeholder interviews we conducted, some nursing home stakeholders noted that many consumers do not know about the website and that consumers collect information from other sources. Obtaining information from consumers who do not access Nursing Home Compare would likely require the dedication of resources to, for instance, consumer-oriented focus groups or broader surveys.

While Improvements Have Been Identified for the Nursing Home Compare Website, CMS Does Not Have a Systematic Process for Implementing Them

We identified five key areas of improvement CMS could make to Nursing Home Compare in order to make it more helpful for consumers. Specifically, we reviewed over 300 individual improvements identified in CMS documents—in part resulting from the mechanisms described above—and in interviews with national and state stakeholders; for example, one internal CMS analysis included over 40 individual recommendations for improvement. Through our analysis, we found the key areas of improvement are: 1) explanation of how to use the website, 2) additional information about the nursing home, 3) community and consumer outreach, 4) clarity of the website, and 5) navigability of the website.

Table 1 below provides more information about these key areas of improvements. For example, the first improvement addresses the fact that the Nursing Home Compare website does not currently have an explanation of how to use the website prominently displayed on its home page; there is not an introduction to the website, or an obvious explanation of how it should be used. According to many stakeholders, Nursing Home Compare is a valuable tool for consumers but a few specified that additional explanatory information is needed; without such information, the usefulness of the website may be limited.

Although CMS has identified the need for improvements to its Nursing Home Compare website, the agency does not have a systematic process that prioritizes recommended website changes and sets a timeline for implementation. In response to a recommendation in our 2012 report, in August 2013, CMS developed a strategic plan for evaluating the usability of Nursing Home Compare. The plan described tasks, including an expert review of the website, an analysis of competitor websites, and usability testing, some of which resulted in the formal mechanisms that CMS now has in place to collect information on the use of Nursing Home Compare, as previously described. However, CMS does not have a documented and systematic approach describing how to prioritize recommended changes to the website and assessing the potential improvements. Instead, officials described a fragmented approach to reviewing and implementing recommended website changes that may include verbal discussions of various factors, such as which changes would provide the broadest impact. CMS officials stated that their current approach to handling website changes had been working well, but since the website has become more complex in recent years, they acknowledged the need for a more formalized approach in addressing identified website changes.

CMS has stated the goal for its Nursing Home Compare website as assisting consumers in finding and comparing information about nursing home quality. In addition, under federal internal control standards, management should address identified program deficiencies on a timely basis and evaluate appropriate actions for improvement. However, in the absence of an established process to evaluate and prioritize implementation of improvements, CMS cannot ensure that it is fully meeting its goal for the website.

Several Factors Inhibit the Ability of the Five-Star System to Help Consumers Understand Nursing Home Quality and Choose Between Homes

Our analysis for the Five-Star System’s ratings data found that its overall rating provided consumers with distinctions between the highest and lowest performing nursing homes for health inspections in most states. Specifically, we found that, in 37 out of 50 states, homes that received an overall rating of 5-stars consistently had higher health inspection scores— the component measure that most significantly contributes to the overall rating—than homes that received an overall rating of 1-star. This means that in the 37 states, consumers can safely assume that, in the case of health inspections, the performance of any nursing home in their state with an overall high 5-star rating is better than the performance of any home with an overall low 1-star rating. Some stakeholders we spoke with agreed that distinctions between nursing homes are clearest at the extremes. For example, one stakeholder noted that the Five-Star System is best at helping consumers identify the poorest performing homes to avoid. Stakeholders also noted the value of having a national resource that uses standardized and objective nursing home quality information.

However, we also identified four factors that may inhibit the ability of consumers to use the Five-Star System ratings as an easy way to understand nursing home quality and identify high- and low- performing homes, CMS’s stated goal for the Five-Star System. 1. Interpreting overall ratings. As previously described, the Five-Star System’s overall rating is calculated using a process that combines three component ratings. However, the formula for combining the components is not intuitive, which can make interpreting overall ratings difficult for consumers by both complicating the comparison of overall ratings and masking the importance of the component ratings. Specifically, the comparison of overall ratings can be complicated because a consumer cannot assume that the performance on a particular component of the higher-rated home is better than that of the lower-rated home. In our review, we generally did not find distinctions in the scores for homes in the same state with adjacent overall ratings—e.g., 2- and 3-star homes—or for homes with middle overall ratings—2-, 3-, and 4-star homes. For example, in one state, 28 percent of homes with a 3-star overall rating had a better health inspection score than the average health inspection score for homes with an overall 4-star rating. Furthermore, the way CMS calculates the overall rating can mask for consumers issues that may be present in the component ratings. A consumer comparing nursing homes will see each home’s overall rating and component ratings, but they may not understand the impact each component score has on the overall rating. This could lead a consumer to rely more on the overall rating when their individual needs may require attention to one specific component more than the others. For example, two nursing homes that both have a 4-star overall rating could have opposite quality measure component ratings—one with a low 1-star quality measure rating and the other with a high 5-star quality measure rating. (See fig. 2).

Many stakeholders stated that it is difficult to distinguish between nursing homes with adjacent or middle ratings. In addition, some stakeholders expressed concern about the overall rating, with one explaining that often consumers make decisions based on the overall rating without understanding it or looking at the underlying components. According to CMS officials, the overall rating provides a summary of complex information to guide consumers—not an explicit report card—in as simple a way as possible. They also added that by providing individual component ratings, consumers have the ability to dig deeper into the source of the overall rating. 2. Timeliness of ratings data. Each of the three rating components— which influence the overall star rating—use a unique source of data that are collected from nursing homes at different time intervals. Specifically, the number of stars assigned to a nursing home is a point-in-time picture of performance based on a prior snapshot of the home’s performance and may not reflect a nursing home’s current status. (See table 2). Some stakeholders we spoke with expressed concern that a consumer may make a decision about a nursing home based on data that does not reflect current conditions in the home. According to CMS officials, a delay is always present due to administrative processes such as validating data prior to being posted. For example, the health inspection component data may be delayed due to additional information from the outcomes of revisits to the nursing home to check that deficiencies have been corrected. CMS officials and stakeholders said the Five-Star System should not be the only source of information a consumer uses; they both encourage consumers to explore additional information including visiting the home. 3. Comparing nursing homes across states. The overall rating and health inspection rating do not allow consumers to compare the quality of homes across states, limiting the ability of the rating system to help consumers who live near state borders or have multistate options where they could place their family members. Because ratings are relative to other nursing homes within a state, homes that receive the highest and lowest ratings in their state may not be the highest or lowest performing homes in another state or nationally. So, a consumer cannot assume that a 5-star nursing home in one state would be rated as a 5-star home in any other state. (See fig. 3). Furthermore, we found that when we recalculated the star ratings using a national distribution rather than a state distribution, homes’ ratings often changed, sometimes dramatically. For example, about 23 percent of nursing homes with a 1-star overall rating in December 2015 had improved ratings when compared nationally and about 30 percent of homes with a 5-star overall rating had decreased ratings when compared nationally. When looking at individual states, we found that the nursing homes in some states would fare better or worse under a national rating. Specifically, the percentage of homes receiving an overall 1-star rating doubled in 4 states and the percentage of homes receiving an overall 5-star rating doubled in 9 states. See appendix III for additional information about the results of our analysis.

According to CMS Five-Star System documentation, the rating system is not designed to compare nursing homes nationally. Instead, ratings are only comparable for homes in the same state. CMS made the decision to base the health inspection component on the relative performance of homes within the same state primarily due to variation across the states in the execution of the standard surveys. Because the health inspection component most significantly contributes to the overall rating, this means that the overall rating also cannot be compared nationally. However, the addition of national ratings would be helpful for consumers and we have previously made recommendations to CMS that would help decrease survey variation across states. CMS has taken action on many of these recommendations. 4. Lack of consumer satisfaction information in ratings. Because the Five-Star System does not include consumer satisfaction information—a key quality performance measure—the rating system is missing important information that could help consumers distinguish between high- and low- performing nursing homes. We believe consumer satisfaction surveys could be a more direct measure of nursing home satisfaction than other available measures. For example, our analysis of consumer satisfaction data shows that nursing homes with higher overall star ratings did not necessarily have higher resident satisfaction scores or fewer complaints. (See fig. 4). Specifically, our analysis found that the Five-Star System overall ratings for each nursing home in two states that conduct resident satisfaction surveys were only slightly correlated with the percentage of residents that would recommend the home to other consumers—an indicator of consumer satisfaction included on the state surveys. Similarly, when analyzing complaint data for all states—a proxy for consumer satisfaction—we also found only a slight correlation between the total number of consumer complaints registered against a home in each state and the home’s overall Five-Star System rating. Many stakeholders told us that they would like to see resident satisfaction included in the Five-Star System. For example, one state stakeholder group explained that they think it is important for a consumer making a nursing home decision to understand how the administration resolves an issue with a resident when one arises. That type of information is not currently captured in the Five-Star System, but could be captured through a resident satisfaction survey, which could strengthen the ratings.

According to CMS officials, they recognize that consumer satisfaction is important information, but collecting the data in a consistent, objective way for all of the nursing homes in the country is a challenge. They acknowledged that some states have been able to overcome these implementation challenges and administer statewide nursing home consumer satisfaction surveys. Until consumer satisfaction information is included in the rating system, consumers will continue to make nursing home decisions without the benefit of this key performance measure and may not be choosing the home that would best meet their needs. While we recognize that gathering this information is challenging, CMS has done so in its hospital rating system. Specifically, CMS developed a hospital consumer satisfaction survey with assistance from HHS’s Agency for Healthcare Research and Quality—an agency that, among other things, focuses on quality measurement and includes consumer satisfaction as one of its National Quality Measures Clearinghouse’s clinical quality measures.

In addition to the items discussed above, presentation of the Five-Star System does not prominently display key explanatory information that could help consumers better understand how to use the ratings. Specifically, we found that CMS does not prominently provide descriptions of how to understand the ratings and what consumers should consider when using the ratings or information on how the overall rating is calculated. In addition, CMS clearly discloses the date of the data used to assign stars for the health inspection component, but not for the staffing or quality measure components, and does not prominently state the previously discussed limitation that homes can only be compared within a state. For example, in order to find descriptions of how the overall rating is calculated, consumers must follow links that take them off of their nursing home search and results webpages and, as noted previously, an average webpage visit is less than 6 minutes. Many stakeholders we spoke with explained that consumers often have very little time to make a nursing home decision, and a few noted that it is also a stressful process, therefore making prominent and readily available information crucial. In addition, many stakeholders expressed concerns that consumers may not understand the ratings and how they are calculated. Further, many stakeholders expressed concern about the timeliness of the data, with some noting that consumers were generally unaware of the timing of the data. CMS officials described the tension between keeping the Five-Star System as simple as possible for consumers so that they can quickly understand the ratings and also providing enough information on how and when the ratings are calculated. Collectively, the four factors that hinder consumers’ ability to use the Five-Star System ratings, along with the lack of explanatory information provided by CMS, may limit the Five-Star System’s ability to meet CMS’s goal of providing consumers with an easy way to understand nursing home quality and make distinctions between high- and low- performing homes.

Conclusions

Nursing Home Compare and the Five-Star System seek to help consumers choose among nursing homes. Nursing home selection can be a stressful and time-sensitive process, so these are important tools that CMS makes available to the public. However, our review found opportunities for improvement in both the website and the ratings. CMS has given much attention to the website since its inception almost 20 years ago. For example, the agency has put in place mechanisms for reviewing the website’s use and has identified numerous improvements that could be made. However, without a systematic process for reviewing options and determining priorities for improvement—currently absent from their efforts—CMS is unable to ensure that the website is meeting its intended goal.

A key component of the website is the Five-Star System, containing important quality information on every nursing home so consumers can differentiate between them and choose those that can best meet their needs. Because the Five-Star System contains multiple types of information, compiled from different sources, and has complexities inherent in ratings systems, it can be challenging for consumers to fully understand how to take advantage of the varied information it contains. Additional capability and information not currently included in the rating system could also benefit consumers trying to differentiate between high- and low- performing nursing homes—such as the ability to compare homes nationally and the addition of consumer satisfaction survey information. In addition, prominently displaying explanatory information on how to use the ratings, which does not require users to navigate off the nursing home search and results pages, could help address challenges consumers face when trying to understand the ratings. Absent such enhancements, CMS cannot ensure that the Five-Star System is fully meeting its stated goal of helping consumers easily understand nursing home quality and distinguish between high- and low- performing homes.

Recommendations

To strengthen CMS’s efforts to improve the usefulness of the Nursing Home Compare website for consumers, we recommend that the Administrator of CMS establish a systematic process for reviewing potential website improvements that includes and describes steps on how CMS will prioritize the implementation of potential website improvements.

To help improve the Five-Star System’s ability to enable consumers to understand nursing home quality and make distinctions between high- and low- performing homes, we recommend that the Administrator of CMS take the following three actions: add information to the Five-Star System that allows consumers to compare nursing homes nationally; evaluate the feasibility of adding consumer satisfaction information to the Five-Star System; and develop and test with consumers introductory explanatory information on the Five-Star System to be prominently displayed on the home page. Such information should explain, for example, how the overall rating is calculated, the importance of the component ratings, where to find information on the timeliness of the data, and whether the ratings can be used to compare nursing homes nationally.

Agency Comments and Our Evaluation

We provided a draft of this report to HHS for its review and comment. HHS provided written comments, which are reproduced in appendix V. In its comments, HHS described the history of the Nursing Home Compare website and the Five-Star System, improvements the agency has made to both, and concurred with three of our four recommendations. In particular, HHS concurred with our recommendations to establish a process for reviewing potential website improvements that describes how it will prioritize their implementation, evaluate the feasibility of adding consumer satisfaction information to the Five-Star System, and develop and test explanatory information on the Five-Star System to be displayed on the home page. HHS did not concur with our recommendation to add information to the Five-Star System that would allow consumers to compare nursing homes nationally. HHS indicated that because of state variation in the execution of standard surveys, it is difficult to compare homes nationally on the health inspection component. They also noted that the Five-Star System is just one of many factors consumers should use when selecting a nursing home. As we describe in this report, efforts have been and should continue to be made to reduce state variation in standard surveys. For example, CMS regional offices are tracking state differences in deficiency citations. We maintain that the ability for consumers to compare nursing homes nationally is critical to making nursing home decisions, especially for those consumers who live near state borders or have multistate options, and that our recommendation remains valid.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from its date. At that time, we will send copies to the Secretary of HHS. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or clowersa@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Additional Detail on GAO’s Data Analyses

This appendix describes additional details of the data analyses we conducted to examine the Five-Star Quality Rating System (Five-Star System). For this examination, we analyzed data from three sources. We analyzed Five-Star System data from the Centers for Medicare & Medicaid Services (CMS). These data provide detailed rating information on over 15,000 nursing homes included in the Five-Star System from the most recent full quarter available at the time of our analysis, which ended December 2015. Additionally, we analyzed CMS consumer complaint data for a six-month period ending in December 2015. We collected and analyzed these data for all 50 states and Washington, D.C. Furthermore, we collected and analyzed 2015 nursing home resident satisfaction survey data from two of our four selected states that collect survey data.

We conducted the following analyses: 1. To determine the extent to which the Five-Star System provides consumers with information distinguishing between high- and low- performing nursing homes, we analyzed December 2015 data from CMS’s publicly available Five-Star Scores and Ratings data. Specifically, for each state, we determined the range of scores that underlie each component star rating: for health inspection we used the weighted all cycles score, for staffing we looked at the total adjusted staffing score, and for quality measures we calculated the total quality measure score—for each nursing home’s overall rating. We then determined whether, within each state and for each component rating, the scores of the worst performing 5-star nursing homes overlapped with the scores of the best performing 1-star nursing homes. We also conducted this analysis for each combination of the star ratings. 2. To determine the timeliness (or age) of the data used for each component Five-Star ratings for consumers viewing the ratings in December 2015, we analyzed data from CMS’s Five-Star Scores and Ratings data for that month downloaded from CMS’s website. Specifically, we calculated the average age of the data for each component rating at that point in time. For the health inspection component, we analyzed the standard survey date, but did not analyze the complaint investigation date because a meaningful average age cannot be calculated. 3. To determine the extent to which nursing homes’ ratings changed when compared nationally rather than compared only within each state, we analyzed December 2015 data from CMS’s Five-Star Score and Ratings data downloaded from CMS’s website. Specifically, we recalculated each nursing home’s health inspection and quality measure scores that are normally assigned based on state distributions so that they were based on a national distribution (new distribution allotments were based on CMS’s state distribution guidelines). We then recalculated each home’s overall rating using our new health inspection component rating, our new quality measure rating, and CMS’s staffing component rating. In addition, we analyzed the change in overall nursing home ratings when applying the methodology nationally. 4. To determine the relationship between nursing home satisfaction data and CMS’s Five-Star ratings, we did the following: a. We used complaints registered against nursing homes by residents, families, ombudsmen, or others as a proxy measurement of satisfaction. Specifically, we analyzed complaint data recorded in CMS’s Automated Survey Processing Environment Complaints/Incidents Tracking System from July 1 through December 31, 2015. We examined the last six months of 2015 to provide a fuller picture of each nursing home’s routine complaint levels. For each state and nationally, we determined the correlation between each nursing home’s total number of registered consumer complaints and its overall Five-Star rating. b. We used the results of 2015 nursing home resident satisfaction surveys from two of our selected states that collect such information. Specifically, we focused on the responses to whether the resident would recommend that nursing home to others as a measure of satisfaction. In one state, this measure was the actual percentage of residents that recommended the home and in the other state it was the ranking of the home based on residents’ responses. For both states, we determined the correlation between each nursing home’s resident response on the state survey and its overall Five-Star rating. The findings from this analysis cannot be generalized to other states.

For all data used in these analyses, we interviewed knowledgeable officials and reviewed related documentation and based on these steps determined that the data were sufficiently reliable to explore the relationship between the overall rating and the component ratings, determine national rating distributions, assess consumer satisfaction information, and describe the age of the data.

Appendix II: Nursing Home Stakeholder Views on Consumer Use of Nursing Home Compare

We held interviews with 30 nursing home stakeholders—eight national stakeholders and 22 state stakeholders from four states (Rhode Island, Georgia, Kansas, and California) we selected based on factors such as variation in geographic region and size (number of nursing homes). These stakeholders represent a range of provider groups, consumer groups, government agencies, and technical experts. We selected organizations in each state and nationally that are relevant to nursing home consumers and providers. Technical experts were identified by their prominence in the nursing home quality research field. In addition, some stakeholders we interviewed identified other groups that would be appropriate to interview. Our interviews included a set of questions regarding consumer use of Nursing Home Compare. Responses to these questions cannot be generalized beyond the stakeholders we interviewed.

We found that stakeholders generally could not quantify the number of consumers who use Nursing Home Compare, but most speculated that consumers use the site “a little” to “somewhat” and a few stakeholders said that consumers use the website “a lot”. Some stakeholders thought the number of people using Nursing Home Compare was growing, and one stakeholder thought this was because people are generally trying to make more educated decisions about nursing home care. Another stakeholder thought this increase could also be a result of people using the Internet to look things up more frequently, nursing homes included.

Some stakeholders noted that use of Nursing Home Compare probably differs depending on whether the patient is searching for care in an urban or a rural setting. Specifically, they stated that they think Nursing Home Compare is used more frequently in urban areas, where more nursing home options are available, compared to rural areas where there may be only one home in a town. See Table 3 below for summary of stakeholder responses.

Some stakeholders stated that they believe the extent to which consumers use Nursing Home Compare may depend on the amount of time that the consumer has to research and pick a nursing home. For example, according to a few stakeholders, if someone’s family member is getting discharged from the hospital and needs to be placed in a nursing home immediately, the consumer is less likely to use Nursing Home Compare. In contrast, one stakeholder noted that if consumers are planning for the future and researching nursing homes before care is needed, then they are more likely to use Nursing Home Compare.

We found that stakeholders were split in their responses about when consumers would typically use Nursing Home Compare – whether as an initial step in beginning their nursing home search or as a way to confirm recommendations obtained from others. Most stakeholders stated that the consumer using the website is most likely a family member—usually an adult child or grandchild—and rarely the individual in need of a nursing home placement.

When asked how valuable the information provided on Nursing Home Compare is to consumers who are researching and choosing a nursing home, most stakeholders stated that it was “somewhat valuable”, and some said that they thought it was “very valuable”. One stakeholder said that the information was “of little value”. See Table 4 below for a summary of stakeholder responses.

Some stakeholders stated that the information on Nursing Home Compare is a good place to start and may help consumers narrow down their search, but ultimately it is not likely to be the only source of information. Many stakeholders agreed that in addition to conducting online research, consumers should also always try to visit nursing homes in person before a making a decision. A few stakeholders stated that observing a nursing home and its current residents firsthand on any given day provides the most valuable information when making a decision. In addition, stakeholders noted that consumers also obtain information about nursing homes through other sources—primarily through word of mouth from friends, family, and neighbors, and from information provided by primary care physicians, hospital discharge planners, and local ombudsmen. Many stakeholders noted that in most cases though, the location of a nursing home is often the main determinant of where a family member is placed.

Stakeholders mentioned that consumers also use third party, private websites, and in some states, such as California and Kansas, consumers may rely on websites with state-specific nursing home information. Some stakeholders thought that consumers used these other sources of information more often than Nursing Home Compare. A couple of stakeholders thought consumers preferred these third-party, private websites because some of them provide a more personalized experience and offer the opportunity to speak with someone on the phone, where CMS’s Nursing Home Compare does not provide either of those options. A couple other stakeholders thought that consumers may be more likely to use and trust Nursing Home Compare compared to these other private websites simply because it is a government website. A couple of stakeholders expressed concern about consumers using third-party, private websites because nursing homes may pay to be included, and so the website may not provide objective information on nursing home options for consumers. Additionally, these private websites often appear on search engine results before Nursing Home Compare, so consumers may use them before seeing and using CMS’s Nursing Home Compare website.

Finally, stakeholders provided mixed responses regarding whether they suggested consumers use Nursing Home Compare when helping them search for nursing homes. For example, several stakeholders told us that they routinely referred consumers to the website, while another said that she would only direct consumers to Nursing Home Compare if they were not familiar with the area, or if did not have any time to spend on the nursing home search process. The stakeholder would ultimately recommend the consumer coming in and talking to her, and then would not use Nursing Home Compare at all.

Appendix III: Potential Changes to Ratings If National Distribution Used Rather than Existing State Distribution, December 2015

Appendix IV: Correlation between Consumer Complaints, July through December 2015, and Overall Ratings, December 2015

Appendix V: Comments from the Department of Health and Human Services

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, Linda Kohn, Director, Karin Wallestad, Assistant Director, Kathryn Richter, Analyst-In-Charge, Amy Andresen, Julianne Flowers, Shannon Smith, and Brienne Tierney made key contributions to this report. Also contributing were Jacques Arsenault, Wesley Dunn, Krister Friday, Rich Lipinski, Dae Park, Vikki Porter, and Steven Putansu.