NURSING HOMES

Consumers Could 

Benefit from 

Improvements to the 

Nursing Home 

Compare Website and 

Five-Star Quality 

Rating System 

Report to Congressional Requesters

November 2016

GAO-17-61

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-17-61, a report to 

congressional requesters

November 2016

NURSING HOMES

Consumers Could Benefit from Improvements 

to the Nursing Home Compare Website and 

Five-Star Quality Rating System 

What GAO Found

GAO found that the Centers for Medicare & Medicaid Services (CMS) collects 

information on the use of the Nursing Home Compare website, which was 

developed with the goal of assisting consumers in finding and comparing nursing 

home quality information. CMS uses three standard mechanisms for collecting 

website information—website analytics, website user surveys, and website 

usability tests. These mechanisms have helped identify potential improvements 

to the website, such as adding information explaining how to use the website. 

However, GAO found that CMS does not have a systematic process for 

prioritizing and implementing these potential improvements. Rather, CMS 

officials described a fragmented approach to reviewing and implementing 

recommended website changes. Federal internal control standards require 

management to evaluate appropriate actions for improvement. Without having an 

established process to evaluate and prioritize implementation of improvements, 

CMS cannot ensure that it is fully meeting its goals for the website.

GAO also found that several factors inhibit the ability of CMS’s Five-Star Quality 

Rating System (Five-Star System) to help consumers understand nursing home 

quality and choose between high- and low- performing homes, which is CMS’s 

primary goal for the system. For example, the ratings were not designed to 

compare nursing homes nationally, limiting the ability of the rating system to help 

consumers who live near state borders or have multistate options. In addition, 

the Five-Star System does not include consumer satisfaction survey information, 

leaving consumers to make nursing home decisions without this important

information. As a result, CMS cannot ensure that the Five-Star System fully

meets its primary goal.

Example of Missing Information Affecting a Consumer’s Nursing Home Decision

View GAO-17-61. For more information, 

contact A. Nicole Clowers at (202) 512-7114 

or clowersa@gao.gov.

Why GAO Did This Study

Approximately 15,600 nursing homes 

participating in the Medicare and 

Medicaid programs provide care to 1.4 

million residents each year. To help 

consumers make informed choices 

about nursing homes, CMS developed

the Nursing Home Compare website,

and on the site made available the 

Five-Star System, which rates homes 

on quality components. 

GAO was asked to assess the website 

and rating system as tools for 

consumers. GAO examined (1) the 

information CMS collects about the use 

of Nursing Home Compare, including 

its usefulness to consumers, and 

potential areas, if any, to improve the 

website, and (2) the extent to which the 

Five-Star System enables consumers 

to understand nursing home quality 

and make distinctions between homes. 

GAO reviewed CMS documents and 

interviewed CMS officials and national 

and a non-generalizable sample of 

state-level stakeholders from four

states, selected on factors such as 

size. GAO also analyzed Five-Star

System and consumer complaint data, 

and analyzed resident satisfaction data 

from two of the four selected states.

What GAO Recommends

GAO is making four recommendations, 

including, that CMS establish a 

process to evaluate and prioritize 

website improvements, add information 

to the Five-Star System that allows 

homes to be compared nationally, and 

evaluate the feasibility of adding 

consumer satisfaction data. HHS 

agreed with three of GAO’s

recommendations, but did not agree to 

add national comparison information. 

GAO maintains this is important 

information, as discussed in the report.
Page i GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Letter 1

Background 5

CMS Collects Information on the Use of the Nursing Home 

Compare Website, but Lacks a Systematic Process for 

Prioritizing and Implementing Improvements 11

Several Factors Inhibit the Ability of the Five-Star System to Help 

Consumers Understand Nursing Home Quality and Choose 

Between Homes 16

Conclusions 25

Recommendations 26

Agency Comments and Our Evaluation 27

Appendix I Additional Detail on GAO’s Data Analyses 29

Appendix II Nursing Home Stakeholder Views on Consumer Use of Nursing 

Home Compare 32

Appendix III Potential Changes to Ratings If National Distribution Used Rather 

than Existing State Distribution, December 2015 36

Appendix IV Correlation between Consumer Complaints, July through December 

2015, and Overall Ratings, December 2015 37

Appendix V Comments from the Department of Health and Human Services 39

Appendix VI GAO Contact and Staff Acknowledgments 42

Tables

Table 1: GAO-Identified Areas of Improvement for Nursing Home 

Compare 14

Contents
Page ii GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Table 2: Timeliness of Data Used in the Five-Star System for 

Nursing Homes 19

Table 3: Stakeholder Responses to How Much Consumers Use 

Nursing Home Compare 33

Table 4: Stakeholder Responses to How Valuable the Information 

Provided on Nursing Home Compare is to Consumers 34

Figures

Figure 1: Calculating the Overall Rating for the Five-Star System 

for Nursing Homes 10

Figure 2: Example of Consumer Making Nursing Home Decision 

Based Only on the Overall Rating 18

Figure 3: Example of Consumer Making Nursing Home Decision 

across Multiple States 21

Figure 4: Example of Consumer Making Nursing Home Decision 

without Consumer Satisfaction Information 24

Abbreviations

CMS Centers for Medicare & Medicaid Services

Five-Star System Five-Star Quality Rating System

HHS Department of Health and Human Services

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

441 G St. N.W.

Washington, DC 20548

November 18, 2016

The Honorable Ron Wyden

Ranking Member

Committee on Finance

United States Senate

The Honorable Elijah E. Cummings

Ranking Member

Committee on Oversight and Government Reform

House of Representatives

The Honorable Bob Casey

United States Senate

Nationwide, approximately 15,600 nursing homes participating in the 

Medicare and Medicaid programs provide care to about 1.4 million 

nursing home residents—a vulnerable population of elderly and disabled 

individuals.1 As the estimated 76 million baby boomers born between 

1946 and 1964 become elderly, the number of nursing home residents 

may increase. When deciding on a nursing home, consumers—current 

and potential nursing home residents and their family or friends—can 

make choices based on such factors as location, fees, services and 

activities for residents, and what they know about the quality of care 

provided in a facility.

Given the various factors to consider when choosing a facility, the 

Centers for Medicare & Medicaid Services (CMS)—an agency within the 

Department of Health and Human Services (HHS) that is responsible for 

establishing and overseeing the quality standards that nursing homes 

must meet in order to participate in the Medicare and Medicaid 

programs—has taken steps to help consumers. Specifically, in 1998 CMS 

began publicly reporting information related to the quality of nursing 

homes on its Nursing Home Compare website.2 CMS’s goals for the 

 1

Medicare is the federal health insurance program for persons aged 65 or over, certain 

disabled individuals, and individuals with end-stage renal disease. Medicaid is the joint 

federal-state health care financing program for certain categories of low-income

individuals. 

2

See https://www.medicare.gov/nursinghomecompare/search.html accessed August 31, 

2016.

Letter
Page 2 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

website include helping consumers compare nursing home quality and 

assisting them in finding other information about nursing homes. In order 

to further aid consumers using Nursing Home Compare, in 2008 CMS 

implemented the Five-Star Quality Rating System (Five-Star System) on 

the website. The Five-Star System assigns each nursing home an overall 

rating and three component ratings based on the extent to which the 

nursing home meets CMS’s quality standards and other measures. 

According to CMS, the primary goal of the Five-Star System is to provide 

consumers with an easy way to understand nursing home quality and 

distinguish between high- and low- performing nursing homes.3

In recent years, we and others have expressed concerns with Nursing 

Home Compare and the Five-Star System, including concerns over 

CMS’s efforts to ensure the usefulness of these tools for consumers. In 

our March 2012 report we reviewed, among other things, how the FiveStar System was developed and how CMS ensures its goals are met, and 

noted that while the Five-Star System was a significant step toward 

increasing the transparency of information important to consumers, to 

sustain the system over time the agency would need to make a concerted 

effort. CMS agreed with our recommendation to establish—through 

planning documents—how its planned efforts would help the agency 

achieve the goals of the Five-Star System.4 Additionally, some nursing 

home stakeholder groups have raised concerns about the usefulness of 

Nursing Home Compare for consumers as well as the accuracy of its 

Five-Star System.5

Given the importance of providing useful information to consumers 

choosing a nursing home, along with the concerns raised by nursing 

home stakeholders and others, you asked us to examine the use of 

 3

The secondary goal of the Five-Star System is to help improve nursing home quality by 

publicly reporting quality of care information, as some research has suggested that 

publishing such information can create an incentive for providers to improve their quality of 

care.

4

See GAO, Nursing Homes: CMS Needs Milestones and Timelines to Ensure Goals for 

the Five-Star Quality Rating System Are Met, GAO-12-390, (Washington, D.C.: March 23, 

2012).

5

For example, see LeadingAge, “What the 5-Star Rating System Changes Mean for 

Nursing Homes,” (Feb. 18, 2015) and The National Consumer Voice for Quality LongTerm Care, “Advocates Welcome Changes to National Nursing Home Rating System: Call 

for Federal Government to Do More to Improve System and Care,” (Feb. 20, 2015).
Page 3 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Nursing Home Compare and the Five-Star System as tools for 

consumers. In this report, we examine:

1. the information CMS collects about the use of the Nursing Home 

Compare website, including its usefulness to consumers, and 

potential areas, if any, to improve the website, and

2. to what extent the Five-Star System on Nursing Home Compare 

enables consumers to understand nursing home quality and make 

distinctions between high- and low- performing nursing homes.

To examine the information CMS collects about the use of Nursing Home 

Compare, including its usefulness to consumers, and CMS’s efforts to 

improve the website, we reviewed documents from CMS, such as reports 

on Nursing Home Compare’s website statistics and assessments of its 

usability. In addition, we interviewed CMS officials, including those 

responsible for the development and oversight of Nursing Home 

Compare. We also conducted interviews with 30 nursing home 

stakeholders. These included advocacy groups representing consumers 

or providers, an organization of government agencies, and nursing home 

quality experts at the national level as well as the state level for four 

selected states—California, Georgia, Kansas, and Rhode Island. We 

selected these four states based on factors such as variation in 

geographic region and size (number of nursing homes).6 We conducted 

interviews with these nursing home stakeholders in order to identify the 

strengths and weaknesses, as well as potential areas of improvement, of 

Nursing Home Compare and the Five-Star System, among other things. 

Using this information and CMS internal documentation, we identified 

areas of improvement for Nursing Home Compare. We also examined the 

information we collected against CMS’s goals for the website and relevant 

standards for internal control in the federal government.7

To examine the extent to which the Five-Star System enables consumers 

to understand nursing home quality and distinguish between high- and 

low- performing nursing homes, we analyzed CMS documents, including 

its February 2015 Five-Star System Technical Users’ Guide, the relevant 

 6

These four selected states are not representative of the nation. 

7

Internal control is a process effected by an entity’s oversight body, management, and 

other personnel that provides reasonable assurance that the objectives of an entity will be 

achieved. See GAO, Standards for Internal Control in the Federal Government, 

GAO-14-704G (Washington, D.C.: Sept. 10, 2014).
Page 4 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

guide at the time of our analysis.

8 We also interviewed CMS officials, 

including those knowledgeable about the methodology for determining 

Five-Star System ratings, and national and state nursing home 

stakeholders, as described above. Furthermore, we analyzed CMS’s 

Five-Star System ratings data from the most recent full quarter of data 

available at the time of our analysis, which ended December 2015. First, 

for nursing homes with adjacent overall ratings (i.e., 4-star homes 

compared to 5-star and 3-star homes) we examined the extent to which 

these homes’ underlying component scores overlapped at the state level. 

For example, we examined the extent to which the component scores for 

nursing homes that received a 4-star overall rating were higher or lower 

than the component scores for nursing homes that received a 5- or 3-star 

overall rating. Second, we determined the average age of the data used 

to calculate each nursing home’s overall rating and component ratings.

Third, we recalculated overall star ratings on a national basis using the 

Five-Star System’s methodology to determine how nursing homes’ statebased ratings compare to their nationally based ratings. In addition, we 

analyzed consumer complaint data collected and recorded in CMS’s 

Automated Survey Processing Environment Complaints/Incidents 

Tracking System from July 2015 through December 2015 to examine the 

extent to which each nursing home’s overall rating correlated with the 

number of consumer complaints registered against it.9 We examined the 

last six months of 2015 to provide a fuller picture of each nursing home’s 

routine complaint levels. We analyzed these data at the national level and 

for all 50 states and Washington, D.C., as appropriate.10 Finally, we 

obtained 2015 nursing home resident satisfaction survey data from two of 

our four selected states and compared these data to each nursing home’s 

 8

CMS, Design for Nursing Home Compare Five-Star Quality Rating System: Technical 

Users’ Guide, (Baltimore, MD: Feb. 2015). Subsequent to our analysis, CMS issued an 

updated technical users guide, see CMS, Design for Nursing Home Compare Five-Star 

Quality Rating System: Technical Users’ Guide, (Baltimore, MD: July, 2016). Based on our 

review of the 2016 update, we would not expect the changes to impact our findings. 

9

State survey agencies are required to enter information about complaints and incidents 

involving nursing homes in their state into CMS’s Automated Survey Processing 

Environment Complaints/Incidents Tracking System. CMS then uses the information 

reported in this database, which includes the date of the alleged event, the name of the 

nursing home involved, and the source of the complaint, to manage enforcement activities 

and provide information to the public on Nursing Home Compare.

The individual nursing home is the unit of analysis. 

10For the purposes of this report, we include Washington, D.C., when we refer to data for 

states. 
Page 5 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

overall Five-Star rating from that same year in order to determine the 

extent to which the overall ratings for nursing homes in those two states 

correlated with the level of consumer satisfaction.11 The findings from this 

final analysis cannot be generalized to other states. For all data used in 

these analyses, we interviewed knowledgeable officials and reviewed 

related documentation, and based on these steps, determined that the 

data were sufficiently reliable to explore the relationship between the 

overall rating and the component ratings, describe the age of the data,

determine national rating distributions, and assess consumer satisfaction 

information. We examined the information contained in the Five-Star 

System against CMS’s goals for the Five-Star System and HHS’s Agency 

for Healthcare Research and Quality’s National Quality Measures 

Clearinghouse.12 For additional information about our methodology for 

conducting these analyses, see appendix I.

We conducted this performance audit from September 2015 to November 

2016 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

CMS’s goals for Nursing Home Compare and its Five-Star System are 

consistent with its strategy to improve the quality of health care by 

providing transparent information about the quality of health care 

services, including those delivered in nursing homes.13 According to the 

strategy, to achieve better care, patients must be given access to 

understandable information and decision support tools that help them 

manage their health and navigate the health care delivery system.

Since 1998, CMS has publicly reported information on nursing home 

quality on its Nursing Home Compare website and has increased the 

 11The individual nursing home is the unit of analysis. 

12The National Quality Measures Clearinghouse provides publically available summaries 

of evidence-based quality measures, among other things. See 

https://www.qualitymeasures.ahrq.gov. 

13CMS, CMS Quality Strategy 2016, (Baltimore, MD: 2016). 

Background

Nursing Home Compare
Page 6 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

amount of information reported on the website over time. CMS initially 

reported information only about nursing home characteristics and nursing 

home health inspection results on Nursing Home Compare. Later, CMS 

began reporting additional information on the website, such as the ratio of 

nursing staff to residents, nursing homes’ performance on various quality 

measures, and the number of complaints registered against nursing 

homes. Additionally, CMS has updated the appearance and functionality 

of the Nursing Home Compare website over time, with the most 

significant change being the introduction of the Five-Star System in 2008.

In December 2008, CMS made the Five-Star System available to the 

public on its Nursing Home Compare website in order to help consumers 

compare nursing homes more easily.14 The Five-Star System assigns 

each nursing home participating in the Medicare or Medicaid programs an 

overall “star” rating, ranging from one to five. Nursing homes with five 

stars are considered to have much above average quality, while nursing 

homes receiving one star are considered to have much below average 

quality. Calculation of the overall star rating is based on separate ratings 

that nursing homes receive for each of three components: health 

inspections, staffing, and quality measures.

• Health inspection rating. CMS contracts with state survey agencies 

to conduct unannounced, on-site nursing home health inspections—

known as standard surveys—to determine whether nursing homes 

meet federal quality standards. Every nursing home receiving 

Medicare or Medicaid payment must undergo a standard survey not 

less than once every 15 months, and the statewide average interval 

for these surveys must not exceed 12 months.15 State surveyors also 

conduct complaint investigations in response to allegations of quality 

problems. If nursing homes are found to be out of compliance with 

any requirements, state surveyors issue deficiency citations that 

reflect the scope (number of residents affected) and severity (level of 

harm to residents) of the deficiency. Surveyors conduct revisits to the 

nursing home to ensure that the deficiencies identified have been 

corrected. A nursing home’s health inspection rating is relative to 

other nursing homes’ health inspection ratings in their state. As such, 

health inspection ratings are assigned to generally achieve the 

 14For more information on how CMS developed the Five-Star System, see GAO-12-390.

15See 42 U.S.C. §§ 1395i-3(g)(2)(A)(iii), 1396r(g)(2)(A)(iii). 

Five-Star System
Page 7 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

following distribution within each state: the top 10 percent of nursing 

homes receive five stars, the bottom 20 percent receive one star, and 

the middle 70 percent of nursing homes receive two, three, or four 

stars.

• Staffing rating. Nursing homes self-report staffing hours worked for a 

2-week period at the time of the standard survey.16 CMS converts the 

reported point-in-time staffing hours for nursing staff—registered 

nurses, licensed practical nurses, and certified nursing assistants—

into measures that indicate the number of registered nurse and total 

nursing hours per resident per day.17 CMS adjusts the staffing levels 

for differences in the level of complexity of nursing services required 

to care for residents across nursing homes—referred to as resident 

acuity. Each nursing home’s staffing rating is assigned based on how 

its total nursing and registered nurse staffing levels compare to the 

distribution of staffing levels for freestanding homes in the nation and 

staffing level thresholds identified by CMS.18

• Quality measure rating. Nursing homes regularly collect assessment 

information on all their residents, including information on the 

residents’ health, physical functioning, mental status, and general 

 16As of July 2016, nursing homes are required to electronically submit staffing hours and 

related information based on payroll and other auditable data on at least a quarterly basis. 

In addition to staffing data, nursing homes are required to submit census data regarding 

their resident population, such as the number of residents whose primary payer is 

Medicaid, and employee turnover and tenure. According to CMS officials, nursing homes 

must electronically submit their first quarter of staffing data by November 14, 2016. CMS 

will then evaluate these data and work toward incorporating them into the Five-Star 

System. At the time of our report, CMS did not have a timeframe for when the Five-Star 

System will reflect updated staffing data.

17Total nursing hours are the sum of registered and licensed practical nurse and certified 

nursing assistant hours. 

18See AM Kramer and R. Fish, Abt Associates, “The Relationship Between Nurse Staffing 

Levels and the Quality of Nursing Home Care,” chapter 2 in Appropriateness of Minimum 

Nurse Staffing Ratios in Nursing Homes: Phase II Final Report, a report prepared at the 

request of CMS, 2001. 

Freestanding nursing homes are those that are not under administrative control of a 

hospital.
Page 8 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

well-being. Nursing homes self-report this information to CMS.19 CMS 

uses some of the assessment information to measure the quality of 

certain aspects of nursing home care, such as the prevalence of 

pressure sores and changes in residents’ mobility. At the time of our 

analysis, CMS calculated this rating for each nursing home based on 

11 of the 18 quality measures posted on Nursing Home Compare. 

Information on the remaining 7 quality measures is posted on the 

website but not used in the calculation of the rating.20 A nursing 

home’s quality measure rating is assigned based on national 

thresholds established by CMS.

The overall star rating is calculated using a process that combines the 

star ratings from the health inspection, staffing, and quality measure 

components—with the greatest weight given to the health inspection 

rating. The overall rating is assigned based on the following steps:

1. Start with the number of stars for the health inspection rating.

2. Add one star if the staffing rating is four or five stars and also greater 

than the health inspection rating. Subtract one star if the staffing rating 

is one star. The overall rating cannot go above five stars or below one 

star.

 19In 2015, we reported that CMS’s ability to use its data to assess nursing home quality is 

complicated by various issues with the data, which make it difficult to determine whether 

observed quality trends over time reflect actual changes in quality, data issues, or both. 

For example, we found that both staffing and quality measure data are self-reported by 

nursing homes and generally not audited by the agency. While CMS has taken steps to 

assure the reliability of this self-reported data—the rollout of focused surveys to evaluate 

the data used in quality measures and plans to use and audit payroll data rather than selfreported data to determine nursing home staffing levels—CMS did not have clear ongoing 

audit plans. We recommended that CMS establish and implement a clear plan for ongoing 

auditing to ensure reliability of data self-reported by nursing homes, including payrollbased staffing data and data used to calculate clinical quality measures. HHS concurred 

with this recommendation. See GAO, Nursing Home Quality: CMS Should Continue to 

Improve Data and Oversight, GAO-16-33 (Washington, D.C.: Oct. 30, 2015).

20In 2016, CMS added new quality measures to Nursing Home Compare and the FiveStar System. Specifically, as of July 2016, CMS reported 24 quality measures on Nursing 

Home Compare, and began to base the quality measure rating for each nursing home on 

its performance in 16 of those measures. Information on the remaining quality measures 

are posted on the website but not used in the calculation of the rating.
Page 9 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

3. Add one star if the quality measure rating is five stars. Subtract one 

star if the quality measure rating is one star. The rating cannot go 

above five stars or below one star.21

See figure 1 for an example of how a nursing home’s overall rating is 

calculated. CMS updates the ratings on a monthly basis; however, a 

particular home’s overall rating will only change if it had new data that 

affected any one of the component ratings. For example, when a home 

has a health inspection survey, either a standard survey or a complaint 

investigation, the deficiency data from the survey will become a part of 

the calculation for the health inspection rating and the overall rating will 

also be adjusted, if necessary.

 21The overall rating is capped in two circumstances. First, if a nursing home’s health 

inspection rating is one star, then the overall rating cannot exceed two stars. Second, 

nursing homes currently in the Special Focus Facility Program—a program that aims to 

remedy noncompliance with federal quality standards in nursing homes with repeated 

cycles of noncompliance with these standards—have their overall rating capped at three 

stars even if they have high ratings in individual components. 
Page 10 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Figure 1: Calculating the Overall Rating for the Five-Star System for Nursing Homes
Page 11 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

We found that CMS utilizes three standard mechanisms for collecting 

information on the use of the Nursing Home Compare website: website 

analytics, website user surveys, and website usability tests.22

• Website analytics. CMS utilizes website analytics to gauge the 

performance of Nursing Home Compare and improve the visibility of 

the website in search engine listings. Through this mechanism, CMS 

is able to track data such as how many users, sessions, bounce rates, 

and page views Nursing Home Compare has per year.23 For example, 

these data show that from 2013 to 2015, Nursing Home Compare 

averaged 1.5 million sessions per year and 914,000 users per year. 

The website analytics also track the average session duration and the 

average number of pages that are viewed per session. For this same 

 22According to DigitalGov’s Digital Analytics Program, agencies should ensure that they 

collect, analyze, and report on a variety of performance and customer satisfaction 

measures, which include, among others, website analytics, website user surveys, and 

website usability testing. The Digital Government Division of the U.S. General Services 

Administration runs a DigitalGov platform that helps agencies build a 21st century digital 

government by, for example, offering services and tools to help agencies meet their digital 

government goals. See www.digitalgov.gov. 

23According to the company that prepares and houses the website analytics, a session is 

the period of time a user is active on a website. By default, if a user is active for 30 

minutes or more, any future sessions are counted as separate sessions. If users leave a 

site before 30 minutes of use and then return within 30 minutes, the return session is 

counted as part of the original session. A bounce rate is the percentage of single-page 

sessions (i.e. sessions in which the person left the site from the entrance page without 

interacting with the page). Nursing Home Compare had an average bounce rate of 27.1 

percent from 2013 to 2015.

CMS Collects 

Information on the 

Use of the Nursing 

Home Compare 

Website, but Lacks a 

Systematic Process 

for Prioritizing and 

Implementing 

Improvements

CMS Collects Information 

on Website Use
Page 12 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

time period, the average session duration was 5.8 minutes and the 

average number of pages that were viewed per session was 4.8.

• Website user surveys. CMS utilizes website user surveys to collect 

information about Nursing Home Compare users, how they use the 

site, and their opinions about the site.24 According to CMS officials, 

these surveys, which CMS began using in 2013, randomly pop up in 

web browsers to 50 percent of the website’s visitors. The surveys ask 

the user to identify themselves (for example, a caregiver or a 

researcher), the primary purpose for visiting the site, and the user’s 

experience in using the site. These surveys provide the only way CMS 

determines the type of users who come to the website, according to 

CMS officials. In October 2015, website survey data showed that 59 

percent of users of Nursing Home Compare identified themselves as 

consumers, and the majority of users report coming to the site to 

research or select nursing homes for themselves or a family 

member.25

• Website usability tests. CMS utilizes usability tests—in the form of 

one-on-one sessions with nine consumer participants—to assess how 

well consumers navigate the website.26 CMS has conducted four 

usability tests from 2011 through 2015. The tests focus on the 

navigability of the website; however, they also include a few 

background questions about consumer use. For example, the usability 

tests ask if participants were previously aware of Nursing Home 

Compare, what factors consumers find most important in searching 

for a nursing home, and what expectations consumers have for a 

nursing home comparison website such as this.

In addition to these three standard mechanisms, CMS officials also told 

us that they gain insight into the use of Nursing Home Compare by 

holding ad-hoc meetings with a variety of stakeholders that are familiar 

 24Our 2012 report noted that, according to a senior CMS official, the agency planned to 

begin conducting a web-based pop-up survey as a part of a more in-depth evaluation of 

Nursing Home Compare usability in fiscal year 2012. See GAO-12-390. CMS has been 

using pop-up website user surveys since January 2013. 

25Other groups of users include nursing home staff members, discharge planners, 

researchers, and health care providers.

26Our 2012 report noted that, according to a senior CMS official, the agency planned to 

conduct more detailed consumer testing of Nursing Home Compare usability in fiscal year 

2012 as a part of a more in-depth evaluation of the website’s usability. We indicated in 

that report that CMS conducted a usability test in December 2011. See GAO-12-390. 

Since 2011, CMS has conducted three additional usability tests in October 2012, 

December 2012, and December 2015. 
Page 13 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

with Nursing Home Compare to discuss the website. CMS held three 

stakeholder meetings from 2010 through 2015. According to CMS 

documents, stakeholders have included groups that represent 

consumers, such as ombudsmen, consumer advocate groups, provider 

advocate groups, and others that are involved in nursing home services. 

Information exchanged during stakeholder meetings includes CMS 

presentations on pending changes, such as the development of new 

quality measures, and stakeholder feedback.

The mechanisms that CMS utilizes to collect information about the use of 

Nursing Home Compare provide the agency with valuable information. 

However, these mechanisms do not provide CMS with information on the 

usefulness of the website to a broader range of consumers. Specifically, 

the usability tests are not designed to assess the website’s usefulness to 

consumers, and the website analytics and user surveys only provide 

information about consumers who access the website. Therefore, the 

mechanisms do not provide CMS with information on nursing home 

consumers who have not used the website because they are unaware of 

it, or choose not to use it, as well as the reasons why. In stakeholder 

interviews we conducted, some nursing home stakeholders noted that 

many consumers do not know about the website and that consumers 

collect information from other sources.27 Obtaining information from 

consumers who do not access Nursing Home Compare would likely 

require the dedication of resources to, for instance, consumer-oriented 

focus groups or broader surveys.

We identified five key areas of improvement CMS could make to Nursing 

Home Compare in order to make it more helpful for consumers. 

Specifically, we reviewed over 300 individual improvements identified in 

CMS documents—in part resulting from the mechanisms described 

above—and in interviews with national and state stakeholders; for 

example, one internal CMS analysis included over 40 individual 

recommendations for improvement. Through our analysis, we found the 

key areas of improvement are: 1) explanation of how to use the website, 

2) additional information about the nursing home, 3) community and 

 27For more information on stakeholders’ views about consumer use of Nursing Home 

Compare, see Appendix II.

While Improvements Have 

Been Identified for the 

Nursing Home Compare 

Website, CMS Does Not 

Have a Systematic 

Process for Implementing 

Them
Page 14 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

consumer outreach, 4) clarity of the website, and 5) navigability of the 

website.28

Table 1 below provides more information about these key areas of 

improvements. For example, the first improvement addresses the fact that 

the Nursing Home Compare website does not currently have an 

explanation of how to use the website prominently displayed on its home

page; there is not an introduction to the website, or an obvious 

explanation of how it should be used. According to many stakeholders, 

Nursing Home Compare is a valuable tool for consumers but a few 

specified that additional explanatory information is needed; without such 

information, the usefulness of the website may be limited.

Table 1: GAO-Identified Areas of Improvement for Nursing Home Compare

Area of Improvement Examples

Explanation of How to Use the Website

Provide clearer explanations of how to find relevant

information and how the rating system is calculated.

• Add an introductory and explanatory video on how to use the 

website, including how to conduct searches and where 

information is located.

• Add a brief “Read this first” section that includes text with 

instructions and information about how to use the website. 

Additional Information About the Nursing Home

Provide additional information about nursing homes that is 

relevant to consumers’ needs and decision processes.

• Add contact information (email addresses and website links) 

for nursing homes.

• Add information about the facility’s specialty care services or 

certain levels and types of care provided.

Community and Consumer Outreach

Raise awareness about Nursing Home Compare to those 

that help consumers make nursing home decisions.

• Communicate, educate, and train local agencies, ombudsman 

programs, and discharge planners about how to use Nursing 

Home Compare and the Five-Star System.

• Enable Nursing Home Compare to appear earlier in the result 

list before other third party, privately-owned websites during 

website searches.

Clarity of the Website

Improve the clarity of the content on the website by ensuring 

that the information is written in reader-friendly language.

• Simplify overall language on website, and present information 

in a way that an average consumer can understand. For 

example, remove technical jargon where possible, and where 

not possible, provide definitions or examples. 

Navigability of Website

Improve the layout of information on the website to make it 

easier for consumers to find information quickly without a lot 

of scrolling or clicking.

• Streamline navigability of website by making it easier for users 

to get to desired results page in as few clicks as possible.

• Allow content to be easily formatted for printing.

Source: GAO analysis of CMS documents and stakeholder interviews. | GAO-17-61

 28In a 2014 report, we found that the Nursing Home Compare website did not use plain 

language that would make sense to consumers who typically are unfamiliar with clinical 

terminology. See GAO, Health Care Transparency: Actions Needed to Improve Cost and 

Quality Information for Consumers, GAO-15-11 (Washington, D.C.: Oct. 20, 2014). 
Page 15 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Although CMS has identified the need for improvements to its Nursing 

Home Compare website, the agency does not have a systematic process 

that prioritizes recommended website changes and sets a timeline for 

implementation. In response to a recommendation in our 2012 report, in 

August 2013, CMS developed a strategic plan for evaluating the usability 

of Nursing Home Compare.29 The plan described tasks, including an 

expert review of the website, an analysis of competitor websites, and 

usability testing, some of which resulted in the formal mechanisms that 

CMS now has in place to collect information on the use of Nursing Home 

Compare, as previously described. However, CMS does not have a 

documented and systematic approach describing how to prioritize 

recommended changes to the website and assessing the potential 

improvements. Instead, officials described a fragmented approach to 

reviewing and implementing recommended website changes that may 

include verbal discussions of various factors, such as which changes 

would provide the broadest impact. CMS officials stated that their current 

approach to handling website changes had been working well, but since 

the website has become more complex in recent years, they 

acknowledged the need for a more formalized approach in addressing 

identified website changes.

CMS has stated the goal for its Nursing Home Compare website as 

assisting consumers in finding and comparing information about nursing 

home quality. In addition, under federal internal control standards, 

management should address identified program deficiencies on a timely 

basis and evaluate appropriate actions for improvement. However, in the 

absence of an established process to evaluate and prioritize 

implementation of improvements, CMS cannot ensure that it is fully 

meeting its goal for the website.

 29See GAO-12-390.
Page 16 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Our analysis for the Five-Star System’s ratings data found that its overall 

rating provided consumers with distinctions between the highest and 

lowest performing nursing homes for health inspections in most states. 

Specifically, we found that, in 37 out of 50 states, homes that received an 

overall rating of 5-stars consistently had higher health inspection scores—

the component measure that most significantly contributes to the overall 

rating—than homes that received an overall rating of 1-star.30 This means 

that in the 37 states, consumers can safely assume that, in the case of 

health inspections, the performance of any nursing home in their state 

with an overall high 5-star rating is better than the performance of any 

home with an overall low 1-star rating. Some stakeholders we spoke with 

agreed that distinctions between nursing homes are clearest at the 

extremes. For example, one stakeholder noted that the Five-Star System 

is best at helping consumers identify the poorest performing homes to

avoid. Stakeholders also noted the value of having a national resource 

that uses standardized and objective nursing home quality information.

However, we also identified four factors that may inhibit the ability of 

consumers to use the Five-Star System ratings as an easy way to 

understand nursing home quality and identify high- and low- performing 

homes, CMS’s stated goal for the Five-Star System.

1. Interpreting overall ratings. As previously described, the Five-Star 

System’s overall rating is calculated using a process that combines 

three component ratings. However, the formula for combining the 

components is not intuitive, which can make interpreting overall 

ratings difficult for consumers by both complicating the comparison of 

overall ratings and masking the importance of the component ratings. 

Specifically, the comparison of overall ratings can be complicated 

because a consumer cannot assume that the performance on a 

particular component of the higher-rated home is better than that of 

the lower-rated home. In our review, we generally did not find 

distinctions in the scores for homes in the same state with adjacent 

overall ratings—e.g., 2- and 3-star homes—or for homes with middle 

overall ratings—2-, 3-, and 4-star homes. For example, in one state, 

28 percent of homes with a 3-star overall rating had a better health 

 30In December 2015, one state did not have any nursing homes with a 1-star overall rating 

and was excluded from our analysis. 

CMS’s Five-Star System methodology uses a point system to assign a score to each 

nursing home for the health inspection component. That score is then used to determine 

each nursing home’s star rating. 

Several Factors 

Inhibit the Ability of 

the Five-Star System 

to Help Consumers 

Understand Nursing 

Home Quality and 

Choose Between 

Homes
Page 17 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

inspection score than the average health inspection score for homes 

with an overall 4-star rating.31 Furthermore, the way CMS calculates 

the overall rating can mask for consumers issues that may be present 

in the component ratings. A consumer comparing nursing homes will 

see each home’s overall rating and component ratings, but they may 

not understand the impact each component score has on the overall 

rating. This could lead a consumer to rely more on the overall rating 

when their individual needs may require attention to one specific 

component more than the others. For example, two nursing homes 

that both have a 4-star overall rating could have opposite quality 

measure component ratings—one with a low 1-star quality measure 

rating and the other with a high 5-star quality measure rating. (See fig. 

2).

 31Due to the way the overall rating is calculated, these nursing homes with a 4-star overall 

rating had better staffing or quality measure ratings than those with a 3-star overall rating. 
Page 18 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Figure 2: Example of Consumer Making Nursing Home Decision Based Only on the Overall Rating

Many stakeholders stated that it is difficult to distinguish between 

nursing homes with adjacent or middle ratings. In addition, some 

stakeholders expressed concern about the overall rating, with one 

explaining that often consumers make decisions based on the overall 

rating without understanding it or looking at the underlying 

components. According to CMS officials, the overall rating provides a 

summary of complex information to guide consumers—not an explicit 

report card—in as simple a way as possible. They also added that by 
Page 19 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

providing individual component ratings, consumers have the ability to 

dig deeper into the source of the overall rating.

2. Timeliness of ratings data. Each of the three rating components—

which influence the overall star rating—use a unique source of data 

that are collected from nursing homes at different time intervals.32

Specifically, the number of stars assigned to a nursing home is a 

point-in-time picture of performance based on a prior snapshot of the 

home’s performance and may not reflect a nursing home’s current 

status. (See table 2). Some stakeholders we spoke with expressed 

concern that a consumer may make a decision about a nursing home 

based on data that does not reflect current conditions in the home. 

According to CMS officials, a delay is always present due to 

administrative processes such as validating data prior to being 

posted. For example, the health inspection component data may be 

delayed due to additional information from the outcomes of revisits to 

the nursing home to check that deficiencies have been corrected. 

CMS officials and stakeholders said the Five-Star System should not 

be the only source of information a consumer uses; they both 

encourage consumers to explore additional information including 

visiting the home.

Table 2: Timeliness of Data Used in the Five-Star System for Nursing Homes

Five-Star component Frequency of data collection

Approximate administrative lag 

before posting data

Average age of 

reported dataa

Health inspectionb

Standard survey Approximately every year, but no longer 

than every 15 months

2-3 months 9 months

Complaint investigation As needed, in response to allegations of 

quality problems

2-3 months N/Ac

Staffing Approximately every year, but not longer 

than every 15 months

2-3 months 9 monthsd

Quality measures Every quarter 3 months 6-9 monthse

Source: GAO analysis of CMS interviews, documents, and data. | GAO-17-61

a

Based on national December 2015 Five-Star Ratings data.

b

The health inspection component rating calculation also takes into account the number of revisits 

needed to ensure a nursing home has corrected certain deficiencies.

c

Complaint investigations are an important complement to the standard surveys, but because the lack 

of a complaint is as important as a complaint, a meaningful average age of the data cannot be 

calculated.

 32CMS updates the Five-Star System ratings on a monthly basis, however, a particular 

home’s rating will only change if it had new data that affected the component or overall 

star rating.
Page 20 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

d

Nursing homes self-report staffing hours for a 2-week period at the time of the standard survey, so 

the staffing data date matches the date of the standard survey. As of July 2016, nursing homes are 

required to electronically submit staffing hours and related information based on payroll and other 

auditable data on at least a quarterly basis. At the time of our report, CMS did not have a timeframe 

for when the Five-Star System will incorporate the new staffing data.

e

The most frequently reported quality measure data are from the second quarter of 2015, which is six 

to nine months prior to the December 2015 cut-off point of our analysis.

3. Comparing nursing homes across states. The overall rating and 

health inspection rating do not allow consumers to compare the 

quality of homes across states, limiting the ability of the rating system 

to help consumers who live near state borders or have multistate 

options where they could place their family members. Because ratings 

are relative to other nursing homes within a state, homes that receive 

the highest and lowest ratings in their state may not be the highest or 

lowest performing homes in another state or nationally. So, a 

consumer cannot assume that a 5-star nursing home in one state 

would be rated as a 5-star home in any other state. (See fig. 3). 

Furthermore, we found that when we recalculated the star ratings 

using a national distribution rather than a state distribution, homes’

ratings often changed, sometimes dramatically. For example, about 

23 percent of nursing homes with a 1-star overall rating in December 

2015 had improved ratings when compared nationally and about 30 

percent of homes with a 5-star overall rating had decreased ratings 

when compared nationally. When looking at individual states, we 

found that the nursing homes in some states would fare better or 

worse under a national rating. Specifically, the percentage of homes 

receiving an overall 1-star rating doubled in 4 states and the 

percentage of homes receiving an overall 5-star rating doubled in 9 

states. See appendix III for additional information about the results of 

our analysis.
Page 21 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Figure 3: Example of Consumer Making Nursing Home Decision across Multiple States

According to CMS Five-Star System documentation, the rating system 

is not designed to compare nursing homes nationally. Instead, ratings 

are only comparable for homes in the same state. CMS made the 

decision to base the health inspection component on the relative 

performance of homes within the same state primarily due to variation 

across the states in the execution of the standard surveys.33 Because 

 33Standard surveys are based on federal regulations, which surveyors implement using 

national interpretive guidance and a federally specified survey process. Federal staff train 

state surveyors and oversee state performance. The federal oversight includes quality 

checks based on a 5 percent sample of the state surveys, in which federal surveyors 

either accompany state surveyors or replicate the survey within 60 days of the state and 

then compare results. These control systems are designed to optimize consistency in the 

survey process.
Page 22 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

the health inspection component most significantly contributes to the 

overall rating, this means that the overall rating also cannot be 

compared nationally. However, the addition of national ratings would 

be helpful for consumers and we have previously made 

recommendations to CMS that would help decrease survey variation 

across states.34 CMS has taken action on many of these 

recommendations.

4. Lack of consumer satisfaction information in ratings. Because the

Five-Star System does not include consumer satisfaction 

information—a key quality performance measure—the rating system 

is missing important information that could help consumers distinguish 

between high- and low- performing nursing homes. We believe 

consumer satisfaction surveys could be a more direct measure of 

nursing home satisfaction than other available measures.35 For 

example, our analysis of consumer satisfaction data shows that 

nursing homes with higher overall star ratings did not necessarily 

have higher resident satisfaction scores or fewer complaints. (See fig. 

4). Specifically, our analysis found that the Five-Star System overall 

ratings for each nursing home in two states that conduct resident 

satisfaction surveys were only slightly correlated with the percentage 

of residents that would recommend the home to other consumers—an 

indicator of consumer satisfaction included on the state surveys.36

Similarly, when analyzing complaint data for all states—a proxy for 

 34For example, in a 2008 report, we recommended that CMS, among other things, require 

its regional offices to determine when state surveyors cite deficiencies at a lower scope 

and severity level than federal surveyors and track that information. CMS concurred with 

and implemented our recommendations. In addition, in 2009, we recommended, among 

other things, that CMS clarify and revise its existing written guidance to make it more 

concise, simplify its action in the field, and reduce confusion. CMS agreed with the 

recommendation in principal, but planned to find an alternative solution. See GAO, 

Nursing Homes: Federal Monitoring Surveys Demonstrate Continued Understatement of 

Serious Care Problems and CMS Oversight Weaknesses, GAO-08-517 (Washington, 

D.C.: May 9, 2008) and GAO, Nursing Homes: Addressing the Factors Underlying 

Understatement of Serious Care Problems Requires Sustained CMS and State 

Commitment, GAO-10-70 (Washington, D.C.: Nov. 24, 2009).

35Nursing home satisfaction surveys are designed to provide information on the 

experiences of residents and their family members. Surveys may cover topics such as the 

environment of the nursing home, the care provided, the communication and respect 

provided to the resident by nursing home staff, and activities available for residents. 

36In one state, which provided the percent of residents recommending each home, we 

found a correlation of 0.24 between the percent of residents recommending the home and 

the overall rating in one state (P<.001). In the second state, which provided rankings 

based on the response ratings of residents recommending the home, we found a 

correlation of 0.27 (P<.05). 
Page 23 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

consumer satisfaction—we also found only a slight correlation 

between the total number of consumer complaints registered against 

a home in each state and the home’s overall Five-Star System 

rating.37 Many stakeholders told us that they would like to see resident 

satisfaction included in the Five-Star System. For example, one state 

stakeholder group explained that they think it is important for a 

consumer making a nursing home decision to understand how the 

administration resolves an issue with a resident when one arises. That 

type of information is not currently captured in the Five-Star System, 

but could be captured through a resident satisfaction survey, which 

could strengthen the ratings.

 37Complaint data are included in the health inspection component rating, but only 

complaints that are substantiated by the state and result in a deficiency for the nursing 

home. We looked at the total volume of complaints registered against a nursing home, 

whether substantiated or not, which can be used as a proxy for consumer satisfaction.

Nationally, we found a -0.27 correlation between the total number of consumer complaints 

registered against a nursing home in each state and the home’s overall rating (P<.0001). 

For information on the correlation for each state, see appendix IV. 
Page 24 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Figure 4: Example of Consumer Making Nursing Home Decision without Consumer Satisfaction Information

According to CMS officials, they recognize that consumer satisfaction 

is important information, but collecting the data in a consistent, 

objective way for all of the nursing homes in the country is a 

challenge. They acknowledged that some states have been able to 

overcome these implementation challenges and administer statewide 

nursing home consumer satisfaction surveys. Until consumer 

satisfaction information is included in the rating system, consumers 

will continue to make nursing home decisions without the benefit of 

this key performance measure and may not be choosing the home 

that would best meet their needs. While we recognize that gathering 

this information is challenging, CMS has done so in its hospital rating 

system. Specifically, CMS developed a hospital consumer satisfaction 

survey with assistance from HHS’s Agency for Healthcare Research 

and Quality—an agency that, among other things, focuses on quality 

measurement and includes consumer satisfaction as one of its 

National Quality Measures Clearinghouse’s clinical quality measures.
Page 25 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

In addition to the items discussed above, presentation of the Five-Star 

System does not prominently display key explanatory information that 

could help consumers better understand how to use the ratings. 

Specifically, we found that CMS does not prominently provide 

descriptions of how to understand the ratings and what consumers should 

consider when using the ratings or information on how the overall rating is 

calculated. In addition, CMS clearly discloses the date of the data used to 

assign stars for the health inspection component, but not for the staffing 

or quality measure components, and does not prominently state the 

previously discussed limitation that homes can only be compared within a 

state. For example, in order to find descriptions of how the overall rating 

is calculated, consumers must follow links that take them off of their 

nursing home search and results webpages and, as noted previously, an 

average webpage visit is less than 6 minutes. Many stakeholders we 

spoke with explained that consumers often have very little time to make a 

nursing home decision, and a few noted that it is also a stressful process, 

therefore making prominent and readily available information crucial. In 

addition, many stakeholders expressed concerns that consumers may not 

understand the ratings and how they are calculated. Further, many 

stakeholders expressed concern about the timeliness of the data, with 

some noting that consumers were generally unaware of the timing of the 

data. CMS officials described the tension between keeping the Five-Star 

System as simple as possible for consumers so that they can quickly 

understand the ratings and also providing enough information on how and 

when the ratings are calculated. Collectively, the four factors that hinder 

consumers’ ability to use the Five-Star System ratings, along with the lack 

of explanatory information provided by CMS, may limit the Five-Star 

System’s ability to meet CMS’s goal of providing consumers with an easy 

way to understand nursing home quality and make distinctions between 

high- and low- performing homes.

Nursing Home Compare and the Five-Star System seek to help 

consumers choose among nursing homes. Nursing home selection can 

be a stressful and time-sensitive process, so these are important tools 

that CMS makes available to the public. However, our review found 

opportunities for improvement in both the website and the ratings. CMS 

has given much attention to the website since its inception almost 20 

years ago. For example, the agency has put in place mechanisms for 

reviewing the website’s use and has identified numerous improvements 

that could be made. However, without a systematic process for reviewing 

options and determining priorities for improvement—currently absent from 

Conclusions
Page 26 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

their efforts—CMS is unable to ensure that the website is meeting its 

intended goal.

A key component of the website is the Five-Star System, containing 

important quality information on every nursing home so consumers can 

differentiate between them and choose those that can best meet their 

needs. Because the Five-Star System contains multiple types of 

information, compiled from different sources, and has complexities 

inherent in ratings systems, it can be challenging for consumers to fully 

understand how to take advantage of the varied information it contains. 

Additional capability and information not currently included in the rating 

system could also benefit consumers trying to differentiate between highand low- performing nursing homes—such as the ability to compare 

homes nationally and the addition of consumer satisfaction survey 

information. In addition, prominently displaying explanatory information on 

how to use the ratings, which does not require users to navigate off the 

nursing home search and results pages, could help address challenges 

consumers face when trying to understand the ratings. Absent such 

enhancements, CMS cannot ensure that the Five-Star System is fully 

meeting its stated goal of helping consumers easily understand nursing 

home quality and distinguish between high- and low- performing homes.

To strengthen CMS’s efforts to improve the usefulness of the Nursing 

Home Compare website for consumers, we recommend that the 

Administrator of CMS establish a systematic process for reviewing 

potential website improvements that includes and describes steps on how 

CMS will prioritize the implementation of potential website improvements.

To help improve the Five-Star System’s ability to enable consumers to 

understand nursing home quality and make distinctions between highand low- performing homes, we recommend that the Administrator of 

CMS take the following three actions:

• add information to the Five-Star System that allows consumers to 

compare nursing homes nationally;

• evaluate the feasibility of adding consumer satisfaction information to 

the Five-Star System; and

• develop and test with consumers introductory explanatory information 

on the Five-Star System to be prominently displayed on the home

page. Such information should explain, for example, how the overall 

rating is calculated, the importance of the component ratings, where 

Recommendations
Page 27 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

to find information on the timeliness of the data, and whether the 

ratings can be used to compare nursing homes nationally.

We provided a draft of this report to HHS for its review and comment. 

HHS provided written comments, which are reproduced in appendix V. In 

its comments, HHS described the history of the Nursing Home Compare 

website and the Five-Star System, improvements the agency has made 

to both, and concurred with three of our four recommendations. In 

particular, HHS concurred with our recommendations to establish a 

process for reviewing potential website improvements that describes how 

it will prioritize their implementation, evaluate the feasibility of adding 

consumer satisfaction information to the Five-Star System, and develop 

and test explanatory information on the Five-Star System to be displayed 

on the home page. HHS did not concur with our recommendation to add 

information to the Five-Star System that would allow consumers to 

compare nursing homes nationally. HHS indicated that because of state 

variation in the execution of standard surveys, it is difficult to compare 

homes nationally on the health inspection component. They also noted 

that the Five-Star System is just one of many factors consumers should 

use when selecting a nursing home. As we describe in this report, efforts 

have been and should continue to be made to reduce state variation in 

standard surveys. For example, CMS regional offices are tracking state 

differences in deficiency citations. We maintain that the ability for 

consumers to compare nursing homes nationally is critical to making 

nursing home decisions, especially for those consumers who live near 

state borders or have multistate options, and that our recommendation 

remains valid. 

As agreed with your office, unless you publicly announce the contents of 

this report earlier, we plan no further distribution of it until 30 days from its 

date. At that time, we will send copies to the Secretary of HHS. In 

addition, the report will be available at no charge on the GAO website at 

http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-7114 or clowersa@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

Agency Comments 

and Our Evaluation
Page 28 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

the last page of this report. GAO staff who made key contributions to this 

report are listed in appendix VI.

A. Nicole Clowers

Managing Director, Health Care
Appendix I: Additional Detail on GAO’s Data 

Analyses

Page 29 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

This appendix describes additional details of the data analyses we 

conducted to examine the Five-Star Quality Rating System (Five-Star 

System). For this examination, we analyzed data from three sources. We 

analyzed Five-Star System data from the Centers for Medicare & 

Medicaid Services (CMS). These data provide detailed rating information 

on over 15,000 nursing homes included in the Five-Star System from the 

most recent full quarter available at the time of our analysis, which ended 

December 2015. Additionally, we analyzed CMS consumer complaint 

data for a six-month period ending in December 2015. We collected and 

analyzed these data for all 50 states and Washington, D.C. Furthermore, 

we collected and analyzed 2015 nursing home resident satisfaction 

survey data from two of our four selected states that collect survey data.

We conducted the following analyses:

1. To determine the extent to which the Five-Star System provides 

consumers with information distinguishing between high- and lowperforming nursing homes, we analyzed December 2015 data from 

CMS’s publicly available Five-Star Scores and Ratings data. 

Specifically, for each state, we determined the range of scores that 

underlie each component star rating: for health inspection we used 

the weighted all cycles score, for staffing we looked at the total 

adjusted staffing score, and for quality measures we calculated the 

total quality measure score—for each nursing home’s overall rating. 

We then determined whether, within each state and for each 

component rating, the scores of the worst performing 5-star nursing 

homes overlapped with the scores of the best performing 1-star 

nursing homes.1 We also conducted this analysis for each 

combination of the star ratings.

2. To determine the timeliness (or age) of the data used for each 

component Five-Star ratings for consumers viewing the ratings in 

December 2015, we analyzed data from CMS’s Five-Star Scores and 

Ratings data for that month downloaded from CMS’s website. 

Specifically, we calculated the average age of the data for each 

component rating at that point in time. For the health inspection 

component, we analyzed the standard survey date, but did not 

analyze the complaint investigation date because a meaningful 

average age cannot be calculated.

 1

In December 2015, one state—Alaska—did not have any one-star nursing homes and 

was excluded from this analysis. 

Appendix I: Additional Detail on GAO’s Data 

Analyses
Appendix I: Additional Detail on GAO’s Data 

Analyses

Page 30 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

3. To determine the extent to which nursing homes’ ratings changed 

when compared nationally rather than compared only within each 

state, we analyzed December 2015 data from CMS’s Five-Star Score 

and Ratings data downloaded from CMS’s website. Specifically, we 

recalculated each nursing home’s health inspection and quality 

measure scores that are normally assigned based on state 

distributions so that they were based on a national distribution (new 

distribution allotments were based on CMS’s state distribution 

guidelines).2 We then recalculated each home’s overall rating using 

our new health inspection component rating, our new quality measure 

rating, and CMS’s staffing component rating.3 In addition, we 

analyzed the change in overall nursing home ratings when applying 

the methodology nationally.

4. To determine the relationship between nursing home satisfaction data 

and CMS’s Five-Star ratings, we did the following:

a. We used complaints registered against nursing homes by 

residents, families, ombudsmen, or others as a proxy 

measurement of satisfaction. Specifically, we analyzed complaint 

data recorded in CMS’s Automated Survey Processing 

Environment Complaints/Incidents Tracking System from July 1 

through December 31, 2015. We examined the last six months of 

2015 to provide a fuller picture of each nursing home’s routine 

complaint levels. For each state and nationally, we determined the 

correlation between each nursing home’s total number of 

registered consumer complaints and its overall Five-Star rating.4

b. We used the results of 2015 nursing home resident satisfaction 

surveys from two of our selected states that collect such 

information. Specifically, we focused on the responses to whether 

the resident would recommend that nursing home to others as a 

 2

CMS determines health inspection ratings by distributing the nursing homes in each state 

by their component score in the following way: (1) Top 10 percent (best health inspection 

scores) of homes receive a five-star rating; (2) Middle 70 percent of homes receive a 

rating of two-, three-, or four-stars, with an equal number (approximately 23.33 percent) 

placed in each rating category; (3) Bottom 20 percent (worst health inspection scores) of 

homes receive a one-star rating. At the time of our analysis, CMS calculated one quality 

measure score based on a state distribution—the percent of long-stay residents whose 

need for help with activities of daily living increased quality measure. 

3

CMS calculates the staffing component ratings on a national basis, so those ratings did 

not need to be recalculated. 

4

The individual nursing home is the unit of analysis. 
Appendix I: Additional Detail on GAO’s Data 

Analyses

Page 31 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

measure of satisfaction. In one state, this measure was the actual 

percentage of residents that recommended the home and in the 

other state it was the ranking of the home based on residents’

responses. For both states, we determined the correlation 

between each nursing home’s resident response on the state 

survey and its overall Five-Star rating.5 The findings from this 

analysis cannot be generalized to other states.

For all data used in these analyses, we interviewed knowledgeable 

officials and reviewed related documentation and based on these steps 

determined that the data were sufficiently reliable to explore the 

relationship between the overall rating and the component ratings, 

determine national rating distributions, assess consumer satisfaction 

information, and describe the age of the data.

 5

The individual nursing home is the unit of analysis. 
Appendix II: Nursing Home Stakeholder Views 

on Consumer Use of Nursing Home Compare

Page 32 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

We held interviews with 30 nursing home stakeholders—eight national 

stakeholders and 22 state stakeholders from four states (Rhode Island, 

Georgia, Kansas, and California) we selected based on factors such as 

variation in geographic region and size (number of nursing homes). 

These stakeholders represent a range of provider groups, consumer 

groups, government agencies, and technical experts. We selected 

organizations in each state and nationally that are relevant to nursing 

home consumers and providers. Technical experts were identified by their 

prominence in the nursing home quality research field. In addition, some 

stakeholders we interviewed identified other groups that would be 

appropriate to interview. Our interviews included a set of questions 

regarding consumer use of Nursing Home Compare. Responses to these 

questions cannot be generalized beyond the stakeholders we 

interviewed.

We found that stakeholders generally could not quantify the number of 

consumers who use Nursing Home Compare, but most speculated that 

consumers use the site “a little” to “somewhat” and a few stakeholders 

said that consumers use the website “a lot”. Some stakeholders thought 

the number of people using Nursing Home Compare was growing, and 

one stakeholder thought this was because people are generally trying to 

make more educated decisions about nursing home care. Another 

stakeholder thought this increase could also be a result of people using 

the Internet to look things up more frequently, nursing homes included.

Some stakeholders noted that use of Nursing Home Compare probably 

differs depending on whether the patient is searching for care in an urban 

or a rural setting. Specifically, they stated that they think Nursing Home 

Compare is used more frequently in urban areas, where more nursing 

home options are available, compared to rural areas where there may be 

only one home in a town. See Table 3 below for summary of stakeholder 

responses.

Appendix II: Nursing Home Stakeholder 

Views on Consumer Use of Nursing Home 

Compare
Appendix II: Nursing Home Stakeholder Views 

on Consumer Use of Nursing Home Compare

Page 33 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Table 3: Stakeholder Responses to How Much Consumers Use Nursing Home 

Compare

Number = 30

A lot Somewhat A little 

Did not 

specify/question 

not asked

How much do consumers use 

Nursing Home Compare to 

make decisions when choosing 

a nursing home?

4 12 11 5

Source: GAO summary of stakeholder interviews. | GAO-17-61

Note: In two cases, stakeholders provided more than one answer to this question based on whether 

the consumer was searching for nursing home care in a rural or urban area. These stakeholders both 

stated that consumers searching for nursing homes in rural areas use Nursing Home Compare “a 

little”. For consumers searching for nursing homes in urban areas, one stakeholder stated that they 

use the website “somewhat’” and the other stakeholder said they use the website “a lot.”

Some stakeholders stated that they believe the extent to which 

consumers use Nursing Home Compare may depend on the amount of 

time that the consumer has to research and pick a nursing home. For 

example, according to a few stakeholders, if someone’s family member is 

getting discharged from the hospital and needs to be placed in a nursing 

home immediately, the consumer is less likely to use Nursing Home 

Compare. In contrast, one stakeholder noted that if consumers are 

planning for the future and researching nursing homes before care is 

needed, then they are more likely to use Nursing Home Compare.

We found that stakeholders were split in their responses about when 

consumers would typically use Nursing Home Compare – whether as an 

initial step in beginning their nursing home search or as a way to confirm 

recommendations obtained from others. Most stakeholders stated that the 

consumer using the website is most likely a family member—usually an 

adult child or grandchild—and rarely the individual in need of a nursing 

home placement.

When asked how valuable the information provided on Nursing Home 

Compare is to consumers who are researching and choosing a nursing 

home, most stakeholders stated that it was “somewhat valuable”, and 

some said that they thought it was “very valuable”. One stakeholder said 

that the information was “of little value”. See Table 4 below for a summary 

of stakeholder responses.
Appendix II: Nursing Home Stakeholder Views 

on Consumer Use of Nursing Home Compare

Page 34 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Table 4: Stakeholder Responses to How Valuable the Information Provided on 

Nursing Home Compare is to Consumers

Number = 30

Very 

valuable

Somewhat 

valuable

Of little 

value

Did not 

specify/question 

not asked

How valuable is the 

information provided on 

Nursing Home 

Compare?

6 19 1 4

Source: GAO summary of stakeholder interviews. | GAO-17-61

Some stakeholders stated that the information on Nursing Home 

Compare is a good place to start and may help consumers narrow down 

their search, but ultimately it is not likely to be the only source of 

information. Many stakeholders agreed that in addition to conducting 

online research, consumers should also always try to visit nursing homes 

in person before a making a decision. A few stakeholders stated that 

observing a nursing home and its current residents firsthand on any given 

day provides the most valuable information when making a decision. In 

addition, stakeholders noted that consumers also obtain information 

about nursing homes through other sources—primarily through word of 

mouth from friends, family, and neighbors, and from information provided 

by primary care physicians, hospital discharge planners, and local 

ombudsmen. Many stakeholders noted that in most cases though, the 

location of a nursing home is often the main determinant of where a 

family member is placed.

Stakeholders mentioned that consumers also use third party, private 

websites, and in some states, such as California and Kansas, consumers 

may rely on websites with state-specific nursing home information. Some 

stakeholders thought that consumers used these other sources of 

information more often than Nursing Home Compare. A couple of 

stakeholders thought consumers preferred these third-party, private 

websites because some of them provide a more personalized experience 

and offer the opportunity to speak with someone on the phone, where 

CMS’s Nursing Home Compare does not provide either of those options. 

A couple other stakeholders thought that consumers may be more likely 

to use and trust Nursing Home Compare compared to these other private 

websites simply because it is a government website. A couple of 

stakeholders expressed concern about consumers using third-party, 

private websites because nursing homes may pay to be included, and so 

the website may not provide objective information on nursing home 
Appendix II: Nursing Home Stakeholder Views 

on Consumer Use of Nursing Home Compare

Page 35 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

options for consumers. Additionally, these private websites often appear 

on search engine results before Nursing Home Compare, so consumers 

may use them before seeing and using CMS’s Nursing Home Compare 

website.

Finally, stakeholders provided mixed responses regarding whether they 

suggested consumers use Nursing Home Compare when helping them 

search for nursing homes. For example, several stakeholders told us that 

they routinely referred consumers to the website, while another said that 

she would only direct consumers to Nursing Home Compare if they were 

not familiar with the area, or if did not have any time to spend on the 

nursing home search process. The stakeholder would ultimately 

recommend the consumer coming in and talking to her, and then would 

not use Nursing Home Compare at all.
Appendix III: Potential Changes to Ratings If 

National Distribution Used Rather than 

Existing State Distribution, December 2015

Page 36 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

n = number of nursing homes

GAO-calculated national overall star ratinga

Five-Star System 

overall star rating

1-star 2-stars 3-stars 4-stars 5-stars Total 

1-star 1,790 445 79 4 0 2,318

2-stars 620 1,547 753 142 7 3,069

3-stars 118 667 1,281 799 75 2,940

4-stars 7 208 833 1,970 624 3,642

5-stars 0 4 142 919 2,486 3,551

Total 2,535 2,871 3,088 3,834 3,192 15,520

Source: GAO analysis of CMS data.| GAO-17-61

Note: Highlighting indicates number of nursing homes with no change in their overall star rating.

a

We recalculated each nursing home’s health inspection and quality measure scores that are 

normally assigned based on state distributions so that they were based on a national distribution (new 

distribution allotments were based on CMS’s state distribution guidelines).

Appendix III: Potential Changes to Ratings If 

National Distribution Used Rather than 

Existing State Distribution, December 2015
Appendix IV: Correlation between Consumer 

Complaints, July through December 2015, and 

Overall Ratings, December 2015

Page 37 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

State Correlation of total number of consumer complaints with 

overall Five-Star System rating

AK -0.42

AL -0.30

AR -0.34

AZ -0.41

CA -0.30

CO -0.26

CT -0.28

DC -0.41

DE -0.04

FL -0.35

GA -0.26

HI -0.21

IA -0.46

ID -0.18

IL -0.34

IN -0.28

KS -0.31

KY -0.31

LA -0.14

MA -0.12

MD -0.40

ME -0.15

MI -0.42

MN -0.26

MO -0.38

MS -0.14

MT -0.43

NC -0.34

ND 0.07

NE -0.37

NH -0.47

NJ -0.27

NM -0.34

NV -0.15

NY -0.28

Appendix IV: Correlation between Consumer 

Complaints, July through December 2015, 

and Overall Ratings, December 2015
Appendix IV: Correlation between Consumer 

Complaints, July through December 2015, and 

Overall Ratings, December 2015

Page 38 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

State Correlation of total number of consumer complaints with 

overall Five-Star System rating

OH -0.26

OK -0.27

OR -0.17

PA -0.38

RI -0.26

SC -0.25

SD -0.32

TN -0.26

TX -0.18

UT -0.34

VA -0.22

VT -0.41

WA -0.36

WI -0.40

WV -0.30

WY -0.59

National -0.27

Source: GAO analysis of CMS data. | GAO-17-61
Appendix V: Comments from the Department 

of Health and Human Services

Page 39 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

Appendix V: Comments from the Department 

of Health and Human Services
Appendix V: Comments from the Department 

of Health and Human Services

Page 40 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System
Appendix V: Comments from the Department 

of Health and Human Services

Page 41 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System
Appendix VI: GAO Contact and Staff 

Acknowledgments

Page 42 GAO-17-61 Nursing Home Compare and Five-Star Quality Rating System

A. Nicole Clowers (202) 512-7114 or clowersa@gao.gov

In addition to the contact name above, Linda Kohn, Director, Karin 

Wallestad, Assistant Director, Kathryn Richter, Analyst-In-Charge, Amy 

Andresen, Julianne Flowers, Shannon Smith, and Brienne Tierney made 

key contributions to this report. Also contributing were Jacques Arsenault, 

Wesley Dunn, Krister Friday, Rich Lipinski, Dae Park, Vikki Porter, and 

Steven Putansu.

Appendix VI: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(100338)
The Government Accountability Office, the audit, evaluation, and investigative 

arm of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to http://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and 

distribution and depends on the number of pages in the publication and whether 

the publication is printed in color or black and white. Pricing and ordering 

information is posted on GAO’s website, http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.