DEPARTMENT OF HEALTH AND HUMAN SERVICES 
OFFICE OF INSPECTOR GENERA.L 
WASHINGTON, DC  2020 1 
MAR 622012 
TO: 	 Mary K. Wakefield, Ph.D., R:N. 
Administrator 
Health Resources and Services Administration 
FROM:  Stuart Wright. 
Deputy Inspector General 
for Evaluation and Inspections 
SUBJECT: 	 Memorandum Report:  Quality Assurance and Care Provided at 
HRSA-Funded Health Centers, OEI-09-06-00420 
This memorandum report provides information about the extent to which health centers 
maintained quality assurance programs and health center patients received primary health 
services in four areas:  preventive health exams, vaccinations, referrals for specialized 
treatment, and dental services.  It also examines the Health Resources and Services 
Administration's (HRSA) oversight of the required primary health services in health 
centers. 
SUMMARY 
The Public Health Service Act authorizes HRSA to award grants to organizations that 
provide care in medically underserved urban or rural areas or to medically underserved 
populations.  Reflecting an increased Federal investment, the number ofheaIth center 
patients grew from 10.3 million in fiscal year (FY) 2001 to 17.1 million in FY 2008. 
Although health center requirements specify which services health centers must make 
available to patients, they do not establish specific quality standards for these services. 
We collected information about the quality assurance programs for a sample of 
147 health center grantees.  To examine the required primary health services, we 
contracted with medical reviewers to assess medical records for a sample of 448 health 
center patients (from the same 147 health center grantees) who had at least 2 patient 
encounters between January 1, 2006, and December 31, 2007, including at least one 
encounter in the last 6 months of2007.  Finally, we held several meetings and calls with 
HRSA staff, including staff from the Bureau of Primary Health Care and the Office of 
Performance Review, to assess HRSA oversight activities.  We found that almost all 
health centers had quality assurance programs.  However, insufficient documentation 
prevented detailed assessments of some medical records.  Health services, generally,  Page 2  Mary K Wakefield, Ph.D., R.N. 
 
 
were appropriate for most health center patients.  During the period of our review, 
HRSAs oversight and review activities provided limited information about the extent to 
which individual health center patients received required primary health services.  HRSA 
agreed with each of our four recommendations.  We recommend that HRSA (1) specify 
what elements should be included in quality assurance programs and, in particular, 
provide more guidance about how grantees conduct their periodic assessments of 
services; (2) provide more specific guidance concerning what information is required in 
patient records at health centers; (3) provide more specificity about patients receipt of 
required primary health services; and (4) establish procedures to independently assess 
patients receipt of primary health services and the adequacy of patients records.   
 
BACKGROUND 
 
The importance of ensuring the quality of care in health centers funded by HRSA is 
growing with the increases in Federal investment in health centers and numbers of 
patients served.  In 1975, Congress authorized funding for neighborhood health centers to 
provide medically underserved communities health and social services.  The Health 
Centers Consolidation Act of 19961 amended the Public Health Service Act (the PHS 
Act)2 to merge separate authorities for community, migrant, homeless, and public 
housing health centers into the Consolidated Health Centers program.  The PHS Act 
authorizes HRSA to award grants to organizations that provide care in medically 
underserved urban or rural areas or to medically underserved populations.3  FY 2002 
marked the start of a multiyear effort to expand the health center program with a goal of 
doubling the patient population within 5 years.  Reflecting the increased Federal 
investment, the number of health center patients grew from 10.3 million in FY 2001 to 
17.1 million in FY 2008.4 
 
The Federal investment in health centers continues to increase.  Health centers received  
$2 billion through the American Recovery and Reinvestment Act of 2009 to expand health 
center capacity and support capital improvements, including the implementation of  
   
                                                 
1 P.L. 104-299. 
2 Section 330 (codified at 42 U.S.C.  254b).    
3 Pursuant to 42 U.S.C.  254b, a medically underserved population includes the population of an urban or 
rural area that has been designated by the Secretary of Health and Human Services (HHS) as an area with a 
shortage of personal health services.  It also may include a population group designated as having a 
shortage of these services, such as migratory and seasonal agricultural workers, homeless individuals, and 
residents of public housing.   
4 HHS, Fiscal Year 2011, Health Resources and Services AdministrationJustification of Estimates for the 
Appropriations Committee.  Accessed at http://hrsa.gov/about/pdf/budgetjust2011.pdf on August 4, 2010. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 3  Mary K Wakefield, Ph.D., R.N. 
 
 
electronic health records.5, 6  In FY 2010, the Patient Protection and Affordable Care Act of 
2010 (ACA) provided a total of $11 billion in new funding for community health centers 
over the 5-year period beginning in FY 2011.  Of this, $9.5 billion is for patient services 
and centers operations.  The remaining $1.5 billion is for health center construction and 
renovation.7   
 
Health Center Quality Assurance and Review Requirements 
Health centers receiving grants from HHS must comply with monitoring and reporting 
requirements established by the awarding HHS agency.8, 9  Health centers are required to 
provide patients with access to certain required primary health services and to maintain a 
quality assurance program.  Through required quality assurance programs, the grantee 
assumes the primary responsibility for ensuring quality of care. 
 
Primary Health Service Requirements.  Health center service requirements address the 
availability and provision of care in health centers.  Health center requirements fall under 
four broad areas:  need, services, management and finance, and governance.10  
Requirements related to services outline the services that health centers must offer to 
patients and to access issues, such as the availability of staff and health center hours.  
Health centers must provide certain required primary health services either directly or 
through contracts or cooperative arrangements with other health care providers.11  Although 
health center requirements specify broad categories of primary health services that health 
centers must offer to patients, they do not establish specific standards for these services, nor  
   
                                                 
5 P.L. 111-5, Division A Title VIII (Feb. 17, 2009). 
6 HHS, American Recovery and Reinvestment ActStrengthening Community Healthcare Services.  
Accessed at  http://www.hhs.gov/recovery/reports/plans/healthcentersservices.pdf on September 11, 2009.   
7 P.L. 111-148, as amended by the Health Care and Education Reconciliation Act of 2010 (P.L. 111-152); 
see, e.g.,  5601, 10503. 
8 45 CFR  74.51 and 42 U.S.C. 254b(k)(3)(I)(ii). 
9 Furthermore, pursuant to 42 U.S.C. 254b(k)(3)(I)(ii), health centers must develop procedures for 
compiling and reporting statistics relating to the costs of operations; the patterns of use of their services; the 
availability, accessibility, and acceptability of their services; and such other matters that the Secretary of 
HHS may require. 
10 These requirements are intended to ensure that (1) health centers target population or area is otherwise 
medically underserved (need); (2) required health services are available, provided by the appropriate staff, 
and provided within the context of a quality assurance program (services); (3) appropriate management, 
billing, budget, and data systems are in place (management and finance); and (4) the composition of the 
board and the conflict-of-interest policies are appropriate (governance).  These requirements are found in 
42 U.S.C.  254b, 42 CFR pt. 51c, 42 CFR  56.20156.604, and 45 CFR pt. 74.  A more detailed 
summary of health center requirements was accessed at http://bphc.hrsa.gov/about/requirements/index.html 
on July 1, 2011.  
11 42 U.S.C.  254b(a)(1). 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 4  Mary K Wakefield, Ph.D., R.N. 
 
 
do they list each individual service that patients might need.  However, health centers are 
required to assess the credentials of their health care practitioners to ensure the adequacy of 
their training and skills.12  Some of the primary health services that health centers must 
offer13 include: 
  basic health services, consisting of 
  health services related to family medicine, internal medicine, pediatrics, and 
o
obstetrics;  
  diagnostic laboratory and radiologic services; and 
o
  preventive services such as prenatal and perinatal services,  
o
well-child services, immunizations against vaccine-preventable diseases, and 
preventive dental services.  
   referrals to providers of medical services, including 
  specialty referral when medically indicated and  
o
other health-related services (including substance abuse and mental health 
services).14   
Quality Assurance Program.  Health center grantees are required to maintain a quality 
assurance program, which provides an organizational structure that supports the provision 
of high-quality patient care.15  Quality assurance programs must provide for periodic 
assessments of utilization and quality of health services.  These assessments must (1) be 
conducted by physicians or other licensed professionals under the supervision of 
physicians, (2) be based on the systematic collection and evaluation of patient records,16 
and (3) identify and document needed changes and the results of such changes.17 
   
                                                 
12 HRSA policy requires all health centers to assess the credentials of each licensed or certified health care 
practitioner.  This assessment must meet the requirement of 42 U.S.C.233(h)(2) that calls for review and 
verification of the professional credentials, references, claims history, fitness, professional review 
organization findings, and license status of its physicians and other licensed or certified health care 
practitioners.  Credentialing & Privileging of Health Center Practitioners, Policy Information Notice 
2001-6.  Accessed at http://bphc.hrsa.gov/policiesregulations/policies/pin200116.html on August 29, 2011. 
13 42 U.S.C.  254b(a)(1).  Health centers must provide these services either through the staff and 
supporting resources of the center or through contracts or cooperative arrangements. 
14 42 U.S.C.  254b (b)(1)(A). 
15 42 CFR  51c.303(c), 42 CFR 56.303(c), and 42 U.S.C. 254b(k)(3)C). 
16 Pursuant to 42 U.S.C. 254b(n), health centers are required to establish and maintain such records as the 
Secretary of HHS shall require. 
17 42 CFR  51c.303(c), 42 CFR 56.303(c), and 42 U.S.C. 254b(k)(3)C). 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 5  Mary K Wakefield, Ph.D., R.N. 
 
 
HRSA Oversight 
Grantees are required to have in place a process for compiling and reporting to HRSA 
information relating to costs, service utilization, and the availability, accessibility, and 
acceptability of [their] services.18  HRSA monitors and assesses grantees performance 
through Uniform Data System (UDS) reporting, grant reviews, and performance reviews. 
 
UDS Reporting.  HRSA requires that health center grantees annually submit a core set of 
data to HRSAs UDS.19  The UDS includes information at the level of each health center 
about patient encounters, staffing, and revenue and expenses; HRSA can use this 
information to monitor and assess grantee performance and to report on overall trends.20, 21  
In 2007 (the period covered by this review), health centers were required to report two 
measures of grant performance (number of patients served and financial stability) and at 
least one measure of clinical performance.  Each grantee selected its clinical measure from 
a standardized set of 14 clinical measures (e.g., percentage of pregnant women beginning 
prenatal care by end of first trimester).  Each of these clinical measureslisted in Appendix 
Apertains to a particular subgroup (e.g., pregnant women, patients with diabetes) within 
the general patient population typically served by health centers.  Grantees performance on 
the one selected clinical measure was examined by HRSA during onsite performance 
reviews (discussed in the section below on grant and performance reviews). 
 
HRSA has modified requirements related to reporting UDS performance measures since 
2007, the period of our review.  For 2011 UDS reporting, all health center grantees are 
required to report data on 11 standardized clinical measures, 2 additional clinical 
measures related to behavioral and oral health, and 5 standardized financial measures, as 
listed in Appendix B.22  
 
Grant and Performance Reviews.  HRSA annually assesses health center grantees 
compliance with health center requirements.23, 24  HRSA reviews grantees Health Care and 
Business Plans (plans) to evaluate their expected and actual grant performance.  In their  
                                                 
18 42 U.S.C.  254b(k)(3)(I)(ii). 
19 Ibid. 
20 Health Center Data.  Accessed at http://www.bphc.hrsa.gov/uds/ on May 6, 2010. 
21 Introduction to the 2007 Uniform Data System Reporting Manual.   Accessed at 
http://bphc.hrsa.gov/healthcenterdatastatistics/reporting/2007udsreportingmanual.pdf on February 15, 2012. 
22  Clinical and Financial Performance Measures.  Updated June 20, 2011.  Accessed at 
http://bphc.hrsa.gov/policiesregulations/performancemeasures/updatedfy2012measures.pdf on August 30, 
2011. 
23 Pursuant to 42 CFR  51c.305 and 56.305. 
24 Pursuant to 42 CFR  51c.303 and 56.303. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 6  Mary K. Wakefield, Ph.D., R.N 
 
 
plans, grantees outline their clinical and management goals (e.g., health outcomes, services 
offered, and recruitment activities) and the objectives; key action steps; expected outcomes; 
data, evaluation and measurement; and points of contact related to these goals.  HRSA 
reviews these plans to evaluate the extent to which prospective grantees will be able to meet 
program requirements and current grantees have achieved established goals.  Starting in 
2009, the required UDS performance measures were included in all grantee plans, along 
with other goals specific to individual grantees. 
 
During the period of our review, HRSAin addition to conducting grant reviewsalso 
conducted onsite performance reviews of health center grantees at least once during the 
grantees 35 year project period.25, 26  During these performance reviews, HRSA examined 
grantees progress on their performance measures including the number of patients served, 
financial stability measures, and the selected clinical measure.27  The review team and the 
grantee discussed, in particular, external and internal factors that influenced the grantees 
performance on clinical and other performance measures.  HRSA and the grantee also 
discussed options for improvement.28  The Performance Review Protocol, used during 
onsite visits, included suggested questions concerning grantees quality 
assurance/improvement plans, but it did not specify how or whether review teams were to 
collect or review information about these plans.29  HRSA stopped conducting performance 
reviews after November 2009.   After that time, onsite HRSA reviews primarily assessed 
grantees ability to comply with health center program requirements, among other issues.30  
According to HRSA staff, site visits also are used to review clinical and financial issues 
raised during quarterly calls between health center grantees and HRSA project officers.  
 
METHODOLOGY 
 
Sample Selection 
Health Center Selection.  Using HRSA data, we selected a stratified random sample of  
150 health center grantees that (1) were funded in 2005, (2) were still operating in 2006, 
and (3) had received a performance review between 2004 and 2006.31, 32  One stratum  
                                                 
25 HRSA, Performance Review Protocol, revised December 2004.  This document is no longer available 
online. 
26 The Performance Review Protocol did not establish required timeframes for reviews.  
27 HRSA, Performance Review Protocol, revised December 2004.   
28 Ibid.   
29 Ibid. 
30 Requirements established pursuant to 42 CFR  51c.303 and 56.303.  Other issues are pursuant to  
42 CFR  51c.305 and 56.305. 
31 HRSA reported 1,002 health center grantees in 2006.  2006 National Aggregate UDS Data:  Table 2.  
Accessed at http://bphc.hrsa.gov/uds/2006data/national/nationaltable2.htm on December 7, 2010. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 7  Mary K. Wakefield, Ph.D., R.N 
 
 
consisted of the population of the 75 largest health centers, based on number of patients; we 
included all of the 75 largest health centers in our sample.  The other stratum consisted of 
the remaining population of 315 health centers, from which we randomly selected 75 health 
centers.33  After excluding three of the sampled health center grantees because of ongoing 
Office of Inspector General (OIG) investigations, we sent letters and information requests 
to the remaining 147 health center grantees. 
 
Patient Selection.  We requested information about patient encounters from the  
147 sampled health center grantees to help us select a sample of patients for medical 
review.  We sent letters to each of the sampled health center grantees to request that they 
provide information about patient encounters in 2006 and 2007.  The letters requested that 
the health center grantees provide the name, identification number, and date of birth for all 
patients with at least one encounter, at any of their health center sites, in the last 6 months 
of 2007.34  For each of these patients, we also requested the number of encounters at the 
health center in 2006 and, separately, in 2007.  We received patient lists from 142 of the 
health center grantees.  We used this information to select a sample of patients who had at 
least two patient encounters between January 1, 2006, and December 31, 2007, including at 
least one encounter in the last 6 months of 2007.  From the patient lists of the 142 health 
centers, we selected a sample of 490 patients.  We collected medical records for the 
sampled patients in September 2008, so services provided through that time could have 
been included in our medical review.     
 
Medical Review of Health Center Records 
We worked with a medical review contractor that contracted with three physicians and three 
dentists, all with current and/or prior health center experience, to review the status of 
primary health services among patients in four categories.  Using available information 
about patients health status and medical history, they determined whether each patient  
   
                                                                                                                                                 
32 Approximately 41 percent of health centers operating in 2006 met these criteria.   
33 The number of patients was based on 2005 UDS data. 
34 When requesting information about patients from health center grantees, we instructed them to use the 
definition of patient encounters included the 2007 Uniform Data System Reporting Manual.  HRSAs 2007 
UDS Reporting Instructions for Section 330 Grantees defines encounters as face-to-face contacts between 
a patient and a provider who exercises independent professional judgment in the provision of services to 
the patient.  To be included as an encounter, services rendered must be documented in the patients record.  
Encounters do not include large-scale efforts such as group immunizations, screening programs, or 
communitywide programs such as health fairs.  Furthermore, the provision of laboratory tests, x-rays, 
immunizations, tuberculosis tests, and prescription refills are not counted as encounters if not accompanied 
by an additional service. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 8  Mary K. Wakefield, Ph.D., R.N 
 
 
received necessary services.35  We developed a medical review instrument,36 using 
feedback from the medical reviewers, to assess:  
  preventive health exams37 (e.g., hearing38 and vision exams, weight and blood pressure 
checks, and cholesterol screenings), 
  vaccinations (e.g., polio; influenza; pneumococcal; measles, mumps, rubella; varicella 
(chicken pox); and diphtheria, tetanus, pertussis),  
  referrals for specialized treatment (e.g., allergy, behavioral health, dermatology, HIV 
testing and/or treatment, health education, nephrology, nutritionist services, 
ophthalmology/optometry, and orthopedics), and 
  dental services (e.g., dental exams, cleanings, fillings, and x-rays).   
 
Medical reviewers made overall assessments based on both the number and potential 
consequence of individual exams or services within each type of service they determined to 
be adequate.  For example, a medical reviewer could have determined that omission of 
vision and blood pressure exams reflected inadequate overall preventive health services for 
a patient with diabetes, but not for a patient without diabetes.  Reviewers generally 
determined that services were inadequate if, based on their judgment: 
  services needed by the patient were not offered,  
  services needed by the patient were not provided in a timely manner, or 
  unnecessary services were provided to the patient.   
 
Analysis.  We conducted all analyses separately for patients in different age groups.  For 
vaccinations, we conducted separate analyses for each of the following age groups:  patients  
                                                 
35 In some cases, preventive health guidelines recommend against certain patients receiving specific 
services, such as a vaccination, given their health status or medical history.  In these cases, medical 
reviewers would have determined that the status of this service was appropriate. 
36 We developed the medical review instrument based on preventive health guidelines issued by 
organizations such as the U.S. Preventive Services Task Force, the American Academy of Pediatrics, and 
the American Diabetes Association, among other national and professional associations. 
37 The term preventive health exams is not listed as a required primary health service pursuant to  
42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR  51c.102(h), or as a supplemental health service pursuant to  
42 CFR  51c.102(j).  We use this term broadly to encompass a variety of preventive health services, 
including screenings, examinations, and laboratory tests.   
38 Some of the health services assessed by our medical reviewers, such as hearing exams for adult patients, 
are not explicitly listed as required primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 
42 CFR  51c.102(h) or supplemental health services pursuant to 42 CFR  51c.102(j).  However, we 
included these services in our review because they have been recommended in guidance issued by national 
and/or professional health care associations for the general population and/or for individuals in certain age 
groups or with particular health conditions. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 9  Mary K. Wakefield, Ph.D., R.N 
 
 
aged 018, 1949, and 50 and older.39  For each of the other service types, we conducted 
separate analyses for pediatric patients (018) and adult patients (19 and older).  The results 
of the medical review are projected to a subset of all health center patients.40  Appendix C 
lists the confidence intervals for all medical review statistics presented in this report.    
 
Limitations 
Assessments of health center services in this report are based on the professional judgment 
of medical reviewers.  By their nature, medical reviews involve some subjectivity.  
Differences in training and certification could result in differences in judgment.  We 
conducted two pretests and made modifications to the review instrument to better 
standardize the medical review process. 
 
Generally, the medical review results are based on documentation of services received or 
needed during encounters at health centers.  In some cases, health center patients may have 
received services from other health providers.  Unless documented in health center records, 
this information was not available to medical reviewers, and, therefore, not incorporated 
into medical reviewers assessments of patients receipt of services. 
 
See Appendix D for a detailed discussion of the methdology. 
 
This inspection was conducted in accordance with the Quality Standards for Inspections 
approved by the Council of the Inspectors General on Integrity and Efficiency. 
 
 
 
 
 
 
 
 
 
 
                                                 
39 The Centers for Disease Control and Prevention (CDC) issues separate Recommended Immunization 
Schedules for patients aged 06, 718, 1949, 5064, and 65 and up.  We combined all pediatric patients 
into a single group and patients aged 50 and older into a single group for analysis because of the relatively 
small number of such patients in our sample. 
40 The results are projected to patients with at least two patient encounters between January 1, 2006, and 
December 31, 2007 (including at least one in the last 6 months of 2007), at health centers that (1) were 
funded in 2005, (2) still operating in 2006, and (3) had received a performance review between 2004 and 
2006. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 10  Mary K. Wakefield, Ph.D., R.N 
 
 
RESULTS 
 
Almost All Health Centers Reported Maintaining Required Quality Assurance 
Programs  
Almost all health center grantees reported that they maintain written quality assurance 
programs.  Only 2 percent of grantees indicated that they did not maintain a written 
quality assurance program.41  Of the 141 grantees that completed surveys, 134 provided 
some documentation of their quality assurance program.  Most of these grantees 
maintained a written quality assurance plan created to describe the grantees quality 
assurance activities.  Others did not maintain a separate quality assurance plan, but 
instead outlined their quality assurance activities in other health center guidance, such as 
policy and procedures manuals, clinical guidance, quality review protocols, and risk 
management guides. 
 
Ninety-Seven Percent of Grantees Reported Conducting Required Periodic Assessments as 
Part of Their Quality Assurance Program.  Most health center grantees reported that they 
conduct periodic assessments of health center services as part of their quality assurance 
program.  Among the health center grantees that indicated they have a written quality 
assurance program, 97 percent indicated that they have conducted a periodic assessment 
(hereafter assessment) of the services provided by their health centers.42   
 
The majority of health center grantees indicated that they conduct assessments at least once 
a year.  Among the health center grantees that reported conducting assessments, 46 percent 
indicated that they conduct assessments once a year and 9 percent indicated that they 
conduct assessments every 6 months.  Other grantees noted that they conduct assessments 
monthly or quarterly, whereas some noted that they continuously assess the services 
provided to patients.  Some health center grantees reported conducting multiple assessments 
staggered throughout the year, with reviews of different types of services or patients 
assigned to specific months. 
 
 
                                                 
41 One health center responded that it did not maintain a written quality assurance program, whereas a 
second health center grantee indicated that although it did not maintain a written quality assurance 
program, it tracked some clinical data and participated in a HRSA disease collaborative.  The information 
provided by the latter grantee did not appear to indicate that the grantee maintained an organized quality 
assurance program.  We asked health center grantees to send documentation of their quality assurance 
programs with their completed surveys.   
42 If a health center grantee indicated that it did not maintain a written quality assurance program, the 
grantee was not required to respond to the rest of the questions on the survey, including those involving 
periodic assessments. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 11  Mary K. Wakefield, Ph.D., R.N 
 
 
Most health center grantees indicated that they reviewed patient medical records as a part of 
their assessments.  Among the health center grantees that reported conducting assessments, 
91 percent indicated that they review patient medical records.  Among grantees that 
reported reviewing medical records as part of their assessments: 
  82 percent select records randomly,  
  47 percent select records from certain types of patients (e.g., pediatric patients),  
  30 percent review records from all health center patients, and 
  22 percent review records of certain provider types (e.g., all dentists, providers being 
credentialed).43   
 
Among health center grantees that conducted assessments, most reported assessing 
medical records and documentation (83 percent) and quality of care (82 percent), with 
fewer assessing patient outcomes (61 percent), service trends (57 percent), and the 
medical necessity of services (35 percent).  
 
Most Quality Assurance Programs Addressed the Primary Health Services Provided by 
Health Center Grantees.  Quality assurance programs addressed primary health services 
through the implementation of clinical protocols and the measurement of patient 
outcomes.  For example, most health center grantees reported that their quality assurance 
programs specifically addressed clinical protocols for diabetes screening44 and/or care  
(89 percent) and immunizations (87 percent).  Grantees also reported addressing patient 
outcomes in their quality assurance programs, including outcomes related to diabetes 
screening and/or care (92 percent) and immunizations (81 percent).  Table 1 highlights 
the clinical protocols and patient outcomes often included in health center grantees 
quality assurance programs. 
 
 
 
 
 
 
 
                                                 
43 These percentages do not sum to 100 percent because some health center grantees reported multiple 
assessments for which they used different methods for selected patient records. 
44 Although diabetes screening is not explicitly included as a required primary health service pursuant to  
42 U.S.C. 254b(b)(1)(A)(i)(I), health centers may have developed protocols for this and other services 
based on guidance issued by professional and/or national health care associations. 
 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 12  Mary K. Wakefield, Ph.D., R.N 
 
 
 
Table 1:  Selected Clinical Protocols and Patient Outcomes Addressed in 
Health Center Grantees Quality Assurance Programs   
Percentage of Health   
Center Grantees   
Element Type  Element 
That Addressed 
 
Element 
 
Diabetes screening/care  88.7% 
 
 
Immunizations  86.9% 
 
Hypertension screening/care  81.4%   
 
Cervical cancer screening  80.3% 
 
Breast cancer screening  79.4%   
Clinical Protocols   
Colon cancer screening  79.4% 
 
 
Cholesterol screening/care  77.2% 
 
Pediatric care  76.8%   
 
Dental care  69.0% 
 
Prenatal care  63.9%   
 
Diabetes screening/care  91.9%   
 
Immunizations  81.4% 
 
Hypertension screening/care  72.8%   
Patient Outcomes   
Cervical cancer screening  68.4% 
 
Breast cancer screening  66.4%   
 
Cholesterol screening/care  66.4% 
 
 Source:  OIG analysis of survey data, 2011.   
 
 
Most quality assurance programs also addressed patients access to the primary health 
services provided by health centers.  Approximately 86 percent of grantees reported 
addressing the availability of clinical staff.  Similarly, 85 percent of grantees reported that 
their quality assurance program specifically addressed the convenience of health center 
hours, and approximately 81 percent of health center grantees reported addressing wait 
times for scheduled appointments.  Most grantees also addressed the availability of 
translation services (82 percent) and the affordability of care (77 percent).   
 
 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 13 Mary K. Wakefield, Ph.D., R.N 
 
 
Most grantees also addressed the adequacy of training and review of clinical staff in their 
quality assurance programs.  For example, 86 percent addressed continuing education for 
medical staff, 86 percent addressed the verification of credentials for health center staff 
providers, and 85 percent addressed staff education on health center policies. 
  
Although Most Health Centers Reported Addressing Medical Records As Part of 
Their Quality Assurance Programs, Medical Reviewers Were Not Able To 
Determine the Adequacy of At Least One Type of Service For 45 Percent of Patients 
Because of Insufficient Documentation 
Most grantees addressed medical records and documentation as a part of their quality 
assurance plans.  Overall, approximately 89 percent of grantees with quality assurance 
programs reported that they addressed the accuracy and completeness of patient medical 
records in their quality assurance programs.  Among all health center grantees that 
conducted assessments as part of their quality assurance programs, 83 percent reported that 
they assessed health center documentation and medical records.   
 
Although most health centers addressed medical records in their quality assurance 
programs, records for approximately 45 percent of health center patients contained 
insufficient documentation for reviewers to assess at least one type of service included in 
our review.45  For each type of service, reviewers either assessed the adequacy of services 
or indicated that they could not do so, primarily because of insufficient documentation on 
the services received by patients or patients health status.  As illustrated in Table 2, 
vaccinations often lacked sufficient documentation for reviewers to determine the 
adequacy of services.46  Approximately 44 percent of health center patients records 
lacked sufficient documentation for reviewers to determine the overall adequacy of 
vaccinations.  Although documentation of vaccinations was sufficient for most pediatric 
patients, it was frequently missing for adults.    
 
 
 
 
 
 
   
                                                 
45 Among the 134 grantees that provided documentation of their quality assurance programs, 95 provided a 
document that was dated from 2007 or earlier, whereas 24 provided a document dated from 2008.  The 
documentation provided by the remaining grantees was undated or contained multiple dates.  As a result, 
the documentation available in some patients medical records could have predated specific documentation-
related guidance included in the quality assurance programs that we reviewed. 
46 If reviewers lacked sufficient information, they did not make determinations about the adequacy of care.   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 14  Mary K. Wakefield, Ph.D., R.N 
 
 
Table 2:  Availability of Documentation in Patients Records 
Percentage of Patients 
With Insufficient 
Type of Service  Patient Age 
Documentation for 
Type of Service 
Preventive health exams  All patients (n=399)  5.3% 
All patients (n=405)  44.0% 
  018 (n=125)  17.3% 
Vaccinations 
  1949 (n=163)  62.5% 
  50 and older (n=117)  46.6% 
Referrals for specialized treatment  All patients (n=407)  2.5% 
Dental services  All dental patients (n=91)  19.0%* 
Source:  OIG analysis of medical review data, 2011. 
 * Among patients who received at least one dental service. 
 
Primary Health Services, Generally, Were Appropriate For Most Health Center 
Patients 
Medical reviewers assessed four types of primary health services among health center 
patients:  preventive health exams, vaccinations, referrals for specialized treatment, and 
dental services.  Health center requirements specify broad categories of primary health 
services that health centers must provide to patients.  However, the requirements do not 
establish specific standards for these services, nor do they list each individual service that 
might be needed by patients.  Therefore, we relied on the professional judgment of 
medical reviewers to assess each type of service.  For each type of service, medical 
reviewers first assessed individual exams and services, such as vision exams, polio 
vaccinations, referrals for allergy treatment, and dental fillings.47, 48  After assessing the  
   
                                                 
47 When assessing each of the individual preventive health exams, vaccinations, referrals, and dental 
services on the medical review instrument, reviewers determined whether care was appropriate, 
inappropriate or, because of the patients age or gender, not applicable.  Inappropriate services were 
those that, in reviewers judgment, were inadequate based on the patients history and health status.  
Reviewers also indicated whether they could not determine the adequacy of individual services, generally 
because documentation in the patients record was incomplete.  If reviewers lacked sufficient information, 
they did not make determinations about the adequacy of care.    
48 Some of the health services assessed by our medical reviewers are not explicitly listed as required 
primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR  51c.102(h) or 
supplemental health services pursuant to 42 CFR  51c.102(j).  However, we included these services in our 
review because they have been recommended in guidance issued by national and/or professional health 
care associations for the general population or for individuals in certain age groups or with particular health 
conditions. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 15  Mary K. Wakefield, Ph.D., R.N 
 
 
individual exams and services, reviewers made an overall assessment of the adequacy of 
each type of service.  Reviewers determined whether services overall in each area were 
adequate or inadequate, or indicated that they could not determine the adequacy of 
services.  Reviewers determination that services were adequate did not necessarily 
indicate that the health center provided these services to the patient, but rather reviewers 
judgment that, based on available documentation, the patient had, at the time of the 
review, received care that was consistent with established guidelines and did not fail to 
receive a necessary service.  Table 3 summarizes medical reviewers overall 
determinations of the adequacy of the four types of primary health services received by 
health center patients.   
 
 Table 3:  Overall Medical Reviewer Determinations of Four Service        
 
 Types 
 
Percentage Determined  Percentage Determined 
Service Type   
Adequate  Inadequate 
 
Preventive health exams  68.4%  26.4% 
 
Vaccinations  36.8%  19.2%   
 
Referrals  86.1%  11.4% 
 
Dental services  74.2%  6.8% 
 
 Source:  OIG analysis of medical record review, 2011.   
 
If reviewers determined that either overall or individual services were inadequate, they 
indicated why.  In most but not all such cases, medical reviewers indicated that patients 
did not receive necessary services.  For a small number of cases, medical reviewers 
determined that services were inadequate because necessary services were not received in 
a timely manner or because patients received medically unnecessary services.  The 
estimates presented in this finding represent the percentage of patients (among all patients 
whose records were reviewed) that, according to medical reviewers, did not receive 
necessary services.  
 
Medical Reviewers Determined That Approximately 25 Percent of Health Center Patients 
Did Not Receive Necessary Preventive Health Exams.  Medical reviewers determined that 
overall preventive health services were inadequate for approximately one in four patients 
because patients did not receive needed preventive health exams.49, 50  Specifically, in the 
professional judgment of medical reviewers, approximately 29 percent of adult patients and  
                                                 
49 These cases represent most, but not all, of the preventive services that reviewers determined to be 
inadequate. 
50 Based on the documentation included in their records. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 16  Mary K. Wakefield, Ph.D., R.N 
 
 
17 percent of pediatric patients did not receive necessary preventive health exams.  Table 4 
shows some of the individual preventive health exams not received by health center 
patients.51  In particular, reviewers reported that approximately 21 percent of adults did not 
receive necessary colon cancer screening exams.  Similarly, approximately 20 percent of 
adults did not receive necessary cholesterol screening exams.  Among pediatric patients, 
approximately 16 percent did not receive necessary hearing exams and 15 percent did not 
receive necessary developmental screenings.   
 
 
Table 4:  Medical Reviewer Assessments of Necessary Preventive Health 
Exams Not Received 
 
Percentage That Did Not 
Preventive Health Exam  Patient Age Group   
Receive Necessary Exams 
Colon screening (i.e., fecal   
occult blood, sigmoidoscopy, or  Adult   21.3%   
colonoscopy)   
Cholesterol screening  Adult  20.3%    
 
Blood sugar level screening  Adult   9.6% 
 
Vision   Adult  7.5%   
 
Hearing   Pediatric  15.6% 
 
Developmental screening   Pediatric  14.7% 
 
Blood pressure   Pediatric  10.7%   
 
Vision   Pediatric  11.5% 
 
 Source:  OIG analysis of medical record review, 2011. 
 
 
Approximately 17 Percent of Patients Did Not Receive Necessary CDC-Recommended 
Vaccinations.  Medical reviewers reported that vaccinations were inadequate for 
approximately 17 percent of patients because they did not receive necessary vaccinations.  
Overall, vaccinations for approximately 24 percent of patients aged 50 and older, 17 percent 
of patients aged 1949, and 10 percent of patients aged 018 were inadequate because the 
patients did not receive necessary vaccinations.  Table 5 displays some of the individual 
vaccinations missing for patients in the different age groups.52   
 
   
                                                 
51 Reviewers assessed the adequacy of 13 individual preventive health exams.  The individual exams 
assessed for adult and pediatric patients varied slightly. 
52 Reviewers assessed the appropriateness of 12 individual vaccinations included on CDCs recommended 
schedules.  The individual vaccinations assessed varied slightly for patients of different ages. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 17  Mary K. Wakefield, Ph.D., R.N 
 
Table 5:  Medical Reviewer Assessments of Necessary Vaccinations Not    
Received      
Percentage That Did Not Receive   
Vaccination  Patient Age 
Necessary Vaccinations   
Influenza53  50 and older  25.0%   
 
Pneumococcal  50 and older  19.1% 
 
Diphtheria, tetanus, pertussis  50 and older  16.1% 
 
Influenza  1949  14.9%   
 
Diphtheria, tetanus, pertussis  1949  14.0% 
 
Hepatitis B  1949  4.4%   
Influenza  018    30.2%   
 
Varicella (chicken pox)  018  12.0% 
 
Hepatitis A  018  8.1%   
 Source:  OIG analysis of medical record review, 2011.   
 
Approximately 10 percent of health center patients did not receive necessary referrals.  
Medical reviewers determined that referrals for specialized treatment were inadequate for 
approximately 10 percent of patients because they did not receive needed referrals.  
Medical reviewers did not assess whether patients actually received services from the 
appropriate specialists, but rather whether a necessary referral to the appropriate specialist 
was documented in patients records.  Approximately 11 percent of adult patients and  
8 percent of pediatric patients did not receive needed referrals.54  Table 6 displays some of 
the referrals that were not provided to adult health center patients.  For example, almost  
8 percent of adult patients did not receive needed referrals for HIV/AIDS testing and/or 
treatment.  Among pediatric patients, referrals for necessary laboratory tests (3 percent) and 
hearing services (2 percent) were not received for a relatively small percentage of patients. 
  
                                                 
53 It is important to note that vaccination rates in the general population are not always consistent with the 
CDC-recommended schedule.  For example, although we cannot make a direct comparison because our 
review assessed the adequacy of vaccinations (as opposed to actual vaccination rates), in 2007, only  
36 percent of adults aged 5064 and 66 percent of adults aged 65 and older reported receiving an influenza 
vaccination.  Self-reported influenza vaccination coverage trends 19892007, National Health Interview 
Survey.  Accessed online at 
http://www.cdc.gov/flu/professionals/vaccination/pdf/NHIS89_07fluvaxtrendtab.pdf on September 8, 2010.  
Similarly, approximately 17 percent of adults aged 5064 and 58 percent of adults aged 65 and older 
reported receiving a pneumococcal vaccination in 2007.  Self-reported pneumococcal vaccination coverage 
trends 19892007, National Health Interview Survey.  Accessed online at 
http://www.cdc.gov/flu/professionals/vaccination/pdf/NHIS89_07ppvvaxtrendtab.pdf on September 8, 
2010. 
54 Reviewers assessed the appropriateness of referrals for 29 types of specialized care. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 18  Mary K. Wakefield, Ph.D., R.N 
 
 
 
Table 6:  Medical Reviewer Assessments of Necessary Referrals Not 
Received by Adult Patients     
 
Percentage That Did Not Receive 
Referral    
Necessary Referrals 
 
HIV/AIDS testing and/or treatment  8.0%    
Laboratory tests  5.1%    
 
Ophthalmology/optometry services  4.9%  
 
Nutritionist services  4.6%    
 
Health education  4.4%  
 
 Source:  OIG analysis of medical record review, 2011. 
 
 
Approximately 2 Percent of Dental Patients Did Not Receive Necessary Dental Services.  
According to medical reviewers assessments, only 2 percent of dental patients received 
inadequate overall dental care because they did not receive necessary services.  In our 
sample of 32 pediatric patients with health center dental records, medical reviewers 
determined that only one patient (3 percent) did not receive necessary dental services.  
Similarly, only one adult patient in our sample of 59 adult dental patients (2 percent) did not 
receive necessary dental services.   
 
HRSA Review Activities Provided Limited Information About the Extent to Which 
Individual Health Center Patients Received Required Primary Health Services 
HRSA oversight in 2007, the period covered by our review, was intended to collect core 
clinical data, ensure grantee compliance with program requirements, and support grantee 
efforts to improve their programs; it was not intended to determine the extent to which 
health center patients received primary health services. 
 
HRSA Reviews Did Not Assess the Required Primary Health Services Received by Health 
Center Patients.  According to HRSA staff, the agency did not independently collect and/or 
review patient medical records or other health information to assess and/or verify the 
services received by individual health center patients.  HRSA onsite reviews assessed 
grantees success at achieving their clinical and management goals, and determined whether 
technical assistance was necessary.  Through its UDS, HRSA received clinical information 
about a subset of health center services and/or patients, but this information did not 
 
 
 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 19  Mary K. Wakefield, Ph.D., R.N 
 
 
necessarily reflect the range of care received by most health center patients.  Even with 
the most recent update to UDS reporting, performance measures might not provide 
HRSA with comprehensive information about individual health center patients care. 
 
The Lack of Detailed Requirements for Quality Assurance Programs May Have Limited 
Their Usefulness in Ensuring That Patients Receive Necessary Primary Health Services.   
Requirements and reviews for quality assurance programs were broad and did not always 
ensure that patients receive necessary services.  Approximately 94 percent of grantees55 
reported that HRSA staff reviewed their overall quality assurance program, and  
86 percent56 reported that HRSA staff reviewed their assessments.  However, beyond the 
requirement that grantees must develop and maintain quality assurance programs that 
provide for assessments of services, the requirements provided few specifics about the 
issues to be addressed in quality assurance programs.  We collected quality assurance 
plans that ranged from four to several hundred pages.  In addition, requirements specified 
that grantees must conduct assessments of services, but did not specify the frequency of 
these assessments, the services to be included, or the review methods to be used.  Most 
grantees reported that they assessed medical record documentation and health services as 
a part of their quality assurance programs and internal assessments.  However, medical 
reviewers often could not determine the adequacy of services because of insufficient 
documentation and, in other cases, judged that patients did not receive necessary services. 
 
RECOMMENDATIONS 
 
Health centers provide a range of health services to medically underserved communities.  
Ninety-seven percent of health center grantees maintained required quality assurance 
programs and conducted internal assessments of their services.  Although 89 percent of 
grantees indicated that their quality assurance programs addressed the accuracy and 
completeness of medical records, medical reviewers determined that 45 percent of patient 
records did not contain sufficient information for them to assess at least one type of service.  
Primary health services, generally, were appropriate for most health center patients.  During 
the period of our review, HRSAs oversight was intended to collect core clinical data, 
ensure grantee compliance with program requirements, and support grantee efforts to  
improve their programs; it was not intended to directly assess the receipt of required 
primary health services by health center patients, and provided limited information about  
 
 
 
                                                 
55 Among grantees that reported maintaining a quality assurance program. 
56 Among grantees that reported conducting assessments. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 20  Mary K. Wakefield, Ph.D., R.N 
 
 
the services actually received by health center patients.  Therefore, we recommend that 
HRSA: 
 
Provide More Specific Guidance About What Grantees Should Address in 
Their Quality Assurance Programs and How They Should Conduct Their 
Periodic Assessments  
Health center grantees must maintain required quality assurance programs and, through 
these quality assurance programs, periodically assess their services.  However, 
requirements provide little detail about what quality assurance programs and assessments 
must address.  HRSA should specify what elements should be included in quality 
assurance programs and, in particular, provide more guidance about how grantees 
conduct their periodic assessments of services.  For example, HRSA could require all 
grantees to review records for a random sample of patients to assess the extent to which 
patients received required primary services as a part of their assessments.  In addition, 
HRSA could require health centers to assess the adequacy of medical records.   
 
Provide More Specific Guidance Concerning What Information is Required 
in Patient Records at Health Centers 
Incomplete medical records limit health providers ability to determine what health services 
their patients need or receive.  HRSA should provide explicit guidance concerning the 
documentation that must be maintained in all health center records.  HRSA could develop 
guidance based on standards issued by national or professional associations or require 
health center grantees to follow established standards and offer technical assistance, as 
appropriate.  
 
HRSA also should continue to support health center grantees efforts to implement 
electronic record systems.  Medical reviewers noted that electronic health records were 
more organized and legible than handwritten paper records.  HRSAs support of health 
center grantees transition to electronic health records, through the American Recovery and 
Reinvestment Act and other agency funding, may improve health centers ability to 
maintain complete medical record documentation.   
 
Provide More Specificity About the Required Primary Health Services  
Although health center requirements indicate the broad primary health services that 
grantees must provide, they do not include specific quality standards for these services.  
HRSA should provide more guidance about the services that should be received by health 
center patients.  To ensure that patients receive necessary services, HRSA could require 
explicitly that health center grantees identify specific preventive health guidance or  
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 21  Mary K. Wakefield, Ph.D., R.N 
 
 
recommendations that their health care practitioners will follow.  HRSA could recommend 
to grantees specific guidance issued by organizations and agencies such as the U.S. 
Preventive Services Task Force, CDC, or other national and/or professional health 
associations. 
 
Establish Procedures To Independently Assess Patients Receipt of Primary 
Health Services and the Adequacy of Patients Records 
Although health center grantees maintain the primary responsibility for the care received by 
their patients, HRSA oversight should provide the agency with more information about the 
extent to which health center patients receive required primary health services.  UDS 
reporting does not provide information about the services received by most health center 
patients.  HRSA should develop a review process that would allow it to assess the primary 
health services received by a cross-section of health center patients.  This process also 
should assess the adequacy of medical record documentation.  For example, HRSA could 
collect and review a representative sample of patient records to determine the extent to 
which patients received adequate care.  In addition, HRSA could examine the extent to 
which expected documentation, such as vaccination records and patient medical histories, is 
maintained in a sample of patient records.  During reviews, HRSA also could determine the 
extent to which grantees implemented elements in their quality assurance programs, in 
particular those related to patients receipt of primary health services and the adequacy of 
medical record documentation.       
 
AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE 
 
In its written comments on our draft, HRSA highlighted the achievements of health centers 
in providing quality primary health care to underserved communities and its own increasing 
focus on quality improvement.  Although our review included health records and services 
through 2007, HRSA cited key initiatives and activities implemented since that time, 
including emphasizing evidence-based quality improvement practices and evaluating the 
quality and impact of health care services.  HRSA also cited recent research highlights 
showing favorable statistics for health center patients in comparison to other populations.   
 
In response to our first recommendationto provide more specific guidance about what 
grantees should address in their quality assurance programsHRSA stated that it is in the 
process of developing specific policy guidance on quality improvement and quality 
assurance plans including periodic assessments, in accordance with the statutory 
requirements.  
 
  
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 22  Mary K. Wakefield, Ph.D., R.N 
 
 
In response to our second recommendationto provide more specific guidance about what 
information is required in patient records at health centersHRSA stated that it fully 
supports the adoption of electronic health records that meet the Meaningful Use criteria  
established by HHSs Office of the National Coordinator for Health Information 
Technology and the Centers for Medicare & Medicaid Services.  OIG supports HRSAs 
efforts to expand the adoption of electronic health records in health centers; these efforts  
will help to establish clear expectations for what information should be documented in 
patient records.  We anticipate receiving documentation from HRSA regarding its 2010 
modification to the UDS to require all health centers to report their specific electronic 
health record activities. 
 
In response to our third recommendationto provide more specificity about the required 
primary health servicesHRSA said that national variation in health centers scope of 
practice and the needs of the target populations limit its ability to specify and/or require 
certain procedures or more detailed aspects of core primary and preventive health services.  
HRSA stated that it will continue to endorse the most current nationally recognized,  
evidence-based clinical practice guidelines and protocols, such as those set forth by the U.S. 
Preventive Services Task Force, to offer as a guide to providers for delivering primary 
health services.  OIG supports HRSAs endorsement of evidence-based practice guidelines 
and encourages the agency to require health centers to include a reference to nationally 
recognized clinical practice protocols in the centers primary health services policies for 
providers.  
 
Finally, in response to our fourth recommendationto establish procedures to 
independently assess patients receipt of primary health services and the adequacy of patient 
recordsHRSA stated that it is in the process of updating its site-visit protocol to add a 
focus on such reviews.  In particular, future site visits will involve reviews/audits of a 
sample of patient records. 
   
For the full text of HRSAs comments, see Appendix E. 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 23  Mary K. Wakefield, Ph.D., R.N  
 
 
APPENDIX A 
Clinical Measures for 2007 Health Center Grantee Performance Reviews57 
 
1.  Perinatal (Effort):  Percentage of pregnant women beginning prenatal care by end of 
first trimester.    
2.  Perinatal (Effort):  Percentage of women who had a postpartum visit within 42 days 
after delivery. 
3.  Perinatal (Effort):  Percentage of newborns who had a followup visit within 14 days 
of birth.  
4.  Child (Effort):  Percentage of children 5 through 18 years of age diagnosed with 
persistent asthma who were prescribed appropriate medications.  
5.  Adolescent (Effort):  Percentage of adolescents who had both a documented 
Behavioral Risk Assessment and a related counseling visit.  
6.  Adult (Outcome):  Percentage of adults diagnosed with an abnormal lipid profile 
whose levels are under control.  
7.  Adult (Effort):  Percentage of adults with abnormal breast, cervical, or colon cancer 
screening results for which referral and/or treatment has been initiated within 30 days 
of test completion.  
8.  Adult (Outcome):  Percentage of adults diagnosed with hypertension whose blood 
pressure is under control.  
9.  Adult (Effort):  Percentage of adults who are tobacco users who have received 
counseling and/or a treatment plan to quit.  
10.  Adult (Outcome):  Percentage of adults with type 1 or type 2 diabetes whose last 
glycated hemoglobin (HbA1c) level was less than 7 percent.  
11.  Oral (Outcome):  Percentage of dental patients with a comprehensive oral exam and 
a treatment plan completed within a 12-month period.  
12.  Mental (Effort & Outcome):  a. Percentage of all adult patients with screening for 
depression (Effort) or b. Percentage of patients with major depression who have a 
50-percent decrease in depression severity score (Outcome).  
13. All Life Cycles (Outcome):  Percentage of child, adolescent, adult, geriatric, or  
HIV/AIDS patients with appropriate immunizations. 
 
                                                 
57 Health Resources and Services Administration guidance.  This information is no longer available online. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 24  Mary K. Wakefield, Ph.D., R.N  
 
 
14.  All Life Cycles (Effort):  Percentage of child/adolescent patients or adult patients 
with a Body Mass Index indicating overweight or obesity who have received healthy 
weight counseling and/or other related interventions or treatment.  
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 25  Mary K. Wakefield, Ph.D., R.N  
 
APPENDIX B 
2011 Clinical and Financial Performance Measures58 
 
 
Outreach/Quality of Care 
1.  Percentage of pregnant women beginning prenatal care in the first trimester 
2.  Percentage of children with 2nd birthday during the measurement year with 
appropriate immunizations 
3.  Percentage of women 2164 years of age who received one or more tests to screen for 
cervical cancer   
4.  Percentage of patients age 2 to 17 years who had a visit during the current year and 
who had Body Mass Index (BMI) Percentile documentation, counseling for nutrition, 
and counseling for physical activity during the measurement year 
5.  Percentage of patients age 18 years or older who had their BMI calculated at the last 
visit or within the last 6 months and, if they were overweight or underweight, had a 
followup plan documented 
6.  Percentage of patients age 18 years and older who were queried about tobacco use 
one or more times within 24 months 
7.  Percentage of patients age 18 years and older who are users of tobacco and who 
received (charted) advice to quit smoking or tobacco use 
8.  Percentage of patients age 5 to 40 years with a diagnosis of persistent asthma  
(mild, moderate, or severe) who were prescribed either the preferred long-term 
control medication or an acceptable alternative pharmacological therapy during the 
current year 
 
Health Outcomes/Disparities 
1.  Percentage of diabetic patients whose HbA1c levels are less than 7 percent, less than  
8 percent, less than or equal to 9 percent, or greater than 9 percent 
2.  Percentage of adult patients with diagnosed hypertension whose most recent blood 
pressure was less than 140/90 
3.  Percentage of births less than 2,500 grams to health center patients 
 
                                                 
58 Clinical and financial performance measures accessed online at 
http://bphc.hrsa.gov/policiesregulations/performancemeasures/index.html on February 22, 2012. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 26  Mary K. Wakefield, Ph.D., R.N  
 
 
Additional Clinical Measures 
In addition to the above Uniform Data System clinical measures, health centers must 
include  
1.  One Behavioral Health (e.g., Mental Health or Substance Abuse) and  
2.  One Oral Health performance measure of their choice in the Health Care Plan 
 
Financial Viability/Costs 
1.  Total cost per patient   
2.  Medical cost per encounter  
3.  Change in net assets to expenses ratio  
4.  Working capital to monthly expense ratio 
5.  Long term debt to equity ratio   
 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 27  Mary K. Wakefield, Ph.D., R.N  
 
 
APPENDIX C 
 
Table C-1:  Confidence IntervalsQuality Assurance Programs and Periodic Assessments 
Sample  Point  95-Percent  
Estimate Description 
Size  Estimate  Confidence Interval 
Percentage of grantees that reported not maintaining a written quality 
141  2.3%  0.77.5% 
assurance program 
Percentage of grantees that reported conducting periodic assessments 
139  97.1%  92.5%98.9% 
as part of their quality assurance program 
Of grantees that conducted assessments, percentage that conduct 
135  45.5%  37.2%54.1% 
assessments once a year 
Of grantees that conducted assessments, percentage that conduct 
135  9.2%  5.4%15.4% 
assessments every 6 months 
Of grantees that conducted assessments, percentage that select patient  
135  91.4%  85.7%95.0% 
medical records as part of the assessment 
Of grantees that reviewed patient medical records, percentage that select  
121  82.1%  74.2%88.0% 
records randomly 
Of grantees that reviewed patient medical records, percentage that select  
121  46.8%  38.1%55.8% 
records from certain types of patients 
Of grantees that reviewed patient medical records, percentage that review 
121  30.2%  22.7%39.0% 
records from all health center patients 
Of grantees that reviewed patient medical records, percentage that review  
121  22.1%  15.6%30.3% 
records from certain provider types 
Of grantees that conducted assessments, percentage that assessed health  
135  82.5%  75.2%87.9% 
center documentation/medical records 
continued on next page 
 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 28  Mary K. Wakefield, Ph.D., R.N 
 
 
   
Table C-1:  Confidence IntervalsQuality Assurance Programs and Periodic Assessments     
(Continued) 
Point  95-Percent  
Estimate Description  Sample Size 
Estimate  Confidence Interval 
Of grantees that conducted assessments, percentage of grantees that 
135  82.1%  74.3%87.9% 
assessed quality of care 
Of grantees that conducted assessments, percentage of grantees that 
135  61.2%  52.5%69.2% 
assessed patient outcomes 
Of grantees that conducted assessments, percentage of grantees that  
135  56.7%  48.0%64.9% 
assessed service trends 
Of grantees that conducted assessments, percentage of grantees that 
135  35.0%    27.5%43.4%   
assessed the medical necessity of care 
Source:  Office of Inspector General (OIG) analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 29  Mary K. Wakefield, Ph.D., R.N 
 
   
Table C-2:  Confidence IntervalsOf Grantees With Quality Assurance Programs, Percentage That 
Addressed Each Element 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Clinical protocols for diabetes screening and/or care  139  88.7%  82.193.1% 
Clinical protocols for immunizations  139  86.9%  79.891.8% 
Patient outcomes related to diabetes screening/care  139  91.9%  85.795.5% 
Patient outcomes related to immunizations  139  81.4%  73.487.4% 
Clinical protocols for hypertension screening/care  139  81.4%  73.787.3% 
Clinical protocols for cervical cancer screening  139  80.3%  72.486.3% 
Clinical protocols for breast cancer screening  139  79.4%  71.485.6% 
Clinical protocols for colon cancer screening  139  79.4%  71.485.6% 
Clinical protocols for cholesterol screening/care   139  77.2%  69.383.5% 
continued on next page 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 30  Mary K. Wakefield, Ph.D., R.N 
 
 
   
Table C-2:  Confidence IntervalsOf Grantees With Quality Assurance Programs, Percentage That 
Addressed Each Element (Continued) 
 
Sample  Point  95-Percent  
Estimate Description 
Size  Estimate  Confidence Interval   
Clinical protocols for pediatric care  139  76.8%  68.883.3% 
Clinical protocols for dental care  139  69.0%  60.576.3% 
Clinical protocols for prenatal care  139  63.9%  55.371.8% 
Patient outcomes related to hypertension screening/care  139  72.8%  64.579.7% 
Patient outcomes related to cervical cancer screening  139  68.4%  59.975.8% 
Patient outcomes related to breast cancer screening  139  66.4%  57.973.9% 
Patient outcomes related to cholesterol screening/care  139  66.4%  57.973.9% 
Availability of clinical staff  139  85.8%  78.890.8% 
Convenience of health center hours  139  84.9%  77.790.1% 
Wait times for scheduled appointments  139  80.8%  72.986.8% 
continued on next page 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 31  Mary K. Wakefield, Ph.D., R.N 
 
 
Table C-2:  Confidence IntervalsOf Grantees With Quality Assurance Programs, Percentage That  
Addressed Each Element (Continued) 
  Sample  Point  95-Percent  
Estimate Description 
Size  Estimate  Confidence Interval 
 
Availability of translation services  139  81.7%  74.087.5% 
 
Affordability of care  139  77.2%  69.383.5% 
Continuing education for medical staff  139  86.4%  79.391.3% 
Verification of credentials for health center providers  139  86.4%  79.391.3% 
Staff education on health center policies  139  84.9%  77.790.1% 
Patient satisfaction  139  95.1%  89.397.8% 
Internal quality-of-care reviewers  139  92.4%  86.196.0% 
Resolution of patient complaints  139  92.2%  85.995.8% 
Source:  OIG analysis of medical record review, 2011. 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 32  Mary K. Wakefield, Ph.D., R.N 
 
 
Table C-3:  Confidence IntervalsMedical Record Documentation 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of patient records that lacked sufficient documentation  
for reviewers to determine the overall appropriateness of at least one type of  436  44.5%  39.149.9% 
care 
Of grantees with quality assurance programs, percentage that addressed the 
139  89.3%  82.693.6% 
accuracy and completeness of patient medical records 
Of grantees that conducted assessments, percentage that assessed health  
135  82.5%  75.287.9% 
center documentation/medical records 
Percentage of patient records that lacked sufficient documentation of 
405  44.0%  38.349.8% 
vaccinations 
Percentage of patient records that lacked documentation for preventive health 
399  5.3%  2.411.0% 
care 
Among patients aged 018, percentage of records that lacked sufficient 
125  17.3%   11.026.0% 
documentation of vaccinations 
Among patients aged 1949, percentage of records that lacked sufficient 
163  62.5%  53.171.0% 
documentation of vaccinations 
Among patients aged 50 and over, percentage of records that lacked sufficient 
117   46.6%   36.157.4% 
documentation of vaccinations 
Percentage of patient records that lacked sufficient documentation of referrals  407  2.5%  1.25.3% 
Among dental patients, percentage of records that lacked sufficient 
91  19.0%  11.130.5%  
documentation of dental services 
Source:  OIG analysis of medical record review, 2011. 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 33  Mary K. Wakefield, Ph.D., R.N    
Table C-4:  Confidence IntervalsMedical Reviewer Assessments of the Four Service Types 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of patients with adequate preventive health exams 
399  68.4%  62.373.9% 
Percentage of patients with inadequate preventive health exams 
399  26.4%  21.432.0% 
Percentage of patients with adequate vaccinations 
405  36.8%  31.642.4% 
Percentage of patients with inadequate vaccinations 
405  19.2%  15.224.0% 
Percentage of patients with adequate referrals 
407  86.1%  81.789.6% 
Percentage of patients with inadequate referrals 
407  11.4%  8.115.7% 
Percentage of dental patients with adequate dental services 
91  74.2%  61.783.7% 
Percentage of dental patients with inadequate dental services 
91  6.8%  2.517.0% 
Source:  OIG analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 34  Mary K. Wakefield, Ph.D., R.N 
   
Table C-5:  Confidence IntervalsMedical Reviewer Assessments of Preventive Health Exams 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of patients who did not receive necessary preventive health  
399  24.9%  20.130.4% 
exams 
Percentage of adult patients who did not receive necessary preventive health 
274  28.7%  22.635.8% 
exams 
Percentage of pediatric patients who did not receive necessary preventive 
125  16.7%  10.525.4% 
health exams 
Percentage of adult patients who did not receive necessary colon cancer 
279  21.3%  16.626.9% 
screening exams 
Percentage of adult patients who did not receive necessary cholesterol 
279  20.3%  15.426.4% 
screening exams 
Percentage of pediatric patients who did not receive necessary hearing  
124  15.6%  9.923.7% 
exams 
Percentage of pediatric patients who did not receive necessary developmental 
123  14.7%  8.923.2% 
screenings 
Percentage of adult patients who did not receive necessary blood sugar level 
280  9.6%  6.314.3% 
screening 
Percentage of adult patients who did not receive necessary vision exams  281  7.5%  4.612.2% 
Percentage of pediatric patients who did not receive necessary blood pressure 
126  10.7%  5.719.0% 
exams 
Percentage of pediatric patients who did not receive necessary vision exams  124  11.5%  6.619.2% 
Source:  OIG analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 35  Mary K. Wakefield, Ph.D., R.N 
 
   
Table C-6:  Confidence IntervalsMedical Reviewer Assessments of Vaccinations 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of patients who received inadequate vaccinations  405  16.7%  12.921.5% 
Percentage of patients aged 50 and older who received inadequate 
117  23.9%  16.034.3% 
vaccinations 
Percentage of patients aged 1949 who received inadequate vaccinations  163  17.1%  11.624.4% 
Percentage of patients aged 018 who received inadequate vaccinations  125  10.0%  5.318.1% 
Percentage of patients aged 50 and older who did not receive necessary 
113  25.0%  17.234.8% 
influenza vaccinations 
Percentage of patients aged 50 and older who did not receive necessary 
113  19.1%  12.029.1% 
pneumococcal vaccinations 
Percentage of patients aged 50 and older who did not receive necessary 
115  16.1%  9.526.1% 
diphtheria, tetanus, pertussis vaccinations 
Percentage of patients aged 1949 who did not receive necessary influenza 
163  14.9%  9.821.9% 
vaccinations 
Percentage of patients aged 1949 who did not receive necessary diphtheria, 
162  14.0%  9.021.0% 
tetanus, pertussis vaccinations 
Percentage of patients aged 1949 who did not receive necessary hepatitis B 
163  4.4%  1.810.1% 
vaccinations 
Percentage of patients aged 018 who did not receive necessary influenza 
125  30.2%  20.941.6% 
vaccinations 
Percentage of patients aged 018 who did not receive necessary varicella 
126  12.0%  6.620.8% 
(chicken pox) vaccinations 
Percentage of patients aged 018 who did not receive necessary hepatitis A 
126  8.1%  4.314.7% 
vaccinations 
Source:  OIG analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 35  Mary K. Wakefield, Ph.D., R.N 
 
 
Table C-7:  Confidence IntervalsMedical Reviewer Assessments of Referrals 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of patients who did not receive necessary referrals  407  9.7%  6.614.0% 
Percentage of adult patients who did not receive necessary referrals  281   10.5%  6.816.0% 
Percentage of pediatric patients who did not receive necessary referrals  126   7.8%  4.014.6% 
Percentage of adult patients who did not receive necessary referrals for 
281  8.0%  4.214.7% 
HIV/AIDS testing/treatment 
Percentage of adult patients who did not receive necessary referrals for 
281  5.1%  2.69.7% 
laboratory tests 
Percentage of adult patients who did not receive necessary referrals for 
281  4.9%  2.68.9% 
ophthalmology/optometry services 
Percentage of adult patients who did not receive necessary referrals for 
281  4.6%  2.48.6% 
nutritionist services 
Percentage of adult patients who did not receive necessary referrals for 
281  4.4%  2.48.1% 
health education 
Percentage of pediatric patients who did not receive necessary referrals 
126  3.3%  1.19.1% 
for laboratory tests 
Percentage of pediatric patients who did not receive necessary referrals 
126  2.1%  0.76.7% 
for hearing services 
Source:  OIG analysis of medical record review, 2011. 
 
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 37  Mary K. Wakefield, Ph.D., R.N 
 
 
   
Table C-8:  Confidence IntervalMedical Reviewer Assessment of Dental Services 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Percentage of dental patients who did not receive necessary dental services  91  2.0%  0.57.7% 
Source:  OIG analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 38  Mary K. Wakefield, Ph.D., R.N 
 
Table C-9:  Confidence IntervalsOversight from the Health Resources and Services Administration 
Sample  Point  95-Percent   
Estimate Description 
Size  Estimate  Confidence Interval 
 
Of grantees with quality assurance programs, percentage that reported that 
staff from the Health Resources and Services Administration (HRSA) reviewed  139  93.7%  88.496.6% 
their overall quality assurance program 
Of grantees that conduct assessments, percentage that reported that HRSA 
135  85.7%  78.990.6% 
staff reviewed their assessments 
Source:  OIG analysis of medical record review, 2011. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 39  Mary K. Wakefield, Ph.D., R.N  
 
APPENDIX D 
Detailed Methdology 
This evaluation assessed the extent to which health centers maintained required quality 
assurance programs and health center patients received necessary primary health services.  
We also examined the Health Resources and Services Administrations (HRSA) 
oversight of the required primary health services provided in health centers.  To examine 
required primary health services, we reviewed medical records for a sample of health 
center patients with at least two patient encounters between January 1, 2006, and 
December 31, 2007, including at least one encounter in the last 6 months of 2007.59  We 
expected that patients with fewer visits were less likely to use the health center as a 
regular source of care and, therefore, would be less likely to have medical records 
available for review.   
 
Data Collection and Analysis 
Our sample of 448 health center patients was selected from a stratified random sample of 
147 health center grantees (explained in more detail in the sample selection section 
below).  We reviewed primary health services that would be relevant for a broad base of 
health center patients regardless of age, gender, or health history, in particular: 
  preventive health exams,  
  vaccinations,  
  referrals for specialized treatment, and 
  dental services. 
 
To assess quality assurance programs, we surveyed the same sample of 147 health center 
grantees about their quality assurance programs.  We requested information about the 
content of quality assurance programs and health center grantees periodic service 
assessments.   
 
Finally, to assess HRSA oversight, we held several meetings and calls with HRSA staff, 
including staff from the Bureau of Primary Health Care and the Office of Performance 
Review, to request Uniform Data System (UDS) data and to discuss UDS reporting, grant 
reviews, and performance reviews. 
 
                                                 
59 Nationally, the average number of encounters per health center patient in 2007 was 3.9.  The Health 
Center Program:  2007 National Aggregate UDS Data.  Accessed at 
http://www.bphc.hrsa.gov/uds/2007data/national/nationaltable2.htm on August 11, 2010. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 40  Mary K. Wakefield, Ph.D., R.N  
 
 
Sample Selection 
Health Center Selection.  Using HRSA data, we selected a stratified random sample of  
150 health center grantees that (1) were funded in 2005, (2) still operating in 2006, and  
(3) had received a performance review between 2004 and 2006.60, 61  HRSAs onsite review 
process for health center grantees changed after 2003.  To ensure that all health centers in 
our sample had participated in HRSAs then-current review and oversight process, we 
limited our review to health centers that received a performance review.  Health center 
grantees were stratified by the size of their patient population.  One stratum consisted of the 
population of the 75 largest health centers, based on number of patients; we included all of 
the 75 largest health centers in our sample.  The other stratum consisted of the remaining 
population of 315 health centers, from which we randomly selected 75.62  After excluding 
three of the sampled health center grantees because of ongoing Office of Inspector General 
(OIG) investigations, we sent letters and information requests to the remaining 147 health 
center grantees. 
   
Patient Selection.  We requested information about patient encounters from the  
147 sampled health center grantees to select a sample of patients for medical review.  We 
sent letters to each of the sampled health center grantees to request that they provide 
information about patient encounters in 2006 and 2007.  The letters requested that the 
health center grantees provide the name, identification number, and date of birth for all 
patients with at least one encounter, at any of their health center sites, in the last 6 months 
of 2007.63  For each of these patients, we also requested the number of encounters at the 
health center in 2006 and, separately, in 2007.  We received patient lists from 142 of the 
health center grantees.  We used this information to select a sample of patients whose 
medical records we collected for review.   
 
 
   
                                                 
60 HRSA reported it had 1,002 health center grantees in 2006.  2006 National Aggregate UDS Data:   
Table 2.  Accessed at http://bphc.hrsa.gov/uds/2006data/national/nationaltable2.htm on December 7, 2010. 
61 Approximately 41 percent of health centers operating in 2006 met these criteria.   
62 The number of patients was based on 2005 UDS data. 
63 When requesting information about patients from health center grantees, we instructed them to use the 
definition of patient encounters included the 2007 Uniform Data System Reporting Manual.  HRSAs 2007 
UDS Reporting Instructions for Section 330 Grantees defines encounters as face-to-face contacts between 
a patient and a provider who exercises independent professional judgment in the provision of services to 
the patient.  To be included as an encounter, services rendered must be documented in the patients record.  
Encounters do not include large-scale efforts such as group immunizations, screening programs, or 
communitywide programs such as health fairs.  Furthermore, the provision of laboratory tests, x-rays, 
immunizations, tuberculosis tests, and prescription refills are not counted as encounters if not accompanied 
by an additional service. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 41  Mary K. Wakefield, Ph.D., R.N  
 
 
From the 142 health centers from which we received patient lists, we selected a sample of 
490 patients.  We used the following procedure for selecting patients with at least two 
health center encounters within 2 years:    
  If the health center grantee was able to provide patient encounter information for 2006 
and 2007, we randomly selected three patients from among all patients with at least 
two encounters between January 1, 2006, and December 31, 2007, including at least 
one encounter in the last 6 months of 2007.  We selected three patients from each of  
78 health center grantees that provided encounter information for a total of  
234 patients.     
  If the health center grantee was not able to provide patient encounter information for 
2006 and 2007, we randomly selected four patients from among all patients with at 
least one encounter in the last 6 months of 2007.  We selected an extra patient from this 
group of grantees to increase the likelihood that at least three patients from each health 
center would have been seen in the health center at least twice between  
January 1, 2006, and December 31, 2007, including at least one encounter in the last  
6 months of 2007.  We selected 4 patients from each of 64 health center grantees that 
did not provide encounter information, for a total of 256 patients.  
 
Patient records.  Of the 490 sampled patients, we determined that 36 were ineligible (not 
part of our intended target population) because their records did not document at least  
2 patient encounters at the health center between January 1, 2006, and December 31, 2007, 
including at least 1 between July 1 and December 31, 2007.  Of the 454 remaining patients, 
we received records for 448 (99 percent).  We included 448 patient records in at least 1 of 
the analyses.  However, we excluded a small number of patients from the overall and 
individual analyses if reviewer notes indicated that the health center offered services, but 
the services were refused by either the patient or the patients parent.  No more than seven 
patients were excluded from any single analysis due to patient refusals.  We collected 
medical records for the sampled patients in September 2008, so services provided through 
that time could have been included in our medical review.     
 
Medical Review of Health Center Records 
We worked with a medical review contractor that contracted with three physicians and three 
dentists, all with current and/or prior health center experience, to review the status of 
primary health services among patients in four categories.  Using information about 
patients health status and medical history, they determined whether each patient  
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 42  Mary K. Wakefield, Ph.D., R.N  
 
 
received necessary services.64  We developed a medical review instrument,65 with feedback 
from the medical reviewers, to assess:  
  preventive health exams66 (e.g., hearing67 and vision exams, weight and blood pressure 
checks, and cholesterol screenings), 
  vaccinations (e.g., polio; influenza; pneumococcal; measles, mumps, rubella; varicella 
(chicken pox); and diphtheria, tetanus, pertussis),  
  referrals for specialized treatment (e.g., allergy, behavioral health, dermatology, HIV 
testing and/or treatment, health education, nephrology, nutritionist services, 
ophthalmology/optometry, and orthopedics), and 
  dental services (e.g., dental exams, cleanings, fillings, and x-rays). 
 
We pretested the review instrument and shared information among medical reviewers to 
improve the consistency of reviews.  Prior to the full sample review, we selected a separate 
sample of grantees from which we requested a sample of patient records, to give medical 
reviewers an opportunity to provide feedback on the review items and to check for 
consistency among the reviewers.  Medical reviewers reviewed the sample patient records 
using a draft medical review instrument.  We made changes to the review instrument 
following the initial pretest and conducted a second pretest to give the medical reviewers an 
opportunity to use the instrument before the full sample review.  None of the pretest patient 
records was included in the full sample review. 
 
 
   
                                                 
64 In some cases, preventive health guidelines recommend against certain patients receiving specific 
services, such as a vaccination, given their health status or medical history.  In these cases, medical 
reviewers would have determined that the status of this service was appropriate if the vaccine was not 
given. 
65 We developed the medical review instrument based on preventive health guidelines issued by 
organizations such as the U.S. Preventive Services Task Force, the American Academy of Pediatrics, and 
the American Diabetes Association, among other national and professional associations. 
66 The term preventive health exams is not listed as a required primary health service pursuant to  
42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR  51c.102(h) or as a supplemental health service pursuant to  
42 CFR  51c.102(j).  We use this term broadly to encompass a variety of preventive health services, 
including screenings, examinations, and laboratory tests.   
67 Some of the health services assessed by our medical reviewers, such as hearing exams for adult patients, 
are not explicitly listed as required primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 
42 CFR  51c.102(h) or as supplemental health services pursuant to 42 CFR  51c.102(j).  However, we 
included these services in our review because they have been recommended in guidance issued by national 
and/or professional health care associations for the general population and/or for individuals in certain age 
groups or with particular health conditions. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 43  Mary K. Wakefield, Ph.D., R.N  
 
 
During the full sample review, a single medical reviewer made the final assessment about 
the services received by each health center patient.  The medical review contractor 
distributed the patient records among the medical reviewers.  We encouraged the medical 
reviewers to seek feedback from one another during the course of the review and instructed 
the medical review contractor to share any new information and guidance with all 
reviewers.   However, the final assessment of each record was made by a single physician 
reviewer and/or dental reviewer, as appropriate.68  
  
Using the medical review instruments, reviewers assessed the services received by patients.  
For each type of service (i.e., preventive health exams, vaccinations, referrals, and dental 
services), reviewers first assessed the adequacy of individual exams and services, such as 
hearing exams, polio vaccinations, referrals for allergy treatment, and dental fillings.69  
After assessing the individual exams and services within each type of service, medical 
reviewers made an overall assessment about the adequacy of each type of service.70  A 
reviewers assessment that services were adequate did not necessarily indicate that the 
health center provided these services to the patient, but rather the reviewers judgment that, 
based on available documentation, the patient had at some point received care that was 
consistent with established guidelines and did not go without a necessary service. 
 
Medical reviewers made overall assessments based on both the number and potential 
consequence of individual exams or services within each type of service they determined to 
be adequate.  For example, a medical reviewer could have determined that omission of 
vision and blood pressure exams reflected inadequate overall preventive health services for 
a patient with diabetes, but not for a patient without diabetes.  Reviewers generally 
determined that services were inadequate if, based on their judgment: 
  services needed by the patient were not offered,  
   
                                                 
68 Only those records that contained documentation of dental services at the health center were distributed 
among the dental reviewers.   
69 On the medical review instrument, when assessing each of the individual preventive health exams, 
vaccinations, referrals, and dental services, reviewers determined whether services were appropriate, 
inappropriate or, because of the patients age or gender, not applicable.  On the medical review 
instrument, inappropriate services were those that, in their judgment, were inadequate based on the 
patients history and health status.  Reviewers also indicated whether they could not determine the 
appropriateness of individual services generally, because documentation in the patients record was 
incomplete.   
70 On the medical review instrument, reviewers indicated whether, overall, each type of service was 
appropriate or inappropriate.  Medical reviewers determined that a type of service was inappropriate 
based on the patients history and health status.  Reviewers also indicated whether they could not determine 
the appropriateness of services overall because documentation in the patients record was incomplete. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 44  Mary K. Wakefield, Ph.D., R.N  
 
 
  services needed by the patient were not provided in a timely manner, or 
  unnecessary services were provided to the patient.   
 
Medical reviewers also indicated whether they were unable to determine the adequacy of 
individual or overall services in any of the four areas.  Reviewers could not determine the 
adequacy of services when documentation of services received or patient health and/or risk 
factors was insufficient.  Cases with insufficient documentation were not included in the 
cases determined to be either adequate or inadequate. 
 
We directed the medical reviewers to base their judgments on their professional experience 
at health centers.  Although we referred to established preventive health guidelines in 
developing the review instrument, we did not direct medical reviewers to use specific 
practice standards (with the exception of the Recommended Immunization Schedules issued 
by the Centers for Disease Control and Prevention (CDC)) to determine whether patients 
received necessary services.  We instructed medical reviewers to be conservative in their 
assessments, and in the event that existing preventive health guidelines were inconsistent, to 
base their assessment on the least-stringent standards.       
  
Analysis.  We conducted all analyses separately for patients in different age groups.  For 
vaccinations, we conducted separate analyses for each of the following age groups:  018, 
1949, and 50 and older.71  For each of the other service types, we conducted separate 
analyses for pediatric patients (aged 018) and adult patients (aged 19 and older).  The 
results of the medical review are projected to a subset of all health center patients.72  
Appendix C lists the confidence intervals for all medical review statistics presented in this 
report.    
 
Survey of Health Center Grantees 
We surveyed health center grantees and collected documentation related to their quality 
assurance programs between January and April 2008.  We sent surveys to the 147 sampled 
health center grantees and received completed surveys from 141 (96 percent).  The first part 
of our survey collected information about quality assurance programs and the periodic  
                                                 
71 CDC issues separate Recommended Immunization Schedules for patients aged 06,  
718, 1949, 5064, and 65 and up.  We combined all pediatric patients into a single group and patients 
aged 50 and older into a single group for analysis because of the relatively small number of patients in our 
sample from the youngest and oldest groups. 
72 The results are projected to patients with at least two patient encounters between January 1, 2006, and 
December 31, 2007 (including at least one in the last 6 months of 2007), at health centers that (1) were 
funded in 2005, (2) still operating in 2006, and (3) had received a performance review between 2004 and 
2006. 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 45  Mary K. Wakefield, Ph.D., R.N  
 
 
service assessments required under these programs.  The survey requested that grantees 
respond to a series of questions about their internal reviews of health center services.  
Grantees provided information about the frequency and focus of their assessments and the 
method of selection of medical records used during the assessments.   
 
The second part of the survey collected information about which, if any, of a list of 
approximately 70 elements was addressed by grantees quality assurance programs.  We 
identified these elements after reviewing HRSA program requirements and expectations, 
materials developed by the National Association of Community Health Centers on quality 
assurance programs, and quality assurance documents collected from health centers during 
pretesting.73, 74   
 
Analysis.  We analyzed health center grantees survey responses about their quality 
assurance programs to determine whether health center grantees: 
  maintained quality assurance programs, 
  conducted periodic assessments of their services, 
  reviewed medical records as part of their periodic assessments, and 
  included specific elements in their quality assurance programs. 
 
Limitations 
Assessments of health center services in this report are based on the professional judgment 
of medical reviewers.  By their nature, medical reviews involve some subjectivity.  
Differences in training and certification could result in differences in judgment.  We 
conducted two pretests and made modifications to the review instrument to better 
standardize the medical review process. 
 
The medical review did not address health services that patients may have received from 
other health providers.  Generally, the medical review results are based on documentation of 
services received or needed during encounters at health centers.  In some cases, health 
center patients may have received services from other health providers.  Unless documented  
                                                 
73 National Association of Community Health Centers, The Role of the Health Centers Board of Directors 
in Establishing a Quality Assurance Program, March 2003. Accessed at 
http://www.nachc.com/client/documents/publications-resources/gov_3_03.pdf on  
August 30, 2011. 
74 Prior to our full sample data collection, we visited two health center grantees to discuss our evaluation 
and to request information about their quality assurance process.  We also mailed our survey to and 
requested documentation related to quality assurance programs from five other health center grantees.   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 46  Mary K. Wakefield, Ph.D., R.N 
 
 
in health center records, this information was not available to medical reviewers and, 
therefore, not incorporated into medical reviewers assessments of patients receipt of 
services. 
 
We did not verify survey responses concerning grantees quality assurance programs or 
their scheduled or completed periodic assessments of health services.  Grantees completed 
survey questions about the development, maintenance, and review of their quality assurance 
programs and their periodic assessments of health services.  Features of health center 
grantees quality assurance programs, such as schedules and summaries of periodic 
assessments, clinical guidance, and risk management practices, often were not included in 
the written quality assurance program documents we collected from health center grantees.  
Verifying information related to quality assurance programs, in many cases, would have 
required reviewing general policy and procedures manuals, which we did not request health 
center grantees to do.   
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 47 - Mary K. Wakefield, Ph.D., R.N 
APPENDIXE 
Agency Comments 
~------------------------------------------~------~~ 
DEPARTMENT OF HEAI.TH &; HUMAN SERVICES  Healtn Resources and Services 
--.- - --.,- - --.~--- -'~- - "- - -- ---.._   ---_.-A-dm-in"is'-tf'a-t"io n-.  - -
Rockville. MD 20857 
TO:  lnsp;:clor General 
FROM:  Administrator 
SI ;BJECT;  OIG Draft Report: "Quality Assurance and Care Provided al JIRSA. 
Funded I lealth Cenler~" (OEI09.0600420) 
AltHched is the lIealth Resources and Servi\;o:s Administration's (llRSA) response to the 
OIO's dmfi rcp0l1. "Quality Assurance and Care Provided at HRSAFunded Health 
Centers" (01:::[.09-06.00420).  If you have any questions, please contact Sandy Seaton ill 
HKSA's Ollicc of Federal Assistance Management at (301) 443-2432. 
Mary K. Wakefield. Ph.D., R.N. 
Allachmelll 
OEI-09-06-00420  Quality Assurance and Care Provided at Health Centers  Page 48 Mary K. Wakefield, Ph.D., R.N 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 49  Mary K. Wakefield, Ph.D., R.N    
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 50 Mary K. Wakefield, Ph.D., R.N  
   
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 51  Mary K. Wakefield, Ph.D., R.N    
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 52  Mary K. Wakefield, Ph.D., R.N    
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 53  Mary K. Wakefield, Ph.D., R.N    
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers  Page 54  Mary K. Wakefield, Ph.D., R.N    
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers   Page 55  Mary K. Wakefield, Ph.D., R.N 
OEI-09-06-00420      Quality Assurance and Care Provided at Health Centers