DEPARTMENT OF HEALTH AND HUMAN SERVICES 

OFFICE OF INSPECTOR GENERA.L 

WASHINGTON, DC 2020 1 

MAR 622012 

TO: 	 Mary K. Wakefield, Ph.D., R:N. 

Administrator 

Health Resources and Services Administration 

FROM: Stuart Wright. 

Deputy Inspector General 

for Evaluation and Inspections 

SUBJECT: 	 Memorandum Report: Quality Assurance and Care Provided at 

HRSA-Funded Health Centers, OEI-09-06-00420 

This memorandum report provides information about the extent to which health centers 

maintained quality assurance programs and health center patients received primary health 

services in four areas: preventive health exams, vaccinations, referrals for specialized 

treatment, and dental services. It also examines the Health Resources and Services 

Administration's (HRSA) oversight of the required primary health services in health 

centers. 

SUMMARY 

The Public Health Service Act authorizes HRSA to award grants to organizations that 

provide care in medically underserved urban or rural areas or to medically underserved 

populations. Reflecting an increased Federal investment, the number ofheaIth center 

patients grew from 10.3 million in fiscal year (FY) 2001 to 17.1 million in FY 2008. 

Although health center requirements specify which services health centers must make 

available to patients, they do not establish specific quality standards for these services. 

We collected information about the quality assurance programs for a sample of 

147 health center grantees. To examine the required primary health services, we 

contracted with medical reviewers to assess medical records for a sample of 448 health 

center patients (from the same 147 health center grantees) who had at least 2 patient 

encounters between January 1, 2006, and December 31, 2007, including at least one 

encounter in the last 6 months of2007. Finally, we held several meetings and calls with 

HRSA staff, including staff from the Bureau of Primary Health Care and the Office of 

Performance Review, to assess HRSA oversight activities. We found that almost all 

health centers had quality assurance programs. However, insufficient documentation 

prevented detailed assessments of some medical records. Health services, generally, 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers

Page 2 – Mary K Wakefield, Ph.D., R.N.

were appropriate for most health center patients. During the period of our review, 

HRSA’s oversight and review activities provided limited information about the extent to 

which individual health center patients received required primary health services. HRSA 

agreed with each of our four recommendations. We recommend that HRSA (1) specify 

what elements should be included in quality assurance programs and, in particular, 

provide more guidance about how grantees conduct their periodic assessments of 

services; (2) provide more specific guidance concerning what information is required in 

patient records at health centers; (3) provide more specificity about patients’ receipt of 

required primary health services; and (4) establish procedures to independently assess 

patients’ receipt of primary health services and the adequacy of patients’ records. 

BACKGROUND

The importance of ensuring the quality of care in health centers funded by HRSA is 

growing with the increases in Federal investment in health centers and numbers of 

patients served. In 1975, Congress authorized funding for neighborhood health centers to 

provide medically underserved communities health and social services. The Health 

Centers Consolidation Act of 19961 amended the Public Health Service Act (the PHS 

Act)2 to merge separate authorities for community, migrant, homeless, and public 

housing health centers into the Consolidated Health Centers program. The PHS Act 

authorizes HRSA to award grants to organizations that provide care in medically 

underserved urban or rural areas or to medically underserved populations.3 FY 2002 

marked the start of a multiyear effort to expand the health center program with a goal of 

doubling the patient population within 5 years. Reflecting the increased Federal 

investment, the number of health center patients grew from 10.3 million in FY 2001 to 

17.1 million in FY 2008.4

The Federal investment in health centers continues to increase. Health centers received 

$2 billion through the American Recovery and Reinvestment Act of 2009 to expand health 

center capacity and support capital improvements, including the implementation of 

 1 P.L. 104-299.

2 Section 330 (codified at 42 U.S.C. § 254b). 

3 Pursuant to 42 U.S.C. § 254b, a medically underserved population includes the population of an urban or 

rural area that has been designated by the Secretary of Health and Human Services (HHS) as an area with a 

shortage of personal health services. It also may include a population group designated as having a 

shortage of these services, such as migratory and seasonal agricultural workers, homeless individuals, and 

residents of public housing. 

4 HHS, Fiscal Year 2011, Health Resources and Services Administration—Justification of Estimates for the 

Appropriations Committee. Accessed at http://hrsa.gov/about/pdf/budgetjust2011.pdf on August 4, 2010.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 3 – Mary K Wakefield, Ph.D., R.N. 

electronic health records.5, 6 In FY 2010, the Patient Protection and Affordable Care Act of 

2010 (ACA) provided a total of $11 billion in new funding for community health centers 

over the 5-year period beginning in FY 2011. Of this, $9.5 billion is for patient services 

and centers’ operations. The remaining $1.5 billion is for health center construction and 

renovation.7 

Health Center Quality Assurance and Review Requirements

Health centers receiving grants from HHS must comply with monitoring and reporting 

requirements established by the awarding HHS agency.8, 9 Health centers are required to 

provide patients with access to certain required primary health services and to maintain a 

quality assurance program. Through required quality assurance programs, the grantee 

assumes the primary responsibility for ensuring quality of care. 

Primary Health Service Requirements. Health center service requirements address the 

availability and provision of care in health centers. Health center requirements fall under 

four broad areas: need, services, management and finance, and governance.10 

Requirements related to services outline the services that health centers must offer to 

patients and to access issues, such as the availability of staff and health center hours. 

Health centers must provide certain required primary health services either directly or 

through contracts or cooperative arrangements with other health care providers.11 Although 

health center requirements specify broad categories of primary health services that health 

centers must offer to patients, they do not establish specific standards for these services, nor 

 

5 P.L. 111-5, Division A Title VIII (Feb. 17, 2009).

6 HHS, American Recovery and Reinvestment Act—Strengthening Community Healthcare Services. 

Accessed at http://www.hhs.gov/recovery/reports/plans/healthcentersservices.pdf on September 11, 2009. 

7 P.L. 111-148, as amended by the Health Care and Education Reconciliation Act of 2010 (P.L. 111-152); 

see, e.g., §§ 5601, 10503.

8 45 CFR § 74.51 and 42 U.S.C. 254b(k)(3)(I)(ii).

9 Furthermore, pursuant to 42 U.S.C. 254b(k)(3)(I)(ii), health centers must develop procedures for 

compiling and reporting statistics relating to the costs of operations; the patterns of use of their services; the 

availability, accessibility, and acceptability of their services; and such other matters that the Secretary of 

HHS may require.

10 These requirements are intended to ensure that (1) health centers’ target population or area is otherwise 

medically underserved (need); (2) required health services are available, provided by the appropriate staff, 

and provided within the context of a quality assurance program (services); (3) appropriate management, 

billing, budget, and data systems are in place (management and finance); and (4) the composition of the 

board and the conflict-of-interest policies are appropriate (governance). These requirements are found in 

42 U.S.C. § 254b, 42 CFR pt. 51c, 42 CFR §§ 56.201–56.604, and 45 CFR pt. 74. A more detailed 

summary of health center requirements was accessed at http://bphc.hrsa.gov/about/requirements/index.html

on July 1, 2011. 

11 42 U.S.C. § 254b(a)(1).
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 4 – Mary K Wakefield, Ph.D., R.N. 

do they list each individual service that patients might need. However, health centers are 

required to assess the credentials of their health care practitioners to ensure the adequacy of 

their training and skills.12 Some of the primary health services that health centers must 

offer13 include: 

• basic health services, consisting of

o health services related to family medicine, internal medicine, pediatrics, and 

obstetrics; 

o diagnostic laboratory and radiologic services; and 

o preventive services such as prenatal and perinatal services, 

well-child services, immunizations against vaccine-preventable diseases, and 

preventive dental services. 

• referrals to providers of medical services, including 

o specialty referral when medically indicated and 

other health-related services (including substance abuse and mental health 

services).14 

Quality Assurance Program. Health center grantees are required to maintain a quality 

assurance program, which provides an organizational structure that supports the provision 

of high-quality patient care.15 Quality assurance programs must provide for periodic 

assessments of utilization and quality of health services. These assessments must (1) be 

conducted by physicians or other licensed professionals under the supervision of 

physicians, (2) be based on the systematic collection and evaluation of patient records,16

and (3) identify and document needed changes and the results of such changes.17

 

12 HRSA policy requires all health centers to assess the credentials of each licensed or certified health care 

practitioner. This assessment must “meet the requirement of 42 U.S.C.§233(h)(2) that calls for review and 

verification of ‘the professional credentials, references, claims history, fitness, professional review 

organization findings, and license status of its physicians and other licensed or certified health care 

practitioners.’ ” Credentialing & Privileging of Health Center Practitioners, Policy Information Notice 

2001-6. Accessed at http://bphc.hrsa.gov/policiesregulations/policies/pin200116.html on August 29, 2011.

13 42 U.S.C. § 254b(a)(1). Health centers must provide these services either through the staff and 

supporting resources of the center or through contracts or cooperative arrangements.

14 42 U.S.C. § 254b (b)(1)(A).

15 42 CFR § 51c.303(c), 42 CFR 56.303(c), and 42 U.S.C. 254b(k)(3)C).

16 Pursuant to 42 U.S.C. 254b(n), health centers are required to establish and maintain such records as the 

Secretary of HHS shall require.

17 42 CFR § 51c.303(c), 42 CFR 56.303(c), and 42 U.S.C. 254b(k)(3)C).
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 5 – Mary K Wakefield, Ph.D., R.N. 

HRSA Oversight

Grantees are required to have in place a process for compiling and reporting to HRSA 

information relating to costs, service utilization, and the “availability, accessibility, and 

acceptability of [their] services.”18 HRSA monitors and assesses grantees’ performance 

through Uniform Data System (UDS) reporting, grant reviews, and performance reviews. 

UDS Reporting. HRSA requires that health center grantees annually submit a core set of 

data to HRSA’s UDS.19 The UDS includes information at the level of each health center 

about patient encounters, staffing, and revenue and expenses; HRSA can use this 

information to monitor and assess grantee performance and to report on overall trends.20, 21 

In 2007 (the period covered by this review), health centers were required to report two 

measures of grant performance (number of patients served and financial stability) and at 

least one measure of clinical performance. Each grantee selected its clinical measure from 

a standardized set of 14 clinical measures (e.g., percentage of pregnant women beginning 

prenatal care by end of first trimester). Each of these clinical measures—listed in Appendix 

A—pertains to a particular subgroup (e.g., pregnant women, patients with diabetes) within 

the general patient population typically served by health centers. Grantees’ performance on 

the one selected clinical measure was examined by HRSA during onsite performance 

reviews (discussed in the section below on grant and performance reviews). 

HRSA has modified requirements related to reporting UDS performance measures since 

2007, the period of our review. For 2011 UDS reporting, all health center grantees are 

required to report data on 11 standardized clinical measures, 2 additional clinical 

measures related to behavioral and oral health, and 5 standardized financial measures, as 

listed in Appendix B.22 

Grant and Performance Reviews. HRSA annually assesses health center grantees’ 

compliance with health center requirements.23, 24

 

 HRSA reviews grantees’ Health Care and 

Business Plans (plans) to evaluate their expected and actual grant performance. In their 

18 42 U.S.C. § 254b(k)(3)(I)(ii).

19 Ibid.

20 Health Center Data. Accessed at http://www.bphc.hrsa.gov/uds/ on May 6, 2010.

21 Introduction to the 2007 Uniform Data System Reporting Manual. Accessed at 

http://bphc.hrsa.gov/healthcenterdatastatistics/reporting/2007udsreportingmanual.pdf on February 15, 2012.

22 Clinical and Financial Performance Measures. Updated June 20, 2011. Accessed at 

http://bphc.hrsa.gov/policiesregulations/performancemeasures/updatedfy2012measures.pdf on August 30, 

2011.

23 Pursuant to 42 CFR §§ 51c.305 and 56.305.

24 Pursuant to 42 CFR §§ 51c.303 and 56.303.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 6 – Mary K. Wakefield, Ph.D., R.N 

plans, grantees outline their clinical and management goals (e.g., health outcomes, services

offered, and recruitment activities) and the objectives; key action steps; expected outcomes; 

data, evaluation and measurement; and points of contact related to these goals. HRSA 

reviews these plans to evaluate the extent to which prospective grantees will be able to meet 

program requirements and current grantees have achieved established goals. Starting in 

2009, the required UDS performance measures were included in all grantee plans, along 

with other goals specific to individual grantees.

During the period of our review, HRSA—in addition to conducting grant reviews—also 

conducted onsite performance reviews of health center grantees at least once during the 

grantees’ 3–5 year project period.25, 26 During these performance reviews, HRSA examined 

grantees’ progress on their performance measures including the number of patients served, 

financial stability measures, and the selected clinical measure.27 The review team and the 

grantee discussed, in particular, external and internal factors that influenced the grantee’s 

performance on clinical and other performance measures. HRSA and the grantee also 

discussed options for improvement.28 The Performance Review Protocol, used during 

onsite visits, included suggested questions concerning grantees’ quality 

assurance/improvement plans, but it did not specify how or whether review teams were to 

collect or review information about these plans.29 HRSA stopped conducting performance 

reviews after November 2009. After that time, onsite HRSA reviews primarily assessed 

grantees’ ability to comply with health center program requirements, among other issues.30 

According to HRSA staff, site visits also are used to review clinical and financial issues 

raised during quarterly calls between health center grantees and HRSA project officers. 

METHODOLOGY

Sample Selection

Health Center Selection. Using HRSA data, we selected a stratified random sample of 

150 health center grantees that (1) were funded in 2005, (2) were still operating in 2006, 

and (3) had received a performance review between 2004 and 2006.31, 32

 

25 HRSA, Performance Review Protocol, revised December 2004. This document is no longer available 

online.

 One stratum 

26 The Performance Review Protocol did not establish required timeframes for reviews. 

27 HRSA, Performance Review Protocol, revised December 2004. 

28 Ibid. 

29 Ibid.

30 Requirements established pursuant to 42 CFR §§ 51c.303 and 56.303. Other issues are pursuant to 

42 CFR §§ 51c.305 and 56.305.

31 HRSA reported 1,002 health center grantees in 2006. 2006 National Aggregate UDS Data: Table 2. 

Accessed at http://bphc.hrsa.gov/uds/2006data/national/nationaltable2.htm on December 7, 2010.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 7 – Mary K. Wakefield, Ph.D., R.N 

consisted of the population of the 75 largest health centers, based on number of patients; we 

included all of the 75 largest health centers in our sample. The other stratum consisted of 

the remaining population of 315 health centers, from which we randomly selected 75 health 

centers.33 After excluding three of the sampled health center grantees because of ongoing

Office of Inspector General (OIG) investigations, we sent letters and information requests 

to the remaining 147 health center grantees.

Patient Selection. We requested information about patient encounters from the 

147 sampled health center grantees to help us select a sample of patients for medical 

review. We sent letters to each of the sampled health center grantees to request that they 

provide information about patient encounters in 2006 and 2007. The letters requested that 

the health center grantees provide the name, identification number, and date of birth for all 

patients with at least one encounter, at any of their health center sites, in the last 6 months 

of 2007.34 For each of these patients, we also requested the number of encounters at the 

health center in 2006 and, separately, in 2007. We received patient lists from 142 of the 

health center grantees. We used this information to select a sample of patients who had at 

least two patient encounters between January 1, 2006, and December 31, 2007, including at 

least one encounter in the last 6 months of 2007. From the patient lists of the 142 health 

centers, we selected a sample of 490 patients. We collected medical records for the 

sampled patients in September 2008, so services provided through that time could have 

been included in our medical review. 

Medical Review of Health Center Records

We worked with a medical review contractor that contracted with three physicians and three 

dentists, all with current and/or prior health center experience, to review the status of 

primary health services among patients in four categories. Using available information 

about patients’ health status and medical history, they determined whether each patient 

 32 Approximately 41 percent of health centers operating in 2006 met these criteria. 

33 The number of patients was based on 2005 UDS data.

34 When requesting information about patients from health center grantees, we instructed them to use the 

definition of patient encounters included the 2007 Uniform Data System Reporting Manual. HRSA’s 2007 

UDS Reporting Instructions for Section 330 Grantees defines encounters as “face-to-face contacts between 

a patient and a provider who exercises independent professional judgment in the provision of services to 

the patient. To be included as an encounter, services rendered must be documented in the patient’s record.” 

Encounters do not include large-scale efforts such as group immunizations, screening programs, or 

communitywide programs such as health fairs. Furthermore, the provision of laboratory tests, x-rays, 

immunizations, tuberculosis tests, and prescription refills are not counted as encounters if not accompanied 

by an additional service.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 8 – Mary K. Wakefield, Ph.D., R.N 

received necessary services.35 We developed a medical review instrument,36 using 

feedback from the medical reviewers, to assess: 

• preventive health exams37 (e.g., hearing38 and vision exams, weight and blood pressure 

checks, and cholesterol screenings), 

• vaccinations (e.g., polio; influenza; pneumococcal; measles, mumps, rubella; varicella 

(chicken pox); and diphtheria, tetanus, pertussis), 

• referrals for specialized treatment (e.g., allergy, behavioral health, dermatology, HIV 

testing and/or treatment, health education, nephrology, nutritionist services, 

ophthalmology/optometry, and orthopedics), and

• dental services (e.g., dental exams, cleanings, fillings, and x-rays). 

Medical reviewers made overall assessments based on both the number and potential 

consequence of individual exams or services within each type of service they determined to 

be adequate. For example, a medical reviewer could have determined that omission of 

vision and blood pressure exams reflected inadequate overall preventive health services for 

a patient with diabetes, but not for a patient without diabetes. Reviewers generally 

determined that services were inadequate if, based on their judgment:

• services needed by the patient were not offered, 

• services needed by the patient were not provided in a timely manner, or 

• unnecessary services were provided to the patient. 

Analysis. We conducted all analyses separately for patients in different age groups. For 

vaccinations, we conducted separate analyses for each of the following age groups: patients 

 

35 In some cases, preventive health guidelines recommend against certain patients receiving specific 

services, such as a vaccination, given their health status or medical history. In these cases, medical 

reviewers would have determined that the status of this service was appropriate.

36 We developed the medical review instrument based on preventive health guidelines issued by 

organizations such as the U.S. Preventive Services Task Force, the American Academy of Pediatrics, and 

the American Diabetes Association, among other national and professional associations.

37 The term “preventive health exams” is not listed as a required primary health service pursuant to 

42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR § 51c.102(h), or as a supplemental health service pursuant to 

42 CFR § 51c.102(j). We use this term broadly to encompass a variety of preventive health services, 

including screenings, examinations, and laboratory tests. 

38 Some of the health services assessed by our medical reviewers, such as hearing exams for adult patients, 

are not explicitly listed as required primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 

42 CFR § 51c.102(h) or supplemental health services pursuant to 42 CFR § 51c.102(j). However, we 

included these services in our review because they have been recommended in guidance issued by national 

and/or professional health care associations for the general population and/or for individuals in certain age 

groups or with particular health conditions.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 9 – Mary K. Wakefield, Ph.D., R.N 

aged 0–18, 19–49, and 50 and older.39 For each of the other service types, we conducted 

separate analyses for pediatric patients (0–18) and adult patients (19 and older). The results 

of the medical review are projected to a subset of all health center patients.40 Appendix C 

lists the confidence intervals for all medical review statistics presented in this report. 

Limitations

Assessments of health center services in this report are based on the professional judgment 

of medical reviewers. By their nature, medical reviews involve some subjectivity. 

Differences in training and certification could result in differences in judgment. We 

conducted two pretests and made modifications to the review instrument to better 

standardize the medical review process.

Generally, the medical review results are based on documentation of services received or 

needed during encounters at health centers. In some cases, health center patients may have 

received services from other health providers. Unless documented in health center records, 

this information was not available to medical reviewers, and, therefore, not incorporated 

into medical reviewers’ assessments of patients’ receipt of services.

See Appendix D for a detailed discussion of the methdology. 

This inspection was conducted in accordance with the Quality Standards for Inspections

approved by the Council of the Inspectors General on Integrity and Efficiency. 

 

39 The Centers for Disease Control and Prevention (CDC) issues separate Recommended Immunization 

Schedules for patients aged 0–6, 7–18, 19–49, 50–64, and 65 and up. We combined all pediatric patients 

into a single group and patients aged 50 and older into a single group for analysis because of the relatively 

small number of such patients in our sample.

40 The results are projected to patients with at least two patient encounters between January 1, 2006, and 

December 31, 2007 (including at least one in the last 6 months of 2007), at health centers that (1) were 

funded in 2005, (2) still operating in 2006, and (3) had received a performance review between 2004 and 

2006.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 10 – Mary K. Wakefield, Ph.D., R.N 

RESULTS

Almost All Health Centers Reported Maintaining Required Quality Assurance 

Programs 

Almost all health center grantees reported that they maintain written quality assurance 

programs. Only 2 percent of grantees indicated that they did not maintain a written 

quality assurance program.41 Of the 141 grantees that completed surveys, 134 provided 

some documentation of their quality assurance program. Most of these grantees 

maintained a written quality assurance plan created to describe the grantees’ quality 

assurance activities. Others did not maintain a separate quality assurance plan, but 

instead outlined their quality assurance activities in other health center guidance, such as 

policy and procedures manuals, clinical guidance, quality review protocols, and risk 

management guides.

Ninety-Seven Percent of Grantees Reported Conducting Required Periodic Assessments as 

Part of Their Quality Assurance Program. Most health center grantees reported that they 

conduct periodic assessments of health center services as part of their quality assurance 

program. Among the health center grantees that indicated they have a written quality 

assurance program, 97 percent indicated that they have conducted a periodic assessment 

(hereafter assessment) of the services provided by their health centers.

42 

The majority of health center grantees indicated that they conduct assessments at least once 

a year. Among the health center grantees that reported conducting assessments, 46 percent 

indicated that they conduct assessments once a year and 9 percent indicated that they 

conduct assessments every 6 months. Other grantees noted that they conduct assessments 

monthly or quarterly, whereas some noted that they continuously assess the services 

provided to patients. Some health center grantees reported conducting multiple assessments 

staggered throughout the year, with reviews of different types of services or patients 

assigned to specific months. 

 

41 One health center responded that it did not maintain a written quality assurance program, whereas a 

second health center grantee indicated that although it did not maintain a written quality assurance 

program, it tracked some clinical data and participated in a HRSA disease collaborative. The information 

provided by the latter grantee did not appear to indicate that the grantee maintained an organized quality 

assurance program. We asked health center grantees to send documentation of their quality assurance 

programs with their completed surveys. 

42 If a health center grantee indicated that it did not maintain a written quality assurance program, the 

grantee was not required to respond to the rest of the questions on the survey, including those involving 

periodic assessments.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 11 – Mary K. Wakefield, Ph.D., R.N 

Most health center grantees indicated that they reviewed patient medical records as a part of 

their assessments. Among the health center grantees that reported conducting assessments, 

91 percent indicated that they review patient medical records. Among grantees that 

reported reviewing medical records as part of their assessments:

• 82 percent select records randomly, 

• 47 percent select records from certain types of patients (e.g., pediatric patients), 

• 30 percent review records from all health center patients, and

• 22 percent review records of certain provider types (e.g., all dentists, providers being 

credentialed).43 

Among health center grantees that conducted assessments, most reported assessing 

medical records and documentation (83 percent) and quality of care (82 percent), with 

fewer assessing patient outcomes (61 percent), service trends (57 percent), and the 

medical necessity of services (35 percent). 

Most Quality Assurance Programs Addressed the Primary Health Services Provided by 

Health Center Grantees. Quality assurance programs addressed primary health services 

through the implementation of clinical protocols and the measurement of patient 

outcomes. For example, most health center grantees reported that their quality assurance 

programs specifically addressed clinical protocols for diabetes screening44 and/or care 

(89 percent) and immunizations (87 percent). Grantees also reported addressing patient 

outcomes in their quality assurance programs, including outcomes related to diabetes 

screening and/or care (92 percent) and immunizations (81 percent). Table 1 highlights 

the clinical protocols and patient outcomes often included in health center grantees’ 

quality assurance programs.

 

43 These percentages do not sum to 100 percent because some health center grantees reported multiple 

assessments for which they used different methods for selected patient records.

44 Although diabetes screening is not explicitly included as a required primary health service pursuant to 

42 U.S.C. 254b(b)(1)(A)(i)(I), health centers may have developed protocols for this and other services 

based on guidance issued by professional and/or national health care associations.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 12 – Mary K. Wakefield, Ph.D., R.N 

Table 1: Selected Clinical Protocols and Patient Outcomes Addressed in 

Health Center Grantees’ Quality Assurance Programs

Element Type Element

Percentage of Health 

Center Grantees 

That Addressed 

Element

Clinical Protocols

Diabetes screening/care 88.7%

Immunizations 86.9%

Hypertension screening/care 81.4%

Cervical cancer screening 80.3%

Breast cancer screening 79.4%

Colon cancer screening 79.4%

Cholesterol screening/care 77.2%

Pediatric care 76.8%

Dental care 69.0%

Prenatal care 63.9%

Patient Outcomes

Diabetes screening/care 91.9%

Immunizations 81.4%

Hypertension screening/care 72.8%

Cervical cancer screening 68.4%

Breast cancer screening 66.4%

Cholesterol screening/care 66.4%

Source: OIG analysis of survey data, 2011.

Most quality assurance programs also addressed patients’ access to the primary health 

services provided by health centers. Approximately 86 percent of grantees reported 

addressing the availability of clinical staff. Similarly, 85 percent of grantees reported that 

their quality assurance program specifically addressed the convenience of health center 

hours, and approximately 81 percent of health center grantees reported addressing wait 

times for scheduled appointments. Most grantees also addressed the availability of 

translation services (82 percent) and the affordability of care (77 percent). 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 13– Mary K. Wakefield, Ph.D., R.N 

Most grantees also addressed the adequacy of training and review of clinical staff in their 

quality assurance programs. For example, 86 percent addressed continuing education for 

medical staff, 86 percent addressed the verification of credentials for health center staff 

providers, and 85 percent addressed staff education on health center policies.

 

Although Most Health Centers Reported Addressing Medical Records As Part of 

Their Quality Assurance Programs, Medical Reviewers Were Not Able To 

Determine the Adequacy of At Least One Type of Service For 45 Percent of Patients 

Because of Insufficient Documentation

Most grantees addressed medical records and documentation as a part of their quality 

assurance plans. Overall, approximately 89 percent of grantees with quality assurance 

programs reported that they addressed the accuracy and completeness of patient medical 

records in their quality assurance programs. Among all health center grantees that 

conducted assessments as part of their quality assurance programs, 83 percent reported that 

they assessed health center documentation and medical records. 

Although most health centers addressed medical records in their quality assurance 

programs, records for approximately 45 percent of health center patients contained 

insufficient documentation for reviewers to assess at least one type of service included in 

our review.45 For each type of service, reviewers either assessed the adequacy of services 

or indicated that they could not do so, primarily because of insufficient documentation on 

the services received by patients or patients’ health status. As illustrated in Table 2, 

vaccinations often lacked sufficient documentation for reviewers to determine the 

adequacy of services.46 Approximately 44 percent of health center patients’ records 

lacked sufficient documentation for reviewers to determine the overall adequacy of 

vaccinations. Although documentation of vaccinations was sufficient for most pediatric 

patients, it was frequently missing for adults. 

 

45 Among the 134 grantees that provided documentation of their quality assurance programs, 95 provided a 

document that was dated from 2007 or earlier, whereas 24 provided a document dated from 2008. The 

documentation provided by the remaining grantees was undated or contained multiple dates. As a result, 

the documentation available in some patients’ medical records could have predated specific documentationrelated guidance included in the quality assurance programs that we reviewed.

46 If reviewers lacked sufficient information, they did not make determinations about the adequacy of care. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 14 – Mary K. Wakefield, Ph.D., R.N 

Table 2: Availability of Documentation in Patients’ Records

Type of Service Patient Age

Percentage of Patients 

With Insufficient 

Documentation for 

Type of Service

Preventive health exams All patients (n=399) 5.3%

Vaccinations

All patients (n=405) 44.0%

 0–18 (n=125) 17.3%

 19–49 (n=163) 62.5%

 50 and older (n=117) 46.6%

Referrals for specialized treatment All patients (n=407) 2.5% 

Dental services All dental patients (n=91) 19.0%*

Source: OIG analysis of medical review data, 2011.

* Among patients who received at least one dental service.

Primary Health Services, Generally, Were Appropriate For Most Health Center 

Patients

Medical reviewers assessed four types of primary health services among health center 

patients: preventive health exams, vaccinations, referrals for specialized treatment, and 

dental services. Health center requirements specify broad categories of primary health 

services that health centers must provide to patients. However, the requirements do not 

establish specific standards for these services, nor do they list each individual service that

might be needed by patients. Therefore, we relied on the professional judgment of 

medical reviewers to assess each type of service. For each type of service, medical 

reviewers first assessed individual exams and services, such as vision exams, polio 

vaccinations, referrals for allergy treatment, and dental fillings.47, 48 After assessing the 

 

47 When assessing each of the individual preventive health exams, vaccinations, referrals, and dental 

services on the medical review instrument, reviewers determined whether care was “appropriate,” 

“inappropriate” or, because of the patients’ age or gender, “not applicable.” Inappropriate services were 

those that, in reviewers’ judgment, were inadequate based on the patients’ history and health status. 

Reviewers also indicated whether they could not determine the adequacy of individual services, generally 

because documentation in the patients’ record was incomplete. If reviewers lacked sufficient information, 

they did not make determinations about the adequacy of care. 

48 Some of the health services assessed by our medical reviewers are not explicitly listed as required 

primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR § 51c.102(h) or 

supplemental health services pursuant to 42 CFR § 51c.102(j). However, we included these services in our 

review because they have been recommended in guidance issued by national and/or professional health 

care associations for the general population or for individuals in certain age groups or with particular health 

conditions.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 15 – Mary K. Wakefield, Ph.D., R.N 

individual exams and services, reviewers made an overall assessment of the adequacy of 

each type of service. Reviewers determined whether services overall in each area were 

adequate or inadequate, or indicated that they could not determine the adequacy of 

services. Reviewers’ determination that services were adequate did not necessarily 

indicate that the health center provided these services to the patient, but rather reviewers’ 

judgment that, based on available documentation, the patient had, at the time of the 

review, received care that was consistent with established guidelines and did not fail to 

receive a necessary service. Table 3 summarizes medical reviewers’ overall 

determinations of the adequacy of the four types of primary health services received by 

health center patients. 

If reviewers determined that either overall or individual services were inadequate, they 

indicated why. In most but not all such cases, medical reviewers indicated that patients 

did not receive necessary services. For a small number of cases, medical reviewers 

determined that services were inadequate because necessary services were not received in 

a timely manner or because patients received medically unnecessary services. The 

estimates presented in this finding represent the percentage of patients (among all patients 

whose records were reviewed) that, according to medical reviewers, did not receive 

necessary services. 

Medical Reviewers Determined That Approximately 25 Percent of Health Center Patients 

Did Not Receive Necessary Preventive Health Exams. Medical reviewers determined that 

overall preventive health services were inadequate for approximately one in four patients 

because patients did not receive needed preventive health exams.49, 50

 

 Specifically, in the 

professional judgment of medical reviewers, approximately 29 percent of adult patients and 

49 These cases represent most, but not all, of the preventive services that reviewers determined to be 

inadequate.

50 Based on the documentation included in their records.

Table 3: Overall Medical Reviewer Determinations of Four Service 

Types

Service Type Percentage Determined 

Adequate

Percentage Determined 

Inadequate

Preventive health exams 68.4% 26.4%

Vaccinations 36.8% 19.2%

Referrals 86.1% 11.4%

Dental services 74.2% 6.8%

Source: OIG analysis of medical record review, 2011.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 16 – Mary K. Wakefield, Ph.D., R.N 

17 percent of pediatric patients did not receive necessary preventive health exams. Table 4 

shows some of the individual preventive health exams not received by health center 

patients.51 In particular, reviewers reported that approximately 21 percent of adults did not 

receive necessary colon cancer screening exams. Similarly, approximately 20 percent of 

adults did not receive necessary cholesterol screening exams. Among pediatric patients, 

approximately 16 percent did not receive necessary hearing exams and 15 percent did not 

receive necessary developmental screenings. 

 

Table 4: Medical Reviewer Assessments of Necessary Preventive Health 

Exams Not Received

Preventive Health Exam Patient Age Group Percentage That Did Not 

Receive Necessary Exams

Colon screening (i.e., fecal 

occult blood, sigmoidoscopy, or 

colonoscopy) 

Adult 21.3%

Cholesterol screening Adult 20.3% 

Blood sugar level screening Adult 9.6%

Vision Adult 7.5%

Hearing Pediatric 15.6%

Developmental screening Pediatric 14.7%

Blood pressure Pediatric 10.7%

Vision Pediatric 11.5%

Source: OIG analysis of medical record review, 2011.

Approximately 17 Percent of Patients Did Not Receive Necessary CDC-Recommended 

Vaccinations. Medical reviewers reported that vaccinations were inadequate for 

approximately 17 percent of patients because they did not receive necessary vaccinations. 

Overall, vaccinations for approximately 24 percent of patients aged 50 and older, 17 percent 

of patients aged 19–49, and 10 percent of patients aged 0–18 were inadequate because the 

patients did not receive necessary vaccinations. Table 5 displays some of the individual 

vaccinations missing for patients in the different age groups.52 

51 Reviewers assessed the adequacy of 13 individual preventive health exams. The individual exams 

assessed for adult and pediatric patients varied slightly.

52 Reviewers assessed the appropriateness of 12 individual vaccinations included on CDC’s recommended 

schedules. The individual vaccinations assessed varied slightly for patients of different ages.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 17 – Mary K. Wakefield, Ph.D., R.N 

 

Approximately 10 percent of health center patients did not receive necessary referrals. 

Medical reviewers determined that referrals for specialized treatment were inadequate for 

approximately 10 percent of patients because they did not receive needed referrals. 

Medical reviewers did not assess whether patients actually received services from the 

appropriate specialists, but rather whether a necessary referral to the appropriate specialist 

was documented in patients’ records. Approximately 11 percent of adult patients and 

8 percent of pediatric patients did not receive needed referrals.54

 

 Table 6 displays some of 

the referrals that were not provided to adult health center patients. For example, almost 

8 percent of adult patients did not receive needed referrals for HIV/AIDS testing and/or 

treatment. Among pediatric patients, referrals for necessary laboratory tests (3 percent) and 

hearing services (2 percent) were not received for a relatively small percentage of patients.

 

53 It is important to note that vaccination rates in the general population are not always consistent with the 

CDC-recommended schedule. For example, although we cannot make a direct comparison because our 

review assessed the adequacy of vaccinations (as opposed to actual vaccination rates), in 2007, only 

36 percent of adults aged 50–64 and 66 percent of adults aged 65 and older reported receiving an influenza 

vaccination. Self-reported influenza vaccination coverage trends 1989–2007, National Health Interview 

Survey. Accessed online at 

http://www.cdc.gov/flu/professionals/vaccination/pdf/NHIS89_07fluvaxtrendtab.pdf on September 8, 2010. 

Similarly, approximately 17 percent of adults aged 50–64 and 58 percent of adults aged 65 and older 

reported receiving a pneumococcal vaccination in 2007. Self-reported pneumococcal vaccination coverage 

trends 1989–2007, National Health Interview Survey. Accessed online at 

http://www.cdc.gov/flu/professionals/vaccination/pdf/NHIS89_07ppvvaxtrendtab.pdf on September 8, 

2010.

54 Reviewers assessed the appropriateness of referrals for 29 types of specialized care.

Table 5: Medical Reviewer Assessments of Necessary Vaccinations Not 

Received 

Vaccination Patient Age Percentage That Did Not Receive 

Necessary Vaccinations

Influenza53 50 and older 25.0%

Pneumococcal 50 and older 19.1%

Diphtheria, tetanus, pertussis 50 and older 16.1%

Influenza 19–49 14.9%

Diphtheria, tetanus, pertussis 19–49 14.0%

Hepatitis B 19–49 4.4%

Influenza 0–18 30.2%

Varicella (chicken pox) 0–18 12.0%

Hepatitis A 0–18 8.1%

Source: OIG analysis of medical record review, 2011.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 18 – Mary K. Wakefield, Ph.D., R.N 

Table 6: Medical Reviewer Assessments of Necessary Referrals Not 

Received by Adult Patients 

Referral Percentage That Did Not Receive 

Necessary Referrals

HIV/AIDS testing and/or treatment 8.0% 

Laboratory tests 5.1% 

Ophthalmology/optometry services 4.9% 

Nutritionist services 4.6% 

Health education 4.4% 

Source: OIG analysis of medical record review, 2011.

Approximately 2 Percent of Dental Patients Did Not Receive Necessary Dental Services. 

According to medical reviewers’ assessments, only 2 percent of dental patients received 

inadequate overall dental care because they did not receive necessary services. In our 

sample of 32 pediatric patients with health center dental records, medical reviewers 

determined that only one patient (3 percent) did not receive necessary dental services. 

Similarly, only one adult patient in our sample of 59 adult dental patients (2 percent) did not 

receive necessary dental services. 

HRSA Review Activities Provided Limited Information About the Extent to Which 

Individual Health Center Patients Received Required Primary Health Services

HRSA oversight in 2007, the period covered by our review, was intended to collect core 

clinical data, ensure grantee compliance with program requirements, and support grantee 

efforts to improve their programs; it was not intended to determine the extent to which 

health center patients received primary health services.

HRSA Reviews Did Not Assess the Required Primary Health Services Received by Health 

Center Patients. According to HRSA staff, the agency did not independently collect and/or 

review patient medical records or other health information to assess and/or verify the 

services received by individual health center patients. HRSA onsite reviews assessed

grantees’ success at achieving their clinical and management goals, and determined whether 

technical assistance was necessary. Through its UDS, HRSA received clinical information 

about a subset of health center services and/or patients, but this information did not 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 19 – Mary K. Wakefield, Ph.D., R.N 

necessarily reflect the range of care received by most health center patients. Even with 

the most recent update to UDS reporting, performance measures might not provide 

HRSA with comprehensive information about individual health center patients’ care. 

The Lack of Detailed Requirements for Quality Assurance Programs May Have Limited 

Their Usefulness in Ensuring That Patients Receive Necessary Primary Health Services. 

Requirements and reviews for quality assurance programs were broad and did not always 

ensure that patients receive necessary services. Approximately 94 percent of grantees55

reported that HRSA staff reviewed their overall quality assurance program, and 

86 percent56 reported that HRSA staff reviewed their assessments. However, beyond the 

requirement that grantees must develop and maintain quality assurance programs that 

provide for assessments of services, the requirements provided few specifics about the 

issues to be addressed in quality assurance programs. We collected quality assurance 

plans that ranged from four to several hundred pages. In addition, requirements specified 

that grantees must conduct assessments of services, but did not specify the frequency of 

these assessments, the services to be included, or the review methods to be used. Most 

grantees reported that they assessed medical record documentation and health services as 

a part of their quality assurance programs and internal assessments. However, medical 

reviewers often could not determine the adequacy of services because of insufficient 

documentation and, in other cases, judged that patients did not receive necessary services.

RECOMMENDATIONS

Health centers provide a range of health services to medically underserved communities. 

Ninety-seven percent of health center grantees maintained required quality assurance 

programs and conducted internal assessments of their services. Although 89 percent of 

grantees indicated that their quality assurance programs addressed the accuracy and 

completeness of medical records, medical reviewers determined that 45 percent of patient 

records did not contain sufficient information for them to assess at least one type of service. 

Primary health services, generally, were appropriate for most health center patients. During 

the period of our review, HRSA’s oversight was intended to collect core clinical data, 

ensure grantee compliance with program requirements, and support grantee efforts to 

improve their programs; it was not intended to directly assess the receipt of required 

primary health services by health center patients, and provided limited information about 

 

55 Among grantees that reported maintaining a quality assurance program.

56 Among grantees that reported conducting assessments.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 20 – Mary K. Wakefield, Ph.D., R.N 

the services actually received by health center patients. Therefore, we recommend that 

HRSA:

Provide More Specific Guidance About What Grantees Should Address in 

Their Quality Assurance Programs and How They Should Conduct Their 

Periodic Assessments 

Health center grantees must maintain required quality assurance programs and, through 

these quality assurance programs, periodically assess their services. However, 

requirements provide little detail about what quality assurance programs and assessments 

must address. HRSA should specify what elements should be included in quality 

assurance programs and, in particular, provide more guidance about how grantees 

conduct their periodic assessments of services. For example, HRSA could require all 

grantees to review records for a random sample of patients to assess the extent to which 

patients received required primary services as a part of their assessments. In addition, 

HRSA could require health centers to assess the adequacy of medical records. 

Provide More Specific Guidance Concerning What Information is Required 

in Patient Records at Health Centers

Incomplete medical records limit health providers’ ability to determine what health services 

their patients need or receive. HRSA should provide explicit guidance concerning the 

documentation that must be maintained in all health center records. HRSA could develop 

guidance based on standards issued by national or professional associations or require 

health center grantees to follow established standards and offer technical assistance, as 

appropriate. 

HRSA also should continue to support health center grantees’ efforts to implement 

electronic record systems. Medical reviewers noted that electronic health records were 

more organized and legible than handwritten paper records. HRSA’s support of health 

center grantees’ transition to electronic health records, through the American Recovery and 

Reinvestment Act and other agency funding, may improve health centers’ ability to 

maintain complete medical record documentation. 

Provide More Specificity About the Required Primary Health Services 

Although health center requirements indicate the broad primary health services that 

grantees must provide, they do not include specific quality standards for these services. 

HRSA should provide more guidance about the services that should be received by health 

center patients. To ensure that patients receive necessary services, HRSA could require 

explicitly that health center grantees identify specific preventive health guidance or 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 21 – Mary K. Wakefield, Ph.D., R.N 

recommendations that their health care practitioners will follow. HRSA could recommend 

to grantees specific guidance issued by organizations and agencies such as the U.S. 

Preventive Services Task Force, CDC, or other national and/or professional health 

associations. 

Establish Procedures To Independently Assess Patients’ Receipt of Primary 

Health Services and the Adequacy of Patients’ Records

Although health center grantees maintain the primary responsibility for the care received by 

their patients, HRSA oversight should provide the agency with more information about the 

extent to which health center patients receive required primary health services. UDS 

reporting does not provide information about the services received by most health center 

patients. HRSA should develop a review process that would allow it to assess the primary 

health services received by a cross-section of health center patients. This process also 

should assess the adequacy of medical record documentation. For example, HRSA could 

collect and review a representative sample of patient records to determine the extent to 

which patients received adequate care. In addition, HRSA could examine the extent to 

which expected documentation, such as vaccination records and patient medical histories, is 

maintained in a sample of patient records. During reviews, HRSA also could determine the 

extent to which grantees implemented elements in their quality assurance programs, in 

particular those related to patients’ receipt of primary health services and the adequacy of 

medical record documentation. 

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE

In its written comments on our draft, HRSA highlighted the achievements of health centers 

in providing quality primary health care to underserved communities and its own increasing 

focus on quality improvement. Although our review included health records and services 

through 2007, HRSA cited key initiatives and activities implemented since that time, 

including emphasizing evidence-based quality improvement practices and evaluating the 

quality and impact of health care services. HRSA also cited recent research highlights 

showing favorable statistics for health center patients in comparison to other populations. 

In response to our first recommendation—to provide more specific guidance about what 

grantees should address in their quality assurance programs—HRSA stated that it is in the 

process of developing specific policy guidance on quality improvement and quality 

assurance plans including periodic assessments, in accordance with the statutory 

requirements. 

 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 22 – Mary K. Wakefield, Ph.D., R.N 

In response to our second recommendation—to provide more specific guidance about what 

information is required in patient records at health centers—HRSA stated that it fully 

supports the adoption of electronic health records that meet the “Meaningful Use” criteria 

established by HHS’s Office of the National Coordinator for Health Information 

Technology and the Centers for Medicare & Medicaid Services. OIG supports HRSA’s 

efforts to expand the adoption of electronic health records in health centers; these efforts 

will help to establish clear expectations for what information should be documented in 

patient records. We anticipate receiving documentation from HRSA regarding its 2010 

modification to the UDS to require all health centers to report their specific electronic 

health record activities.

In response to our third recommendation—to provide more specificity about the required 

primary health services—HRSA said that national variation in health centers’ scope of 

practice and the needs of the target populations limit its ability to specify and/or require 

certain procedures or more detailed aspects of core primary and preventive health services. 

HRSA stated that it will continue to endorse the most current nationally recognized, 

evidence-based clinical practice guidelines and protocols, such as those set forth by the U.S. 

Preventive Services Task Force, to offer as a guide to providers for delivering primary 

health services. OIG supports HRSA’s endorsement of evidence-based practice guidelines 

and encourages the agency to require health centers to include a reference to nationally 

recognized clinical practice protocols in the centers’ primary health services policies for 

providers. 

Finally, in response to our fourth recommendation—to establish procedures to 

independently assess patients’ receipt of primary health services and the adequacy of patient 

records—HRSA stated that it is in the process of updating its site-visit protocol to add a 

focus on such reviews. In particular, future site visits will involve reviews/audits of a 

sample of patient records. 

 

For the full text of HRSA’s comments, see Appendix E. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 23 – Mary K. Wakefield, Ph.D., R.N 

APPENDIX A 

Clinical Measures for 2007 Health Center Grantee Performance Reviews57

1. Perinatal (Effort): Percentage of pregnant women beginning prenatal care by end of 

first trimester. 

2. Perinatal (Effort): Percentage of women who had a postpartum visit within 42 days 

after delivery.

3. Perinatal (Effort): Percentage of newborns who had a followup visit within 14 days 

of birth. 

4. Child (Effort): Percentage of children 5 through 18 years of age diagnosed with 

“persistent” asthma who were prescribed appropriate medications. 

5. Adolescent (Effort): Percentage of adolescents who had both a documented 

Behavioral Risk Assessment and a related counseling visit. 

6. Adult (Outcome): Percentage of adults diagnosed with an abnormal lipid profile 

whose levels are under control. 

7. Adult (Effort): Percentage of adults with abnormal breast, cervical, or colon cancer 

screening results for which referral and/or treatment has been initiated within 30 days 

of test completion. 

8. Adult (Outcome): Percentage of adults diagnosed with hypertension whose blood 

pressure is under control. 

9. Adult (Effort): Percentage of adults who are tobacco users who have received 

counseling and/or a treatment plan to quit. 

10. Adult (Outcome): Percentage of adults with type 1 or type 2 diabetes whose last 

glycated hemoglobin (HbA1c) level was less than 7 percent. 

11. Oral (Outcome): Percentage of dental patients with a comprehensive oral exam and 

a treatment plan completed within a 12-month period. 

12. Mental (Effort & Outcome): a. Percentage of all adult patients with screening for 

depression (Effort) or b. Percentage of patients with major depression who have a 

50-percent decrease in depression severity score (Outcome). 

13. All Life Cycles (Outcome): Percentage of child, adolescent, adult, geriatric, or 

HIV/AIDS patients with appropriate immunizations.

 

57 Health Resources and Services Administration guidance. This information is no longer available online.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 24 – Mary K. Wakefield, Ph.D., R.N 

14. All Life Cycles (Effort): Percentage of child/adolescent patients or adult patients 

with a Body Mass Index indicating overweight or obesity who have received healthy 

weight counseling and/or other related interventions or treatment. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 25 – Mary K. Wakefield, Ph.D., R.N 

APPENDIX B 

2011 Clinical and Financial Performance Measures58

Outreach/Quality of Care

1. Percentage of pregnant women beginning prenatal care in the first trimester

2. Percentage of children with 2nd birthday during the measurement year with 

appropriate immunizations 

3. Percentage of women 21–64 years of age who received one or more tests to screen for 

cervical cancer 

4. Percentage of patients age 2 to 17 years who had a visit during the current year and 

who had Body Mass Index (BMI) Percentile documentation, counseling for nutrition, 

and counseling for physical activity during the measurement year

5. Percentage of patients age 18 years or older who had their BMI calculated at the last 

visit or within the last 6 months and, if they were overweight or underweight, had a 

followup plan documented 

6. Percentage of patients age 18 years and older who were queried about tobacco use 

one or more times within 24 months 

7. Percentage of patients age 18 years and older who are users of tobacco and who 

received (charted) advice to quit smoking or tobacco use 

8. Percentage of patients age 5 to 40 years with a diagnosis of persistent asthma 

(mild, moderate, or severe) who were prescribed either the preferred long-term 

control medication or an acceptable alternative pharmacological therapy during the 

current year

Health Outcomes/Disparities

1. Percentage of diabetic patients whose HbA1c levels are less than 7 percent, less than 

8 percent, less than or equal to 9 percent, or greater than 9 percent 

2. Percentage of adult patients with diagnosed hypertension whose most recent blood 

pressure was less than 140/90 

3. Percentage of births less than 2,500 grams to health center patients

 

58 Clinical and financial performance measures accessed online at 

http://bphc.hrsa.gov/policiesregulations/performancemeasures/index.html on February 22, 2012.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 26 – Mary K. Wakefield, Ph.D., R.N 

Additional Clinical Measures

In addition to the above Uniform Data System clinical measures, health centers must 

include 

1. One Behavioral Health (e.g., Mental Health or Substance Abuse) and 

2. One Oral Health performance measure of their choice in the Health Care Plan

Financial Viability/Costs

1. Total cost per patient 

2. Medical cost per encounter 

3. Change in net assets to expenses ratio 

4. Working capital to monthly expense ratio 

5. Long term debt to equity ratio 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 27 – Mary K. Wakefield, Ph.D., R.N 

APPENDIX C 

Table C-1: Confidence Intervals—Quality Assurance Programs and Periodic Assessments

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of grantees that reported not maintaining a written quality 

assurance program

141 2.3% 0.7–7.5%

Percentage of grantees that reported conducting periodic assessments

as part of their quality assurance program

139 97.1% 92.5%–98.9%

Of grantees that conducted assessments, percentage that conduct 

assessments once a year

135 45.5% 37.2%–54.1%

Of grantees that conducted assessments, percentage that conduct 

assessments every 6 months

135 9.2% 5.4%–15.4%

Of grantees that conducted assessments, percentage that select patient 

medical records as part of the assessment

135 91.4% 85.7%–95.0%

Of grantees that reviewed patient medical records, percentage that select 

records randomly

121 82.1% 74.2%–88.0%

Of grantees that reviewed patient medical records, percentage that select 

records from certain types of patients

121 46.8% 38.1%–55.8%

Of grantees that reviewed patient medical records, percentage that review 

records from all health center patients

121 30.2% 22.7%–39.0%

Of grantees that reviewed patient medical records, percentage that review 

records from certain provider types

121 22.1% 15.6%–30.3%

Of grantees that conducted assessments, percentage that assessed health 

center documentation/medical records

135 82.5% 75.2%–87.9%

continued on next page
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 28 – Mary K. Wakefield, Ph.D., R.N 

Table C-1: Confidence Intervals—Quality Assurance Programs and Periodic Assessments 

(Continued)

Estimate Description Sample Size

Point 

Estimate

95-Percent 

Confidence Interval

Of grantees that conducted assessments, percentage of grantees that 

assessed quality of care

135 82.1% 74.3%–87.9%

Of grantees that conducted assessments, percentage of grantees that 

assessed patient outcomes

135 61.2% 52.5%–69.2%

Of grantees that conducted assessments, percentage of grantees that 

assessed service trends

135 56.7% 48.0%–64.9%

Of grantees that conducted assessments, percentage of grantees that 

assessed the medical necessity of care

135 35.0% 27.5%–43.4% 

Source: Office of Inspector General (OIG) analysis of medical record review, 2011.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 29 – Mary K. Wakefield, Ph.D., R.N 

Table C-2: Confidence Intervals—Of Grantees With Quality Assurance Programs, Percentage That 

Addressed Each Element 

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Clinical protocols for diabetes screening and/or care 139 88.7% 82.1–93.1%

Clinical protocols for immunizations 139 86.9% 79.8–91.8%

Patient outcomes related to diabetes screening/care 139 91.9% 85.7–95.5%

Patient outcomes related to immunizations 139 81.4% 73.4–87.4%

Clinical protocols for hypertension screening/care 139 81.4% 73.7–87.3%

Clinical protocols for cervical cancer screening 139 80.3% 72.4–86.3%

Clinical protocols for breast cancer screening 139 79.4% 71.4–85.6%

Clinical protocols for colon cancer screening 139 79.4% 71.4–85.6%

Clinical protocols for cholesterol screening/care 139 77.2% 69.3–83.5%

continued on next page
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 30 – Mary K. Wakefield, Ph.D., R.N 

Table C-2: Confidence Intervals—Of Grantees With Quality Assurance Programs, Percentage That 

Addressed Each Element (Continued)

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Clinical protocols for pediatric care 139 76.8% 68.8–83.3%

Clinical protocols for dental care 139 69.0% 60.5–76.3%

Clinical protocols for prenatal care 139 63.9% 55.3–71.8%

Patient outcomes related to hypertension screening/care 139 72.8% 64.5–79.7%

Patient outcomes related to cervical cancer screening 139 68.4% 59.9–75.8%

Patient outcomes related to breast cancer screening 139 66.4% 57.9–73.9%

Patient outcomes related to cholesterol screening/care 139 66.4% 57.9–73.9%

Availability of clinical staff 139 85.8% 78.8–90.8%

Convenience of health center hours 139 84.9% 77.7–90.1%

Wait times for scheduled appointments 139 80.8% 72.9–86.8%

continued on next page
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 31 – Mary K. Wakefield, Ph.D., R.N 

Table C-2: Confidence Intervals—Of Grantees With Quality Assurance Programs, Percentage That 

Addressed Each Element (Continued)

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Availability of translation services 139 81.7% 74.0–87.5%

Affordability of care 139 77.2% 69.3–83.5%

Continuing education for medical staff 139 86.4% 79.3–91.3%

Verification of credentials for health center providers 139 86.4% 79.3–91.3%

Staff education on health center policies 139 84.9% 77.7–90.1%

Patient satisfaction 139 95.1% 89.3–97.8%

Internal quality-of-care reviewers 139 92.4% 86.1–96.0%

Resolution of patient complaints 139 92.2% 85.9–95.8%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 32 – Mary K. Wakefield, Ph.D., R.N 

Table C-3: Confidence Intervals—Medical Record Documentation

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of patient records that lacked sufficient documentation 

for reviewers to determine the overall appropriateness of at least one type of 

care

436 44.5% 39.1–49.9%

Of grantees with quality assurance programs, percentage that addressed the 

accuracy and completeness of patient medical records

139 89.3% 82.6–93.6%

Of grantees that conducted assessments, percentage that assessed health 

center documentation/medical records

135 82.5% 75.2–87.9%

Percentage of patient records that lacked sufficient documentation of 

vaccinations

405 44.0% 38.3–49.8%

Percentage of patient records that lacked documentation for preventive health 

care 399 5.3% 2.4–11.0%

Among patients aged 0–18, percentage of records that lacked sufficient 

documentation of vaccinations

125 17.3% 11.0–26.0%

Among patients aged 19–49, percentage of records that lacked sufficient 

documentation of vaccinations

163 62.5% 53.1–71.0%

Among patients aged 50 and over, percentage of records that lacked sufficient 

documentation of vaccinations

117 46.6% 36.1–57.4%

Percentage of patient records that lacked sufficient documentation of referrals 407 2.5% 1.2–5.3%

Among dental patients, percentage of records that lacked sufficient 

documentation of dental services

91 19.0% 11.1–30.5% 

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 33 – Mary K. Wakefield, Ph.D., R.N

Table C-4: Confidence Intervals—Medical Reviewer Assessments of the Four Service Types

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of patients with adequate preventive health exams 399 68.4% 62.3–73.9%

Percentage of patients with inadequate preventive health exams 399 26.4% 21.4–32.0%

Percentage of patients with adequate vaccinations 405 36.8% 31.6–42.4%

Percentage of patients with inadequate vaccinations 405 19.2% 15.2–24.0%

Percentage of patients with adequate referrals 407 86.1% 81.7–89.6%

Percentage of patients with inadequate referrals 407 11.4% 8.1–15.7%

Percentage of dental patients with adequate dental services 91 74.2% 61.7–83.7%

Percentage of dental patients with inadequate dental services 91 6.8% 2.5–17.0%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 34 – Mary K. Wakefield, Ph.D., R.N 

Table C-5: Confidence Intervals—Medical Reviewer Assessments of Preventive Health Exams

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of patients who did not receive necessary preventive health 

exams 399 24.9% 20.1–30.4%

Percentage of adult patients who did not receive necessary preventive health 

exams 274 28.7% 22.6–35.8%

Percentage of pediatric patients who did not receive necessary preventive 

health exams

125 16.7% 10.5–25.4%

Percentage of adult patients who did not receive necessary colon cancer 

screening exams

279 21.3% 16.6–26.9%

Percentage of adult patients who did not receive necessary cholesterol 

screening exams

279 20.3% 15.4–26.4%

Percentage of pediatric patients who did not receive necessary hearing 

exams 124 15.6% 9.9–23.7%

Percentage of pediatric patients who did not receive necessary developmental 

screenings

123 14.7% 8.9–23.2%

Percentage of adult patients who did not receive necessary blood sugar level 

screening

280 9.6% 6.3–14.3%

Percentage of adult patients who did not receive necessary vision exams 281 7.5% 4.6–12.2%

Percentage of pediatric patients who did not receive necessary blood pressure 

exams 126 10.7% 5.7–19.0%

Percentage of pediatric patients who did not receive necessary vision exams 124 11.5% 6.6–19.2%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 35 – Mary K. Wakefield, Ph.D., R.N 

Table C-6: Confidence Intervals—Medical Reviewer Assessments of Vaccinations

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of patients who received inadequate vaccinations 405 16.7% 12.9–21.5%

Percentage of patients aged 50 and older who received inadequate 

vaccinations

117 23.9% 16.0–34.3%

Percentage of patients aged 19–49 who received inadequate vaccinations 163 17.1% 11.6–24.4%

Percentage of patients aged 0–18 who received inadequate vaccinations 125 10.0% 5.3–18.1%

Percentage of patients aged 50 and older who did not receive necessary 

influenza vaccinations

113 25.0% 17.2–34.8%

Percentage of patients aged 50 and older who did not receive necessary 

pneumococcal vaccinations

113 19.1% 12.0–29.1%

Percentage of patients aged 50 and older who did not receive necessary 

diphtheria, tetanus, pertussis vaccinations

115 16.1% 9.5–26.1%

Percentage of patients aged 19–49 who did not receive necessary influenza 

vaccinations

163 14.9% 9.8–21.9%

Percentage of patients aged 19–49 who did not receive necessary diphtheria, 

tetanus, pertussis vaccinations

162 14.0% 9.0–21.0%

Percentage of patients aged 19–49 who did not receive necessary hepatitis B 

vaccinations

163 4.4% 1.8–10.1%

Percentage of patients aged 0–18 who did not receive necessary influenza 

vaccinations

125 30.2% 20.9–41.6%

Percentage of patients aged 0–18 who did not receive necessary varicella 

(chicken pox) vaccinations

126 12.0% 6.6–20.8%

Percentage of patients aged 0–18 who did not receive necessary hepatitis A 

vaccinations

126 8.1% 4.3–14.7%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 35 – Mary K. Wakefield, Ph.D., R.N 

Table C-7: Confidence Intervals—Medical Reviewer Assessments of Referrals

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of patients who did not receive necessary referrals 407 9.7% 6.6–14.0%

Percentage of adult patients who did not receive necessary referrals 281 10.5% 6.8–16.0%

Percentage of pediatric patients who did not receive necessary referrals 126 7.8% 4.0–14.6%

Percentage of adult patients who did not receive necessary referrals for 

HIV/AIDS testing/treatment

281 8.0% 4.2–14.7%

Percentage of adult patients who did not receive necessary referrals for 

laboratory tests

281 5.1% 2.6–9.7%

Percentage of adult patients who did not receive necessary referrals for 

ophthalmology/optometry services

281 4.9% 2.6–8.9%

Percentage of adult patients who did not receive necessary referrals for 

nutritionist services

281 4.6% 2.4–8.6%

Percentage of adult patients who did not receive necessary referrals for 

health education

281 4.4% 2.4–8.1%

Percentage of pediatric patients who did not receive necessary referrals 

for laboratory tests

126 3.3% 1.1–9.1%

Percentage of pediatric patients who did not receive necessary referrals 

for hearing services

126 2.1% 0.7–6.7%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 37 – Mary K. Wakefield, Ph.D., R.N 

Table C-8: Confidence Interval—Medical Reviewer Assessment of Dental Services

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Percentage of dental patients who did not receive necessary dental services 91 2.0% 0.5–7.7%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 38 – Mary K. Wakefield, Ph.D., R.N 

Table C-9: Confidence Intervals—Oversight from the Health Resources and Services Administration

Estimate Description Sample 

Size

Point 

Estimate

95-Percent 

Confidence Interval

Of grantees with quality assurance programs, percentage that reported that 

staff from the Health Resources and Services Administration (HRSA) reviewed 

their overall quality assurance program

139 93.7% 88.4–96.6%

Of grantees that conduct assessments, percentage that reported that HRSA 

staff reviewed their assessments

135 85.7% 78.9–90.6%

Source: OIG analysis of medical record review, 2011. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 39 – Mary K. Wakefield, Ph.D., R.N 

APPENDIX D

Detailed Methdology 

This evaluation assessed the extent to which health centers maintained required quality 

assurance programs and health center patients received necessary primary health services. 

We also examined the Health Resources and Services Administration’s (HRSA)

oversight of the required primary health services provided in health centers. To examine 

required primary health services, we reviewed medical records for a sample of health

center patients with at least two patient encounters between January 1, 2006, and 

December 31, 2007, including at least one encounter in the last 6 months of 2007.59 We 

expected that patients with fewer visits were less likely to use the health center as a 

regular source of care and, therefore, would be less likely to have medical records 

available for review. 

Data Collection and Analysis 

Our sample of 448 health center patients was selected from a stratified random sample of 

147 health center grantees (explained in more detail in the sample selection section 

below). We reviewed primary health services that would be relevant for a broad base of 

health center patients regardless of age, gender, or health history, in particular: 

• preventive health exams, 

• vaccinations, 

• referrals for specialized treatment, and

• dental services.

To assess quality assurance programs, we surveyed the same sample of 147 health center 

grantees about their quality assurance programs. We requested information about the 

content of quality assurance programs and health center grantees’ periodic service 

assessments. 

Finally, to assess HRSA oversight, we held several meetings and calls with HRSA staff, 

including staff from the Bureau of Primary Health Care and the Office of Performance 

Review, to request Uniform Data System (UDS) data and to discuss UDS reporting, grant 

reviews, and performance reviews.

 

59 Nationally, the average number of encounters per health center patient in 2007 was 3.9. The Health 

Center Program: 2007 National Aggregate UDS Data. Accessed at 

http://www.bphc.hrsa.gov/uds/2007data/national/nationaltable2.htm on August 11, 2010.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 40 – Mary K. Wakefield, Ph.D., R.N 

Sample Selection

Health Center Selection. Using HRSA data, we selected a stratified random sample of 

150 health center grantees that (1) were funded in 2005, (2) still operating in 2006, and 

(3) had received a performance review between 2004 and 2006.60, 61 HRSA’s onsite review 

process for health center grantees changed after 2003. To ensure that all health centers in 

our sample had participated in HRSA’s then-current review and oversight process, we 

limited our review to health centers that received a performance review. Health center 

grantees were stratified by the size of their patient population. One stratum consisted of the 

population of the 75 largest health centers, based on number of patients; we included all of 

the 75 largest health centers in our sample. The other stratum consisted of the remaining 

population of 315 health centers, from which we randomly selected 75.62 After excluding 

three of the sampled health center grantees because of ongoing Office of Inspector General 

(OIG) investigations, we sent letters and information requests to the remaining 147 health 

center grantees.

 

Patient Selection. We requested information about patient encounters from the 

147 sampled health center grantees to select a sample of patients for medical review. We 

sent letters to each of the sampled health center grantees to request that they provide 

information about patient encounters in 2006 and 2007. The letters requested that the 

health center grantees provide the name, identification number, and date of birth for all 

patients with at least one encounter, at any of their health center sites, in the last 6 months 

of 2007.63 For each of these patients, we also requested the number of encounters at the 

health center in 2006 and, separately, in 2007. We received patient lists from 142 of the 

health center grantees. We used this information to select a sample of patients whose 

medical records we collected for review. 

 

60 HRSA reported it had 1,002 health center grantees in 2006. 2006 National Aggregate UDS Data: 

Table 2. Accessed at http://bphc.hrsa.gov/uds/2006data/national/nationaltable2.htm on December 7, 2010.

61 Approximately 41 percent of health centers operating in 2006 met these criteria. 

62 The number of patients was based on 2005 UDS data.

63 When requesting information about patients from health center grantees, we instructed them to use the 

definition of patient encounters included the 2007 Uniform Data System Reporting Manual. HRSA’s 2007 

UDS Reporting Instructions for Section 330 Grantees defines encounters as “face-to-face contacts between 

a patient and a provider who exercises independent professional judgment in the provision of services to 

the patient. To be included as an encounter, services rendered must be documented in the patient’s record.” 

Encounters do not include large-scale efforts such as group immunizations, screening programs, or 

communitywide programs such as health fairs. Furthermore, the provision of laboratory tests, x-rays, 

immunizations, tuberculosis tests, and prescription refills are not counted as encounters if not accompanied 

by an additional service.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 41 – Mary K. Wakefield, Ph.D., R.N 

From the 142 health centers from which we received patient lists, we selected a sample of 

490 patients. We used the following procedure for selecting patients with at least two 

health center encounters within 2 years: 

• If the health center grantee was able to provide patient encounter information for 2006 

and 2007, we randomly selected three patients from among all patients with at least 

two encounters between January 1, 2006, and December 31, 2007, including at least 

one encounter in the last 6 months of 2007. We selected three patients from each of 

78 health center grantees that provided encounter information for a total of 

234 patients. 

• If the health center grantee was not able to provide patient encounter information for 

2006 and 2007, we randomly selected four patients from among all patients with at 

least one encounter in the last 6 months of 2007. We selected an extra patient from this 

group of grantees to increase the likelihood that at least three patients from each health 

center would have been seen in the health center at least twice between 

January 1, 2006, and December 31, 2007, including at least one encounter in the last 

6 months of 2007. We selected 4 patients from each of 64 health center grantees that 

did not provide encounter information, for a total of 256 patients. 

Patient records. Of the 490 sampled patients, we determined that 36 were ineligible (not 

part of our intended target population) because their records did not document at least 

2 patient encounters at the health center between January 1, 2006, and December 31, 2007, 

including at least 1 between July 1 and December 31, 2007. Of the 454 remaining patients, 

we received records for 448 (99 percent). We included 448 patient records in at least 1 of 

the analyses. However, we excluded a small number of patients from the overall and 

individual analyses if reviewer notes indicated that the health center offered services, but 

the services were refused by either the patient or the patient’s parent. No more than seven 

patients were excluded from any single analysis due to patient refusals. We collected 

medical records for the sampled patients in September 2008, so services provided through 

that time could have been included in our medical review. 

Medical Review of Health Center Records

We worked with a medical review contractor that contracted with three physicians and three 

dentists, all with current and/or prior health center experience, to review the status of 

primary health services among patients in four categories. Using information about 

patients’ health status and medical history, they determined whether each patient 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 42 – Mary K. Wakefield, Ph.D., R.N 

received necessary services.64 We developed a medical review instrument,65 with feedback 

from the medical reviewers, to assess: 

• preventive health exams66 (e.g., hearing67 and vision exams, weight and blood pressure 

checks, and cholesterol screenings), 

• vaccinations (e.g., polio; influenza; pneumococcal; measles, mumps, rubella; varicella 

(chicken pox); and diphtheria, tetanus, pertussis), 

• referrals for specialized treatment (e.g., allergy, behavioral health, dermatology, HIV 

testing and/or treatment, health education, nephrology, nutritionist services, 

ophthalmology/optometry, and orthopedics), and

• dental services (e.g., dental exams, cleanings, fillings, and x-rays).

We pretested the review instrument and shared information among medical reviewers to 

improve the consistency of reviews. Prior to the full sample review, we selected a separate 

sample of grantees from which we requested a sample of patient records, to give medical 

reviewers an opportunity to provide feedback on the review items and to check for 

consistency among the reviewers. Medical reviewers reviewed the sample patient records 

using a draft medical review instrument. We made changes to the review instrument 

following the initial pretest and conducted a second pretest to give the medical reviewers an 

opportunity to use the instrument before the full sample review. None of the pretest patient 

records was included in the full sample review. 

 

64 In some cases, preventive health guidelines recommend against certain patients’ receiving specific 

services, such as a vaccination, given their health status or medical history. In these cases, medical 

reviewers would have determined that the status of this service was appropriate if the vaccine was not 

given.

65 We developed the medical review instrument based on preventive health guidelines issued by 

organizations such as the U.S. Preventive Services Task Force, the American Academy of Pediatrics, and 

the American Diabetes Association, among other national and professional associations.

66 The term “preventive health exams” is not listed as a required primary health service pursuant to 

42 U.S.C. 254b(b)(1)(A)(i)(I) and 42 CFR § 51c.102(h) or as a supplemental health service pursuant to 

42 CFR § 51c.102(j). We use this term broadly to encompass a variety of preventive health services, 

including screenings, examinations, and laboratory tests. 

67 Some of the health services assessed by our medical reviewers, such as hearing exams for adult patients, 

are not explicitly listed as required primary health services pursuant to 42 U.S.C. 254b(b)(1)(A)(i)(I) and 

42 CFR § 51c.102(h) or as supplemental health services pursuant to 42 CFR § 51c.102(j). However, we 

included these services in our review because they have been recommended in guidance issued by national 

and/or professional health care associations for the general population and/or for individuals in certain age 

groups or with particular health conditions.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 43 – Mary K. Wakefield, Ph.D., R.N 

During the full sample review, a single medical reviewer made the final assessment about 

the services received by each health center patient. The medical review contractor 

distributed the patient records among the medical reviewers. We encouraged the medical 

reviewers to seek feedback from one another during the course of the review and instructed 

the medical review contractor to share any new information and guidance with all 

reviewers. However, the final assessment of each record was made by a single physician 

reviewer and/or dental reviewer, as appropriate.68

 

 

Using the medical review instruments, reviewers assessed the services received by patients. 

For each type of service (i.e., preventive health exams, vaccinations, referrals, and dental 

services), reviewers first assessed the adequacy of individual exams and services, such as 

hearing exams, polio vaccinations, referrals for allergy treatment, and dental fillings.69 

After assessing the individual exams and services within each type of service, medical 

reviewers made an overall assessment about the adequacy of each type of service.70 A 

reviewer’s assessment that services were adequate did not necessarily indicate that the 

health center provided these services to the patient, but rather the reviewer’s judgment that, 

based on available documentation, the patient had at some point received care that was 

consistent with established guidelines and did not go without a necessary service. 

Medical reviewers made overall assessments based on both the number and potential 

consequence of individual exams or services within each type of service they determined to 

be adequate. For example, a medical reviewer could have determined that omission of 

vision and blood pressure exams reflected inadequate overall preventive health services for 

a patient with diabetes, but not for a patient without diabetes. Reviewers generally 

determined that services were inadequate if, based on their judgment:

• services needed by the patient were not offered, 

 

68 Only those records that contained documentation of dental services at the health center were distributed 

among the dental reviewers. 

69 On the medical review instrument, when assessing each of the individual preventive health exams,

vaccinations, referrals, and dental services, reviewers determined whether services were “appropriate,” 

“inappropriate” or, because of the patients’ age or gender, “not applicable.” On the medical review 

instrument, inappropriate services were those that, in their judgment, were inadequate based on the 

patients’ history and health status. Reviewers also indicated whether they could not determine the 

appropriateness of individual services generally, because documentation in the patient’s record was 

incomplete. 70 On the medical review instrument, reviewers indicated whether, overall, each type of service was 

“appropriate” or “inappropriate.” Medical reviewers determined that a type of service was “inappropriate” 

based on the patients’ history and health status. Reviewers also indicated whether they could not determine 

the appropriateness of services overall because documentation in the patient’s record was incomplete. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 44 – Mary K. Wakefield, Ph.D., R.N 

• services needed by the patient were not provided in a timely manner, or 

• unnecessary services were provided to the patient. 

Medical reviewers also indicated whether they were unable to determine the adequacy of

individual or overall services in any of the four areas. Reviewers could not determine the 

adequacy of services when documentation of services received or patient health and/or risk 

factors was insufficient. Cases with insufficient documentation were not included in the 

cases determined to be either adequate or inadequate.

We directed the medical reviewers to base their judgments on their professional experience 

at health centers. Although we referred to established preventive health guidelines in 

developing the review instrument, we did not direct medical reviewers to use specific 

practice standards (with the exception of the Recommended Immunization Schedules issued 

by the Centers for Disease Control and Prevention (CDC)) to determine whether patients 

received necessary services. We instructed medical reviewers to be conservative in their 

assessments, and in the event that existing preventive health guidelines were inconsistent, to 

base their assessment on the least-stringent standards. 

 

Analysis. We conducted all analyses separately for patients in different age groups. For 

vaccinations, we conducted separate analyses for each of the following age groups: 0–18, 

19–49, and 50 and older.71 For each of the other service types, we conducted separate 

analyses for pediatric patients (aged 0–18) and adult patients (aged 19 and older). The 

results of the medical review are projected to a subset of all health center patients.72 

Appendix C lists the confidence intervals for all medical review statistics presented in this 

report. 

Survey of Health Center Grantees

We surveyed health center grantees and collected documentation related to their quality 

assurance programs between January and April 2008. We sent surveys to the 147 sampled 

health center grantees and received completed surveys from 141 (96 percent). The first part 

of our survey collected information about quality assurance programs and the periodic 

 

71 CDC issues separate Recommended Immunization Schedules for patients aged 0–6, 

7–18, 19–49, 50–64, and 65 and up. We combined all pediatric patients into a single group and patients 

aged 50 and older into a single group for analysis because of the relatively small number of patients in our 

sample from the youngest and oldest groups.

72 The results are projected to patients with at least two patient encounters between January 1, 2006, and 

December 31, 2007 (including at least one in the last 6 months of 2007), at health centers that (1) were 

funded in 2005, (2) still operating in 2006, and (3) had received a performance review between 2004 and 

2006.
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 45 – Mary K. Wakefield, Ph.D., R.N 

service assessments required under these programs. The survey requested that grantees 

respond to a series of questions about their “internal reviews of health center services.” 

Grantees provided information about the frequency and focus of their assessments and the 

method of selection of medical records used during the assessments. 

The second part of the survey collected information about which, if any, of a list of 

approximately 70 elements was addressed by grantees’ quality assurance programs. We 

identified these elements after reviewing HRSA program requirements and expectations, 

materials developed by the National Association of Community Health Centers on quality 

assurance programs, and quality assurance documents collected from health centers during 

pretesting.73, 74 

Analysis. We analyzed health center grantees’ survey responses about their quality 

assurance programs to determine whether health center grantees:

• maintained quality assurance programs,

• conducted periodic assessments of their services,

• reviewed medical records as part of their periodic assessments, and

• included specific elements in their quality assurance programs.

Limitations

Assessments of health center services in this report are based on the professional judgment 

of medical reviewers. By their nature, medical reviews involve some subjectivity. 

Differences in training and certification could result in differences in judgment. We 

conducted two pretests and made modifications to the review instrument to better 

standardize the medical review process.

The medical review did not address health services that patients may have received from 

other health providers. Generally, the medical review results are based on documentation of 

services received or needed during encounters at health centers. In some cases, health 

center patients may have received services from other health providers. Unless documented 

 

73 National Association of Community Health Centers, The Role of the Health Center’s Board of Directors 

in Establishing a Quality Assurance Program, March 2003. Accessed at 

http://www.nachc.com/client/documents/publications-resources/gov_3_03.pdf on 

August 30, 2011.

74 Prior to our full sample data collection, we visited two health center grantees to discuss our evaluation 

and to request information about their quality assurance process. We also mailed our survey to and 

requested documentation related to quality assurance programs from five other health center grantees. 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 46 – Mary K. Wakefield, Ph.D., R.N 

in health center records, this information was not available to medical reviewers and, 

therefore, not incorporated into medical reviewers’ assessments of patients’ receipt of 

services.

We did not verify survey responses concerning grantees’ quality assurance programs or 

their scheduled or completed periodic assessments of health services. Grantees completed 

survey questions about the development, maintenance, and review of their quality assurance 

programs and their periodic assessments of health services. Features of health center 

grantees’ quality assurance programs, such as schedules and summaries of periodic 

assessments, clinical guidance, and risk management practices, often were not included in 

the written quality assurance program documents we collected from health center grantees. 

Verifying information related to quality assurance programs, in many cases, would have 

required reviewing general policy and procedures manuals, which we did not request health 

center grantees to do. 
Page 47 - Mary K. Wakefield, Ph.D., R.N 

APPENDIXE 

Agency Comments 

~------------------------------------------~------~~ 

DEPARTMENT OF HEAI.TH &; HUMAN SERVICES Healtn Resources and Services 

Administfation --.- - -- .,- - --. -- '~ - "- - - - ---.. _ ---_.- --"'-'-" -. - -

Rockville. MD 20857 

TO: lnsp;:clor General 

FROM: Administrator 

SI ;BJECT; OIG Dra ft Report: "Quality Assurance and Care Prov ided al JIRSA. 

Funded I lealth Cenler~" (OEI·09.06·00420) 

AltHched is the lIealth Resources and Servi\;o:s Administration's (llRSA) response to the 

OIO's dmfi rcp0l1. "Quality Assurance and Care Provided at HRSA·Funded Health 

Centers" (01:::[.09-06.00420). If you have any questions, please contact Sandy Seaton ill 

HKSA's Ollicc of Federal Assistance Management at (301) 443-2432. 

Mary K. Wakefield. Ph.D., R.N. 

Allachmelll 

OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 48– Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 49 – Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 50– Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 51 – Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 52 – Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 53 – Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 54 – Mary K. Wakefield, Ph.D., R.N
OEI-09-06-00420 Quality Assurance and Care Provided at Health Centers 

Page 55 – Mary K. Wakefield, Ph.D., R.N