Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

JUNE GIBBS BROWN 

Inspector General 

AUGUST 1999 

OEI-05-98-00390 

RYAN WHITE EVALUATION SYSTEMS 

AIDS Drug Assistance Program 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through a 

nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Chicago Regional Office prepared this report under the direction of William Moran, 

Regional Inspector General. Principal OEI staff included: 

REGION 

William Moran, Project Leader

Ann Maxwell, Lead Analyst

Victoria Jacobs, Analyst

Nora Leibowitz, Analyst

Vince Greiber, New York Regional Office

HEADQUARTERS 

Elise Stein, Program Specialist 

Rada Spencer, Program Specialist 

Alan Levine, Program Specialist 

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SUMMARY 

PURPOSE 

To examine the evaluation systems established by Ryan White AIDS Drug Assistance 

Programs (ADAPs) to monitor program compliance, measure client outcomes and 

measure overall effectiveness. 

BACKGROUND 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resource Emergency 

(CARE) Act as a comprehensive response to the HIV epidemic and its impact on 

individuals, families, communities, cities and States. The AIDS Drug Assistance Program 

provides medications to low-income people living with HIV/AIDS that are uninsured or 

under-insured and lack coverage for medications. In fiscal year (FY) 1999, Congress 

appropriated $461 million of Title II funds to finance drug assistance programs. This is an 

increase of 61 percent over FY 1998 when States received $285.5 million in Title II 

ADAP funds. 

The Ryan White CARE Act will be considered for re-authorization in the year 2000. As 

Congress debates this re-authorization, questions are certain to be asked about the 

effectiveness and impact of the programs funded. This inspection examines the capacity of 

AIDS Drug Assistance Programs to answer those questions. 

We conducted an examination of the evaluation mechanisms used by eight Title II ADAPs 

through an extensive documentation review and on-site interviews. The States visited 

represent 68 percent of the living AIDS cases reported in 1997 and 61 percent of the Title 

II funding for FY 1998. 

FINDINGS 

The AIDS Drug Assistance Programs in the Eight States We Visited Have 

Sufficient Programmatic and Fiscal Monitoring Mechanisms in Place 

The ADAP administrators closely monitor program and fiscal activity through the review 

of routine reports. Fiscal control over expenditures is further maintained by employing a 

variety of cost containment strategies to purchase the drugs at the lowest possible price. 

Program enrollment is monitored through the use of standardized enrollment procedures 

Evaluation Systems: ADAP 1 OEI-05-98-00390 
and centralized processing. All ADAPs rely on regular, automated matches with Medicaid 

databases as the primary tool to assure that ADAP funds are being used as the payor of 

last resort. 

While Most ADAPs Rely on Process Measures to Gauge Program Progress, a Few 

are Developing Ways to Measure Client Outcomes 

Most ADAP administrators reported that they rely on program utilization data to assess 

the success of their program. However, an increasing number of administrators are taking 

on the responsibility of monitoring program impact to improve program efficiency and 

effectiveness and to respond to demands for greater accountability. On the other hand, 

some ADAP administrators are of the opinion that measuring outcomes is not an 

appropriate way to expend limited resources. They argue that it is best to rely on medical 

research professionals to measure clinical outcomes and that requiring ADAPs to measure 

outcomes would duplicate work already being done in the medical field. 

ADAPs are Conducting Minimal Assessments of Whether They are Reaching 

Populations in Need 

All ADAPs indicated that they match the demographics of enrolled clients to the 

demographic profile of the epidemic as a measure of how they are meeting population 

needs. Despite the understood disadvantages of relying on this method to assess program 

effectiveness, ADAPs have not found other, more sophisticated means to assess whether 

they are reaching populations in need. 

RECOMMENDATIONS 

To Promote Outcome Measurement, HRSA Should Encourage ADAPs to Compare 

Utilization Data to Recommended Clinical Guidelines 

The Health Resources and Services Administration (HRSA) should require drug assistance 

programs to review whether drugs are dispensed in accordance with the Guidelines for the 

Use of Antiretroviral Agents in HIV Infected Adults and Adolescents. Requiring ADAPs 

to compare their utilization data to these recommended clinical guidelines provides a 

means to extrapolate medically indicated client health outcomes from utilization data. 

Given that ADAP clients may also be receiving HIV-related drugs from other sources, 

HRSA should encourage ADAPs to expand their relationships with Medicaid to enable an 

annual exchange of information on prescription drugs dispensed to ADAP clients.. 

While not a rigorous scientific measure of program impact, with or without secondary 

data sources, this comparison would provide a general sense of ADAP’s contribution to 

Evaluation Systems: ADAP 2 OEI-05-98-00390 
client health outcomes. It would also allow for the analysis of patterns and trends that 

would suggest more in-depth explorations of client outcomes and program impact. 

Finally, it is a logical next step in understanding the impact of the ADAP program. The 

current Ryan White application guidance requires ADAPs to report the number of clients 

receiving protease inhibitors. The HRSA could simply expand this requirement to include 

a broader range of drugs along with a narrative comparison of how utilization patterns 

match the recommended clinical guidelines. 

The HRSA Should Encourage Collaborative Efforts among Ryan White Titles to 

Monitor Client Adherence to Drug Regimens 

Since positive health outcomes cannot be expected unless clients adhere to prescribed 

drug regimens, HRSA should work with adherence initiatives where they exist and 

encourage other Title I and Title II grantees, in consultation with ADAP, to promote and 

evaluate adherence. Given the difficulty of this complex behavioral and socio-economic 

issue, initiatives will likely require a combination of local resources. 

The HRSA Should Instruct ADAPs to Collaborate with Title I and Title II Needs 

Assessment Efforts to Assure That They Explicitly Evaluate Medication Therapy 

Needs 

To better assess unmet need, ADAP liaisons should work with and support current Title I 

or Title II needs assessment efforts. The ADAP liaisons should also ensure that needs 

assessment efforts actively solicit information from current ADAP clients as well as 

HIV-infected persons currently not enrolled in ADAP. 

AGENCY COMMENTS 

The HRSA provided comments on the draft report in which they did not concur with the 

first and third recommendations. The complete text of HRSA’s comments can be found in 

Appendix A. 

Regarding the first recommendation, HRSA stated that they did not have the statutory 

authority to require ADAPs to match client utilization data to NIH/PHS guidelines. The 

HRSA also stated that the data available to ADAPs would be insufficient to make this 

analysis meaningful. 

In response to HRSA’s concern regarding appropriate authority, the recommendation was 

altered to remove the implication of a statutory requirement. However, developing 

outcome measures for ADAP remains an important step in the evolution of the program. 

Matching utilization patterns to the National Institutes for Health/Public Health Service 

(NIH/PHS) Guidelines for the Use of Antiretroviral Agents in HIV Infected Adults and 

Adolescents would provide a general sense of ADAP’s contribution to client health 

Evaluation Systems: ADAP 3 OEI-05-98-00390 
outcomes. It is also a logical first step in outcome development. The HRSA already 

requires ADAPs to present a break-out of the utilization of protease inhibitors in their 

annual application for funds. Our recommendation would simply broaden this requirement 

to include an analysis of utilization by nationally recommended drug therapies. 

We appreciate HRSA’s concerns regarding the impact of incomplete utilization data. It is 

for this reason that we suggested ADAPs expand their current relationships with Medicaid 

to include a data match on drug utilization. This data sharing agreement could be based 

on the model data sharing agreement disseminated as part of the 1998 HRSA, Health 

Care Financing Administration (HCFA), and Centers for Disease Control (CDC) interagency agreement to promote the development of integrated public health information 

systems. 

Even with data limitations, the comparison of utilization data to drug therapy guidelines 

provides a sense of program contribution. Our recommendation to use this limited 

measure of program outcome builds on the United Way model which allows programs to 

take credit for their contribution to client outcomes while recognizing the practical 

limitations of assessing attribution in the absence of complete data. 

The HRSA also did not concur with the third recommendation that ADAPs should 

collaborate with Title I and Title II needs assessment efforts. They stated that ADAP was 

not the appropriate entity to take the lead in collaborative efforts with the other Ryan 

White programs. 

It was not our intention to imply that ADAP should take the lead in collaborating between 

the Ryan White programs. Rather, our recommendation directed ADAPs to more closely 

participate in the required Title I and Title II needs assessment efforts in order to have 

their evaluation needs better met. The HRSA’s stated intention to include instructions in 

the Title I application guidance regarding collaboration with ADAP reflects our intention 

to promote the level of meaningful collaboration between ADAP and the other CARE Act 

programs. 

Evaluation Systems: ADAP 4 OEI-05-98-00390 
TABLE OF CONTENTS 

PAGE

EXECUTIVE SUMMARY ............................................... 1

INTRODUCTION ..................................................... 6

FINDINGS

Sufficient Monitoring Mechanisms ........................................ 10

Most use Utilization Data, Few Developing Outcome Measures .................. 13

Minimal Assessments of Unmet Need .................................. 16

RECOMMENDATIONS ............................................... 17

AGENCY COMMENTS ................................................ 22

APPENDIX

A: Agency Comments ............................................. 24

B: Related Office of Inspector General Reports ........................... 32

Evaluation Systems: ADAP 5 OEI-05-98-00392 
INTRODUCTION 

PURPOSE 

To examine the evaluation systems established by Ryan White AIDS Drug Assistance 

Programs (ADAPs) to monitor program compliance, measure client outcomes and 

measure overall effectiveness. 

BACKGROUND 

This study complements our reports entitled Ryan White Evaluation Systems, Title I: 

Grants to Metropolitan Areas (OEI-05-98-00392) and Ryan White Evaluation Systems, 

Title II: Grants to States (OEI-05-98-00393). 

Ryan White CARE Act 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resource Emergency 

(CARE) Act as a comprehensive response to the HIV epidemic and its impact on 

individuals, families, communities, cities and States. The Ryan White programs provide 

health care and support services to persons with HIV/AIDS who would otherwise not 

have access to care. The Health Resources and Services Administration (HRSA) directs 

Ryan White resources to various local entities through the Act's four titles and Part F. 

The Ryan White programs constitute slightly over $1 billion of the $6.8 billion in 

HIV/AIDS expenditures included in the U.S. Department of Health and Human 

Services (DHHS) fiscal year (FY) 1998 budget. The bulk of Ryan White funds are 

allocated through Titles I and II of the Act. In FY 1998, nearly 90 percent of the Act’s 

funds were awarded to Title I and Title II grantees. 

The Ryan White CARE Act was re-authorized in 1996 through the year 2000. Changes to 

the Act in the re-authorization included a directive to Title I and Title II grantees to 

prioritize a portion of their funds for emerging populations. 

AIDS Drug Assistance Program 

Title II of the Ryan White CARE Act provides grants to States to improve the quality, 

availability and organization of health care and support services for individuals and 

families with HIV/AIDS. Section 2616(a) of the CARE Act requires States to use a 

portion of their Title II funds to establish an AIDS Drug Assistance Program. States are 

allowed to determine unique financial and medical eligibility for the ADAP in their State. 

Evaluation Systems: ADAP 6 OEI-05-98-00390 
States are also allowed to determine what drugs will be included in their formularies, how 

the drugs will be purchased and how the drugs will be distributed to clients. 

In FY 1999, Congress appropriated $461 million in Title II funds to be used for ADAP. 

This is an increase of 61 percent over FY 1998 when States received $285.5 million in 

Title II ADAP funds. These ADAP funds are typically augmented by contributions from 

State general revenue funds and, to a lesser extent, Title I grants. The mission of ADAP is 

to serve low-income people living with HIV/AIDS that are uninsured or under-insured and 

lack coverage for medications. 

The need for financial assistance to support expensive HIV/AIDS drug regimens was first 

identified in 1987 when Congress provided an emergency appropriation of $30 million to 

States. Beginning in late 1995, combination antiretroviral therapies, which include 

protease inhibitors, emerged as a new standard of care and ADAPs experienced explosive 

growth in enrollment and utilization. In November 1997, DHHS published Guidelines for 

the Use of Antiretroviral Agents in HIV Infected Adults and Adolescents developed by the 

Panel on Clinical Practices for Treatment of HIV Infection convened by the Department 

and the Henry J. Kaiser Family Foundation. These guidelines encouraged aggressive 

antiretroviral therapy for a broad range of HIV infected individuals, creating even greater 

demand for the program. In 1998, ADAPs served approximately 108,000 people with 

HIV/AIDS. 

Ryan White CARE Act Evaluation Requirements 

Section 2616(c)(5) of the Ryan White CARE Act requires that States document the 

progress made in making therapeutics available to individuals eligible for assistance. 

Section 2616(d) requires that the Secretary review the current status of State drug 

reimbursement programs, assess the barriers to the expanded availability of treatments, 

and examine the extent to which States coordinate with other Ryan White grantees to 

reduce barriers to the expanded availability of the treatments. 

The Ryan White CARE Act includes caps on the amount of funds Title II grantees, 

inclusive of ADAP, can spend on administrative and planning/evaluation functions. 

Grantees cannot exceed a 10 percent administrative cap and a 10 percent 

planning/evaluation cap, the combined total of which cannot exceed 15 percent. Grantees 

can pay for needs assessments, program evaluation and service delivery assessments out of 

their planning/evaluation cap. 

HRSA’s Current Evaluation Requirements 

In the FY 1998 Application Guidance, States were asked, for the first time, to complete an 

ADAP State Profile. The FY 1998 guidance also required a narrative description of 

Evaluation Systems: ADAP 7 OEI-05-98-00390 
agency administration, funding sources, formulary, eligibility criteria, client utilization, 

cost-savings strategies and coordination with other Ryan White funded services. 

The FY 1998 guidance also asked ADAPs to: 1) describe their system for tracking 

utilization and expenditure information on a monthly basis, 2) describe any significant 

changes in client and drug utilization, 3) compare the demographic characteristics of 

ADAP clients to the demographic characteristics of the HIV epidemic of their State and 

discuss any discrepancies, and 4) describe the barriers to access and plans to enroll underserved populations. The FY 1999 application asks that ADAPs provide a brief overview 

of client enrollment and utilization trends. It does not provide direction on how this 

should be achieved. 

The ADAPs are also required, as a condition of their award, to file monthly and annual 

reports with HRSA. The monthly reports track enrollment and average monthly 

utilization. They specify a break-out for the utilization of protease inhibitors. The annual 

administrative report requires ADAPs to document the services they provide and the 

characteristics of clients receiving services. 

As a program administered by the State under Title II, ADAPs are required to outline 

goals and objectives for the upcoming year in an Implementation Plan. These must be 

accompanied by the service or outcome measure used to evaluate goal attainment. The 

application guidance includes people served and units of service as examples of 

outcome/service measures. The achievement of these goals must be reported in the next 

year’s application. 

Re-authorization 

The Ryan White CARE Act of 1990 will be considered for its second re-authorization in 

2000. As Congress debates this re-authorization, questions are certain to be asked about 

the effectiveness and impact of the Ryan White programs. Grantees and administrators 

will also be asked to account for whether and how the programs have evolved to meet the 

shifting needs of the communities served. 

SCOPE AND METHODOLOGY 

This inspection is an examination of the evaluation systems established by State AIDS 

Drug Assistance Programs. Specifically, this report documents the mechanisms ADAPs 

use to monitor their program compliance, measure client outcomes, and assess their 

effectiveness in meeting the evolving needs of the population affected by the epidemic. 

This evaluation does not judge the compliance, measure the outcomes or assess the 

effectiveness of the ADAPs, but rather judges whether the drug assistance programs have 

Evaluation Systems: ADAP 8 OEI-05-98-00390 
and appropriately utilize the tools to perform these evaluation functions. This study also 

does not examine HRSA’s monitoring and evaluation of the ADAPs. 

For the purposes of this analysis, we define compliance as serving the population, 

providing the services and spending the resources as agreed to in return for the award of 

program funds. Outcomes are defined as measures of the benefits clients derived as a 

result of the program. Finally, we define overall effectiveness as how well the mix of 

services meets the evolving needs of the population affected by the epidemic. 

A purposive sample of eight Title II drug assistance programs were selected for this 

evaluation. The sampled ADAPs consist of New York, New Jersey, Maryland, Florida, 

Illinois, Texas, California and the District of Columbia. The sites were selected based on 

the number of reported AIDS cases, FY 1998 Ryan White funding, high incidence of 

intravenous drug use transmission, the demographics of the AIDS/HIV population, and 

the regional diversity of the sites. These seven States and DC represent 68 percent of the 

living AIDS cases reported in 1997 and 61 percent of the Title II funding for FY 1998. 

For each of the eight drug assistance programs selected, we conducted an extensive 

document review. We reviewed their FY 1998 grant application, templates of their 

programmatic and fiscal reports, enrollment guidelines, client satisfaction surveys and any 

other documents pertaining to their use of outcomes or evaluations of the overall 

effectiveness of the program. 

We also conducted on-site interviews with the sampled ADAPs between October and 

December 1998. We interviewed the ADAP administrator and any staff responsible for 

monitoring, outcomes or evaluation. 

In addition to our survey work, we interviewed knowledgeable organizations such as the 

Center for Disease Control (CDC), the HIV/AIDS Bureau of the Health Resources and 

Services Administration and the Office of HIV/AIDS Policy. These meetings were 

complimented by background materials on the Ryan White program provided by HRSA. 

We conducted our review in accordance with the Quality Standards for Inspections issued 

by the President’s Council on Integrity and Efficiency. 

Evaluation Systems: ADAP 9 OEI-05-98-00390 
FINDINGS 

The AIDS Drug Assistance Programs in the eight States we 

visited have sufficient programmatic and fiscal monitoring 

mechanisms in place 

Routine program and fiscal reports 

The ADAP administrators monitor program activity through the review of routine 

programmatic reports. These activity reports are typically reviewed on a monthly and, 

sometimes weekly, basis. They contain aggregate data on program processes such as the 

number of clients enrolled, newly approved applications, the number of clients served, the 

number of prescriptions filled by drug and the number of clients receiving each covered 

drug. One ADAP receives a standard set of 13 tables every month that includes an 

unduplicated client count, new enrollees by ethnic group and county, CD4 count by age 

group, client income by county, insurance status, etc. Typically, the point-in-time data 

provided on a monthly basis is analyzed longitudinally to detect enrollment and utilization 

trends and project utilization needs for the future. 

The ADAPs ensure the fiscal integrity of their programs by monitoring regular reports of 

their program expenditures. Weekly batches of pharmacy claims are bundled into monthly 

expenditure reports. These financial reports usually provide an accounting of expenditures 

during the reporting month and year-to-date figures. Some ADAPs produce more 

complex financial reports that include items such as average cost per prescription, average 

cost per client and average fills per client. Expenditure reports are typically accompanied 

by supporting documentation such as the payroll register, pharmacy claims for drugs 

dispensed and orders for drugs. 

Centralized databases facilitate routine reporting, but remain under-utilized 

All of the programmatic and fiscal reports used to monitor program performance are 

created off of centralized computer data management systems. These administrative data 

systems track confidential client demographic and service information as well as claims 

information. One State maintains 1,000 fields on each client including basic 

demographics, income information, insurance, CD4 count, viral load, disease staging 

information, treatment history, risk factor, and so forth. 

Despite the rich resource of data contained in these centralized data systems, some 

administrators do not fully utilize the capabilities at their disposal. Most of the ADAP 

Evaluation Systems: ADAP 10 OEI-05-98-00390 
administrators indicated that their centralized data systems allow instant and easy access to 

a variety of up-to-the-minute, client level program data and analyses. Further, they stated 

that the databases provide the capacity to respond to emerging program and policy issues 

with quantitative analysis. However, only half of the ADAPs mentioned actually using 

their databases in this manner. For example, one of these ADAPs recently used their 

database to respond to concerns regarding the number of pediatric cases, the number of 

women served, and high utilization rates in particular counties and pharmacies. Other 

administrators feel that while they are collecting all of the necessary programmatic data, 

they have yet to realize the full potential of the data to assist in program management and 

evaluation. 

Standard eligibility criteria are uniformly enforced 

Standard enrollment forms, and training on how to appropriately complete them, are used 

to ensure that all required eligibility information is obtained in a consistent manner. 

Standard forms enhance the efficiency of enrollment procedures and the ability to evaluate 

the accuracy of the data. These forms are extensive and typically request demographic, 

financial and medical information in accordance with State eligibility rules. Appropriate 

support documentation is also required such as personal identification cards, income 

verification statements and a physician signed medical certification of HIV diagnosis. One 

State AIDS Drug Assistance Program trimmed 2,000 duplicates from its rolls last year 

when it standardized its eligibility screening and enrollment practices. 

All ADAPs centrally review enrollment forms and the supporting documentation for 

completeness and accuracy. The central screening and processing of enrollment 

applications helps ensure a uniform enforcement of the program eligibility rules. It also 

helps prevent duplication of services and other fraud. Some ADAPs have further 

centralized the enrollment process by requiring all application forms to be sent to the 

central administration office for initial eligibility determination. 

Seven of the ADAPs visited redetermine eligibility on an annual or semi-annual basis. One 

AIDS Drug Assistance Program only initiates redetermination procedures if alerted to a 

potential change in Medicaid status or other eligibility criteria. 

Payor of last resort status ensured mainly through automated Medicaid matches 

All ADAPs rely on regular, automated matches with Medicaid databases as the primary 

tool to assure that ADAP funds are being used as the payor of last resort. The ADAPs 

focus their enforcement energies on Medicaid since they feel it is the most likely source of 

overlapping funding and they have the greatest ability to enforce compliance. Other 

techniques used to enforce payor of the last resort status include explicit instructions 

Evaluation Systems: ADAP 11 OEI-05-98-00390 
written into pharmacy contracts and highlighted on enrollment forms and soliciting 

information regarding other third-party insurance on client enrollment forms. 

Medicaid matches are a part of the standard ADAP enrollment process. In the majority of 

ADAPs we visited, Medicaid receipt disqualifies applicants from ADAP eligibility. In 

order to enforce this, ADAPs automatically cross check all applicants against the Medicaid 

rolls. The ADAPs we visited all have on-line capacity to perform this check. The ease 

and speed with which this automated match occurs allows ADAPs to perform this check 

with every application at the time of application and without compromising a client’s 

access to necessary drugs. 

Periodic matches of enrollment files to Medicaid membership files are also conducted to 

monitor client Medicaid eligibility status which can fluctuate over time. It is typical 

procedure to approve applicants for ADAP who appear to be Medicaid eligible while also 

helping them apply for Medicaid benefits. Medicaid eligible HIV-infected persons can also 

receive ADAP to aid in spend down or cost sharing requirements of Medicaid or if they 

have limited Medicaid coverage. The periodic checks against the Medicaid rolls allows 

the ADAPs to monitor on-going client transitions from ‘Medicaid Pending’ status to full 

Medicaid benefits so the appropriate program is billed. The ADAPs vary tremendously in 

how often they perform this on-going, automated check. Some ADAPs check on a 

monthly basis, some weekly, and others check with every prescription request. 

One weakness in the current system is the difficulty of recouping payments from Medicaid 

for those ADAP clients who cycle onto Medicaid while still receiving benefits through 

ADAP. Since the matches do not occur on a daily basis, and some ADAPs go as long as a 

month before checking the Medicaid rolls, there is the potential for ADAPs to incur costs 

for Medicaid-covered services. Of the eight ADAPs interviewed, only one mentioned 

having a way to bill Medicaid retroactively. Upon discovering an overpayment, this drug 

assistance program withholds future payments to the respective pharmacy for the amount 

billable to Medicaid. One other ADAP indicated in their application for funds that they 

have negotiated a way to retroactively bill Medicaid. 

Cost containment strategies are used to limit drug expenses 

Fiscal control over expenditures is further maintained by employing a variety of cost 

containment strategies to purchase the drugs at the lowest possible price. Six of the eight 

ADAPs purchase drugs through Section 602 of the Veteran’s Health Care Act. This 

allows them to purchase drugs at a discounted Public Health Service (PHS) price or 

access a rebate option. The Office of Drug Pricing that administers the Federal Drug 

Pricing Program for purchasing drugs at the PHS price requires ADAPs to follow 

established regulations in order to qualify. These regulations were designed to avoid the 

diversion of drugs purchased under this program to non-eligible patients. Thus, for these 

drug assistance programs, there is an additional level of Federal control. 

Evaluation Systems: ADAP 12 OEI-05-98-00390 
For those States where the Federal drug discount pricing program is not an available 

option due to program structure, multiple other strategies are pursued. These include 

negotiating savings off of the Average Wholesale Price (AWP) and voluntary rebate 

programs. The States we visited have negotiated from 10 to 15 percent off of the AWP. 

These States also typically search for further savings through avoiding or reducing 

infrastructure costs potentially incurred through storing and distributing the drugs. 

Despite the efforts individual State ADAPs are making to ensure they are purchasing 

drugs at the lowest possible price, variations in the purchase prices States are paying 

remain. Each AIDS Drug Assistance Program is allowed to craft their own cost 

containment strategy. This allows for maximum State flexibility, but it also means States 

have only their own resources and ingenuity to rely upon to overcome the barriers to 

accessing the lowest drug prices. Some ADAPs suggested that HRSA and the Health 

Care Financing Administration (HCFA) assume a leadership role at the national level in 

order to obtain the best possible drug prices for the ADAP program. 

While most ADAPs rely on process measures to gauge 

program progress, a few are developing ways to measure 

client outcomes 

The AIDS drug assistance program was conceived as a means to provide prescription 

drugs to HIV infected individuals who were uninsured or underinsured. In response to 

this and tremendous funding pressures, some ADAPs have limited their activities to the 

process of supplying prescription drugs to HIV-infected persons in need. They view the 

program primarily as a financing and distribution mechanism. However, an increasing 

number of administrators are taking on the responsibility of monitoring program impact to 

improve program efficiency and effectiveness and to respond to demands for greater 

accountability. 

Most ADAPs track client utilization of drugs to monitor program success 

Most ADAP administrators reported that they rely on the utilization data reported in their 

regular program reports to assess the success of their program. The utilization reports 

typically provide, in aggregate, the number of clients actively enrolled and the number of 

drugs prescribed by drug class or specific drug. Often they will break out utilization by 

gender, race/ethnicity and age. A few ADAPs also regularly track enrollment and 

utilization trends by sub-population or targeted neighborhoods. Utilization goals are 

outlined in the Implementation Plan in each year’s application for funding. 

A few of the AIDS Drug Assistance Programs currently, or plan to, take this review of 

utilization data a step further by using the data as a basis to extrapolate health outcomes. 

Evaluation Systems: ADAP 13 OEI-05-98-00390 
This analysis involves reviewing drug utilization data to determine how closely the drugs 

dispensed match the aggressive drug regimens recommended by the National Institutes of 

Health/Public Health Service (NIH/PHS) Guidelines for the Use of Antiretroviral Agents 

in HIV Infected Adults and Adolescents1

. If the analysis reveals that ADAP prescriptions 

accurately mirror NIH/PHS guidelines, these ADAP administrators assume their clients 

are benefitting from the positive health outcomes attributed to these drug therapies in the 

medical research. One ADAP conducts a quarterly analysis comparing drug usage to 

NIH/PHS guidelines. Another ADAP is planning on examining prescribing patterns as 

well as utilization patterns by reviewing a random selection of ADAP clients’ medical 

charts. 

A few ADAPs are planning on tracking individual client health outcomes to 

determine program success 

While the majority of AIDS Drug Assistance Programs are currently using process 

measures to track program functioning and outcomes, a few ADAPs are planning on 

incorporating client health outcomes into their tracking systems in order to document 

program impact. Outcome measures under consideration include CD4 counts, viral loads, 

opportunistic infections, and hospital admissions. One ADAP is struggling to move 

beyond strict measures of health outcomes by developing broader measures of quality of 

life. They are interested in tracking how long their clients are ambulatory, to what extent 

their clients can take care of themselves, the number and length of hospitalizations, and the 

development and duration of opportunistic infections. 

Most ADAPs developing outcome measures plan to track the data for these measures by 

augmenting their current client-level, administrative data systems. They plan to request 

pertinent clinical information on the original enrollment form and subsequent forms 

submitted for re-eligibility determination. A few ADAPs are already capturing CD4 

counts in this manner, but none are currently tracking the variations over time as a 

measure of client health. The clinical information obtained on the enrollment form will be 

input into the current database where it can be linked to existing client utilization data for 

analysis. One ADAP indicated they would be tracking outcome measures in another year. 

Other ADAPs felt the development of outcome measures would take a few years. 

One of the ADAPs planning on developing outcome measures also plans to assess client 

outcomes by means of an independent evaluation. Over the next year and a half, they plan 

1 These guidelines contain recommendations for the clinical use of antiretroviral agents in the 

treatment of HIV-infected adults and adolescents. A separate document contains 

pediatric-specific issues related to antiretroviral therapy. These guidelines were developed to 

serve as a companion document to the therapeutic principles formulated by the National Institute 

of Health Panel to Define Principles of Therapy of HIV infection. 

Evaluation Systems: ADAP 14 OEI-05-98-00390 
to review a random selection of clients. This special project will focus on client’s clinical 

outcomes, length of participation in the ADAP program and the extent to which nonadherence to the prescribed drug regimen negatively impacts health outcomes. 

Some ADAP administrators feel that assessing client health outcomes is beyond 

the scope of ADAP 

Some ADAP administrators feel that measuring outcomes is not an appropriate way to 

expend limited resources. They believe the process measures tracked in their utilization 

reports are sufficient to judge their program’s performance against the goal of efficiently 

distributing prescription drugs on the formulary to uninsured or underinsured HIVinfected individuals. As one ADAP administrator stated, “We are just responsible for 

getting drugs to folks who need them and this is where our limited money should go.” 

Some ADAP administrators also argue that it is best to rely on medical research 

professionals to measure clinical outcomes. According to these respondents, in order to 

get reliable results on clinical outcomes, outcome evaluations should be controlled clinical 

studies. These types of studies are best directed by skilled researchers who have the 

expertise to properly conduct such evaluations and access to the necessary resources. 

Finally, several ADAP administrators feel that requiring ADAPs to measure client 

outcomes would duplicate work completed by the medical evaluation field. Medical 

studies already exist on the effects of the drugs since all approved drugs are put through 

extensive clinical trials prior to Food and Drug Administration approval. These studies 

demonstrate what the expected outcomes for a given drug regimen should be. Only drugs 

that are proven to reduce HIV/AIDS symptoms and prolong life are placed on the 

formulary to be distributed by ADAP. Thus, the ADAP administrators reason, as long as 

the program ensures that it is fulfilling prescrip`tions according to the prescribed 

guidelines, they should be able to assume that their medical outcomes reflect the outcomes 

demonstrated in medical research. As one administrator put it, “ADAP outcomes can only 

be the outcomes predicted by medical studies.” 

Evaluation Systems: ADAP 15 OEI-05-98-00390 
ADAPs are conducting minimal assessments of whether they 

are reaching populations in need 

While all of the sample ADAPs conduct periodic matches of client and 

epidemiological data, they conduct little additional research to determine whether 

they are reaching the populations in need 

All ADAPs indicated that they match client demographics to the demographic profile of 

the epidemic to measure how they are meeting population needs. The majority of ADAPs 

perform this analysis annually. Some ADAPs perform this analysis more frequently, on an 

“as needed basis.” Typically, the analysis consists of comparing the demographics of 

enrolled clients to AIDS surveillance data broken out by gender, race/ethnicity and age. A 

few of the ADAPs we visited make use of HIV prevalence estimates. The two data sets 

are compared in order to locate any discrepancies. These discrepancies are explored as 

possible areas of unmet need to address in the future. 

However, many ADAP administrators expressed great reservations about judging the 

effectiveness of their programs by this simple methodology. Their major concern is that 

Ryan White AIDS Drug Assistance Programs are not the only source of funding for 

HIV/AIDS medication. A certain proportion of HIV/AIDS cases access drugs through 

State Medicaid programs, private insurance or clinical trials. One ADAP study found that 

90 percent of the HIV-infected residents in an area with disproportionately low ADAP 

enrollment rates were enrolled in Medicaid. 

Despite the disadvantages of relying on an epidemiological match to assess program 

effectiveness and growing pressure to better assess the needs of traditionally underserved 

populations, ADAPs have not found other, more sophisticated means to assess whether 

they are reaching populations in need. The sources ADAPs mentioned they rely on for 

information regarding unmet need and access to services include activist groups, the 

ADAP advisory boards, and the Statewide Coordinated Statement of Need’s assessment 

of overall State needs. Only three of the ADAPs we visited indicated receiving 

information from planning council or consortia needs assessment efforts. 

Evaluation Systems: ADAP 16 OEI-05-98-00390 
RECOMMENDATIONS 

To promote outcome measurement, HRSA should encourage 

ADAPs to compare utilization data to recommended clinical 

guidelines 

As a first step in gathering a better sense of program impact, HRSA should encourage 

drug assistance programs to review whether drugs are dispensed in accordance with the 

NIH/PHS Guidelines for the Use of Antiretroviral Agents in HIV Infected Adults and 

Adolescents. To the extent that the comparison reveals that drug distribution accurately 

mirrors the guidelines, ADAP administrators and HRSA can assume that clients are 

benefitting from the positive health outcomes attributed to specific drug therapies 

evaluated in medical research studies. 

To create a more meaningful picture of program impact, HRSA should also encourage 

ADAPs to expand their relationships with Medicaid to enable an annual exchange of 

information on HIV-related drugs dispensed to ADAP clients. The resulting data match 

would render a more complete picture of clients’ access to prescription drugs. It would 

also be helpful, where possible, to establish data exchanges with the other drug 

reimbursement sources such as private insurance companies and other pharmacy assistance 

programs. 

Most ADAPs already have data relationships with State Medicaid offices in order to verify 

payor of last resort status. The HRSA could encourage ADAPs to expand these data 

sharing arrangements to include an interchange of utilization data. The model data sharing 

agreement disseminated as part of the 1998 inter-agency agreement between HRSA, 

HCFA, and the CDC could be used. This inter-agency effort was undertaken to facilitate 

data sharing between State Medicaid and other health agencies to support program 

monitoring, promote the development of common performance measures across multiple 

programs and improve the quality of care for vulnerable populations. 

While matching ADAP distribution patterns with the NIH/PHS guidelines would 

inevitably lack some information regarding drug utilization, it would, nonetheless, provide 

a general sense of ADAP’s contribution to client health outcomes. This approach to 

outcome measurement corresponds with a highly regarded method of outcome 

development promoted by the United Way. This approach recognizes the practical 

limitations faced by program administrators searching for a way to gage program impact. 

These limitations typically mean that rigorous program impact evaluations are beyond the 

means and scope of these agencies. Thus the United Way endorses using outcome 

measures of contribution as the first step in building meaningful outcome measures. 

Evaluation Systems: ADAP 17 OEI-05-98-00390 
Beyond providing a broad measure of contribution, this comparison would also allow for 

the analysis of patterns and trends that would suggest more in-depth explorations of client 

outcomes and program impact. For example, if disparities in the quality of care are 

detected in the client population through this broad analysis, ADAPs could engage in a 

more in-depth examination of utilization in specific sub-populations. This data and 

analysis could be provided to clinical providers as a form of feedback. The ADAPs might 

also want to take the next step of surveying physician prescribing practices. 

This approach to outcome measurement also integrates the United Way philosophy of 

using data sources readily available to agencies. Most AIDS drug assistance programs are 

already relying on client utilization data as a means to gage program success. Asking 

them to analyze these utilization patterns against a national standard is a logical next step 

to a more sophisticated understanding of program impact. In fact, some ADAPs already 

routinely perform this analysis. All of the ADAPs that we interviewed have utilization 

data at their disposal and the capacity to perform this analysis. Further, the current Ryan 

White application guidance already requires ADAPs to report the number of clients 

receiving protease inhibitors. The HRSA could simply expand this requirement to include 

a broader range of drugs along with a narrative comparison of how utilization patterns 

match the recommended clinical guidelines. 

HRSA should encourage collaborative efforts among Ryan 

White Titles to monitor client adherence to drug regimens 

Since positive health outcomes cannot be expected unless clients adhere to prescribed 

drug regimens, HRSA should work with adherence initiatives where they exist and 

encourage other Title I and Title II grantees, in consultation with ADAPs, to promote and 

evaluate adherence. Given the difficulty of this complex behavioral and socio-economic 

issue, initiatives will likely require a combination of resources and a collaboration of 

organizations. Medical providers funded by Title I and Title II are responsible for 

prescribing the most appropriate drug regimen for the patient. Medical and other support 

service providers that have on-going personal interaction with the clients have the capacity 

to educate and monitor patients on adherence issues. The ADAP program could 

contribute utilization trend data and information on which drugs clients are obtaining. 

The HRSA could encourage collaborative adherence initiatives by disseminating the latest 

medical research on the impact of non-adherence and offering technical assistance on how 

to promote and evaluate adherence. Recommendations, best practices and implementation 

models could stem from adherence studies currently underway. 

Recognizing that adherence ensures their clients will obtain the maximum benefit from 

advanced drug therapy, several locations are already grappling with this difficult issue. 

One planning council outlined their plans to make adherence a service category in order to 

Evaluation Systems: ADAP 18 OEI-05-98-00390 
allocate funds to an adherence initiative. One of the grantees we visited has been awarded 

funds through the Special Projects of National Significance program to study adherence. 

Two of the ADAPs we interviewed are involved in adherence initiatives. One is 

monitoring patient adherence by generating a list of clients who do not pick up their refills 

from the pharmacy and are past due to receive their medications. Pharmacists perform a 

weekly manual review of refill logs to see which patients have not picked up their 

prescriptions. The file is flagged for further follow-up initiatives including contacting the 

patient directly or contacting the patient’s case manager. 

Techniques for adherence promotion include the creation and dissemination of patient 

education materials and behavior modification programs. Monitoring methods include 

supervised drug intake, on-going contact through beepers and regularly convened peersupport groups. In cases where limited resources do not allow for such intense 

monitoring, adherence information could be collected through limited adherence studies 

using random sampling techniques. 

The HRSA should facilitate the spread of these efforts to help ensure that clients are 

reaping the benefits of advanced drug therapy. Gathering adherence information is the 

only way that ADAP programs can assure their clients are benefitting from the drugs the 

program is providing and that the drugs are having the intended impact. 

The HRSA should instruct ADAPs to collaborate with Title I 

and Title II needs assessment efforts to assure that they 

explicitly evaluate medication therapy needs 

The ADAPs need a more sophisticated way to assess whether or not they are meeting the 

medication needs of the HIV/AIDS population. The ADAPs are being called to greater 

accountability for their efforts to reach and serve traditionally underserved populations. 

The current method of matching the demographics of ADAP clients with the 

demographics of local HIV cases provides useful information on the needs of infected 

persons. However, as ADAP administrators indicated, it is a simple method that paints an 

incomplete picture. Several ADAP administrators pointed out that relying on a match of 

ADAP cases to existing public health data overestimates unmet need since it does not take 

into account other programs providing drug therapy. Other shortcomings include lack of 

HIV prevalence data in some States and the reliance on enrollment data rather than active 

cases to estimate program utilization. 

Not only is the currently utilized methodology unreliable, but it only provides a general 

picture of unmet need. It does not provide the ADAPs with a reliable way to pinpoint 

specific subpopulations in various regions of the State. It also does not present a 

Evaluation Systems: ADAP 19 OEI-05-98-00390 
qualitative picture of unmet need. Barriers to care, historic mistrust of the medical 

profession and the stigma of HIV add a complexity to the needs of various subpopulations of infected persons which are often not reflected through a quantitative 

analysis of unmet need. 

Finally, properly judging unmet need is essential not only to ensure that medications are 

equitably distributed to populations in need, but also to better target current ADAP 

outreach efforts. The U. S. Surgeon General David Satcher has called for ADAP 

programs to do a better job of targeting minority populations. The ADAP programs must 

have a reliable way to determine the subpopulations in their State that require further 

attention in order to better target their outreach initiatives. 

While ADAPs may lack the resources and funding to conduct intensive outreach efforts 

aimed at specific subpopulations, they do have the responsibility to make sure that all 

those in need of HIV medication are aware of the program in order to ensure that existing 

access to health care disparities are not simply perpetuated by ADAP. One way to 

accomplish this would be to target outreach efforts to specific organizations that focus on 

traditionally underserved populations. 

To better assess unmet need, HRSA should instruct ADAPs to appoint liaisons or create a 

mechanism to work with and support Title I and Title II needs assessment efforts to assure 

a comprehensive assessment of prescription drug needs. Title I and Title II grantees are 

required to conduct routine needs assessments as part of their priority setting and 

allocation processes. The surveys and focus groups used to collect information for these 

needs assessments typically capture information regarding unmet need for prescription 

drugs. However, since the goal of the assessment is to gather information on primary 

medical care and support services, the information on medication therapy needs tends to 

be very basic. Furthermore, it does not appear that this information is routinely relayed to 

ADAP administrators. 

The ADAPs should also collaborate with Title I and Title II needs assessment efforts to 

make sure that the Title I and Title II grantees actively solicit respondents from the 

currently enrolled ADAP population as well as those HIV-infected persons who are 

currently not enrolled in the ADAP. This would allow ADAPs to access information from 

a broader spectrum of HIV-infected individuals and provide a richer understanding of 

unmet need and barriers to care. 

The ADAP needs assessment liaison could be appointed from the administrative staff or 

the ADAP advisory boards. In some States, these advisory boards already take on 

responsibilities beyond determining the ADAP formulary. In particular, two of the 

ADAPs visited have advisory bodies that provide overall program and policy guidance and 

recommendations. Like Title I planning councils and Title II consortia, they advise on all 

programmatic issues and strive to be representative of the epidemic. The liaison could 

Evaluation Systems: ADAP 20 OEI-05-98-00390 
work with the Title I and Title II planning bodies to assure that they collect needed 

information to assist the ADAPs in fulfilling their mission of supplying drug therapy to 

HIV-infected individuals in need. 

Evaluation Systems: ADAP 21 OEI-05-98-00390 
AGENCY COMMENTS 

The HRSA provided comments on the draft report in which they did not concur with the 

first and third recommendations. Where appropriate, we changed the report to reflect 

their comments, concerns and technical corrections. The complete text of HRSA’s 

comments can be found in Appendix A. 

Regarding the first recommendation to encourage ADAPs to match client utilization data 

to NIH/PHS guidelines as a broad measure of program impact, HRSA stated that they did 

not have the statutory authority to require ADAPs to perform this analysis. The HRSA 

also stated that the data available to ADAPs would be insufficient to make meaningful 

comparisons and extrapolate client health outcomes. 

In response to HRSA’s concern regarding their ability to require ADAPs to perform 

evaluative functions, the recommendation was altered to remove the implication of a 

statutory requirement. However, developing outcome measures for ADAP remains an 

important step in the evolution of the program. In our interviews, we found that many 

ADAPs have moved beyond a limited sense of mission and accountability. They do not 

view the drug assistance program as merely a financing mechanism and recognize the 

importance of documenting program impact. They recognize that measuring outcomes 

provides a means to promote the achievements of the program in terms of its most 

valuable contributions - the quality and prolonged life of its clients. 

Further, the match is a logical first step in outcome development. The HRSA already 

requires ADAPs to present a break-out of the utilization of protease inhibitors in their 

annual application for funds. Our recommendation would simply broaden this requirement 

to include an analysis of utilization of nationally recommended drug therapies. 

We appreciate HRSA’s concerns regarding the impact of incomplete utilization data. 

Since ADAP is only one of many reimbursement mechanisms for HIV drug therapies, it is 

true that any picture of client drug utilization constructed by ADAP would not accurately 

represent total client access to HIV medication. It is for this reason that we suggested 

that ADAP expand their current relationships with Medicaid to include a data match on 

drug utilization. This data sharing agreement could build on the relationships already 

established with Medicaid in order to verify payor of last resort status and the 1998 

HRSA, HCFA and CDC inter-agency agreement to promote the development of 

integrated public health information systems. 

While matching ADAP distribution patterns with the NIH/PHS guidelines would 

inevitably lack some information regarding drug utilization, it would, nonetheless, provide 

a general sense of ADAP’s contribution to client health outcomes. In fact, the measure 

Evaluation Systems: ADAP 22 OEI-05-98-00390 
was never intended to serve as a rigorous measure of program impact. We recognize that 

this type of evaluation is beyond the scope of ADAP agencies and that leading medical 

institutions around the country are involved in efforts to understand the impact of various 

drug combinations on client health outcomes. Our recommendation is based on the 

United Way model which allows programs to take credit for their contribution to client 

outcomes while recognizing the practical limitations on assessing attribution in the absence 

of complete data. A few ADAPs are already engaged in this type of analysis and have 

found it a useful tool for understanding program impact. 

The HRSA also did not concur with the recommendation that ADAPs should collaborate 

with Title I and Title II needs assessment efforts. They stated that the drug assistance 

programs were not the appropriate entities to take the lead in collaborative efforts with the 

other Ryan White programs. They were also concerned by the suggestion that the ADAP 

advisory boards could take on the responsibility of collaboration since ADAP advisory 

boards are typically clinical bodies supplying ADAP formulary advice. 

It was not our intention to imply that ADAP should take the lead in collaborating between 

the Ryan White programs. Rather, our recommendation directed ADAPs to more closely 

participate in the required Title I and Title II needs assessment efforts in order to have 

their evaluation needs better met. We offered the ADAP advisory boards as a possible 

liaison source only where the board was already functioning in a larger capacity than 

formulary advisement. The specific designation of liaison responsibilities would be the 

responsibility of each ADAP administration. Of ultimate importance is that meaningful 

collaboration happens on a continuing basis. The HRSA’s stated intention to include 

instructions in the Title I application guidance regarding collaboration with ADAP reflects 

our intention to promote the level of meaningful collaboration between ADAP and the 

other Ryan White programs. 

Evaluation Systems: ADAP 23 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 24 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 25 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 26 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 27 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 28 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 29 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 30 OEI-05-98-00390 
APPENDIX A 

Agency Comments

Evaluation Systems: ADAP 31 OEI-05-98-00390 
APPENDIX B 

Related Office of Inspector General Reports

The Ryan White CARE Act: Local Implementation Issues (OEI-05-93-00336) 

The Ryan White CARE Act: Examples of Local Coordination (OEI-05-93-00335) 

The Ryan White CARE Act: Special Projects of National Significance (OEI-05-93-00332) 

The Ryan White CARE Act: Technical Report of 1992 Expenditures (OEI-05-93-00334) 

The Ryan White CARE Act: Funding Formulas (OEI-05-93-00330) 

Medicaid Managed Care and HIV/AIDS (OEI-05-97-00210) 

Audit of State AIDS Drug Assistance Programs’ Use of Drug Price Discounts (A-01-97-01501) 

Review of Maximizing Drug Discounts Under the Missouri Ryan White Program 

(A-01-97-00926) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the Health Resources and Services Administration (A-01-97-01500) 

Audit of Eligibility Under Title I of the Ryan White Comprehensive AIDS Resource Emergency 

Act of 1990, New York Metropolitan Area (A-02-95-02517) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the State of Connecticut (A-01-96-01501) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990 in the Boston 

Metropolitan Area for Fiscal Year 1994 (A-01-95-01504) 

Evaluation Systems: ADAP 32 OEI-05-98-00390