Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

JUNE GIBBS BROWN 

Inspector General 

AUGUST 1999 

OEI-05-98-00393 

RYAN WHITE EVALUATION SYSTEMS 

Title II: Grants to the States 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through a 

nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Chicago Regional Office prepared this report under the direction of William Moran, 

Regional Inspector General. Principal OEI staff included: 

REGION 

William Moran, Project Leader

Ann Maxwell, Lead Analyst

Victoria Jacobs, Analyst

Nora Leibowitz, Analyst

Vince Greiber, New York Regional Office

HEADQUARTERS 

Elise Stein, Program Specialist 

Rada Spencer, Program Specialist 

Alan Levine, Program Specialist 

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SUMMARY 

PURPOSE 

To examine the evaluation systems used by Title II Ryan White grantees to monitor the 

compliance, measure the outcomes and measure the overall effectiveness of the programs 

they fund. 

BACKGROUND 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 

(CARE) Act as a comprehensive response to the HIV epidemic and its impact on 

individuals, families, communities, cities, and States. The Ryan White CARE Act was reauthorized in fiscal year (FY) 1996 through FY 2000. Title II of the Act provides grants 

to States to improve the quality, availability and organization of health care and support 

services for individuals and families with HIV disease. In FY 1999, the States received 

$710 million in Title II funds. Most grantees disburse a portion of these funds to service 

providers through regional fiscal agents called lead agencies and planning bodies called 

consortia. 

The Ryan White CARE Act will be considered for its second re-authorization in 2000. As 

Congress debates this re-authorization, questions are certain to be asked about the 

effectiveness and impact of the programs funded. This inspection examines the capacity of 

Title II Ryan White grantees to answer those questions. 

Through an extensive documentation review and on-site interviews, we examined the 

evaluation mechanisms used in 8 States, 12 consortia/lead agencies and 5 consortia 

responsible for planning only. The eight States visited represent 61 percent of FY 1998 

Ryan White Title II funds and 68 percent of reported AIDS cases in FY 1997. 

FINDINGS 

In Most Sampled States, Program and Fiscal Compliance of Service Providers Is 

Monitored Through Fiscal Intermediaries; Intermediaries Have Monitoring 

Mechanisms in Place 

In the eight States we visited, seven of the grantees use regionally based lead agencies to 

serve as fiscal agents for the service providers. In six of the States, the lead agencies have 

primary responsibility for monitoring the program and fiscal compliance of the service 

providers. The lead agencies receive regular program and fiscal reports from the 

providers and conduct frequent site visits to providers. Likewise, the grantees receive 

Ryan White Evaluation Systems: Title II 1 OEI-05-98-00393 
regular program and fiscal reports from the lead agencies and conduct periodic site visits 

to them. Most of the grantees and lead agencies have mechanisms in place to ensure that 

Ryan White funds are used only for eligible clients and as the payor of last resort. 

Few Grantees or Lead Agencies Currently Require Outcome Measurement of 

Their Service Providers; Most Are in the Early Stages of Discussion Regarding 

Outcome Measures 

Currently, only 1 of the 8 State grantees and 1 of the 12 lead agencies we visited, require 

client outcomes to be measured. The majority of grantees and lead agencies are in the 

early stages of planning for outcome measurement. Concerns regarding outcomes 

measures and a desire for technical assistance from the Health Resources and Services 

Administration (HRSA) were expressed by over half of the grantees and lead agencies. 

Grantees Rely on Consortia to Assess Needs; Most Consortia Routinely Assess 

the Needs of Clients but Not of Persons Outside of HIV/AIDS Care 

All of the State grantees we visited rely on regional consortia to assess the needs of 

persons with HIV/AIDS and require them to do so on a routine basis. Slightly over half 

of the grantees require the consortia to submit the needs assessment results as part of the 

consortium’s annual application for funds. 

Most of the consortia or local planning bodies use both qualitative and quantitative data 

sources in their needs assessment work, including public health data, consumer and 

provider surveys, focus groups and public forums. Outside of the public health data, 

information on the needs of infected individuals not receiving HIV/AIDS care is typically 

not obtained. Of the 17 planning bodies we interviewed, only 3 reported that they have 

successfully solicited needs information from persons with HIV/AIDS not in care. 

RECOMMENDATIONS 

The HRSA Should Implement a Multi-year Outcomes Initiative 

The HRSA should build upon their efforts to promote outcome measurement by 

establishing a multi-year implementation plan. Outcome measurement should start on the 

provider level and lead through subsequent steps to consortia-wide analysis of client 

outcomes by the responsible fiscal agent. Outcome measurement would foster program 

improvements and greater accountability for expenditures and promote the achievements 

of the Ryan White programs. 

Ryan White Evaluation Systems: Title II 2 OEI-05-98-00393 
The HRSA Should Provide Technical Assistance to Improve the Ability of 

Grantees and Consortia to Estimate the Needs of Infected Persons Not in 

HIV/AIDS Care 

The HRSA currently requires grantees to estimate unmet need of persons not in 

HIV/AIDS care by comparing service utilization data to the HIV epidemic in their area. 

The HRSA should require the grantees and consortia to improve their knowledge of 

unmet need by soliciting information directly from persons not in HIV/AIDS care. 

Grantee solicitation of the input of persons unconnected to HIV/AIDS care would assist 

the grantees in complying with the re-authorized CARE Act’s directive to prioritize a 

portion of funds for emerging populations. Grantee solicitation of this input would also 

contribute to the Department’s efforts to address the greater burden of HIV/AIDS on 

racial and ethnic minorities. 

Several respondents indicated a need for technical assistance on using outreach strategies 

to solicit information from persons not in HIV/AIDS care. The HRSA could provide the 

grantees and consortia with a technical assistance document detailing effective outreach 

strategies. Particular attention should also be paid to reaching persons who receive 

HIV/AIDS services but may not be receiving primary health care services. 

AGENCY COMMENTS 

While HRSA concurred with the findings and recommendations in our draft report, they 

offered suggestions for clarifying the report. Where appropriate, we changed the report to 

reflect their comments. The HRSA’s complete comments can be found in Appendix B. 

The HRSA briefly described a few activities in support of our recommendation to 

implement a multi-year outcomes initiative. We commend HRSA on these activities and 

plans. Despite their efforts in this area, HRSA expressed a few concerns regarding data 

limitations, funding limitations, and confidentiality concerns. 

We do not expect the outcomes initiative to lead to rigorous scientific evaluations 

conducted by every grantee. By examining the client-level impact of the services they 

fund, the grantees can assess in general terms the effectiveness of these services. 

Although we expect the costs involved to be minimized by emphasizing the use of data 

resources already available to the providers, many grantees may need additional financial 

resources to develop an outcomes system. We recommend that HRSA work with the 

Department to secure funds for these initiatives. The FY 2000 re-authorization of the 

CARE Act provides an optimal opportunity to consider funding for infrastructure 

development vital to ensuring efficient and effective service delivery in the future. 

The HRSA’s third area of concern - assuring client confidentiality, can be addressed by 

using unique identifiers and staff confidentiality agreements. 

Ryan White Evaluation Systems: Title II 3 OEI-05-98-00393 
TABLE OF CONTENTS 

EXECUTIVE SUMMARY ............................................... 1

INTRODUCTION ..................................................... 5

FINDINGS

Fiscal Intermediaries Perform Monitoring ................................ 9

Few Require Outcomes ............................................ 11

Adequately Assessing Client Needs, Not Those Outside of Care ............... 14

RECOMMENDATIONS ................................................... 16

AGENCY COMMENTS .................................................... 20

APPENDICES 

A: Title II Administrative Structure by State ................................ 21

B: Agency Comments ................................................. 24

C: Related Office of Inspector General Reports .............................. 32

Ryan White Evaluation Systems: Title II 4 OEI-05-98-00393 
INTRODUCTION 

PURPOSE 

To examine the evaluation systems used by Title II Ryan White grantees to monitor the 

compliance, measure the outcomes and measure the overall effectiveness of the programs 

they fund. 

BACKGROUND 

This study complements our reports entitled, Ryan White Evaluation Systems, Title I: 

Grants to Metropolitan Areas (OEI-05-98-00392) and Ryan White Evaluation Systems, 

AIDS Drug Assistance Programs (OEI-05-98-00390). 

The Ryan White CARE Act 

In 1990, Congress passed the Ryan White Comprehensive AIDS Resources Emergency 

(CARE) Act as a response to the HIV epidemic and its impact on individuals, families, 

communities, cities and States. The Ryan White programs provide health and support 

services to persons with HIV/AIDS who would otherwise not have access to care. The 

Ryan White programs constitute slightly over $1 billion of the $6.8 billion in HIV/AIDS 

expenditures in the U.S. Department of Health and Human Services’ Fiscal Year (FY) 

1998 budget. 

The Ryan White CARE Act was re-authorized in FY 1996 through FY 2000. Changes to 

the Act included a directive to Title II grantees to prioritize a portion of their funds for 

emerging populations. 

The Health Resources and Services Administration (HRSA) directs Ryan White resources 

to various local entities through the Act's four titles and Part F and allows grantees 

maximum flexibility in the use of funds. In FY 1998, nearly 90 percent of the Act’s funds 

were awarded to Title I and Title II grantees. 

Title II of the Ryan White CARE Act 

Title II provides grants to States to improve the quality, availability and organization of 

health care and support services for individuals and families with HIV disease. Grants are 

awarded to States in accordance with the number of living AIDS cases in the State or 

territory and the estimated number of living AIDS cases outside of Title I funded eligible 

metropolitan areas. In addition to the base award, States receive earmarked funds for the 

Ryan White Evaluation Systems: Title II 5 OEI-05-98-00393 
AIDS Drug Assistance Programs (ADAPs) through which States provide medications to 

low-income individuals with HIV disease with no or limited insurance coverage. 

Outside of the earmarked funds, States can use Title II funds to support home and 

community based health and support services, continuation of health insurance coverage, 

pharmaceutical treatments through the ADAP, HIV care consortia, and direct health and 

support services. This inspection focuses on evaluation mechanisms within the consortia 

program, as this program receives the largest share of Title II non-ADAP funds. 

Consortia are community-based regional planning entities established by Title II grantees. 

The consortia plan and prioritize Title II funds in their area, promote coordination of 

services and serve as a community forum. Representatives of local public and non-profit 

health and support service providers serve as consortium members. Incorporated 

consortium may serve as the local fiscal agent for Title II funds. In other cases, 

consortium designate a member agency to be the “lead agency,” responsible for grants 

administration and provider monitoring of Title II funds in the region. In some areas, the 

lead agency is the same agency that is responsible for the administration of Title I funds. 

(See Appendix A for a description of the consortia/lead agency structure in visited States) 

OIG Evaluation Recommendations in 1995 

In a June 1995 Office of Inspector General (OIG) report, “The Ryan White CARE Act: 

Local Implementation Issues”(OEI-05-93-00336), the OIG found an absence of program 

outcome evaluation to measure the CARE Act’s impact. Beyond surveys of client 

satisfaction undertaken by providers at the local level, grantees and consortia were not 

assessing the impact of CARE Act funds on their service area. 

The OIG recommended that the Health Resources and Services Administration and the 

Assistant Secretary for Planning and Evaluation (ASPE) fund appropriate studies to assess 

the overall effectiveness of the Ryan White programs. The OIG also recommended that 

HRSA ensure grantees evaluate their own programs. The OIG suggested local 

evaluations include provider compliance reviews, outcome measure studies and overall 

program effectiveness reviews. The OIG asked HRSA to provide grantees with technical 

assistance and evaluation instruments that guarantee consistency in data collection and 

evaluation. 

The ASPE did not conduct any evaluations of the Ryan White program as recommended 

in the 1995 OIG report. However, ASPE is currently funding the HIV Costs and Services 

Utilization Study (HCSUS) with HRSA and the Agency for Health Care Policy and 

Research (AHCPR). The HCSUS examines who is treated for HIV, the health services 

HIV patients receive, the costs of the services and means of payment. The ASPE and 

AHCPR are also sponsoring a related study on the impact of health care financing 

arrangements on access to and use of new and emerging HIV therapies. 

Ryan White Evaluation Systems: Title II 6 OEI-05-98-00393 
The Office of Science and Epidemiology, within the HIV/AIDS Bureau of HRSA, has a 

national Ryan White evaluation agenda which includes client-level data studies, links with 

secondary data sets, topic specific evaluations and the aggregate reporting system. The 

office is focusing on evaluating the following areas: 1) Enrolling under-served populations; 

2) Providing clients with care equal to that received by the insured; 

3) Providing services that remove barriers to receiving primary care; 4) Reducing 

morbidity, including opportunistic infections and perinatal transmission, and maintaining or 

increasing CD4 levels; 5) Reducing mortality; and 6) Adapting to a changing service and 

cost environment. 

HRSA’s Current Evaluation Requirements 

In recent years, HRSA’s Grant Application Guidances have placed a growing emphasis on 

the ability of grantees to document and quantify the impact of the services they fund. The 

FY 1999 application guidance asks grantees to begin developing outcome measures in 

relation to four program principles: responding to the needs of the underserved and hardto-serve, assuring access to emerging therapies, adapting to changes in the health care 

market, and documenting client-level outcomes. Suggested client-level outcomes include: 

decreased viral load, declines in perinatal transmission of HIV, and morbidity and 

mortality reductions. In the absence of defined outcome measures, the grantee is asked to 

describe activities they will undertake to measure impact in these areas. 

The FY 1999 Application Guidance instructs applicants to estimate unmet need for those 

in care and those not currently in the system of care in terms of their demographics, 

geographic variation across the State and their unmet need. Applicants are also instructed 

to provide quantitative data on co-morbidities, poverty status and insurance status of the 

State’s living HIV/AIDS cases. 

The FY 1999 Application Guidance asks the grantees for more information on unmet need 

to be obtained through comparisons of HIV prevalence and service utilization data. The 

Guidance emphasizes the importance of documenting unmet need and indicates future 

HRSA plans to work with the States in this endeavor. 

HRSA’s Technical Assistance on Evaluation 

The HRSA has provided technical assistance to Title II grantees on conducting 

evaluations through a variety of means including: self assessment protocols, meetings to 

share evaluation strategies, and a series of monograph reports on evaluation and 

assessment. The most recent report emphasizes the importance of outcome measurement 

and provides some guidance to grantees on how to approach it. The report does not 

suggest outcomes to measure or steps to develop outcome measures. More recently, 

HRSA has suggested grantees focus on primary health care outcomes and will soon 

disseminate a publication focusing on primary health care outcome measures. 

Ryan White Evaluation Systems: Title II 7 OEI-05-98-00393 
Re-authorization in 2000 

The Ryan White CARE Act will be considered for its second re-authorization in 2000. As 

Congress debates this re-authorization, questions are certain to be asked about the 

effectiveness and impact of the programs funded. This inspection examines the capacity of 

Title II Ryan White grantees to answer those questions. 

SCOPE AND METHODOLOGY 

In this inspection, we examined the mechanisms Title II grantees use to monitor 

sub-grantee program and fiscal compliance, measure the outcomes of the programs they 

fund and assess their effectiveness in meeting the evolving needs of the affected 

population. This evaluation does not judge the compliance, measure the outcomes or 

assess the effectiveness of the Title II grantees, but rather judges whether the grantees 

have and appropriately utilize the tools to perform these evaluation functions. We did not 

review HRSA’s evaluation of the grantees. 

For the purposes of this analysis, we define compliance as serving the population, 

providing the services and spending the resources as agreed to in return for the award of 

program funds. Outcomes are defined as measures of the benefits clients derive as a result 

of the program. Finally, we define overall effectiveness as how well the mix of services 

meets the evolving needs of the population affected by the epidemic. 

We conducted an on-site examination of these evaluation mechanisms in eight States: New 

York, New Jersey, Maryland, Florida, Illinois, Texas, California and the District of 

Columbia. We selected these States as a purposive sample representative of 

comparatively high rates of incidence through intravenous drug use transmission, 

comparatively high rates of infections among minority populations, and regional diversity. 

In addition, our selected States contain 68 percent of the living AIDS cases reported in 

1997 and 61 percent of the Title II funding for FY 1998. 

We conducted an extensive review of documentation provided by the grantees, consortia 

and providers. We examined States’ FY 1998 Title II grant applications, service provider 

contracts, samples of all program, fiscal, outcome and needs assessment reports, site visit 

protocols and additional evaluation materials. 

We supplemented the documentation review with on-site interviews from October through 

December 1998. In each State we interviewed: the Title II grantee, the Title II 

consortia/lead agency in up to three different regions, and at least 10 percent of the Title II 

providers. The number of providers interviewed ranged from 4 to 15. They were chosen 

to represent various program sizes, service delivery purposes and locations. 

We conducted our review in accordance with the Quality Standards for Inspections issued 

by the President’s Council on Integrity and Efficiency. 

Ryan White Evaluation Systems: Title II 8 OEI-05-98-00393 
FINDINGS 

In most sampled States, program and fiscal compliance of 

service providers is monitored through fiscal intermediaries; 

intermediaries have monitoring mechanisms in place 

In the eight States we visited, seven of the grantees use regionally based lead agencies to 

serve as fiscal agents for the service providers. The grantees monitor the lead agencies 

and the lead agencies monitor the providers. The lead agencies determine funding awards 

in accordance with a service plan developed by the regional consortium or planning body. 

In six of the States, the lead agencies have primary responsibility for monitoring the 

program and fiscal compliance of the service providers. In two of these six States, the 

grantees keep abreast of individual provider performance by reviewing the reports 

providers submit to the lead agency or by accompanying the lead agency on site visits. In 

the other four States, the grantees leave all provider monitoring responsibilities to the lead 

agencies. 

Grantees receive regular program and fiscal reports from lead agencies and most 

conduct periodic site visits to lead agencies 

All of the grantees we visited receive monthly or quarterly program and fiscal reports from 

the lead agencies. Program reports typically contain compiled program data, such as the 

units of service provided and the number and demographics of clients served, as well as a 

narrative section on program accomplishments and barriers. Fiscal reports are typically 

expense vouchers or invoices of line item costs. 

Most of the grantees we visited reimburse the lead agencies based on a line item budget. 

A couple of the grantees reimburse lead agencies on a fee-for-service or unit cost basis. 

Some grantees advance the lead agencies a portion of their funds to facilitate the flow of 

the grant making process. All but one of the grantees requires the lead agencies to have 

an independent audit performed. The majority of grantees require the audits to be 

submitted for review on an annual basis. 

Most of the grantees conduct formal site visits to the lead agencies at least once every 2 

years. Two of the grantees have not routinely visited lead agencies but plan to conduct 

routine visits in the future. One of these grantees plans to visit lead agencies once every 

3 years. 

Ryan White Evaluation Systems: Title II 9 OEI-05-98-00393 
Lead agencies receive regular program and fiscal reports from providers and 

conduct frequent site visits to providers 

All of the lead agencies we visited receive monthly or quarterly program and fiscal reports 

from their service providers. The program and fiscal reports contain similar information to 

that which the lead agencies report to the grantees. While most of the grantees reimburse 

lead agencies based on a line item budget, the majority of lead agencies reimburse 

providers on a fee-for-service or unit cost basis or have plans to do so. Nearly all of the 

lead agencies require and review annual audits of the providers. 

The majority of sampled lead agencies conduct formal site visits to service providers at 

least once a year and as often as quarterly. Site visits typically include a review of a 

sample of client files, financial records and other support documentation used for 

reporting. Most of the lead agencies also informally monitor providers through ongoing 

communication and individualized technical assistance. 

Most grantees and lead agencies have mechanisms to ensure that Ryan White 

funds are used only for eligible clients and as the payor of last resort 

All of the grantees and lead agencies have mechanisms in place to ensure that providers 

verify client eligibility for Ryan White services. Six of the eight grantees and most of the 

lead agencies also have mechanisms to ensure that providers use Ryan White funding as 

the payor of last resort. The grantees and lead agencies include eligibility and payor 

requirements in written policy and procedural manuals and check relevant documentation 

on site visits. A few lead agencies do not routinely verify provider use of Ryan White 

funds as the payor of last resort. These lead agencies indicated awareness of the need to 

devote more attention to this area in the future. 

Most of the service providers we visited have mechanisms in place to ensure that they use 

Ryan White funds as the payor of last resort, such as documenting Medicaid and private 

insurance coverage in client files and billing accordingly. However, a few of the ancillary 

service providers we visited appeared unaware of the rules or their responsibility in this 

area. A few also appeared to be unclear of the services paid for by Medicaid in their 

State. In addition, service providers have varying capacity to check Medicaid eligibility. 

Ryan White Evaluation Systems: Title II 10 OEI-05-98-00393 
Few Grantees or Lead Agencies Currently Require Outcome 

Measurement of Their Service Providers; Most Are in the 

Early Stages of Discussion Regarding Outcome Measures 

Currently, Only One of the Eight State Grantees and One of the Twelve Lead 

Agencies We Visited, Require Client Outcomes to Be Measured 

The New York grantee currently requires outcome measures for three service categories, 

primary care, dentistry and case management. Primary care and dentistry are measured 

according to professional clinical guidelines and expected clinical health outcomes. An 

independent peer review organization monitors providers’ performance on a set of core 

indicators such as CD4 count and gynecological health. These are tracked by a common 

software package that aggregates individual level clinical data entered to manage patient 

care. The grantee is also piloting outcomes in case management in the following areas: 

optimal health, financial/entitlements, independent living, adult education/employment, 

family stability, mental health, substance abuse and client empowerment. 

The Sacramento County Department of Health and Human Services, the lead agency for 

Title II funds in the Sacramento area, requires providers to report outcome measures on a 

quarterly basis. Since reporting compliance was low in the initial 6 months, the lead 

agency is developing a standard format for outcome reporting. Outcome measures are a 

required component of the provider quarterly reports as of FY 1999. 

The rest of the grantees and lead agencies rely on process measures to measure the 

success of service providers. These measures, along with annual goals, are usually 

outlined in the scope of work section of a provider’s contract. The process measures 

typically consist of the number of clients served and the number of service units delivered 

in the contract year. Some contracts require that the measures be broken down by subpopulation, with a special emphasis on women and children. 

A few providers currently report outcomes; many others have client outcomes in case files 

A few providers in several localities have developed aggregate outcomes to measure the 

impact of the services they provide. These providers have taken the initiative due to the 

requirements of other funding sources or their own desire for greater program 

accountability. Though they are not required to, these agencies typically submit their 

outcome information to their lead agencies as part of their quarterly or annual progress 

reports. 

The outcome measures developed by the Mount Carmel Guild in Mercer County New 

Jersey provide an example of the type of outcome measures some providers are using. The 

Ryan White Evaluation Systems: Title II 11 OEI-05-98-00393 
Mount Carmel Guild provides nutritional counseling, nutritional food packages and 

personal care packages for HIV infected clients. They have a set of outcome measures 

that are tracked for each client and aggregated to evaluate overall program performance. 

The measures include weight loss, weight gain, weight stabilization, cholesterol level, 

albumin level and routine vitamin intake. The nutritionist routinely tracks progress on 

these outcomes by client and aggregates them annually through a random audit of 50 

client charts. The organization has plans underway to refine this process by analyzing 

outcomes across sub-populations. 

Many other providers we visited collect client information which could be used to measure 

program outcomes if aggregated. For example, case managers often establish client goals 

at intake and document progress towards these goals in a client’s case file. Physicians also 

typically record clinical progress in patient charts. In these situations, a client’s progress 

or the progress of the caseload in general may be discussed at organizational meetings, but 

agencies do not aggregate the client outcomes and are not able to compare outcomes 

across client groups, service strategies or over time. Nor do they report outcomes to the 

lead agency or grantee. 

The Majority of Grantees and Lead Agencies We Visited Are in the Early Stages of 

Planning for Outcome Measurement 

Most State grantees are in various stages of planning for outcome measures. Two State 

grantees are discussing the development of outcome measures in the near future. Both are 

planning on developing a core set of outcomes at the State level with provider input. One 

has begun by introducing the concept of outcome measurement to its providers to solicit 

their feedback. The other grantee is setting up a task force to focus on outcome 

measures. This task force is soliciting information from other Ryan White programs and is 

searching for a consultant to help them devise a strategy to develop outcome measures in 

a participatory manner. They are also considering contracting a local university to define 

the measures once they are selected. 

Other State grantees, while not yet engaged in outcomes discussions, are laying the 

groundwork for future outcome measurement. A few are developing standards of care. It 

is their belief that this standardization of service provision is necessary in order to hold 

providers accountable to a common set of outcome measures. Several grantees are 

working to improve their databases which will facilitate the collection of outcome 

measures. The improvements include projects such as developing the capacity to collect 

client-level data and devising standard data definitions to assure accurate and reliable 

reporting. 

Half of the lead agencies are also engaged in the development of outcome measures. Half 

of these agencies indicated that they are actively engaged in putting outcome measures in 

place by the FY 1999 grant cycle. The other half are in the very early stages of discussion 

surrounding the implementation of outcome measurement. 

Ryan White Evaluation Systems: Title II 12 OEI-05-98-00393 
Agencies actively pursuing outcome measures are doing so by formulating a core set of 

outcome measures. One agency has, based on the United Way model1

, met with all of the 

providers to assess the data they are collecting and their capacity to aggregate outcome 

measures with minimal additional burden. This information will be used to create a set of 

outcomes for each service category. Another lead agency, already receiving voluntary 

outcome measures from its providers, has compiled a list of all of the outcome measures 

reported and is in the process of selecting a core set of measures. Another agency has 

hired a consultant to develop a set of quality indicators in consultation with staff and 

providers. This agency plans to recommend a set of outcome measures and require that 

providers incorporate a minimum of three outcome measures per service category. 

Other lead agencies are aware of the management and accountability benefits of utilizing 

outcome measures, but are still “figuring out their approach.” While they indicate plans to 

have outcomes in place by FY 2000, they have no specific plans underway. 

More than Half of the Grantees and Lead Agencies Expressed Concerns 

Regarding Outcomes Measures 

Grantees and lead agencies share a variety of concerns regarding the development of 

outcome measures. Several respondents noted that the administrative cap limits the 

amount of money available to track outcomes. Additional grant funds that can be set 

aside for outcome measurement would be in direct competition with direct service funds. 

There are also capacity concerns. A number of grantees and lead agencies are concerned 

that they do not possess the technical expertise to develop outcome measures that would 

appropriately reflect the diversity of service delivery and which could be attributed solely 

to the infusion of Ryan White funds. Respondents also questioned the ability of planning 

bodies to properly understand and utilize outcome measures. Providers voiced similar 

concerns regarding outcome measures. In particular, they felt that they did not have the 

resources or skills to properly develop or monitor client outcomes. 

Respondents also questioned whether the State and local levels are the appropriate place 

to develop and conduct outcome measurement. Many respondents feel that outcome 

measures should be uniform across Ryan White Titles, the State and the country in order 

to reduce the reporting burden and to allow for meaningful comparisons between 

programs. Some also feel that Ryan White outcome measures should relate to the 

outcomes measures of other Federal agencies that support HIV services, such as the 

Department of Housing and Urban Development, and the Public Health Service goals for 

the year 2000. In addition, several respondents suggested that tracking outcomes at the 

national level would be more cost-effective. Respondents indicated that with uniform 

measures HRSA could take advantage of sampling methodologies to reduce the amount of 

1 The United Way approach emphasizes developing outcomes reflective of an agency’s goals and 

recommends using data sources readily available to agencies such as client records and responses 

to routinely administered standardized tests and measurements. 

Ryan White Evaluation Systems: Title II 13 OEI-05-98-00393 
labor involved. The HRSA could also tap into the capacity and resources that already 

exist in the medical research field. 

Majority of grantees desire technical assistance from HRSA on outcome 

measurement 

Grantees requested that HRSA clearly delineate and communicate the direction they are 

headed with regards to outcome measures. Grantees asked for a definition of what is 

meant by outcome measures and what outcome measures are appropriate for specific 

service categories. 

Several grantees also want HRSA to offer guidance on how to design an inclusive process 

of developing outcomes, how to measure service outcomes and how to build 

organizational capacity to interpret and analyze outcomes. They want answers to 

questions such as: How do you create indicators that provide the information you are 

after? How do you track outcome measures over time? How do you get the necessary 

information from the client or their provider? What about confidentiality issues? The 

respondents suggested that the answers to these questions could be provided through 

workshops, dissemination of best practices and national conferences, designed to promote 

information exchange among the grantees. 

Grantees rely on consortia to assess needs; most consortia 

routinely assess the needs of clients but not of persons 

outside of HIV/AIDS care 

All of the State grantees we visited rely on regional consortia to assess the needs of 

persons with HIV/AIDS and require them to conduct periodic assessments. Slightly over 

half of the grantees we visited require the consortia to submit needs assessment results as 

part of the consortium’s annual application for funds. A couple of grantees supplement 

the local needs assessments with limited Statewide assessments of specific issues or subpopulations. None of the grantees currently conduct comprehensive assessments of the 

needs of the HIV/AIDS population in their State. However, in compliance with Federal 

law, all of the State grantees review local needs assessments to identify Statewide needs as 

part of the Statewide Coordinated Statement of Need (SCSN) process.2 Building on the 

momentum of the SCSN process, two of the grantees are planning to conduct a 

Statewide needs assessment in FY 1999. 

2 The 1996 CARE Act legislation requires all Ryan White grantees to participate in a Statewide 

Coordinated Statement of Need (SCSN) process to collaborate in identifying and addressing HIV 

care issues. The SCSN is to be based on existing information from local needs assessments. 

Starting in FY 1998 Title II grantees are required to submit a copy of the SCSN with their grant 

applications and update the SCSN every three years, at a minimum. 

Ryan White Evaluation Systems: Title II 14 OEI-05-98-00393 
Most grantees do not specify what to include in the needs assessment 

Most of the grantees we visited do not specify what the consortia must include in their 

needs assessment or provide a standard tool for conducting the needs assessment. A few 

of the grantees provide some guidance to the consortia on how to conduct the needs 

assessment and on the types of information they are expected to collect. For example, the 

Florida grantee specifies what types of entities and population groups the consortium 

should consult with in the assessment. Lead agencies in Florida are also directed to 

include an epidemiological profile, a demographic survey, focus groups, a resource 

inventory and a profile of provider capacity. Two of the State grantees are in the process 

of developing a standard needs assessment tool for use by their consortium. 

Most consortia do not assess the needs of infected persons not in HIV/AIDS care 

We interviewed 17 planning bodies responsible for conducting local Title II needs 

assessments, including 5 “HIV/AIDS care networks” in New York and several Title I 

planning councils who assess needs for both Title I and Title II services. Most of the 

consortia or planning bodies use both qualitative and quantitative data sources in their 

needs assessment work, including public health data, consumer and provider surveys, 

focus groups and public forums. 

Outside of the public health data, information on the needs of infected individuals not 

receiving HIV care is usually not obtained. Of the 17 planning bodies interviewed, only 3 

reported that they have successfully solicited needs information from persons with 

HIV/AIDS who are not in care. A few others indicated that they may have obtained needs 

information from infected persons not in care through public forums, but these forums 

were not targeted to this population. Two of the other needs assessment bodies tried to 

solicit information from persons not in HIV/AIDS care through advertising in local 

newspapers and through substance abuse facilities, but were reportedly unsuccessful. 

The three consortia successful in obtaining the input of persons not in HIV/AIDS care 

used peers and street outreach to solicit input. Two of the consortia trained persons living 

with HIV to survey infected persons in non-HIV/AIDS provider locations such as shelters 

and community groups. One care network in New York conducted focus groups in 

shooting galleries and crack houses to solicit information on needs from persons outside of 

the HIV/AIDS care system. 

Ryan White Evaluation Systems: Title II 15 OEI-05-98-00393 
RECOMMENDATIONS 

The HRSA should implement a multi-year outcomes initiative 

The HRSA should build upon their efforts to promote outcome measurement by 

establishing a multi-year implementation plan. Outcome measurement should start on the 

provider level and lead through subsequent steps to consortia-wide analysis of client 

outcomes by the responsible lead agency.3 Outcome measurement would foster program 

improvements and greater accountability for expenditures and promote the achievements 

of the Ryan White program. 

Here is one option of how HRSA could proceed: 

< In stage one, HRSA could develop one or more core outcomes for each service 

category and require the grantees to direct the lead agencies of the Title II consortia 

to develop three additional outcomes for each service category. The HRSA’s core 

outcomes could be used for both Title II and Title I grantees and should be aligned 

with the performance goals outlined in HRSA’s annual performance plan. The 

HRSA should urge Title II lead agencies to work with co-located Title I grantees 

and planning bodies in establishing the consortia-level outcome measures, so as to 

avoid imposing a greater measurement burden on providers funded under both 

Titles. 

< In stage two, HRSA could require the grantees to direct the lead agencies of the 

consortia to collect the established outcome measures from the providers on a 

quarterly basis. The HRSA should also urge the grantees and the lead agencies to 

require providers to report outcome measures as client-level data. These agencies 

will need client-level outcome data in order to analyze client outcomes across the 

region in stage three. 

< In stage three, HRSA could require the grantees to direct the lead agencies of the 

consortia to analyze the reported outcome measures across services and report 

consortia-wide outcomes to HRSA via the grantees on an annual basis. 

In establishing the core outcomes, HRSA should place a greater emphasis on primary 

health care outcomes since the ultimate goal of the Ryan White CARE Act is improved 

3 In States where grantees, rather then regional consortia, have direct oversight over providers, the 

responsibility of developing, collecting and analyzing outcomes would fall to the State. 

Ryan White Evaluation Systems: Title II 16 OEI-05-98-00393 
health status. However, at least one core outcome should be developed for each support 

service category. The outcomes for the support services should reflect the contribution 

the support services make to improved health status such as facilitating access to primary 

care or creating conditions conducive to following and benefitting from a structured health 

care regimen. Measuring these incremental outcomes would provide valuable information 

on the contribution of these services to health outcomes. This information could be used 

for accountability and program improvement purposes. Confidentiality concerns can be 

overcome using staff confidentiality agreements and unique identification numbers, rather 

than client name, to track clients. 

The outcome measurement initiative can serve as a means for HRSA to further promote 

continuous quality improvement (CQI) efforts. Through CQI, information on the 

effectiveness and quality of program delivery is fed back to program managers on a 

continuous basis, serving as a tool for managers to improve program delivery. While 

quality assurance and process indicators are used to gauge the quality of services 

delivered, outcome measures can be used to gauge service effectiveness. The HIV/AIDS 

service providers could use outcome measures along with quality of care indicators to 

improve their performance through program alterations and further self-evaluation under 

CQI. 

As the Ryan White CARE Act faces its second decade of providing care to persons 

infected with HIV/AIDS, it is time to ensure that the grantees have the capacity to 

evaluate the outcomes of the services they deliver. Outcome evaluation as a CQI tool is 

vital to strengthening the effectiveness of the services delivered and the efficiency of the 

funds expended to provide the services. This basic need for appropriate funding should be 

addressed in the re-authorization of the CARE Act in FY 2000. 

The HRSA should provide the grantees with technical assistance to implement the core 

outcome measures and to develop and implement their own additional outcome measures 

within a CQI framework. The technical assistance could emphasize soliciting provider 

input in the development of the additional outcomes, using the United Way approach. 

The United Way approach to outcome measurement emphasizes using outcome measures 

for program improvement purposes. To this end, the United Way emphasizes soliciting 

provider input in the development of outcome measures to make sure that outcome 

measures reflect program goals. The United Way also stresses the importance of 

considering the data resources already available to the providers when developing 

outcome measures so as not to impose too much additional administrative or cost burden. 

The United Way’s approach recognizes the limited capacity of health and human service 

providers to engage in rigorous outcome evaluation. Hence, the United Way model 

emphasizes contribution rather than attribution. Using outcome measures, providers can 

determine if clients achieve an expected result after receiving their service intervention. 

While the provider is not able to scientifically prove that the result is due solely to their 

Ryan White Evaluation Systems: Title II 17 OEI-05-98-00393 
intervention, they can see whether their service is contributing to positive outcomes on a 

regular basis. 

In recent years, HRSA’s application guidance to Title II grantees has stressed the 

importance of documenting outcomes and has asked grantees to include outcome data in 

their applications. All of our respondents have received the message from HRSA that 

outcome measures are important and that HRSA expects them to move ahead in this area. 

However, respondents stated that they would like more specificity from HRSA on what 

outcomes they are expected to measure and on HRSA’s implementation plans. 

Stronger leadership from HRSA on outcome measurement would help coordinate efforts 

that remain fragmented at various levels of the Title II program. Further, outlining a 

multi-year initiative would provide grantees and the lead agencies with the necessary 

perspective to strategically plan for the future. As HIV/AIDS has evolved from an 

emergency epidemic to a chronic disease in a tight health care market, it is critical that the 

agencies serving infected individuals build the capacity to provide care efficiently and 

effectively through the productive use of outcome and impact studies. 

The HRSA should provide technical assistance to improve 

the ability of grantees to estimate the needs of infected 

persons not in HIV/AIDS care 

In the FY 1999 Application Guidance, HRSA emphasizes the importance of obtaining 

information on unmet need, acknowledges the difficulty of obtaining this information and 

pledges to work with States to address this area in the future. The HRSA should build on 

this effort by requiring the grantees and consortia to solicit needs information from 

persons not in HIV/AIDS care and by providing technical assistance on how to do this. In 

turn, the grantees should provide the consortia and other Title II planning bodies with 

guidance on needs assessment expectations and on methods to obtain the required 

information. 

Currently, grantees are required to use public health data to estimate unmet need in their 

applications to HRSA. This data provides useful information on the needs of infected 

persons. However, the input of persons outside of the care system would add a richness 

of understanding regarding their needs and the barriers to service they encounter. 

Grantee solicitation of the input of persons un-connected to HIV/AIDS care would assist 

the grantees in complying with the re-authorized CARE act’s directive to prioritize a 

portion of funds for emerging populations. Grantee solicitation of this input would also 

contribute to the Department’s efforts to address the greater burden of HIV/AIDS on 

racial and ethnic minorities. 

Ryan White Evaluation Systems: Title II 18 OEI-05-98-00393 
In their attempt to fulfill the CARE Act’s requirement to prioritize funds for emerging 

populations, HRSA should urge the grantees to strive to reach persons unconnected to 

HIV/AIDS care, in addition to efforts to reach persons who demographically reflect the 

emerging populations. The HRSA should urge the grantees to recognize the complexity 

and disproportionate costs which may be involved in serving persons unconnected to 

HIV/AIDS care. Persons infected by HIV/AIDS are increasingly individuals who are 

traditionally not connected to the health care system for reasons such as poverty and 

historic mistrust of institutional health care. A focused outreach and planning strategy is 

required to address their needs. 

The HRSA’s technical assistance materials on conducting needs assessments urge grantees 

to solicit information from persons not in care. The materials suggest avenues for 

soliciting this information, such as substance abuse clinics and homeless shelters. Outside 

of suggesting that consortia provide incentives like food vouchers or use the media to 

solicit participation, the materials do not describe specific outreach strategies. As these 

ideas are discussed only generally, we suggest HRSA provide more detailed information 

on steps to solicit needs information. Several consortia representatives indicated that they 

would like more guidance on how to reach infected persons not in care. 

The HRSA could provide the grantees and the consortia with a technical assistance 

document devoted to soliciting needs information from people living with HIV/AIDS who 

are not in HIV/AIDS care, providing steps for conducting outreach and examples of 

effective strategies. The HRSA could highlight outreach strategies which have been used 

by other Ryan White grantees, Centers for Disease Control and Prevention HIV 

prevention community planning groups and service providers in other arenas. Particular 

attention should also be paid to reaching persons who receive HIV/AIDS services but may 

not be receiving primary health care services. 

Ryan White Evaluation Systems: Title II 19 OEI-05-98-00393 
AGENCY COMMENTS 

The HRSA provided comments to the draft report. While HRSA concurred with the 

report’s findings and recommendations, they offered suggestions for clarifying the report 

and making other technical changes. Where appropriate, we changed the report to reflect 

their comments. The complete text of HRSA’s comments can be found in Appendix B. 

The HRSA briefly described a few activities planned or underway in support of our 

recommendation to implement a multi-year outcomes initiative. We commend HRSA on 

these activities and plans. Despite their efforts in this area, HRSA expressed a few 

concerns regarding their ability to carry out this recommendation: 1) limitations on 

accessibility and compatibility of data sets, 2) restrictions on administrative costs 

impacting a Grantee’s ability to use complex data sets for outcome studies, and 

3) confidentiality concerns. 

These limitations are to be expected in a large endeavor but are not insurmountable. For 

this reason we recommend an outcomes approach designed with feasibility as a 

cornerstone. We do not expect the outcomes initiative to lead to rigorous scientific 

evaluations conducted by every grantee. The HRSA is funding such projects on a limited 

basis where the capacity exists. It is not feasible, nor is it necessary, for every grantee to 

participate in this work. What is feasible and necessary for every grantee to undertake is 

an assessment of the effectiveness of the services they fund. By examining the 

contributions made by the services to client outcomes, the grantees can assess, in general 

terms, the effectiveness of these services. We highlight the United Way approach which 

emphasizes contribution, rather than attribution, for this reason. 

Although we expect the costs involved to be minimized by emphasizing the use of data 

resources already available to the providers, many grantees may need additional financial 

resources to develop an outcomes system, depending upon the size of their grant and 

competing demands on limited administrative funds. We recommend that HRSA work 

with the Department to secure funds for these initiatives. The FY 2000 re-authorization 

of the CARE Act provides an optimal opportunity to consider funding for infrastructure 

development vital to ensuring efficient and effective service delivery in the future. 

The HRSA’s third area of concern - assuring client confidentiality, can be addressed by 

using unique identifiers and staff confidentiality agreements. Grantees collecting clientlevel information have used these measures successfully to address confidentiality 

concerns. 

We commend HRSA’s efforts to address our recommendation to improve the ability of 

grantees to estimate the needs of infected persons not in HIV/AIDS care. We urge HRSA 

to disseminate technical assistance information to grantees in this area as soon as possible, 

as the population of infected persons outside of care continues to grow. 

Ryan White Evaluation Systems: Title II 20 OEI-05-98-00393 
APPENDIX A 

Structure of Title II Programs

CALIFORNIA 

The California Department of Public Health is the Title II grantee agency. Funds are 

dispersed through 37 regionally-based consortium. Each consortium has a lead agency 

which is responsible for the fiscal and administrative functions of the Title II consortia 

program in their region. The State monitors the lead agencies and requires them to 

monitor the programmatic and fiscal compliance of the service providers they fund. 

In Eligible Metropolitan Areas, the lead agency is typically the same as the Title I grantee. 

For example, the Los Angeles County Department of Health Services receives Title I 

funds and also acts as the fiscal intermediary for the Title II consortium funds. In these 

situations, a planning council that has merged with the local consortia is responsible for 

advising the lead agency regarding needs and service priorities. In non-EMA areas, the 

lead agencies are counseled by a regional consortia as to the service priorities of that area. 

THE DISTRICT OF COLUMBIA 

The District of Columbia Department of Health is the Title II grantee. The Department of 

Health has designated the DC CARE Consortium to be responsible for conducting needs 

assessments, planning service delivery, promoting service coordination and monitoring the 

program and fiscal activities of the service providers. The Department of Health maintains 

some oversight over provider-level activities by reviewing individual provider reports 

submitted to the consortium and by accompanying the consortium on some provider site 

visits. 

FLORIDA 

The Florida Department of Health is the Title II grantee agency. The Department of 

Health has established 14 regionally based consortia to provide a continuum of care for 

HIV infected persons throughout the State. The consortia plan and prioritize Title II 

funds in their area, promote coordination of services and serve as a community forum. 

Each consortium has a lead agency which is responsible for the fiscal and administrative 

functions for the Title II consortia program in their region. In some consortia areas, the 

Department of Health serves as the lead agency for the consortium. In other areas, the 

Ryan White Evaluation Systems: Title II 21 OEI-05-98-00393 
APPENDIX A 

Structure of Title II Programs

local consortia designate a lead agency. All of the lead agencies in the State act as the fiscal 

conduit and data coordinator for the service providers in their area. 

ILLINOIS 

The Illinois Department of Public Health HIV/AIDS Section administers the Title II 

program in Illinois. The Department relies on 11 lead agencies to disperse the funds 

regionally and to monitor service providers. The State monitors the lead agencies. 

Regional consortium plan and prioritize Title II funds in their area and promote 

coordination of services. 

MARYLAND 

The Maryland Department of Health and Mental Hygiene AIDS Administration is the Title 

II grantee agency. There are now five lead agencies that disperse the Title II funds 

regionally in accordance with a consortia developed service priority and allocation plan. 

The State has oversight over the lead agency’s funding decisions. The State also retains 

control over monitoring the service providers. Both the lead agencies and service 

providers are required to submit their routine program and fiscal reports to the State 

Title II staff. 

NEW YORK 

The New York State Department of Health is the designated Title II grantee. The AIDS 

Institute of the Department of Health operates the Title II program. The AIDS Institute 

contracts with 17 Consortia, 16 of which are regionally based and one which is a 

Statewide consortium for special populations. The consortia, otherwise known as “HIV 

care networks” are responsible for conducting local needs assessments, identifying local 

service priorities and developing service delivery plans. The AIDS Institute determines 

funding allocations and awards contracts directly to service providers, in accordance with 

the networks’ service delivery plans. The Institute maintains fiscal and administrative 

oversight of the service providers funded through the Title II program, rather than use 

regionally based lead agencies. 

Ryan White Evaluation Systems: Title II 22 OEI-05-98-00393 
APPENDIX A 

Structure of Title II Programs

NEW JERSEY 

The New Jersey Department of Health and Senior Services is the Title II grantee. The 

Department allocates its Title II consortia funds to 10 consortia, 9 of which are regionally 

based and one of which is a Statewide network of hospital sites serving HIV-infected 

women and children. New Jersey is in the process of phasing out the six regional 

consortia which are located in Title I EMAs. The New Jersey consortia are responsible 

for planning and coordination activities. Each consortium designates a lead agency to 

handle fiscal and administrative responsibilities for the region. Although monitoring 

responsibilities primarily fall to the lead agencies, the State grantee maintains some 

oversight over the activities of local service providers. The lead agencies determine 

contract awards but provider contracts are subject to the review of the grantee. The 

grantee also conducts annual site visits to all service providers. 

TEXAS 

The Texas Department of Health is the Title II grantee agency. They allocate funds to 26 

regionally-based lead agencies which are either a local health department or a community 

organization. The State monitors the lead agencies and relies on the lead agencies to 

monitor the programmatic and fiscal compliance of the service providers they fund. 

Funding decisions made at the local level are directed by regional consortia. 

The Title II structure in Austin differs from the rest of Texas. In Austin, the Austin/Travis 

County Health and Human Services Department is the lead agency for the Title I funds 

and Title II consortium funds. The local planning council performs the priority setting 

and allocation for both Title I and Title II funds to all 10 counties. 

Ryan White Evaluation Systems: Title II 23 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 24 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 25 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 26 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 27 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 28 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 29 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 30 OEI-05-98-00393 
APPENDIX B 

Agency Comments

Ryan White Evaluation Systems: Title II 31 OEI-05-98-00393 
APPENDIX C 

Related Office of Inspector General Reports

The Ryan White CARE Act: Local Implementation Issues (OEI-05-93-00336)

The Ryan White CARE Act: Examples of Local Coordination (OEI-05-93-00335)

The Ryan White CARE Act: Special Projects of National Significance (OEI-05-93-00332)

The Ryan White CARE Act: Technical Report of 1992 Expenditures (OEI-05-93-00334)

The Ryan White CARE Act: Funding Formulas (OEI-05-93-00330)

Medicaid Managed Care and HIV/AIDS (OEI-05-97-00210)

Audit of State AIDS Drug Assistance Programs’ Use of Drug Price Discounts (A-01-97-01501)

Review of Maximizing Drug Discounts Under the Missouri Ryan White Program 

(A-01-97-00926) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the Health Resources and Services Administration (A-01-97-01500) 

Audit of Eligibility Under Title I of the Ryan White Comprehensive AIDS Resource Emergency 

Act of 1990, New York Metropolitan Area (A-02-95-02517) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990, Title II, 

Administered by the State of Connecticut (A-01-96-01501) 

Audit of the Ryan White Comprehensive AIDS Resource Emergency Act of 1990 in the Boston 

Metropolitan Area for Fiscal Year 1994 (A-01-95-01504) 

Ryan White Evaluation Systems: Title II 32 OEI-05-98-00393